You can join the discussion about their removal at the bottom of the talk page of the article (also see the discussion here). Do Wikipedia readers benefit from learning about the existence of the generative fixation hypothesis? Does mentioning the generative fixation hypothesis uphold or violate Wikipedia’s neutral point of view (NPOV) policy? If you feel strongly one way or the other about these issues, please add your voice to the discussion.

Enjoy!

The generative fixation hypothesis essentially states that genetic algorithms work by efficiently implementing a decimation strategy called hyperclimbing.

The chief barrier to the wide and enthusiastic use of hyperclimbing is that it seems to scale very poorly with the number of attributes. If one heeds the seemingly high cost of applying hyperclimbing to large search spaces, this heuristic quickly looses its shine. A key conclusion of my doctoral work is that this seemingly high cost is illusory. I have uncovered evidence that strongly suggests that genetic algorithms can implement hyperclimbing extraordinarily efficiently.

As readers of this blog probably know, genetic algorithms are search algorithms that mimic natural evolution. These algorithms have been used in a wide range of engineering and scientific fields to quickly procure useful solutions to poorly understood (i.e. black-box) optimization problems. Unfortunately, despite the routine use of genetic algorithms for over three decades, their adaptive capacity has not been adequately accounted for. Given the evidence that genetic algorithms can implement efficient hyperclimbing, I’ve proposed a new explanation for the adaptive capacity of these algorithms. This new account—the generative fixation hypothesis—promises to spark significant advances in the fields of genetic algorithmics and discrete optimization.

The discovery that hyperclimbing is efficiently implementable also promises to have a non-negligible impact on the ecology of machine learning research. Optimization and machine learning are, after all, intimately related. Overlooking a few exceptions, the practice of machine learning research, can be characterized as the effective reduction of difficult learning problems to optimization problems for which efficient algorithms exist. In other words, the machine learning problems that can effectively be tackled are in large part those that can in practice be reduced to optimization problems that can be tackled efficiently. Currently, this largely limits the class of tractable machine learning problems to the class of learning problems that can in practice be reduced to convex optimization problems [1] . The identification of general-purpose non-convex optimization heuristics with efficient implementations (e.g. hyperclimbing), thus, has the potential to significantly extend the reach of machine learning.

For a description of hyperclimbing, and evidence that genetic algorithms can implement this heuristic efficiently, please see my dissertation

[1]  Kristin P. Bennett and Emilio Parrado-Hernandez. The interplay of optimization and machine  learning research. Journal of Machine Learning Research, 7:1265–1281, 2006.

http://groups.google.com/group/generativefixation

Click here to see the final version (single spaced for easy reading).

The foundations of most computer engineering disciplines are almost entirely mathematical. There is, for instance, almost no question about the  soundness of the foundations of such engineering disciplines as graphics, machine learning, programming languages, and databases. An exception to this general rule is the field of genetic algorithmics, whose foundation includes a significant scientific component.

The existence of a science at the heart of this computer engineering discipline is  regarded with nervousness. Science traffics in provisional truth; it requires one to adopt a form of skepticism that is more nuanced, and hence more difficult to master than the radical sort of skepticism that suffices in mathematics and theoretical computer science. Many, therefore, would be happy to see science excised from the foundations of genetic algorithmics. Indeed, over the past decade and a half, much effort seems to have been devoted to turning genetic algorithmics into just another field of computer engineering, one with an entirely mathematical foundation.

Broadening one’s perspective beyond computer engineering, however, one cannot help wondering if much of this effort is not a little misplaced. Clearly, as fields of engineering go, genetic algorithmics is not the exception—the foundations of most engineering fields include large scientific components. What seems to matter is, not the  existence of a science within the foundation of an engineering discipline, but the state of that science. The advanced state of physics and chemistry is, for example, a significant part of the reason for the advanced state of such fields as mechanical, chemical, civil, aeronautical and electrical engineering.

Historically, the blossoming of a field of engineering has typically had to await the maturation of certain underlying field(s) of science. Consider for a moment the improbability of  constructing an internal combustion engine based on the phlogiston theory of combustion. Even if one somehow succeeds in actually building a prototype, further advances within the rubric of phlogiston theory would probably be limited. Combustion engine engineering would be a black art.

I trust that the scenario just described will give users of genetic algorithms and would-be inventors of new genetic algorithms pause, and reason for hope. Pause because even after decades of research, “black art” about sums up the process of applying current genetic algorithms and inventing viable new ones. Hope because it is conceivable that just as Lavoisier’s oxygen based theory of combustion stimulated rapid advances in the construction of internal combustion engines, fundamental upheavals in the science of genetic algorithmics might stimulate rapid advances in the ways in which genetic algorithms are applied and improved.

Given the above, the following question seems to get at  the heart of the matter: What should a science of genetic algorithmics, one capable of stimulating advances in the construction and application of genetic algorithms, look like? I submit that such a science should be organized around the search for a minimal set of computational efficiencies possessed by the simple genetic algorithm such that when considered together these efficiencies explain the adaptive capacity of the simple genetic algorithm on a very broad range of fitness functions. Roughly, computational efficiencies should play the part played by scientific laws in the physical sciences. The challenge is to identify the minimal set with the widest possible explanatory power.

There are two important reasons for making the simple genetic algorithm the object of attention. The first is precedence. There already exists a well known body of science with this algorithm as its focus. This pre-existing work, specifically the theory that goes by the name of the building block hypothesis, provides a point of reference against which future theories may be compared. The second reason is biological plausibility. Unlike many genetic algorithms currently in use, the simple genetic algorithm contains no biologically implausible mechanisms and is, therefore, a legitimate model of sexually evolving biological populations. Such populations have been the subject of intense scientific scrutiny for well over a century and have generated an enormous amount of scientific work. This body of work can serve as a second point of reference.

The aforementioned outline for a science of genetic algorithmics is hardly novel. Until about the mid 1990s, the study of genetic algorithms was organized roughly along the lines just described, with implicitly parallel building block discovery, and implicitly parallel hierarchical assembly being the core computational efficiencies that the simple genetic algorithm supposedly parlayed into a powerful capacity for general purpose adaptation. Problems arose when researchers were unsuccessful in their attempts to rigorously derive complexity theoretic bounds that showcased these purported core efficiencies. Much more seriously, efforts to demonstrate these efficiencies experimentally also proved unsuccessful. The consequence for the building block hypothesis in theoretical circles was severe—rightfully so.

Unfortunately, so was the consequence for the overarching scientific program described above. If there is just one thing readers take away from this dissertation, I hope it’s the sense that this program is viable.

Dissertation webpage

Consider the following stochastic function which takes a bitstring of length as input and returns a real value as output.

fitness(bitstring)
  accum = 0
  for i = 1 to 4
     accum = accum + bitstring[pivotalLoci[i]]
  end
  if accum is odd
     return a random value from normal distribution N(+0.25,1)
  else
     return a random value from normal distribution N(-0.25,1)
  end

The variable pivotalLoci is an array of four distinct integers between 1and which specifies the location of  four loci—let’s call them A, B, C, D—of an input bitstring that matter in the determination the bitstring’s fitness. These four loci are said to be pivotal. The other bits of the input bitstring do not matter in the determination of the bitstring’s fitness, and are said to be non-pivotal. Given some input bitstring, if the parity of the bits at the pivotal loci is odd, then the fitness of the bitstring is drawn from a normal distribution with mean 0.25, and variance 1 (the magenta distribution; see below). Otherwise the fitness is drawn from a normal distribution with mean -0.25, and variance 1 (the black distribution).

The expected fitness of each of the 16  “genotypes” of ABCD is shown below.

The following figure depicts the result of querying the fitness function with randomly generated bitstrings of length .

The next figure shows the locations of the pivotal loci A, B, C, D in this hypothetical scenario.

Consider the problem of classifying loci as pivotal or non-pivotal given only query access to the fitness function. This problem is closely related to the problem of finding the effective attributes of a parity function studied by Uehara et al.  [1,2]. One big difference is the presence of a stochastic element in the problem currently under consideration. Before reading further, I invite you to think of an algorithm that can solve this problem relatively robustly (with not more than, say, a 0.005 chance of misclassification per locus).

The naive approach would be to adopt a scanning strategy in which all combinations of four loci are visited. (Observe that visiting loci in combinations of three or less will not work). Suppose = 500,000; that’s approximately 6.25×10²² combinations. Even if it were possible to visit a million combinations per second, it would still take approximately two billion years to visit all such combinations.

It turns out that a genetic algorithm can be used to tackle this problem far more cheaply.

Suppose = 200 and pivotalLoci = [7 90 131 198]. The animation below shows the behavior of a simple genetic algorithm with uniform crossover (described in the materials and methods section of my dissertation) on the fitness function just described. Each frame in this animation shows the one-frequency of each locus (i.e. the frequency of the bit 1 at each locus) in a single generation. By extension, the frequency of the bit 0 at each locus is also on display. The red dots mark the positions of the pivotal loci. The blue dots mark the positions of non-pivotal loci.

Observe that the red dots diverge to the top or bottom of the plot, whereas the blue dots remain in the middle. After 500 generations, the location of the pivotal loci A, B, C, and D can simply be “read off” by examining the one-frequency of each of the 200 loci in the final population. Note also that the genotype of ABCD that goes to fixation is 1011. This genotype has odd parity, which explains the increase in the average fitness of the population shown in the following figure.

Now for the two-part punchline. First, there is nothing special about the location of the four pivotal loci. In other words, the expected number of generations required for the four red dots to diverge to the top or bottom of the plot remains the same regardless of the location of these dots. Second, there is nothing special about = 200. In other words, the behavior of the red dots will be unaffected by the number of blue dots present, and each blue dot will have the same behavior regardless of the total number of blue dots. Both these conclusions can be arrived at by appreciating the symmetries present when is applied to the fitness function described above (or rather, to the class of fitness functions described). Readers looking for a more rigorous treatment are referred to chapter 3 of my dissertation.

The animation below shows the result of applying to the fitness function when = 1000. The array pivotalLoci remained unchanged at [7 90 131 198].

The average fitness of the population over 500 generations is shown below.

The final animation shows the result of applying to the fitness function when = 10000. In this case, the array pivotalLoci was set to [2000 2681 6892 9520].

The average fitness of the population over 500 generations is shown below.

Note that in keeping with the assertions made above, the number of generations it takes for the red dots to diverge remains approximately the same despite an increase in by two orders of magnitude.

I hope the experience of watching evolutionary computation in action sparks your curiosity about the generative fixation hypothesis. Feel free to email me your questions and comments.

Click here to see the Matlab script used to generate the results presented above.

[1] Uehara, Tsuchida, and Wegener. Optimal attribute-efficient learning of disjunction, parity and threshold functions. In EUROCOLT: EUROCOLT, European Conference on Computational Learning Theory, EuroCOLT,. LNCS, 1997.

[2] Uehara, Tsuchida, and Wegener. Identication of partial disjunction, parity, and threshold functions. TCS: Theoretical Computer Science, 230, 2000.