Enjoy!

The generative fixation hypothesis essentially states that genetic algorithms work by efficiently implementing a decimation strategy called hyperclimbing.

The chief barrier to the wide and enthusiastic use of hyperclimbing is that it seems to scale very poorly with the number of attributes. When one heeds the seemingly high cost of applying hyperclimbing to large search spaces, this heuristic quickly looses its shine. A key conclusion of my doctoral work is that this seemingly high cost is illusory. I have uncovered evidence that strongly suggests that genetic algorithms can implement hyperclimbing extraordinarily efficiently.

As readers of this blog probably know, genetic algorithms are search algorithms that mimic natural evolution. These algorithms have been used in a wide range of engineering and scientific fields to quickly procure useful solutions to poorly understood (i.e. black-box) optimization problems. Unfortunately, despite the routine use of genetic algorithms for over three decades, their adaptive capacity has not been adequately accounted for. Given the evidence that genetic algorithms can implement efficient hyperclimbing, I’ve proposed a new explanation for the adaptive capacity of these algorithms. This new account—the generative fixation hypothesis—promises to spark significant advances in the fields of genetic algorithmics and discrete optimization.

The discovery that hyperclimbing is efficiently implementable also promises to have a non-negligible impact on the ecology of machine learning research. Optimization and machine learning are, after all, intimately related. Overlooking a few exceptions, the practice of machine learning research, can be characterized as the effective reduction of difficult learning problems to optimization problems for which efficient algorithms exist. In other words, the machine learning problems that can effectively be tackled are in large part those that can in practice be reduced to optimization problems that can be tackled efficiently. Currently, this largely limits the class of tractable machine learning problems to the class of learning problems that can in practice be reduced to convex optimization problems [1] . The identification of general-purpose non-convex optimization heuristics with efficient implementations (e.g. hyperclimbing), thus, has the potential to significantly extend the reach of machine learning.

For a description of hyperclimbing, and evidence that genetic algorithms can implement this heuristic efficiently, please see my dissertation

[1]  Kristin P. Bennett and Emilio Parrado-Hernandez. The interplay of optimization and machine  learning research. Journal of Machine Learning Research, 7:1265–1281, 2006.

http://groups.google.com/group/generativefixation

Click here to see the final version (single spaced for easy reading).

The foundations of most computer engineering disciplines are almost entirely mathematical. There is, for instance, almost no question about the  soundness of the foundations of such engineering disciplines as graphics, machine learning, programming languages, and databases. An exception to this general rule is the field of genetic algorithmics, whose foundation includes a significant scientific component.

The existence of a science at the heart of this computer engineering discipline is  regarded with nervousness. Science traffics in provisional truth; it requires one to adopt a form of skepticism that is more nuanced, and hence more difficult to master than the radical sort of skepticism that suffices in mathematics and theoretical computer science. Many, therefore, would be happy to see science excised from the foundations of genetic algorithmics. Indeed, over the past decade and a half, much effort seems to have been devoted to turning genetic algorithmics into just another field of computer engineering, one with an entirely mathematical foundation.

Broadening one’s perspective beyond computer engineering, however, one cannot help wondering if much of this effort is not a little misplaced. Clearly, as fields of engineering go, genetic algorithmics is not the exception—the foundations of most engineering fields include large scientific components. What seems to matter is, not the  existence of a science within the foundation of an engineering discipline, but the state of that science. The advanced state of physics and chemistry is, for example, a significant part of the reason for the advanced state of such fields as mechanical, chemical, civil, aeronautical and electrical engineering.

Historically, the blossoming of a field of engineering has typically had to await the maturation of certain underlying field(s) of science. Consider for a moment the improbability of  constructing an internal combustion engine based on the phlogiston theory of combustion. Even if one somehow succeeds in actually building a prototype, further advances within the rubric of phlogiston theory would probably be limited. Combustion engine engineering would be a black art.

I trust that the scenario just described will give users of genetic algorithms and would-be inventors of new genetic algorithms pause, and reason for hope. Pause because even after decades of research, “black art” about sums up the process of applying current genetic algorithms and inventing viable new ones. Hope because it is conceivable that just as Lavoisier’s oxygen based theory of combustion stimulated rapid advances in the construction of internal combustion engines, fundamental upheavals in the science of genetic algorithmics might stimulate rapid advances in the ways in which genetic algorithms are applied and improved.

Given the above, the following question seems to get at  the heart of the matter: What should a science of genetic algorithmics, one capable of stimulating advances in the construction and application of genetic algorithms, look like? I submit that such a science should be organized around the search for a minimal set of computational efficiencies possessed by the simple genetic algorithm such that when considered together these efficiencies explain the adaptive capacity of the simple genetic algorithm on a very broad range of fitness functions. Roughly, computational efficiencies should play the part played by scientific laws in the physical sciences. The challenge is to identify the minimal set with the widest possible explanatory power.

There are two important reasons for making the simple genetic algorithm the object of attention. The first is precedence. There already exists a well known body of science with this algorithm as its focus. This pre-existing work, specifically the theory that goes by the name of the building block hypothesis, provides a point of reference against which future theories may be compared. The second reason is biological plausibility. Unlike many genetic algorithms currently in use, the simple genetic algorithm contains no biologically implausible mechanisms and is, therefore, a legitimate model of sexually evolving biological populations. Such populations have been the subject of intense scientific scrutiny for well over a century and have generated an enormous amount of scientific work. This body of work can serve as a second point of reference.

The aforementioned outline for a science of genetic algorithmics is hardly novel. Until about the mid 1990s, the study of genetic algorithms was organized roughly along the lines just described, with implicitly parallel building block discovery, and implicitly parallel hierarchical assembly being the core computational efficiencies that the simple genetic algorithm supposedly parlayed into a powerful capacity for general purpose adaptation. Problems arose when researchers were unsuccessful in their attempts to rigorously derive complexity theoretic bounds that showcased these purported core efficiencies. Much more seriously, efforts to demonstrate these efficiencies experimentally also proved unsuccessful. The consequence for the building block hypothesis in theoretical circles was severe—rightfully so.

Unfortunately, so was the consequence for the overarching scientific program described above. If there is just one thing readers take away from this dissertation, I hope it’s the sense that this program is viable.

Dissertation webpage

Consider the following stochastic function which takes a bitstring of length as input and returns a real value as output.

fitness(bitstring)
  accum = 0
  for i = 1 to 4
     accum = accum + bitstring[pivotalLoci[i]]
  end
  if accum is odd
     return a random value from normal distribution N(+0.25,1)
  else
     return a random value from normal distribution N(-0.25,1)
  end

The variable pivotalLoci is an array of four distinct integers between 1and which specifies the location of  four loci—let’s call them A, B, C, D—of an input bitstring that matter in the determination the bitstring’s fitness. These four loci are said to be pivotal. The other bits of the input bitstring do not matter in the determination of the bitstring’s fitness, and are said to be non-pivotal. Given some input bitstring, if the parity of the bits at the pivotal loci is odd, then the fitness of the bitstring is drawn from a normal distribution with mean 0.25, and variance 1 (the magenta distribution; see below). Otherwise the fitness is drawn from a normal distribution with mean -0.25, and variance 1 (the black distribution).

The expected fitness of each of the 16  “genotypes” of ABCD is shown below.

The following figure depicts the result of querying the fitness function with randomly generated bitstrings of length .

The next figure shows the locations of the pivotal loci A, B, C, D in this hypothetical scenario.

Consider the problem of classifying loci as pivotal or non-pivotal given only query access to the fitness function. This problem is closely related to the problem of finding the effective attributes of a parity function studied by Uehara et al.  [1,2]. One big difference is the presence of a stochastic element in the problem currently under consideration. Before reading further, I invite you to think of an algorithm that can solve this problem relatively robustly (with not more than, say, a 0.005 chance of misclassification per locus).

The naive approach would be to adopt a scanning strategy in which all combinations of four loci are visited. (Observe that visiting loci in combinations of three or less will not work). Suppose = 500,000; that’s approximately 6.25×10²² combinations. Even if it were possible to visit a million combinations per second, it would still take approximately two billion years to visit all such combinations.

It turns out that a genetic algorithm can be used to tackle this problem far more cheaply.

Suppose = 200 and pivotalLoci = [7 90 131 198]. The animation below shows the behavior of a simple genetic algorithm with uniform crossover (described in the materials and methods section of my dissertation) on the fitness function just described. Each frame in this animation shows the one-frequency of each locus (i.e. the frequency of the bit 1 at each locus) in a single generation. By extension, the frequency of the bit 0 at each locus is also on display. The red dots mark the positions of the pivotal loci. The blue dots mark the positions of non-pivotal loci.

Observe that the red dots diverge to the top or bottom of the plot, whereas the blue dots remain in the middle. After 500 generations, the location of the pivotal loci A, B, C, and D can simply be “read off” by examining the one-frequency of each of the 200 loci in the final population. Note also that the genotype of ABCD that goes to fixation is 1011. This genotype has odd parity, which explains the increase in the average fitness of the population shown in the following figure.

Now for the two-part punchline. First, there is nothing special about the location of the four pivotal loci. In other words, the expected number of generations required for the four red dots to diverge to the top or bottom of the plot remains the same regardless of the location of these dots. Second, there is nothing special about = 200. In other words, the behavior of the red dots will be unaffected by the number of blue dots present, and each blue dot will have the same behavior regardless of the total number of blue dots. Both these conclusions can be arrived at by appreciating the symmetries present when is applied to the fitness function described above (or rather, to the class of fitness functions described). Readers looking for a more rigorous treatment are referred to chapter 3 of my dissertation.

The animation below shows the result of applying to the fitness function when = 1000. The array pivotalLoci remained unchanged at [7 90 131 198].

The average fitness of the population over 500 generations is shown below.

The final animation shows the result of applying to the fitness function when = 10000. In this case, the array pivotalLoci was set to [2000 2681 6892 9520].

The average fitness of the population over 500 generations is shown below.

Note that in keeping with the assertions made above, the number of generations it takes for the red dots to diverge remains approximately the same despite an increase in by two orders of magnitude.

I hope the experience of watching evolutionary computation in action sparks your curiosity about the generative fixation hypothesis. Feel free to email me your questions and comments.

Click here to see the Matlab script used to generate the results presented above.

[1] Uehara, Tsuchida, and Wegener. Optimal attribute-efficient learning of disjunction, parity and threshold functions. In EUROCOLT: EUROCOLT, European Conference on Computational Learning Theory, EuroCOLT,. LNCS, 1997.

[2] Uehara, Tsuchida, and Wegener. Identication of partial disjunction, parity, and threshold functions. TCS: Theoretical Computer Science, 230, 2000.

Download the latest version of SpeedyGA (version 1.3) from Matlab Central’s File Exchange or from Google Code.

Version 1.3 of speedyGA is faithful to the specification for a simple GA given on page 10 of Melanie Mitchell’s book “Introduction to Genetic Algorithms”; that is if none of the bells and whistles (mask repositories, stochastic universal sampling, and sigma scaling) are used. SpeedyGA has also been changed from a function to a script. This makes it easier to inspect variables and plot data after a run has completed.

Here’s a visualization of the bit dynamics of an SGA with 1pt crossover when applied to the the Royal Roads fitness function. Going by the building block hypothesis one expects to see the dots marching orderly to the top of the plot in groups of eight or more.

That’s not what happens. Instead, one gets to see hitchhiking in action—look for a swift downward movement of certain dots in tandem with the swift upward movement of other dots at close by loci.

The maximum and average fitness in each generation of this run are shown below

The matlab code used to generate these and other figures in this blog post can be found here.

Let’s visualize the bit dynamics of a population when an SGA with uniform-crossover is applied to the Royal Roads function.

The maximum and average fitness in each generation of this run are shown below

Finally, let’s visualize the bit dynamics of a mutation-only SGA on the Royal Roads function.

The maximum and average fitness in each generation of this run are shown below

As an exercise you might examine the bit dynamics of the SGA on the oneMax fitness function included in the distribution.

Comments welcome.

Enjoy!

Posted in genetic algorithms, visualization Tagged: empirical

Download SpeedyGA from Matlab Central’s File Exchange or from Google Code.

Release notes:

Download manuscript

In a draft of one of my dissertation chapters I identify a hard computational problem and show that a GA can solve it robustly and scalably. Remarkably, this problem is closely related to a hairy statistical problem in computational biology. How might a GA leverage this kind of computational ability to perform adaptation? I’ll be presenting my theory about this in future chapters. The idea behind this theory is delightfully simple. Presenting it formally, however, is a another story. Stay tuned.

See http://code.google.com/p/vector-ga/

vectorGA is written in Matlab, which is a proprietary platform. The first order of business is to port vectorGA to the open-source platform Octave so that everyone can use it.

http://arxiv.org/abs/0711.1401

The biosphere is replete with organisms that are exquisitely well adapted to the environmental niches they inhabit. Natural sexual evolution has been crucial to the generation of what are arguably the most highly adapted of these organisms — cheetahs, owls, humans etc. A deeply intriguing idea is that we can build adaptation algorithms which, at an abstract level, mimic the behavior of natural sexual evolution, and in doing so, “harness” something of the adaptive power of this incredibly effective process. But what is the abstract level at which natural sexual evolution should be mimicked? In other words given everything we know about natural sexual evolutionary systems, how do we distinguish between aspects of these systems which are essential to their adaptive power, and those which can be viewed as “mere biological detail” and need not be simulated, especially when taking a first swipe at building an artificial evolutionary systems which harnesses the power of natural sexual evolution? For instance is it necessary for such “firstorder” artificial evolutionary systems to simulate hydrogen bonding between the bases of DNA strands? What about diploidy, or the way genomes of organisms are comprised of multiple chromosomes? And how crucial is the fact that crossover takes place between homologous chromosomes during meiosis?

Engineers of yesteryear faced a similar quandary when trying to ascertain just what it is about birds that gives them their capacity for flight? Stories of inventors in feathered suits jumping to their deaths off cliffs and buildings bear testament to the fact that that our initial answers to such questions are often incorrect. It was only after the realization that birds were “using” Bernoulli’s principle to stay aloft that researchers began to make any real progress towards building machines that successfully harnessed the principle underlying birds’ capacity for soaring. One can infer the following general rule from this example: without a good understanding of exactly why a natural system exhibits a certain useful phenomenon, efforts to build artificial systems that exhibit the same phenomenon by mimicking the natural system will be misguided and are unlikely to be successful.

The field of Population Genetics stems from the efforts of its founders — Fisher, Wright and Haldane — to reconcile Darwin’s theory of adaptation by natural selection with Mendel’s theory of genetics (odd as it may now seem, these two theories were once thought to contradict each other. See Okasha, 2006). The literature of this field seems like the most reasonable place to look for answers to questions about how and why adaptation occurs in natural sexually reproducing populations. Unfortunately Population Genetics does not hold ready answers to these questions. Indeed the differing theories of Fisher and Wright regarding exactly this issue has been the subject of a longstanding and ongoing debate (Wade and Goodnight, 1998; Brodie III, 2000). Significant empirical evidence has been gathered by both sides in support of their respective positions yet a definitive answer has not emerged. The absence of a definitive answer makes it difficult to make principled decisions about the level of detail which must be present in an artificial system which seeks, through mimicry, to harness something of the adaptive power of natural sexual evolution.

Let us return to the analogy with the field of aviation that we introduced above. For the sake of argument let us suppose that in the age before the discovery of Bernoulli’s principle someone had, for some reason or the other, succeeded in inventing a simple winged machine which a) mimicked birds at some relatively abstract level, and b) was capable of soaring long distances (even if slowly, or inefficiently). Such a machine would immediately be incredibly interesting because when compared to the complex body of a bird, such a machine would be much more amenable to analysis. The principle underlying this machine’s ability to soar, once derived, could then be used to build “better” soaring machines. This underlying principle would also play a very important part in the development of an accurate theory of why birds can soar (If the inventor of the winged machine gives an incorrect reason for why his machine can soar, that would probably slow down the progression described above, but it would take away nothing from the importance of the winged machine itself.). The implications of this vignette for the importance of the SGA to the fields of Evolutionary Computation and Evolutionary Biology should be evident. The SGA should thus be viewed as a ‘lucky break’, one that can and should be exploited for its potential to advance theories and applications of the adaptive capacity of sexual evolution.

Let us spell out the importance of studying the SGA’s capacity for adaptation. As models of sexual evolutionary systems go, the SGA is arguably the simplest one which has regularly been observed to adapt high-quality solutions despite the almost certain presence of non-trivial epistasis between genomic loci, in other words, despite its application to problems whose representations are in all likelihood riddled with local optima. Because of its effectiveness in spite of its simplicity, the SGA is a model of sexual evolution that is highly likely to a) yield an explanation for the incredible adaptive capacity of sexual evolution and b) precipitate the identification of classes of non-trivial epistasis which do not pose much of a problem for sexual evolution. The SGA is of course not the last word in evolution inspired adaptive systems. Efforts to extend this algorithm in ways that “increase its adaptive power” should and have been made. However if, while attempting to extend the SGA, one works within a flawed paradigm, one is unlikely to capitalize on, and may even compromise, whatever “power” the SGA derives, by virtue of imitation, from natural sexual evolution. A non-dogmatic study of how SGAs perform adaptation has the potential to yield a theory which accurately explains the reasons behind the SGA’s frequent success. Such a theory will probably usher in a new paradigm within which fruitful research into the construction of more “powerful” evolutionary algorithms can proceed. If such a theory differs significantly from those of Fisher and Wright it is likely to have deep implications for the field of Population Genetics and the larger field of Evolutionary Biology (within which several basic questions — why sex? Why punctuated equilibrium? Why diploidy and polyploidy? Why speciation? What is the unit of selection? — have yet to receive satisfactory answers). Finally, a study that reveals the SGA’s capacity for adaptation will also undoubtedly reveal classes of problems that SGAs can efficiently solve. These classes of problems may prove useful to machine learning researchers in their efforts to find semiprincipled reductions of difficult learning problems to problems for which robust and efficient solvers exist.

This paper makes two concrete contributions. Firstly, in sections 2–10 we have derived results which we believe are relevant to the riddle of the SGA’s capacity for adaptation. Secondly in section 11 we have presented results which show that an oft perceived shortcoming of the SGA is misplaced. The following two paragraphs expand upon these contributions.

As we discussed in section 1.2, a promising way to understand the effect of selection and variation on the composition of the evolving population of an SGA is by understanding the multi-generational effect of these operations on the search distribution of the SGA. One way to study an evolving high-dimensional distribution is to study its evolving multivariate marginals. In this paper we derive conditions under which a multivariate marginal of the search distribution of an SGA, with an infinite population of long genomes, can be approximated over multiple generations. In other words we derive conditions under which the frequency dynamics of some family of schemata under the action of an SGA, with an infinite population of long genomes, can be approximated over multiple generations. The conditions we derived in this paper are much weaker than those derived by Wright et al. (2003). This makes our result more useful. The conclusions reached in section 9 are reached by making small leaps of intuition. In section 10 we experimentally validated these conclusions. Our validation, though indirect, is based on assumptions and modeling decisions which are, in our opinion, uncontroversial.

Besides being incorrectly used to support an outlandish hypothesis about what the SGA can do (hierarchical building block assembly), Holland’s Schema Theorem has also heavily shaped opinion about what the SGA cannot do. Following the experiments by Mitchell et al. (1992), and Forrest and Mitchell (1993), the perceived abilities of the SGA stand compromised, yet the perceived limitations of the SGA have remained unchanged. The SGA is currently thought to be incapable of increasing the frequency of a low-order schema with higher than average fitness when the defining length of that schema is large (Goldberg et al., 1989; Goldberg, 2002). In section 11 we argued that this perception is just plain wrong — an SGA can increase the frequency of a low-order schema with higher than average fitness even when the defining length of that schema is large.

In closing we briefly mention that we have recently obtained a new, simple, and (in our opinion) satisfying theory which explains the SGA’s remarkable capacity for adaptation. We have also identified a class of hard statistical problems such that a) the problems in this class can be solved efficiently and robustly by an SGA, and b) this class is likely to be a useful target of machine learning reductions. All of this will soon appear in our forthcoming dissertation. Our theory relies crucially on the SGA’s ability to increase the frequency of schemata of low order and above average fitness, even when the defining bits of those schemata are widely dispersed.

Perceptions of the abilities and limitations of the SGA (and hence the kinds of problems that it can and cannot solve) have been heavily influenced by a theory of adaptation called the building block hypothesis (Goldberg, 1989; Mitchell, 1996; Holland, 1975, 2000). This theory of adaptation has its genesis in the following idea: maybe small groups of closely located co-adaptive alleles propagate within an evolving population of genomes in much the same way that single adaptive alleles do in Fisher’s theories of sexual evolution (Fisher, 1958). Holland called such groups of alleles building blocks. This idea can be taken one step further: maybe small groups of co-adaptive building blocks propagate within an evolving population of genomes in much the same way that single building blocks do. Such groups can be thought of as higher-level building blocks. Pursuing this idea to the fullest extent, maybe co-adaptive groups of higher-level building blocks propagate in much the same way as ordinary building blocks do to yield building blocks of an even higher level, and so on and so forth in hierarchical fashion with the building-blocks of higher levels being comprised of co-adaptive groups of lower-level building blocks. Let us call this this idea hierarchical building block assembly.

Holland (1975) saw in hierarchical building block assembly a way out from the problem that epistasis (Wolf et al., 2000) poses for Fisher’s theory of sexual evolution. He also believed that hierarchical building block assembly, if implemented efficiently, could serve as a useful problem solving technique. He argued that a genetic algorithm that he called a genetic plan can implement hierarchical building block assembly, and moreover does so efficiently. He offered the genetic plan as a model of natural sexual evolution and also as a useful technique for finding solutions to adaptation problems with non-convex objective functions. The main theoretical tool that he used in his argument has come to be called the schema theorem (Goldberg, 1989; Mitchell, 1996). However neither the schema theorem, nor any of Holland’s other theoretical analyses fully support his claim that simple genetic algorithms are capable of efficiently implementing hierarchical building block assembly. Given the boldness of his claim and the large leaps of intuition that Holland makes in order to support it, the absence of experimental support in (Holland, 1975) is rather conspicuous (even more so given that simple, computationally unintensive, proof-of-concept experiments are not difficult to conceive of. See, Mitchell et al., 1992, and Forrest and Mitchell, 1993) . It would not have been surprising therefore if the genetic plan had been relegated to the history books as an algorithm that did not fulfill its raison d’etre — to support its inventor’s hunch about the utility of hierarchical building block assembly as a theory of adaptation for natural sexual evolutionary systems, and to support its inventor’s hunch that hierarchical building block assembly can be efficiently implemented. What seems to have saved the SGA from this fate is the curious matter of its utility.

In the years following the publication of Holland’s seminal work (Holland, 1975), the SGA was successfully used to adapt high-quality solutions to different sorts of real world and toy problems with non-convex objective functions. In an unfortunate twist of reasoning hierarchical building block assembly became the de-facto explanation for the success of the SGA. This explanation came to be called the building block hypothesis. Despite its name, the building block hypothesis was treated more as an assumption than as a hypothesis. Hierarchical building block assembly had aesthetic appeal, and the building block hypothesis had Holland’s unqualified endorsement (Holland, 1992). Therefore the building block hypothesis was readily accepted by most within the GA community. Some even went so far as to tout the success of SGAs as evidence of the veracity of the building block hypothesis or as evidence that hierarchical building block assembly is a useful search technique for a wide variety of search problems. Consider the following confused passage from one of the first text books on genetic algorithms:

“…the building block hypothesis has held up in many different problem domains. Smooth, unimodal problems, noisy multimodal problems, and combinatorial optimization problems have all been attacked successfully using virtually the same reproduction-crossover-mutation [S]GA.”(Goldberg, 1989)

The early support that the building block hypothesis enjoyed accounts for the deep impact it has had and continues to have on the course of research in genetic algorithms as well as other fields of evolutionary computation such as genetic programming. Recently the building block hypothesis has been sharply criticized for lacking adequate theoretical support. The most forceful criticism that we are aware of has been levied by Wright et al. (2003): “The various claims about [S]GAs that are traditionally made under the name of the building block hypothesis have, to date, no basis in theory, and, in some cases,are simply incoherent”. On the empirical side experimental results have been obtained which straightforwardly cast doubt upon the ability of a simple genetic algorithm to efficiently implement hierarchical building block assembly (Mitchell et al., 1992; Forrest and Mitchell, 1993). In response to these experimental results a silent transition has occurred within the field of genetic algorithms: hierarchical building block assembly has gone from being thought of as the abstract process that SGAs implement to being thought of as a normative process that SGAs mis-implement. Even though this transition between intellectual positions is completely specious it is now widely assumed that SGAs work because they manage to “fudge” hierarchical building block assembly. Many new genetic algorithms have been constructed to compensate for the perceived short-comings of the GA —e.g. messy GA, (Goldberg et al., 1989; Goldberg, 1989, 2002), LLGA (Harik and Goldberg, 1997; Goldberg, 2002), CGA (Harik et al., 1999), ECGA (Harik, 1999), cohort GA (Holland, 2000), FDA (M¨uhlenbein and Mahnig, 1999), LFDA (M¨uhlenbein and Mahnig, 2001), BOA (Pelikan et al., 1999; Goldberg, 2002), hBOA (Pelikan and Goldberg, 2001), SEAM (Watson, 2002, 2006),etc. The inventors of these algorithms claim, or at least imply, that their algorithms are better than the SGA at its own game — hierarchical building block assembly. In many circles within the GA community the curious matter of frequent utility of SGAs is now considered closed.

For a case in point of the kind of sleight of hand that we are discussing consider the following: conceding that there is little evidence that SGAs can efficiently and robustly implement hierarchical building block assembly, Holland (2000) remarks, “Are [S]GA’s, then, a robust approach to all problems in which building blocks play a key role? By no means! After years of investigation we still have only limited information about the [S]GA’s capabilities for exploiting building blocks”. Later he asserts that “the very essence of good GA design is retention of diversity, furthering exploration, while exploiting building blocks already discovered”, and presents a new genetic algorithm, the Cohort Genetic Algorithm, and argues that it implements this essence (see (Pei and Goodman, 2001) for evidence that it does not).

The field of genetic algorithms is both a scientific field as well as an engineering domain. Heedful science and meticulous engineering can often work synergistically. However when the boundary between science and engineering begins to blur, dogma and misplaced faith can beleaguer the practice of both, to wit, a system that is useful in practice, but does not implement a hypothetical mechanism may receive reduced attention, whereas the mechanism, far from being dismissed according to the basic norms of science may become the holy grail of the engineering goals of the field.

A theory that explains why a system exhibits a particular behavior can influence perceptions of how the system can behave, and also of how it cannot. Of the two kinds of perceptions, the latter kind is often judged in retrospect to be the greater impediment to the discovery of a new theory that can explain and predict the behavior of the system with greater accuracy. This is because by influencing perceptions of how the system cannot behave a theory implicitly determines the “domain of the impossible” and in doing so it steers researchers away from considering certain possibilities. Yet it is precisely amongst these ”impossibilities” that the seeds of a new more accurate theory often lie.

One of the two goals of this paper is to challenge the widespread belief that the SGA cannot increase the frequency of a low order schema with above-average fitness when the defining length of that schema is high (i.e. when the defining bits of that schema are widely dispersed). This belief can be traced back to Holland’s original treatise on genetic algorithms (Holland, 1975) and goes hand in hand with belief in the building block hypothesis (and variations thereof). In section 11 we provide an argument based on experimental evidence that this belief is misplaced. We believe that this errant belief will be judged in retrospect to have been a significant impediment to the discovery of a sound theory of adaptation for the SGA.

The practice of Machine Learning research can be characterized as the effective semiprincipled reduction of learning problems to problems for which robust and efficient solution techniques exist – ideally ones with provable bounds on their use of time and space. In a recent paper Bennett and Parrado-Hern´andez (2006) describe the synergistic relationship between the fields of machine learning (ML) and mathematical programming (MP). They remark:

“Optimization lies at the heart of machine learning. Most machine learning problems reduce to optimization problems. Consider the machine learning analyst in action solving a problem for some set of data. The modeler formulates the problem by selecting an appropriate family of models and massages the data into a format amenable to modeling. Then the model is typically trained by solving a core optimization problem that optimizes the variables or parameters of the model with respect to the selected loss function and possibly some regularization function. In the process of model selection and validation, the core optimization problem may be solved many times. The research area of mathematical programming theory intersects with machine learning through these
core optimization problems” (Bennett and Parrado-Hern´andez, 2006).

Later Bennett and Parrado-Hern´andez imply that when the targets of ML reductions have been optimization problems, they have for the most part been the convex optimization problems within the MP pantheon.

“Convexity plays a key role in mathematical programming. Convex programs minimize convex optimization functions subject to convex constraints ensuring that every local minimum is always a global minimum. In general, convex problems are much more tractable algorithmically and theoretically. The complexity of nonconvex problems can grow enormously. General nonconvex programs are NP-hard.” (Bennett and Parrado-Hern´andez, 2006).

The close relationship between ML and MP arguably exists because MP provides ML with a set of crisp, well-defined problems along with algorithmic solvers that come with guarantees on their use of time and space. To state this using metaphors from software engineering, the well-defined convex optimization problems are interfaces that MP publishes, and the provably efficient and robust algorithmic solvers of MP implement these interfaces.

Let us differentiate, in this paper, between optimization and adaptation. We define optimization as the procurement of one or more points of optimal or close-to-optimal value, and adaptation as the generation of points of increasing value over time. Given this definition, to say that the target problems of Machine Learning reductions are optimization problems is to fudge the truth somewhat. While the Mathematical Programming community indeed seems to be almost completely concerned with the procurement of optimal or close to optimal points, ML researchers aren’t interested in optimization per se but in the means by which it is achieved in most MP algorithms, i.e. adaptation. In fact optimization is often prevented in machine learning algorithms – using a “technique” named early-stopping – to prevent overfitting. In other words, robust, efficient adaptation is the modus operandi of most convex optimization algorithms, and for the most part, it is this modus operandi that makes these algorithms interesting to Machine Learning researchers.

The interface-problems published by the MP community give ML researchers useful targets to hit; if a ML researcher works out a semi-principled reduction of a class of learning problems to one of MP’s interface-problems, there are off-the-shelf algorithms within MP which allow her to quickly test whether her reduction is effective. Because of the emphasis that the ML community place on guarantees of robustness and efficiency. when the targets of ML reductions have been optimization problems, they have for the most part, been restricted to being convex optimization problems within MP. These problems are rather simple as adaptation problems go – every local optimum is also a global optimum, or stated differently, there are no local optima. Rather heroic feats of ingenuity are therefore necessary in order to obtain effective semi-principled reductions
of hard problems to these simple optimization problems. The difficulty of obtaining such
reductions is currently a fundamental limitation on the pace of progress within ML.

The SGA (Mitchell, 1996) is an adaptation algorithm which mimics natural sexual evolution. It has been directly applied to a large number of hard real-world problems and has often succeeded in generating solutions of remarkably high-quality. To be sure, some amount of thought is required to “massage” these problems into a form which allows the SGA to operate successfully on them (e.g. choices must be made about the fitness function used and the way solutions are encoded as bitstrings), but unlike the case in machine learning this massaging is largely ad-hoc, an outcome more of trial and error than principled reasoning. The resulting problems are almost certainly hard ones (non-convex), with objective functions that are riddled with local optima. It is a testament to the adaptive power of the SGA that it nevertheless often produces solutions of remarkably high quality. Given these successes one might expect a great deal of interest in SGAs from the machine learning community. That this is not the case speaks to an unfortunate shortcoming of GA research. There is no dearth of one-off problems that SGAs have adequately solved. However GA researchers have yet to publish a single class of problems such that a) SGAs are likely to perform robust, efficient adaptation when applied to problems in this class, and b) the class is likely to be useful as the target of ML reductions. For the sake of brevity we will loosely define such a class of problems as an SGA-Easy/ML-Useful class. We believe that when such problem classes are found the ML community will begin to take a greater interest in GA research. The future relationship between the GA and ML communities might then be similar to the one that currently exists between MP and ML. As mentioned above SGAs commonly adapt high-quality solutions to problems which are almost certainly contain large numbers of local optima. It is reasonable therefore to suspect that there exists an SGA-Easy/ML-Useful class of hard non-convex problems and that the identification of this class will significantly ease the burden of obtaining novel ML reductions. We believe that the identification of such a problem class will go hand in hand with the discovery of a theory which can give a satisfying explanation of the adaptive capacity of the SGA. Such a theory does not currently exist.

In this paradigm, each post-selection population is thought to harbor “good” genetic material. Recombination operators, and estimation of distribution procedures, are thought drive adaptation by composing this material to produce good or better individuals in the next generation.When adaptation stalls it is thought to be because “good” genetic material is unavailable, or because recombination of this material was not performed effectively.

Let us call this general set of beliefs the Compositional Paradigm. This paradigm draws its support from Holland’s Building Block Hypothesis. Its widespread acceptance in the GA community signals the implicit acceptance of the BBH as an accurate description of how the simple GA performs adaptation. It also signals the acceptance of a generalization of the BBH, namely that all varieties of recombinative GAs perform adaptation by recombining pe-adapted genetic material.

The Building Block Hypothesis however has never been conclusively validated. It has in fact been sharply criticized by a few (e.g. see section 5 of Wright et. al. Implicit Parallelism, GECCO 2003), and many have suggested that it be treated with more skepticism. Nevertheless, it and its generalization – what I call the Compositional Hypothesis – remain widely accepted. I believe that this is so because of two reasons:

1. The Compositional Hypothesis seems right. Recombination of genetic material occurs at the individual level, so the recombination of “good” genetic material must be the mechanism by which adaptation occurs at the system level. Right?
2. Real paradigmatic scrutiny only occurs when an alternate hypothesis is submitted for consideration. Despite the publication of a large number of studies of the foundations of genetic algorithms no alternate hypothesis that explains adaptation has been proposed so far.

In my own studies I’ve been fortunate to obtain theoretical and empirical results that suggest an alternate non-compositional hypothesis for adaptation in selecto-recombinative genetic systems. Right now I’m in the process of finding the best way to present it.

Bennett and Parrado-Hernandez remark:

“Optimization lies at the heart of machine learning. Most machine learning problems reduce to optimization problems. Consider the machine learning analyst in action solving a problem for some set of data. The modeler formulates the problem by selecting an appropriate family of models and massages the data into a format amenable to modeling. Then the model is typically trained by solving a core optimization problem that optimizes the variables or parameters of the model with respect to the selected loss function and possibly some regularization function. In the process of model selection and validation, the core optimization problem may be solved many times. The research area of mathematical programming theory intersects with machine learning through these core optimization problems”

and later,

“Convexity plays a key role in mathematical programming [MP]. Convex programs minimize convex optimization functions subject to convex constraints ensuring that every local [optimum] is always a global [optimum]. In general, convex problems are much more tractable algorithmically and theoretically. … General nonconvex programs are NP-hard”

This paper lends support to my sense that the optimization techniques used in machine learning are largely limited to convex optimization algorithms. Successes in machine learning have so far been a result of the application of human ingenuity to carry out effective semi-principled reductions of certain classes of problems to convex optimization problems. I use the qualifier semi-principled because there is usually some solid rationale behind an ML reduction that leads one to expect that the reduction makes the original problem amenable to solution by convex optimization techniques. This however is never formally proven. i.e. the reduction is not formal in the sense that the traveling salesman problem formally reduces to SAT. In all likelihood the optimization problems that the redutions yield are non-convex. However in practice, the application of convex optimiaztion techniques to these optimization problems often gives satisfactory solutions to the original problems. Hence the qualifer effective.

For example researchers in the support vector machine community use the kernel trick in an effective semi-principle reduction of classification and regression problems to a class of convex optimization problems called quadratic programming problems. These problems are then solved using quadratic programing algorithms such as Platt’s SMO algorthm.

To summarize the discussion so far: convex optimization problems are a class of optimization problems in which every local optimum is also a global optimum. Many useful algorithms for efficiently solving convex optimization problems have been constructed by the mathematical programming community. One of the reasons for the success of Machine Learning techniques is the use of human ingenuity to achieve effective semi-principled reductions of certain classes of problems to convex optimization problems for which there exist efficient solvers within MP.

Another reason, as authors mention, is that machine learning researchers have been tweaking the convex optimization algorithms of mathematical programming to create new algorithms which are better suited for their purposes. “Mathematical Programming puts a premium on accuracy, speed, and robustness. Since generalization is the bottom line in machine learning and training is normally done off-line, accuracy and small speed improvements are of little concern in machine learning. Machine learning prefers simpler algorithms that work in reasonable computational time for specific classes of problems” [1]. The paper later mentions that optimization is often cut short (!) in Machine Learning to prevent overfitting This “technique” is called early-stopping. And later ‘Thus not only is “good” optimization not necessary, but “bad” optimization algorithms can lead to better machine learning models’.

Here are some reformulations of this paper and some of my thoughts.

Let me differentiate between optimization and adaptation. I’ll define optimization as the procurement of one or more points of optimal or close-to-optimal value, and adaptation as the generation of points of increasing value over time. Things become quite clear in light of this distinction. Mathematical Programming seems to be almost completely concerned with optimization whereas machine learning seems to require relatively quick adaptation.

The love affair between mathematical programming and Machine learning exists because MP provides the world with a set of crisp, well-defined problems — e.g. a linear programming problem, a quadratic programming problem, a second-order cone programming problem, a semdefinite programming problem, a semi-infinite programming problem (see the appendix in [1] for a description of each of these convex optimization problems) — and then provides programmatic techniques for solving those problems. To state this using metaphors from software engineering (particularly object oriented programming), these well-defined problems are the interfaces that MP publishes, and the efficient algorithmic solvers within that field implement these interfaces.

The interface-problems published by the MP community give ML researchers a rough target to hit in their efforts to carry out effective semi-principled reductions of difficult problems to ones for which efficient solvers exist. The interface-problems of MP are similar to the optimization problems that ML researchers ultimately end up with, but there are important differences: for example, MPers seek fast optimization whereas MLers merely want a high degree of adaptation, and have additional requirements such as scalability. Nevertheless, if a ML researcher works out a semi-principled reduction of a class problems to one of MP’s interface-problems, there are off-the-shelf algorithms within MP which allow her to quickly test whether her reduction is effective. MLers put great stock in the fact that each convex optimization algorithm in the MP pantheon has robust performance over its problem class and is theoretically proven to converge to the optimum.

Machine learning researchers spend their time trying to achieve effective reductions of difficult problems to problem classes for which efficient and robust solvers exist. An effective reduction may be a multistep affair (effectively reduce problem A to problem B, effectively reduce problem B to problem C, ….) but it most often bottoms out in an effective reduction to an optimization problem, most often a convex optimization problem. Different techniques and tricks may be used at each step of an effective reduction (e.g. probabilistic models, the kernel trick, etc.).

Convex optimization problems are rather easy as optimization problems go. Therefore a large amount of human ingenuity is required to carry out effective semi-principled reductions of classes of real world problems to convex optimization problems. These reductions often involves the use of “heavy machinary” from computer science, mathematics and statistics.

It seems to me that fundamental advances by the machine learning community are currently being limited by two factors (besides computational speed). The first is the sheer level of human ingenuity that is required to realize novel reductions of real world problems to convex optimization problems. The second is that the targets of these reductions have so far been limited to convex-optimization problems. The first factor cannot be changed. With regard to the second, I believe that the field of evolutionary computation may have a lot to contribute . Selecto-recombinative evolutionary algorithms are widely believed to perform efficient adaptation even on non-convex problems. Unfortunately there are currently no crisp descriptions of the classes problems that such algorithms can efficiently ’solve’. I believe that if and when such interface-problems are determined the ML community will take much greater interest in Evolutionary Computation. EC might then play the same role w.r.t. ML that MP currently plays, i.e. ML researchers might then seek effective semi-principled reductions of real-world problems to interface-problems in the EC pantheon.

References:
[1] Bennett, K.P., Parrado-Hernandez, E. The interplay of optimization and machine learning research, Journal of Machine Learning Research 7 (2006) 1265-1281

Matlab is optimized for performing operations on arrays. Loops, especially nested loops, tend to run slowly in Matlab. It is possible to significantly improve the performance of Matlab programs by converting loops into array operations. This process is called vectorization. Matlab provides a rich set of functions and many expressive indexing schemes that make it possible to vectorize code. Such code not only runs faster, it is also shorter, and simpler to understand and change (provided that you know a little about Matlab of course).

Genetic Algorithms that are implemented in C/C++ or Java typically have multiple nested loops. Therefore direct ports of such implementations to Matlab will run very slowly. Many of the nested loops found in a typical GA implementation have been eliminated from VectorGA. The resulting code is short, fast and simple. It is a neat coincidence when the constructs of a programming language match a programming task so well that a program can be written this succintly.

VectorGA is proof that Matlab is a useful language for the rapid prototyping of Genetic Algorithms. This, in addition to Matlab’s extensive data visualization capabilities, make Matlab an extremely useful platform for the experimental analysis of GAs.

VectorGA has been created and tested under Matlab 7 (R14). Out of the box it evolves a population against the one-max fitness function. The royal-roads fitness function has also been included but is not currently being called. If you find vectorGA useful or find any bugs please let me know.

Enjoy!

Original vectorGA release site

The latst version of VectorGA is SpeedyGA

In a sense, we know exactly what GAs ‘do’ (the algorithm after all is open for inspection). This is the same sense in which it can be said that we know what the set of assembly language instructions of a large executable application (e.g. Firefox), will ‘do’. In order to understand why the application exhibits some particular behavior, it is typically necessary to have a description of the behavior of the application in terms of concepts that are more abstract than those associated with assembly language. Similarly, in order to understand how genetic algorithms perform adaptation (i.e. to obtain a theory of adaptation for GAs) I believe that it will be necessary to express the behavior of a GA at a higher level of abstraction than the one at which the algorithm is specified. Chikofsky and Cross write that “reverse engineering is the process of analyzing a subject system to create representations of the system at a higher level of abstraction” [1]. Given this definition, another way of expressing the above is to say that in order to obtain a viable theory of adaptation for GAs it is necessary to reverse-engineer this class of algorithms.

Experimentation is very important to the reverse-engineering process. Information about the high-level description of some system can be gleaned by performing well-conceived experiments and by making inferences based on the results. Well-chosen experiments can also be used to resolve disputes between competing hypothetical high-level descriptions of the system.

It is also very useful to know the abstract constructs that may have been used at each level of abstraction involved in the the forward engineering of the system. In the software domain, reverse engineering is oftentimes an iterative process that produces descriptions of a system at higher and higher levels of abstraction. For instance when reverse engineering the bytecode of some Java application, a first step is to reverse engineer the application to the level of Java code. A subsequent step is to reverse engineer the application to the level of design patterns (e.g. factory, flyweight, façade etc.)

Genetic Algorithms however are not the end-products of human engineering, so unfortunately we have no apriori knowledge about what abstract concepts may be invoked in adaptation-relevant higher-level descriptions of their behavior. Adaptation focused mathematical analyses of genetic algorithms are based on the implicit assumption that concepts within mathematics will prove useful in an adaptation-relevant high-level description of GA behavior. This assumption is reasonable. Mathematics has proven to be an extremely useful framework within which to derive and express useful descriptions of natural systems. Yet, when it comes to the analysis of complex systems like GAs, mathematical approaches may be limited. It is not just that current mathematical techniques might not allow for the derivation of adaptation-relevant high-level mathematical descriptions of GA behavior, but that such descriptions might be best expressed algorithmically (i.e. programmatically) rather than mathematically (i.e. declaratively)

References:
[1] Chikofsky, E.J.; J.H. Cross II (January 1990). “Reverse Engineering and Design Recovery: A Taxonomy in IEEE Software“. IEEE Computer Society: 13–17