Though not as ubiquitously recognizable as hierarchical assembly is, generative fixation underlies progress in many areas. Like in the video industry for instance, which only really took off once the VHS/Betamax war had run its course. The “fixation” of VHS within this industry in essence generated new opportunities for advancement. For example, it permitted the development of the video rental business, which brought films from the back rooms of studio houses into living rooms everywhere. Had the tussle between VHS and Betamax continued, many of these opportunities might not have presented themselves. The economics of supporting two formats simultaneously would have been economically crippling for the fledgling industry.

VHS v. Betamax was a case where two contestants competed for dominance along a single dimension—the format accepted by video players. Consider a scenario consisting of multiple dimensions and multiple contestants in each dimension. Suppose, moreover, that no contestant in any dimension is outright superior. That is, the superioriority/non-superiority of the contestants in each dimension is dependent on the state of the contests in the other dimensions. This scenario describes the most commonly arising situation in natural evolution. Here, the “dimensions” are genetic loci, and the “contestants” are alleles. (Statistically, the scenario just described is more likely than one where one or more locus has an allele that is superior regardless of the frequencies of the alleles at other loci.)

The danger in such cases is that progress will stall because no contestant will come to dominate its dimension. The generative fixation hypothesis holds that in a system undergoing recombinative evolutionary dynamics, progress will continue. Although no locus has an allele that is outright superior, a small number of  alleles belonging to different dimensions that play nice together (i.e. confer above average fitness as a group) will come to dominate their respective dimensions. In doing so, they will set the stage for the next multi-dimensional contest over the remaining dimensions. And so on.

I’ve demonstrated the genetic algorithm’s ability to scaleably identify and fix synergistic sets of unlinked non-competing alleles in a recent manuscript and in my dissertation.

Manuscript: http://s3.amazonaws.com/burjorjee/www/hyperclimbing_hypothesis.pdf

Abstract:
We submit the hyperclimbing hypothesis—an explanation for adaptation in genetic algorithms with uniform crossover (UGAs). Hyperclimbing is a stochastic search heuristic that works by decimating a search space, i.e. by iteratively fixing the values of small numbers of search space attributes. Global decimation is known to be an effective way to approach large instances of hard constraint satisfaction problems. The hyperclimbing hypothesis holds that UGAs work by implicitly implementing efficient global decimation. Proof of concept for this hypothesis comes from the use of a novel analytic technique involving the exploitation of algorithmic symmetry. We also present experimental results that show that a simple tweak inspired by the hyperclimbing hypothesis significantly improves the performance of a UGA on an instance of Uniform Random MAX-3SAT . The hyperclimbing hypothesis suggests that other kinds of evolutionary algorithms may also work by implicitly implementing efficient global decimation.

Talk abstract

My slides

Click here to see the final version (single spaced for easy reading).

The foundations of most computer engineering disciplines are almost entirely mathematical. There is, for instance, almost no question about the  soundness of the foundations of such engineering disciplines as graphics, machine learning, programming languages, and databases. An exception to this general rule is the field of genetic algorithmics, whose foundation includes a significant scientific component.

The existence of a science at the heart of this computer engineering discipline is  regarded with nervousness. Science traffics in provisional truth; it requires one to adopt a form of skepticism that is more nuanced, and hence more difficult to master than the radical sort of skepticism that suffices in mathematics and theoretical computer science. Many, therefore, would be happy to see science excised from the foundations of genetic algorithmics. Indeed, over the past decade and a half, much effort seems to have been devoted to turning genetic algorithmics into just another field of computer engineering, one with an entirely mathematical foundation.

Broadening one’s perspective beyond computer engineering, however, one cannot help wondering if much of this effort is not a little misplaced. Clearly, as fields of engineering go, genetic algorithmics is not the exception—the foundations of most engineering fields include large scientific components. What seems to matter is, not the  existence of a science within the foundation of an engineering discipline, but the state of that science. The advanced state of physics and chemistry is, for example, a significant part of the reason for the advanced state of such fields as mechanical, chemical, civil, aeronautical and electrical engineering.

Historically, the blossoming of a field of engineering has typically had to await the maturation of certain underlying field(s) of science. Consider for a moment the improbability of  constructing an internal combustion engine based on the phlogiston theory of combustion. Even if one somehow succeeds in actually building a prototype, further advances within the rubric of phlogiston theory would probably be limited. Combustion engine engineering would be a black art.

I trust that the scenario just described will give users of genetic algorithms and would-be inventors of new genetic algorithms pause, and reason for hope. Pause because even after decades of research, “black art” about sums up the process of applying current genetic algorithms and inventing viable new ones. Hope because it is conceivable that just as Lavoisier’s oxygen based theory of combustion stimulated rapid advances in the construction of internal combustion engines, fundamental upheavals in the science of genetic algorithmics might stimulate rapid advances in the ways in which genetic algorithms are applied and improved.

Given the above, the following question seems to get at  the heart of the matter: What should a science of genetic algorithmics, one capable of stimulating advances in the construction and application of genetic algorithms, look like? I submit that such a science should be organized around the search for a minimal set of computational efficiencies possessed by the simple genetic algorithm such that when considered together these efficiencies explain the adaptive capacity of the simple genetic algorithm on a very broad range of fitness functions. Roughly, computational efficiencies should play the part played by scientific laws in the physical sciences. The challenge is to identify the minimal set with the widest possible explanatory power.

There are two important reasons for making the simple genetic algorithm the object of attention. The first is precedence. There already exists a well known body of science with this algorithm as its focus. This pre-existing work, specifically the theory that goes by the name of the building block hypothesis, provides a point of reference against which future theories may be compared. The second reason is biological plausibility. Unlike many genetic algorithms currently in use, the simple genetic algorithm contains no biologically implausible mechanisms and is, therefore, a legitimate model of sexually evolving biological populations. Such populations have been the subject of intense scientific scrutiny for well over a century and have generated an enormous amount of scientific work. This body of work can serve as a second point of reference.

The aforementioned outline for a science of genetic algorithmics is hardly novel. Until about the mid 1990s, the study of genetic algorithms was organized roughly along the lines just described, with implicitly parallel building block discovery, and implicitly parallel hierarchical assembly being the core computational efficiencies that the simple genetic algorithm supposedly parlayed into a powerful capacity for general purpose adaptation. Problems arose when researchers were unsuccessful in their attempts to rigorously derive complexity theoretic bounds that showcased these purported core efficiencies. Much more seriously, efforts to demonstrate these efficiencies experimentally also proved unsuccessful. The consequence for the building block hypothesis in theoretical circles was severe—rightfully so.

Unfortunately, so was the consequence for the overarching scientific program described above. If there is just one thing readers take away from this dissertation, I hope it’s the sense that this program is viable.

Dissertation webpage

Download manuscript

http://arxiv.org/abs/0711.1401

Perceptions of the abilities and limitations of the SGA (and hence the kinds of problems that it can and cannot solve) have been heavily influenced by a theory of adaptation called the building block hypothesis (Goldberg, 1989; Mitchell, 1996; Holland, 1975, 2000). This theory of adaptation has its genesis in the following idea: maybe small groups of closely located co-adaptive alleles propagate within an evolving population of genomes in much the same way that single adaptive alleles do in Fisher’s theories of sexual evolution (Fisher, 1958). Holland called such groups of alleles building blocks. This idea can be taken one step further: maybe small groups of co-adaptive building blocks propagate within an evolving population of genomes in much the same way that single building blocks do. Such groups can be thought of as higher-level building blocks. Pursuing this idea to the fullest extent, maybe co-adaptive groups of higher-level building blocks propagate in much the same way as ordinary building blocks do to yield building blocks of an even higher level, and so on and so forth in hierarchical fashion with the building-blocks of higher levels being comprised of co-adaptive groups of lower-level building blocks. Let us call this this idea hierarchical building block assembly.

Holland (1975) saw in hierarchical building block assembly a way out from the problem that epistasis (Wolf et al., 2000) poses for Fisher’s theory of sexual evolution. He also believed that hierarchical building block assembly, if implemented efficiently, could serve as a useful problem solving technique. He argued that a genetic algorithm that he called a genetic plan can implement hierarchical building block assembly, and moreover does so efficiently. He offered the genetic plan as a model of natural sexual evolution and also as a useful technique for finding solutions to adaptation problems with non-convex objective functions. The main theoretical tool that he used in his argument has come to be called the schema theorem (Goldberg, 1989; Mitchell, 1996). However neither the schema theorem, nor any of Holland’s other theoretical analyses fully support his claim that simple genetic algorithms are capable of efficiently implementing hierarchical building block assembly. Given the boldness of his claim and the large leaps of intuition that Holland makes in order to support it, the absence of experimental support in (Holland, 1975) is rather conspicuous (even more so given that simple, computationally unintensive, proof-of-concept experiments are not difficult to conceive of. See, Mitchell et al., 1992, and Forrest and Mitchell, 1993) . It would not have been surprising therefore if the genetic plan had been relegated to the history books as an algorithm that did not fulfill its raison d’etre — to support its inventor’s hunch about the utility of hierarchical building block assembly as a theory of adaptation for natural sexual evolutionary systems, and to support its inventor’s hunch that hierarchical building block assembly can be efficiently implemented. What seems to have saved the SGA from this fate is the curious matter of its utility.

In the years following the publication of Holland’s seminal work (Holland, 1975), the SGA was successfully used to adapt high-quality solutions to different sorts of real world and toy problems with non-convex objective functions. In an unfortunate twist of reasoning hierarchical building block assembly became the de-facto explanation for the success of the SGA. This explanation came to be called the building block hypothesis. Despite its name, the building block hypothesis was treated more as an assumption than as a hypothesis. Hierarchical building block assembly had aesthetic appeal, and the building block hypothesis had Holland’s unqualified endorsement (Holland, 1992). Therefore the building block hypothesis was readily accepted by most within the GA community. Some even went so far as to tout the success of SGAs as evidence of the veracity of the building block hypothesis or as evidence that hierarchical building block assembly is a useful search technique for a wide variety of search problems. Consider the following confused passage from one of the first text books on genetic algorithms:

“…the building block hypothesis has held up in many different problem domains. Smooth, unimodal problems, noisy multimodal problems, and combinatorial optimization problems have all been attacked successfully using virtually the same reproduction-crossover-mutation [S]GA.”(Goldberg, 1989)

The early support that the building block hypothesis enjoyed accounts for the deep impact it has had and continues to have on the course of research in genetic algorithms as well as other fields of evolutionary computation such as genetic programming. Recently the building block hypothesis has been sharply criticized for lacking adequate theoretical support. The most forceful criticism that we are aware of has been levied by Wright et al. (2003): “The various claims about [S]GAs that are traditionally made under the name of the building block hypothesis have, to date, no basis in theory, and, in some cases,are simply incoherent”. On the empirical side experimental results have been obtained which straightforwardly cast doubt upon the ability of a simple genetic algorithm to efficiently implement hierarchical building block assembly (Mitchell et al., 1992; Forrest and Mitchell, 1993). In response to these experimental results a silent transition has occurred within the field of genetic algorithms: hierarchical building block assembly has gone from being thought of as the abstract process that SGAs implement to being thought of as a normative process that SGAs mis-implement. Even though this transition between intellectual positions is completely specious it is now widely assumed that SGAs work because they manage to “fudge” hierarchical building block assembly. Many new genetic algorithms have been constructed to compensate for the perceived short-comings of the GA —e.g. messy GA, (Goldberg et al., 1989; Goldberg, 1989, 2002), LLGA (Harik and Goldberg, 1997; Goldberg, 2002), CGA (Harik et al., 1999), ECGA (Harik, 1999), cohort GA (Holland, 2000), FDA (M¨uhlenbein and Mahnig, 1999), LFDA (M¨uhlenbein and Mahnig, 2001), BOA (Pelikan et al., 1999; Goldberg, 2002), hBOA (Pelikan and Goldberg, 2001), SEAM (Watson, 2002, 2006),etc. The inventors of these algorithms claim, or at least imply, that their algorithms are better than the SGA at its own game — hierarchical building block assembly. In many circles within the GA community the curious matter of frequent utility of SGAs is now considered closed.

For a case in point of the kind of sleight of hand that we are discussing consider the following: conceding that there is little evidence that SGAs can efficiently and robustly implement hierarchical building block assembly, Holland (2000) remarks, “Are [S]GA’s, then, a robust approach to all problems in which building blocks play a key role? By no means! After years of investigation we still have only limited information about the [S]GA’s capabilities for exploiting building blocks”. Later he asserts that “the very essence of good GA design is retention of diversity, furthering exploration, while exploiting building blocks already discovered”, and presents a new genetic algorithm, the Cohort Genetic Algorithm, and argues that it implements this essence (see (Pei and Goodman, 2001) for evidence that it does not).

The field of genetic algorithms is both a scientific field as well as an engineering domain. Heedful science and meticulous engineering can often work synergistically. However when the boundary between science and engineering begins to blur, dogma and misplaced faith can beleaguer the practice of both, to wit, a system that is useful in practice, but does not implement a hypothetical mechanism may receive reduced attention, whereas the mechanism, far from being dismissed according to the basic norms of science may become the holy grail of the engineering goals of the field.

A theory that explains why a system exhibits a particular behavior can influence perceptions of how the system can behave, and also of how it cannot. Of the two kinds of perceptions, the latter kind is often judged in retrospect to be the greater impediment to the discovery of a new theory that can explain and predict the behavior of the system with greater accuracy. This is because by influencing perceptions of how the system cannot behave a theory implicitly determines the “domain of the impossible” and in doing so it steers researchers away from considering certain possibilities. Yet it is precisely amongst these ”impossibilities” that the seeds of a new more accurate theory often lie.

One of the two goals of this paper is to challenge the widespread belief that the SGA cannot increase the frequency of a low order schema with above-average fitness when the defining length of that schema is high (i.e. when the defining bits of that schema are widely dispersed). This belief can be traced back to Holland’s original treatise on genetic algorithms (Holland, 1975) and goes hand in hand with belief in the building block hypothesis (and variations thereof). In section 11 we provide an argument based on experimental evidence that this belief is misplaced. We believe that this errant belief will be judged in retrospect to have been a significant impediment to the discovery of a sound theory of adaptation for the SGA.

In this paradigm, each post-selection population is thought to harbor “good” genetic material. Recombination operators, and estimation of distribution procedures, are thought drive adaptation by composing this material to produce good or better individuals in the next generation.When adaptation stalls it is thought to be because “good” genetic material is unavailable, or because recombination of this material was not performed effectively.

Let us call this general set of beliefs the Compositional Paradigm. This paradigm draws its support from Holland’s Building Block Hypothesis. Its widespread acceptance in the GA community signals the implicit acceptance of the BBH as an accurate description of how the simple GA performs adaptation. It also signals the acceptance of a generalization of the BBH, namely that all varieties of recombinative GAs perform adaptation by recombining pe-adapted genetic material.

The Building Block Hypothesis however has never been conclusively validated. It has in fact been sharply criticized by a few (e.g. see section 5 of Wright et. al. Implicit Parallelism, GECCO 2003), and many have suggested that it be treated with more skepticism. Nevertheless, it and its generalization – what I call the Compositional Hypothesis – remain widely accepted. I believe that this is so because of two reasons:

1. The Compositional Hypothesis seems right. Recombination of genetic material occurs at the individual level, so the recombination of “good” genetic material must be the mechanism by which adaptation occurs at the system level. Right?
2. Real paradigmatic scrutiny only occurs when an alternate hypothesis is submitted for consideration. Despite the publication of a large number of studies of the foundations of genetic algorithms no alternate hypothesis that explains adaptation has been proposed so far.

In my own studies I’ve been fortunate to obtain theoretical and empirical results that suggest an alternate non-compositional hypothesis for adaptation in selecto-recombinative genetic systems. Right now I’m in the process of finding the best way to present it.