Manuscript: http://s3.amazonaws.com/burjorjee/www/hyperclimbing_hypothesis.pdf

Abstract:
We submit the hyperclimbing hypothesis—an explanation for adaptation in genetic algorithms with uniform crossover (UGAs). Hyperclimbing is a stochastic search heuristic that works by decimating a search space, i.e. by iteratively fixing the values of small numbers of search space attributes. Global decimation is known to be an effective way to approach large instances of hard constraint satisfaction problems. The hyperclimbing hypothesis holds that UGAs work by implicitly implementing efficient global decimation. Proof of concept for this hypothesis comes from the use of a novel analytic technique involving the exploitation of algorithmic symmetry. We also present experimental results that show that a simple tweak inspired by the hyperclimbing hypothesis significantly improves the performance of a UGA on an instance of Uniform Random MAX-3SAT . The hyperclimbing hypothesis suggests that other kinds of evolutionary algorithms may also work by implicitly implementing efficient global decimation.

Talk abstract

My slides

We recently proposed a new explanation for the adaptive capacity of simple recombinative genetic algorithms. This explanation proceeds from evidence that the simple genetic algorithm with uniform crossover (UGA) can implement a stochastic non-local search heuristic called hyperclimbing extraordinarily efficiently. To showcase the core computational efficiency involved we take up the problem of learning a classifier for the attributes of an unknown parity function over n attributes, k of which are effective. We consider the case where the learning algorithm can make adaptive queries against a membership query oracle. Given a bitstring of length n, the oracle returns a boolean value indicating the parity of the bitstring under the unknown parity function. For certain small, but otherwise arbitrarily chosen, values of k, we “show” that a UGA that uses the oracle as its fitness function can learn a classifier that classifies any attribute of the parity function—as effective or non-effective—with arbitrary accuracy; the learning occurs in time that is linear in n, and with query complexity that is constant in n, even when the oracle is “moderately” noisy.

Related blog post: Red Dots, Blue Dots

Update (June 11, 2010): Had a back and forth with Vitaly Feldman about the “angle” I take in this paper.  He suggested that it may not be the best. For small values of k, and particular regimes of the noise parameter, a GA based learning algorithm performs at par (in an asymptotic sense) with the best known algorithms for solving the learning parities problem. Vitaly cautioned, however, that the problem of learning parities with an adaptive memebership oracle is currently not of practical interest. And since the GA based learning algorithm does not improve upon an existing computational bound, he thinks that from a pure computational learning perspective, this result it is unlikely to be of interest.

So, back to the drawing board. Taking it from the top, the goal is to draw the attention of the machine learning community to the hyperclimbing heuristic, and the GA’s ability to implement this heuristic extraordinarily efficiently. One way to do this is to showcase the core computational efficiency at play in this implementation . And one way to do this is by showing how this core efficiency can be used to efficiently solve a problem that members of the computational learning community care about—presuming that such a problem currently exists.

Enjoy!

The generative fixation hypothesis essentially states that genetic algorithms work by efficiently implementing a decimation strategy called hyperclimbing.

The chief barrier to the wide and enthusiastic use of hyperclimbing is that it seems to scale very poorly with the number of attributes. If one heeds the seemingly high cost of applying hyperclimbing to large search spaces, this heuristic quickly looses its shine. A key conclusion of my doctoral work is that this seemingly high cost is illusory. I have uncovered evidence that strongly suggests that genetic algorithms can implement hyperclimbing extraordinarily efficiently.

As readers of this blog probably know, genetic algorithms are search algorithms that mimic natural evolution. These algorithms have been used in a wide range of engineering and scientific fields to quickly procure useful solutions to poorly understood (i.e. black-box) optimization problems. Unfortunately, despite the routine use of genetic algorithms for over three decades, their adaptive capacity has not been adequately accounted for. Given the evidence that genetic algorithms can implement efficient hyperclimbing, I’ve proposed a new explanation for the adaptive capacity of these algorithms. This new account—the generative fixation hypothesis—promises to spark significant advances in the fields of genetic algorithmics and discrete optimization.

The discovery that hyperclimbing is efficiently implementable also promises to have a non-negligible impact on the ecology of machine learning research. Optimization and machine learning are, after all, intimately related. Overlooking a few exceptions, the practice of machine learning research, can be characterized as the effective reduction of difficult learning problems to optimization problems for which efficient algorithms exist. In other words, the machine learning problems that can effectively be tackled are in large part those that can in practice be reduced to optimization problems that can be tackled efficiently. Currently, this largely limits the class of tractable machine learning problems to the class of learning problems that can in practice be reduced to convex optimization problems [1] . The identification of general-purpose non-convex optimization heuristics with efficient implementations (e.g. hyperclimbing), thus, has the potential to significantly extend the reach of machine learning.

For a description of hyperclimbing, and evidence that genetic algorithms can implement this heuristic efficiently, please see my dissertation

[1]  Kristin P. Bennett and Emilio Parrado-Hernandez. The interplay of optimization and machine  learning research. Journal of Machine Learning Research, 7:1265–1281, 2006.

Click here to see the final version (single spaced for easy reading).