The chief barrier to the wide and enthusiastic use of hyperclimbing is that it seems to scale very poorly with the number of attributes. If one heeds the seemingly high cost of applying hyperclimbing to large search spaces, this heuristic quickly looses its shine. A key conclusion of my doctoral work is that this seemingly high cost is illusory. I have uncovered evidence that strongly suggests that genetic algorithms can implement hyperclimbing extraordinarily efficiently.

As readers of this blog probably know, genetic algorithms are search algorithms that mimic natural evolution. These algorithms have been used in a wide range of engineering and scientific fields to quickly procure useful solutions to poorly understood (i.e. black-box) optimization problems. Unfortunately, despite the routine use of genetic algorithms for over three decades, their adaptive capacity has not been adequately accounted for. Given the evidence that genetic algorithms can implement efficient hyperclimbing, I’ve proposed a new explanation for the adaptive capacity of these algorithms. This new account—the generative fixation hypothesis—promises to spark significant advances in the fields of genetic algorithmics and discrete optimization.

The discovery that hyperclimbing is efficiently implementable also promises to have a non-negligible impact on the ecology of machine learning research. Optimization and machine learning are, after all, intimately related. Overlooking a few exceptions, the practice of machine learning research, can be characterized as the effective reduction of difficult learning problems to optimization problems for which efficient algorithms exist. In other words, the machine learning problems that can effectively be tackled are in large part those that can in practice be reduced to optimization problems that can be tackled efficiently. Currently, this largely limits the class of tractable machine learning problems to the class of learning problems that can in practice be reduced to convex optimization problems [1] . The identification of general-purpose non-convex optimization heuristics with efficient implementations (e.g. hyperclimbing), thus, has the potential to significantly extend the reach of machine learning.

For a description of hyperclimbing, and evidence that genetic algorithms can implement this heuristic efficiently, please see my dissertation

[1]  Kristin P. Bennett and Emilio Parrado-Hernandez. The interplay of optimization and machine  learning research. Journal of Machine Learning Research, 7:1265–1281, 2006.

Click here to see the final version (single spaced for easy reading).

http://arxiv.org/abs/0711.1401

The practice of Machine Learning research can be characterized as the effective semiprincipled reduction of learning problems to problems for which robust and efficient solution techniques exist – ideally ones with provable bounds on their use of time and space. In a recent paper Bennett and Parrado-Hern´andez (2006) describe the synergistic relationship between the fields of machine learning (ML) and mathematical programming (MP). They remark:

“Optimization lies at the heart of machine learning. Most machine learning problems reduce to optimization problems. Consider the machine learning analyst in action solving a problem for some set of data. The modeler formulates the problem by selecting an appropriate family of models and massages the data into a format amenable to modeling. Then the model is typically trained by solving a core optimization problem that optimizes the variables or parameters of the model with respect to the selected loss function and possibly some regularization function. In the process of model selection and validation, the core optimization problem may be solved many times. The research area of mathematical programming theory intersects with machine learning through these
core optimization problems” (Bennett and Parrado-Hern´andez, 2006).

Later Bennett and Parrado-Hern´andez imply that when the targets of ML reductions have been optimization problems, they have for the most part been the convex optimization problems within the MP pantheon.

“Convexity plays a key role in mathematical programming. Convex programs minimize convex optimization functions subject to convex constraints ensuring that every local minimum is always a global minimum. In general, convex problems are much more tractable algorithmically and theoretically. The complexity of nonconvex problems can grow enormously. General nonconvex programs are NP-hard.” (Bennett and Parrado-Hern´andez, 2006).

The close relationship between ML and MP arguably exists because MP provides ML with a set of crisp, well-defined problems along with algorithmic solvers that come with guarantees on their use of time and space. To state this using metaphors from software engineering, the well-defined convex optimization problems are interfaces that MP publishes, and the provably efficient and robust algorithmic solvers of MP implement these interfaces.

Let us differentiate, in this paper, between optimization and adaptation. We define optimization as the procurement of one or more points of optimal or close-to-optimal value, and adaptation as the generation of points of increasing value over time. Given this definition, to say that the target problems of Machine Learning reductions are optimization problems is to fudge the truth somewhat. While the Mathematical Programming community indeed seems to be almost completely concerned with the procurement of optimal or close to optimal points, ML researchers aren’t interested in optimization per se but in the means by which it is achieved in most MP algorithms, i.e. adaptation. In fact optimization is often prevented in machine learning algorithms – using a “technique” named early-stopping – to prevent overfitting. In other words, robust, efficient adaptation is the modus operandi of most convex optimization algorithms, and for the most part, it is this modus operandi that makes these algorithms interesting to Machine Learning researchers.

The interface-problems published by the MP community give ML researchers useful targets to hit; if a ML researcher works out a semi-principled reduction of a class of learning problems to one of MP’s interface-problems, there are off-the-shelf algorithms within MP which allow her to quickly test whether her reduction is effective. Because of the emphasis that the ML community place on guarantees of robustness and efficiency. when the targets of ML reductions have been optimization problems, they have for the most part, been restricted to being convex optimization problems within MP. These problems are rather simple as adaptation problems go – every local optimum is also a global optimum, or stated differently, there are no local optima. Rather heroic feats of ingenuity are therefore necessary in order to obtain effective semi-principled reductions
of hard problems to these simple optimization problems. The difficulty of obtaining such
reductions is currently a fundamental limitation on the pace of progress within ML.

The SGA (Mitchell, 1996) is an adaptation algorithm which mimics natural sexual evolution. It has been directly applied to a large number of hard real-world problems and has often succeeded in generating solutions of remarkably high-quality. To be sure, some amount of thought is required to “massage” these problems into a form which allows the SGA to operate successfully on them (e.g. choices must be made about the fitness function used and the way solutions are encoded as bitstrings), but unlike the case in machine learning this massaging is largely ad-hoc, an outcome more of trial and error than principled reasoning. The resulting problems are almost certainly hard ones (non-convex), with objective functions that are riddled with local optima. It is a testament to the adaptive power of the SGA that it nevertheless often produces solutions of remarkably high quality. Given these successes one might expect a great deal of interest in SGAs from the machine learning community. That this is not the case speaks to an unfortunate shortcoming of GA research. There is no dearth of one-off problems that SGAs have adequately solved. However GA researchers have yet to publish a single class of problems such that a) SGAs are likely to perform robust, efficient adaptation when applied to problems in this class, and b) the class is likely to be useful as the target of ML reductions. For the sake of brevity we will loosely define such a class of problems as an SGA-Easy/ML-Useful class. We believe that when such problem classes are found the ML community will begin to take a greater interest in GA research. The future relationship between the GA and ML communities might then be similar to the one that currently exists between MP and ML. As mentioned above SGAs commonly adapt high-quality solutions to problems which are almost certainly contain large numbers of local optima. It is reasonable therefore to suspect that there exists an SGA-Easy/ML-Useful class of hard non-convex problems and that the identification of this class will significantly ease the burden of obtaining novel ML reductions. We believe that the identification of such a problem class will go hand in hand with the discovery of a theory which can give a satisfying explanation of the adaptive capacity of the SGA. Such a theory does not currently exist.

Bennett and Parrado-Hernandez remark:

“Optimization lies at the heart of machine learning. Most machine learning problems reduce to optimization problems. Consider the machine learning analyst in action solving a problem for some set of data. The modeler formulates the problem by selecting an appropriate family of models and massages the data into a format amenable to modeling. Then the model is typically trained by solving a core optimization problem that optimizes the variables or parameters of the model with respect to the selected loss function and possibly some regularization function. In the process of model selection and validation, the core optimization problem may be solved many times. The research area of mathematical programming theory intersects with machine learning through these core optimization problems”

and later,

“Convexity plays a key role in mathematical programming [MP]. Convex programs minimize convex optimization functions subject to convex constraints ensuring that every local [optimum] is always a global [optimum]. In general, convex problems are much more tractable algorithmically and theoretically. … General nonconvex programs are NP-hard”

This paper lends support to my sense that the optimization techniques used in machine learning are largely limited to convex optimization algorithms. Successes in machine learning have so far been a result of the application of human ingenuity to carry out effective semi-principled reductions of certain classes of problems to convex optimization problems. I use the qualifier semi-principled because there is usually some solid rationale behind an ML reduction that leads one to expect that the reduction makes the original problem amenable to solution by convex optimization techniques. This however is never formally proven. i.e. the reduction is not formal in the sense that the traveling salesman problem formally reduces to SAT. In all likelihood the optimization problems that the redutions yield are non-convex. However in practice, the application of convex optimiaztion techniques to these optimization problems often gives satisfactory solutions to the original problems. Hence the qualifer effective.

For example researchers in the support vector machine community use the kernel trick in an effective semi-principle reduction of classification and regression problems to a class of convex optimization problems called quadratic programming problems. These problems are then solved using quadratic programing algorithms such as Platt’s SMO algorthm.

To summarize the discussion so far: convex optimization problems are a class of optimization problems in which every local optimum is also a global optimum. Many useful algorithms for efficiently solving convex optimization problems have been constructed by the mathematical programming community. One of the reasons for the success of Machine Learning techniques is the use of human ingenuity to achieve effective semi-principled reductions of certain classes of problems to convex optimization problems for which there exist efficient solvers within MP.

Another reason, as authors mention, is that machine learning researchers have been tweaking the convex optimization algorithms of mathematical programming to create new algorithms which are better suited for their purposes. “Mathematical Programming puts a premium on accuracy, speed, and robustness. Since generalization is the bottom line in machine learning and training is normally done off-line, accuracy and small speed improvements are of little concern in machine learning. Machine learning prefers simpler algorithms that work in reasonable computational time for specific classes of problems” [1]. The paper later mentions that optimization is often cut short (!) in Machine Learning to prevent overfitting This “technique” is called early-stopping. And later ‘Thus not only is “good” optimization not necessary, but “bad” optimization algorithms can lead to better machine learning models’.

Here are some reformulations of this paper and some of my thoughts.

Let me differentiate between optimization and adaptation. I’ll define optimization as the procurement of one or more points of optimal or close-to-optimal value, and adaptation as the generation of points of increasing value over time. Things become quite clear in light of this distinction. Mathematical Programming seems to be almost completely concerned with optimization whereas machine learning seems to require relatively quick adaptation.

The love affair between mathematical programming and Machine learning exists because MP provides the world with a set of crisp, well-defined problems — e.g. a linear programming problem, a quadratic programming problem, a second-order cone programming problem, a semdefinite programming problem, a semi-infinite programming problem (see the appendix in [1] for a description of each of these convex optimization problems) — and then provides programmatic techniques for solving those problems. To state this using metaphors from software engineering (particularly object oriented programming), these well-defined problems are the interfaces that MP publishes, and the efficient algorithmic solvers within that field implement these interfaces.

The interface-problems published by the MP community give ML researchers a rough target to hit in their efforts to carry out effective semi-principled reductions of difficult problems to ones for which efficient solvers exist. The interface-problems of MP are similar to the optimization problems that ML researchers ultimately end up with, but there are important differences: for example, MPers seek fast optimization whereas MLers merely want a high degree of adaptation, and have additional requirements such as scalability. Nevertheless, if a ML researcher works out a semi-principled reduction of a class problems to one of MP’s interface-problems, there are off-the-shelf algorithms within MP which allow her to quickly test whether her reduction is effective. MLers put great stock in the fact that each convex optimization algorithm in the MP pantheon has robust performance over its problem class and is theoretically proven to converge to the optimum.

Machine learning researchers spend their time trying to achieve effective reductions of difficult problems to problem classes for which efficient and robust solvers exist. An effective reduction may be a multistep affair (effectively reduce problem A to problem B, effectively reduce problem B to problem C, ….) but it most often bottoms out in an effective reduction to an optimization problem, most often a convex optimization problem. Different techniques and tricks may be used at each step of an effective reduction (e.g. probabilistic models, the kernel trick, etc.).

Convex optimization problems are rather easy as optimization problems go. Therefore a large amount of human ingenuity is required to carry out effective semi-principled reductions of classes of real world problems to convex optimization problems. These reductions often involves the use of “heavy machinary” from computer science, mathematics and statistics.

It seems to me that fundamental advances by the machine learning community are currently being limited by two factors (besides computational speed). The first is the sheer level of human ingenuity that is required to realize novel reductions of real world problems to convex optimization problems. The second is that the targets of these reductions have so far been limited to convex-optimization problems. The first factor cannot be changed. With regard to the second, I believe that the field of evolutionary computation may have a lot to contribute . Selecto-recombinative evolutionary algorithms are widely believed to perform efficient adaptation even on non-convex problems. Unfortunately there are currently no crisp descriptions of the classes problems that such algorithms can efficiently ‘solve’. I believe that if and when such interface-problems are determined the ML community will take much greater interest in Evolutionary Computation. EC might then play the same role w.r.t. ML that MP currently plays, i.e. ML researchers might then seek effective semi-principled reductions of real-world problems to interface-problems in the EC pantheon.

References:
[1] Bennett, K.P., Parrado-Hernandez, E. The interplay of optimization and machine learning research, Journal of Machine Learning Research 7 (2006) 1265-1281