Public

Convolutional Nets and SVMs were developed within a few years of each other (between 1988 and 1992) in the Adaptive Systems Research Department at Bell Labs in Holmdel, NJ. Larry Jackel was the head of the department whose research staff included Vladmir Vapnik and me, along with Bernhardt Boser, Léon Bottou, John Denker, Hans-Peter Graf, Isabelle Guyon, Patrice Simard, and Sara Solla,

In 1995, Vladimir Vapnik and Larry Jackel made two bets (I was the witness, though admittedly not and entirely impartial one).

In the first bet, Larry claimed that by 2000 we will have a theoretical understanding of why big neural nets work well (in the form of a bound similar to what we have for SVMs). He lost.

In the second bet, Vladimir Vapnik claimed that by 2000 no one in their right mind would use neural nets of the type we had in 1995 (he claimed that everyone would be using SVM). Not only Vladimir lost that one in 2000, but recent deployments of neural nets by Google and Microsoft are proving him wrong in 2012.

In 1995, Vladimir Vapnik and Larry Jackel made two bets (I was the witness, though admittedly not and entirely impartial one).

In the first bet, Larry claimed that by 2000 we will have a theoretical understanding of why big neural nets work well (in the form of a bound similar to what we have for SVMs). He lost.

In the second bet, Vladimir Vapnik claimed that by 2000 no one in their right mind would use neural nets of the type we had in 1995 (he claimed that everyone would be using SVM). Not only Vladimir lost that one in 2000, but recent deployments of neural nets by Google and Microsoft are proving him wrong in 2012.

View 8 previous comments

- +Yann LeCun The goal is not to achieve a zero training error per se, which allows a lot of trivial solutions, like to make probability non-zero on training samples and zero everywhere else. The goal is to get minimum errors on test set, that is not seen in training. To do that we want to maximize probability of training set Prob(training) which is defined by Bayesian integral over parameters of Prob(training|params)*Prior(params) for our model. Those trivial solutions with zero errors contribute nearly nothing to the Bayesian integral. Instead you will find that non-trivial solutions that minimize errors and also have optimal regularization will contribute most, including non-trivial zero-error solutions if they exist.

You are right that SVM with kernels localized on training samples will give zero error, but it is rather a trivial solution.

However, it is possible to prove that

1. most significant contributions to Bayesian integral come from solutions with optimal regularization parameters;

2. there are possible "perfect" solutions that have zero training errors with optimal regularization parameters going to zero;

3. in the limit of zero regularization "perfect" solutions are maximum margin solutions that are SVMs;

4. maximum contribution to Bayesian integral comes from a "perfect" solution that has minimum number of support vectors, however, some "perfect" solutions with large number of support vectors could make lower contributions to Bayesian integral than "non-perfect" solution with optimal non-zero regularization;

5. it is possible to design an iterative process that will start from "perfect" solutions with a big number of support vectors and converge to a solution with smaller number of support vectors;

6. that perfect solution defines a non-trivial kernel SVM with a computable kernel.Oct 21, 2012 - +Yaroslav Bulatov For corrupt data you do not need to strive for zero errors, you can learn features from dirty data and train on a clean, whatever small set you can get, along with finding optimal regularization.

In any case zero-error classifier is not an ultimate goal, as clarified in my post above, it could be a best solution if ranked by Bayesian principle. This is not related to a corrupt data case.Oct 21, 2012 - researchbets.com would be greatOct 27, 2012
- Did they pay up?Nov 2, 2012
- Yes, +Kevin R. Vixie , they both paid up. I was the only one to get a free dinner.Nov 2, 2012
- Nov 11, 2012

Add a comment...