Who introduced the term “deep learning” to the field of Machine Learning (ML) and Neural Networks (NNs)? Just a few days ago we had an interesting discussion about this on the connectionists mailing list http://www.cnbc.cmu.edu/connectionists

Although Ivakhnenko had working, deep learning nets in the 1960s (still used in the new millennium), and Fukushima had them in the 1970s, and backpropagation also was invented back then, nobody called this “deep learning.”

In other contexts, the term has been around for centuries, but apparently it was first introduced to the field of Machine Learning in a paper by Rina Dechter at AAAI 1986 (thanks to Brian Mingus for pointing this out). She wrote not only about “deep learning,” but also “deep first-order learning” and “second-order deep learning.” Her paper was not about NNs though: http://www.aaai.org/Papers/AAAI/1986/AAAI86-029.pdf

To my knowledge, the term was introduced to the NN field by Aizenberg & Aizenberg & Vandewalle's book (2000): "Multi-Valued and Universal Binary Neurons: Theory, Learning and Applications.” They wrote about “deep learning of the features of threshold Boolean functions, one of the most important objects considered in the theory of perceptrons …” http://www.researchbooks.org/0792378245/MULTI-VALUED-UNIVERSAL-BINARY-NEURONS/

The Google-generated graph seems to indicate that the term’s popularity went up right after Aizenberg et al.’s book came out in 2000. However, this graph is not limited to NN-specific usage. (Thanks to Antoine Bordes and Yoshua Bengio for pointing this out.)

Although my own team has published on deep learning for a quarter-century, we adopted the terminology only in the new millennium. Our first paper with the word combination “learn deep” in the title appeared at GECCO 2005

ftp://ftp.idsia.ch/pub/juergen/gecco05gomez.pdf

Of course, all of this is just syntax, not semantics. The real deep learning pioneers did their work in the 1960s and 70s!

I also mentioned this somewhere deep down in the AMA at reddit: http://www.reddit.com/r/MachineLearning/comments/2xcyrl/i_am_j%C3%BCrgen_schmidhuber_ama/cpfrrnr

Although Ivakhnenko had working, deep learning nets in the 1960s (still used in the new millennium), and Fukushima had them in the 1970s, and backpropagation also was invented back then, nobody called this “deep learning.”

In other contexts, the term has been around for centuries, but apparently it was first introduced to the field of Machine Learning in a paper by Rina Dechter at AAAI 1986 (thanks to Brian Mingus for pointing this out). She wrote not only about “deep learning,” but also “deep first-order learning” and “second-order deep learning.” Her paper was not about NNs though: http://www.aaai.org/Papers/AAAI/1986/AAAI86-029.pdf

To my knowledge, the term was introduced to the NN field by Aizenberg & Aizenberg & Vandewalle's book (2000): "Multi-Valued and Universal Binary Neurons: Theory, Learning and Applications.” They wrote about “deep learning of the features of threshold Boolean functions, one of the most important objects considered in the theory of perceptrons …” http://www.researchbooks.org/0792378245/MULTI-VALUED-UNIVERSAL-BINARY-NEURONS/

The Google-generated graph seems to indicate that the term’s popularity went up right after Aizenberg et al.’s book came out in 2000. However, this graph is not limited to NN-specific usage. (Thanks to Antoine Bordes and Yoshua Bengio for pointing this out.)

Although my own team has published on deep learning for a quarter-century, we adopted the terminology only in the new millennium. Our first paper with the word combination “learn deep” in the title appeared at GECCO 2005

ftp://ftp.idsia.ch/pub/juergen/gecco05gomez.pdf

Of course, all of this is just syntax, not semantics. The real deep learning pioneers did their work in the 1960s and 70s!

I also mentioned this somewhere deep down in the AMA at reddit: http://www.reddit.com/r/MachineLearning/comments/2xcyrl/i_am_j%C3%BCrgen_schmidhuber_ama/cpfrrnr

- Exactly. It's just a marketing term for something that has existed since the inception of neural networks. Multi-layer feed forward nets have existed forever, that was a technical term in textbooks IIRC. 'Deep learning' seems slightly misleading, implying this was not the case... also imprecise because it was initially used for something else altogether. Technical terms must be precise and disambiguous.Mar 18, 2015
- FWIW, this happens often in research when a researcher wants to "own" an approach, they make up new terms and appropriate old research. Reviewers should be more careful about the origin of ideas, I think.Mar 18, 2015
- +Brian Mingus good to see you have such a sharp eye. ;)Mar 18, 2015
- https://www.reddit.com/r/IAmA/comments/1xj56q/hello_reddit_im_bill_gates_cochair_of_the_bill

"The ultimate is computers that learn. So called deep learning which started at Microsoft and is now being used by many researchers looks like a real advance that may finally learn."

/troll :)Mar 19, 2015