There's a couple things about Jeff Dean's recent talk ( that I wanted to emphasize. First, I absolutely love the way these huge networks and their learning algorithms are robust enough that they can not only handle dropout (randomly dropping nodes, Geoffrey Hinton's crazy but remarkably effective idea) but also the learning or values having extra noise from precision errors or race conditions. That kind of noise tolerance is something that's been missing from AI, but is a core part of biological intelligence, and it's exciting to see it in real world, deployed, state-of-the-art speech and object recognition systems.

Second, I want to reiterate how remarkable it is that these systems use raw data (raw pixels and raw waveforms) as input and have to build their own representations. That's a little like taking a person who's never heard a sound before, not to mention ever hearing language before, and trying to have them learn how to transcribe English speech. It's super hard, crazy difficult, as in almost impossible to believe they are solving that problem that way. Even our own brains appear to have structure that predisposes them to be able to perform at tasks like language understanding. The idea that these networks are able to succeed at this impossibly hard task -- no preprocessing of data, no existing structure to the network, everything from the representation to how to use the representation is learned -- is remarkable.
Shared publiclyView activity