AI startup Vicarious claims to have a system that can solve CAPTCHAs with "succes rate up to 90%".
Beware: It's a textbook example of AI hype of the worst kind
Hype is dangerous to AI. Hype killed AI four times in the last five decades. AI Hype must be stopped.
Perhaps Vicarious can get "up to 90%" accuracy on some CAPTCHA dataset they cooked up, but
- (1) breaking CAPTCHAs is hardly an interesting task, unless you are a spammer;
- (2) it's easy to claim success on a dataset you cooked up yourself. There is no risk someone else will beat you.
- (3) recognizing object in images is much, much harder than breaking CAPTCHAs. Some deep learning systems can already do this with decent accuracy. Some such systems have been deployed by Google and Baidu.
- (4) doing simultaneous segmentation and recognition of character strings is hardly a breakthrough. See demos of a 20 year-old system here: http://yann.lecun.com/exdb/lenet/index.html
The sad thing is that this announcement is being picked up by a number publications, including MIT Tech Review, Forbes, etc.Here is an advice to scientific/tech journalists
: please, please do not believe vague claims by AI startups unless
they produce state of the art results on widely accepted benchmarks.
This is particularly true for claims in image and speech recognition for which good benchmarks exists. For image recognition, a good example of such benchmark would be the ImageNet Large Scale Visual Recognition Challenge.
Whenever a startup claims "90% accuracy" on some random task, do not consider this newsworthy. If the company also makes claims like "we are developing machine learning software based on the computational principles of the human brain" or uses impressive-sounding names like "Recursive Cortical Network", be even more suspicious.
There are extremely impressive applications of deep learning out there (e.g. deployed by Google, Baidu, Microsoft, IBM, and a few startups), but this is not one of them.
Google's automatic photo tagger and Baidu's image retrieval system are much, much more impressive than the system in this announcement. Even if we just talk about challenging character recognition tasks, Google's system for picking out house numbers in StreetView images is way more impressive than this.
AI "died" about four times in five decades because of hype: people made wild claims (often to impress potential investors or funding agencies) and could not deliver. Backlash ensued. It happened twice with neural nets already: once in the late 60's and again in the mid-90's.Don't let it happen again. Beware of hype.
And by the way, no one is interested in breaking CAPTCHAs except spammers and computer security researchers. That's why you won't find many computer vision papers on the topic. That's also why it would be easy to break records, even if a standard dataset existed.