Images and the Future of Google Semantic Search

Despite leaving out links to his other posts (+Sergey Andrianov can you fix that), this is a very important post if you're following the growth of semantic search.

Until recently, text ruled in search. Search engines can easily index and learn now to associate text with human meaning. But images are a whole different level. Human brains can look at an image and have it spawn an entire web of meanings and associations. But how can a machine do that?

As Sergey says in his post, we were given an amazing look at how far that is coming during the Google+ product update event on Tuesday. +Vic Gundotra told us that now I can search my photos for things like "karyn in snow" and get all my photos of my wife taken with snow in the photo. How does that happen? We already know Google can recognize faces in photos. Now Vic told us that they are teaching the photo algorithm to recognize thousands of new objects.

Imagine the possibilities here. This is the new web of "things instead of strings." Entity recognition in images opens up whole new worlds of semantic associations.

cc: +David Amerland +Eric Enge 
The Rise of the Image in Semantic Search
Vic Gundotra's announcement of recent changes at Google+ didn't go unnoticed. People all over G+verse went and shared this video ( The 1,000 words Google added to it's G+ Image Highlighting tool are probably bigger in the world of semantic search than the numbers Vic spelled out about the growth of Google+. So how is this important?

Contrary to what I said in my post recently (, Google has machine vision and it is being improved. The "deep learning" mechanism Google+ is using to search images has just added 1,000 words.


1. Images Form a New Language. Look at Facebook with 300 million images each day, or 100 billion photos a year ( Already most of the world's data is in the form of media (images, sounds/music, and videos) -- media that work directly with the human perceptual senses of vision and hearing. (from the link just below). 

2. Machines learned to see. Google's vision is probably not perfect. However, this vision might already look similar to what you see on this cover image. Want to read more? Here is what Google does to teach machines see better every day ( 

3. Machines will learn much more about us. In social media, we share in the form of ... media! Only a fraction of people share elaborated textual content. How far can Google and other search engines get by learning the meaning behind our media shares? Very far. Search engines will know what we eat, what we dress, where we go, how we brush our hair (if any), what sports we do, what weather we enjoy and etc. 

4. Ads will become more relevant. Google knows what we search and what we read. In other words - Google knows what we consume on the web. But there is a whole other side of us - what we share. By building semantics connections between us, what we consume a, what we share and the rest of the world, Google will be able to provide marketers with much better ad serving platform.

5. The standard for visual communication raises. Can we already see how "just an image" is not doing well in getting our attention? With the number of images on the internet growing exponentially, the bar to what's deemed visually appealing will also raise. For your website, it takes only 50 milliseconds (that's 0.05 seconds) - and sometimes, even 17ms - to make the first impression, Google Says in it's research ( And if your website is unattractive, it will stay ... well, unattractive no matter how relevant your ads are. I foresee the same apply to your social media posts. And here is a conversation +David Amerland started that confirms this thought (

6. Visual communication will become less metaphorical and more direct. Unfortunately, semantic connections established by search engines pale in comparison to the semantics in our brains. Search engines won't understand metaphorical relevance of images to our textual content (at least any time soon). This means that in order to be better seen by search engines, marketers, journalists, bloggers and anyone else who understands the value of search will have to learn to communicate graphically in a way that is understandable by search engines. This will probably help the image become actually more meaningful rather than just be a beautiful graphic.

Image Credit: DOMOKOS II, Semantic Uproar (
Shared publiclyView activity