Shared publicly  - 
 
Freakonomics looked into Microsoft's "Bing It On" challenge. From the blog post: "tests indicate that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results. .... The upshot: Several of Microsoft’s claims are a little fishy.  Or to put the conclusion more formally, we think that Google has a colorable deceptive advertising claim."

I have to admit that I never bothered to debunk the Bing It On challenge, because the flaws (small sample size; bias in query selection; stripping out features of Google like geolocation, personalization, and Knowledge Graph; wording of the site; selective rematches) were pretty obvious.
447
106
Jeff Neilson's profile photoThomas Hawk's profile photoShaun Snee's profile photoRicardo Blanco's profile photo
64 comments
 
OK just took the test and chose Bing 3 out of 5 times... i Still use Google Chrome +Matt Cutts & always will
 
For a company where the common assessment is that it's more about marketing than technology. Microsoft is astoundingly bad at marketing (Bing, Zune, Kin, Windows 8 RT, etc. etc.). I do really like some of the things out of Microsoft Research though.
 
+Matt Cutts 

For some reason when searching for my new site on Google searching for: "The Bandie" the site does not come up, whereas on Bing and Yahoo it does.  I can't figure out why it wouldn't be top of the list.  I am trying to understand the webmaster tools to figure out if there was an indexing error, but I am a bit lost.

Just an anomaly that will hopefully get ironed out soon.
 
You kind of feel sorry for them a little bit.
 
I continue to prefer Google no matter how hard I try to like Bing...
 
It's almost as if Microsoft are behind Bing.
They're certainly the only company I've ever heard of that skews the statistics to show their product in a better light.
 
i just took the test and gave up... its a pointless test if you are a web/graphic designer :)

still, google had better results than boing, bang, whatever that shit is called..
 
As a former MSFT employee, I can say this is all par for the course.

I will also say that when searching MSFT sites, most 'softies use Google because it's better at finding what they need. Bing is targeted towards a mythical end user.
 
Google is peoples coca cola. Even if pepsi tastes better they find themselves buying coca cola...
 
+Matt Cutts The quote you ended your first paragraph with is truncated.  It changes the meaning in a way you probably did not intend.
 
+Matt Cutts google drives 90% of the traffic to my site (in the organic search department) but in bing a search on Tolkien puts me on a second place... wished the same was going on in Google. Used to be so, but after hummingbird (as of the 24th) I was kicked back to the second page in favor of wikipedia and some "old timer / well established" sites... of well... see movement still and while Hummingbird claims to affect 90% of all search I get the feeling a lot is moving still. With result in .co.uk taking 3rd place while same keyword result in .com a 24th place // this is not normal behavior. Or am I mistaken?
James S
+
1
2
1
 
<Siri voice>This is not going to end well for MS 
Kay Passa
+
9
10
9
 
I keep wishing for the "Bing it on Challenge" to come test me because every single time I've done it, I end up with Google 5, Bing 0. I want a free Xbox!
 
I got the same results as +Kay Passa.  The problem is that if you're tech savvy or an experienced searcher, you'll be able to spot the Google results vs. the Bing ones without any effort.  Bing returned its own results in a smaller font.  

If they really want to do a blind taste test, they need to make the results appear identical.  They'd also have to show plain text results only, because there are certain searches that give Google or Bing away beyond just the font difference.  Search for "Leap Frog Phonics Farm" (my daughter loves this cartoon) and you'll probably see that Google shows YouTube videos, as opposed to Bing showing shopping results.  Even if you don't know that about Google, you'll probably sense it instinctively.
 
+Matt Cutts: Ian wrote he was annoyed because the study used "... just 1,000 participants..." however if you read the actual Bing blog post it states they used "... nearly 1,000 people." I don't know how many nearly is, so I would be curious to see the actual data used...
 
+Nick Altmann Microsoft have been poor at targeting the consumer (with the exception of Windows 95 and the Xbox). They are much better at marketing to businesses and to developers.
 
I think Microsoft has always lagged behind in web technologies. Their browser is a prime example. As web designers, we always have to make special rules so their pages display properly.
 
google a technical question about microsoft, then

bing the same question,

with bing you'll never find your answer
 
+Barry Craft just to have them stripped out in later versions while still providing no comprehensive solution to the original problem. It's like they do it and laugh at us!!
 
+Jon Hedges :  "nearly" comes after "almost" and immediately before "just about".  So "almost", "nearly", "just about", 1000.  Hope this helps!
 
+Adam "The problem is that if you're tech savvy or an experienced searcher, you'll..." always choose google over bing simply because it produces the results you're looking for.
 
+Kay Passa :  that too.  But either way, the "blind test" is no longer truly blind.  You know exactly which is which.
 
Microsoft is a crappy company which does not know how to compete. They come up with all false campaigns to fool users into using their products... This strategy may work for sometime but not forever.
 
Microft advertising has been leaning toward unethical practices with their windows phone challenge and now this. Makes me want to boycott their products.
 
"Did you find this blog post through Bing?  Probably not —" Oooo, big burn right off the get go.
 
Desktops, tablets, laptops, and phones are all PCs.
 
Multinational uses answer data from biased questions to promote product - shock!
 
This is me being surprised exactly zero.
 
Hi Matt, please see my message I have got a manual spam message for my website. After getting it i had started link removal processes. I have remove all the link which is unnatural and low quality links. I have share all the links with Google spam team through disavow tool. In webmaster tool i have got a following message.

"The file containing disavowed links to site has been updated. If this is unexpected, it may have been updated by another site owner. For more information, visit the Disavow links https://www.google.com/webmasters/tools/disavow-links?siteUrl=mydomain.com/ page in Webmaster Tools."

This activity had done by me on 27-Sep-2013. After doing this i am tring to send the reconsideration request to Google with detailed information about the links. But i am getting following error

"Hm. Something isn't right. We're checking into it now."    

Please suggest what should i do to submit my reconsideration so that i am able to get my previous rank back.

Looking forwards for better response from you

Thanks in Advance.

Regards
Brajesh
 
I'm sure that just got placed as a top priority for Matt there Brajesh. In fact I bet he has no less than 4 people actively working on a solution for your website and you should expect a written apology on a Google Letterhead shortly.

- But back to the point of the conversation, I've always found Microsoft's ads to be extremely questionable. I think they are offering a good product, but the initial 2:1 claim was certainly unrealistic. The results being much closer as the article on Freakonomics states seem much more reasonable.

However, there is certainly something to be said for the fact that the results Google was returning are being unfairly stripped of value adding elements that a marketer may not appreciate but a casual user might.

My personal preference has switched back and forth between the two products as there is a substantial difference in the quality of returns based on the intention of the query. It would be easy for me to throw things like Panda & Penguin under the bus here - but I can't ignore that they have had possibility unintended side effects that have made the differences in the products more noticeable and therefore created a greater demand of Microsoft's product depending on the query.

Of course, I'm sure that creating that separation between the products wasn't wholly unintentional.
 
Why would anyone want to use Bung when we have Google???
 
In SurveyMonkey's Bing vs Google study, age was a huge factor in whether people favored Bing or Google (ours had the headers and not the just results). Younger people nearly always chose Google while older people were more discerning on which results were better. All the data is linked in the blog post, so feel free to do your own analysis.  http://blog.surveymonkey.com/blog/2013/06/26/google-bing-seo-surveys/
 
I preferred Bing until I got new glasses and discovered that it wasn't "Bling." No gold, no glory.
 
+Eli Schwartz "64% of 30 to 44-year-olds selected actual Google results but when the branding was swapped, 78% still selected Google results when masked as Bing. This suggests that perhaps the quality of Bing’s results were actually better than Google’s."

Not sure how 78% choosing Google when masked as Bing suggests that Bing's results are better than Google's. It seems that Google's would be better because they chose it over Bing even when masked....
 
Bing does (or did) a G Vs B challange when signing up for their rewards system thing. It was a wash, as far as I could tell.

G does some very interesting stuff, and is far and away the big dog on the block, but the personalization bits don't work well for me. 

I use it because it works well and Big is close, but not quite there. Here's hoping they figure it out. G can use the competition. 
 
+Kay Passa You need to account for some sort of brand bias. Most people are biased towards Google, so they are automatically going to choose Google under all scenarios. The percentage of users choosing "Google" when it was actually Bing is the sum of those biased towards the brand + those that are just choosing the most relevant results.
 
I'd love to see Matt get in the mix and reply directly to their response. I'm sure it's unrealistic to hope for, but it would make for some very interesting reading.
 
+Eli Schwartz So you're saying those 78% who chose what they thought was Bing (even though it was really Google masked as Bing) are brand-biased toward Bing? It still doesn't show how Bing has "better quality results" than Google; it just shows that the 30-44 age set prefers the Bing brand over Google.
 
Do the Google issued macbooks have bing as the default search?
 
+Barry Schwartz I saw that response. It was weird to read a Microsoft employee say "we simply don’t track [Bing It On] results, because the tracking itself would be incredibly unethical," when a Microsoft employee did exactly that a few years ago with http://blindsearch.fejus.com/ .


Personally, it also seemed strange that a company would run a comparison test website and then throw the results away, not even keeping the data in some aggregate form. Lots of other stuff I could say, but it's probably not worth getting into.
 
+Matt Cutts yeah I'm guessing that's either an outright lie from the Microsoft employee, or someone higher up told him to write that. If I were ms, I would he using the site to try to figure out how to make my product better.
 
+Matt Cutts
I would hope that everyone would know the excuse about not keeping data reflects one of two things.

A) They don't want to have to produce the results later as they might be embarrassing, (even incriminating?) It's much easier to say they aren't collected then to face the nightmare of that situation.

B) It's purely a marketing based attack to try to capitalize on somewhat erroneous reports of Google playing a Boogie-Man character with collecting user information. Note I said user as it's data driven to provide information on the usage of a service/product as opposed to some kind of spy agency activity.

None the less - It would be great to see you get in there and kick some ass. If for no other reason than to state the obvious about their reply.
 
Funny - we wrote a blog post about Bing not learning marketing lessons from Pepsi about that campaign.  Now we see they didn't even have an accurate test. - Kudos to +Freakonomics 
 
Funny sidenote: I wanted to try the challenge, so I went to bing.com and searched for [bing it on]. The bingiton.com site was not returned as a result until I refined my search to [bing it on challenge]. Google, meanwhile, easily found the site when I searched for [bing it on].
However, since I live in Denmark I cannot access the site and actually take the challenge. Anyway I think my search for the site gave a quite clear result...
 
I find that my customers catering to low cost consumers are found more often on Bing than those catering to a higher $ crowd. One possible reason could be "always lower prices" for PC's and mobile devices pre-loaded with the Bing searchbar
 
Yes, well it's not surprising, all corporations cheat, for example, google algorithms favoritizes big corporation and kills small businesses on a daily basis.
 
Microsoft is doing so much of this crap that they start to look like the wannabe that can't get what they want. It's really sad to watch.
 
+Jean Peuplu No no no no,... Small businesses have to learn that they have to market their business as a small business and not the same way big corporations do. You do that and Google will send you all the visitors you need.
Add a comment...