Profile cover photo
Profile photo
Bryan Larson
Multimedia nerd and thrill seeker.
Multimedia nerd and thrill seeker.

Bryan's posts

Post has shared content
How Selection Bias is Hurting the Web

Pop  Quiz: Does the average American spend more time watching TV or browsing Facebook?  The answer is watching TV. By a mile.  Americans spend 159 hours a month watching TV (, but only 8 hours on Facebook.  In fact, It turns out that the average American only spends 30 hours per month browsing the web (  

I bet that doesn’t describe your habits.  If you’re anything like me (and if you’re reading this, you probably are), you spend both your working hours and a substantial chunk of your free time on the internet.

You probably watch some TV, but not necessarily on a TV.  You own a smartphone. A tablet. A laptop running OS X.  Maybe more than one of each.  You’d only ever consider using Chrome.  Or maybe you’ve switched back Firefox because Chrome has gotten “all bloated and slow”. 

The average American is just getting their first smartphone, doesn’t own a tablet, and runs Windows XP and uses Internet Explorer 8.  About the only person you know who fits that description is your parents. The average American is less educated, poorer, and less tech savvy than you. But it sure doesn’t feel like it because you work for Google, Facebook or a hip SF startup or maybe you’re finishing your bachelors in CS at Stanford, MIT or CMU surrounded by the nation’s best and brightest.

You are up to date on all the latest happenings in the Valley.  Things like natural language processing, digital wallets, promoted ads, and Pinstagram are everyday conversations.  You don’t really talk about sports.  Sports are silly.  Professional Starcraft is pretty cool though.

This is a problem.  We software engineers and entrepreneurs are out of touch with normal people.  We hace a strong filter bubble.  We suffer from a logical fallacy known as selection bias.  Selection bias is the tendency to make false conclusions from non-representative samples.

Here are some examples of selection bias:

- Believing all CGI looks fake, because you only notice fake looking CGI.

- Believing all gay people have lisps because you only ask guys with lisps if they’re gay.

- Believing that the results of studies that sample western college students are representative of all humankind.

- Believing you’re universally attractive because all your girlfriends told you you were.

- Believing a real names policy on a social network is OK because everybody you know has no reason to hide their real name.

The first four examples are relatively harmless for a product manager at Google to believe.  But the fifth is what led to the Google+ real names fiasco. Similarly, a belief that the everybodywould be OK with their contact list being made public led to the Google Buzz fiasco.  

Why does this keep happening to Google?  Because at the end of the day, the people who make the product decisions look to their own life and experiences for guidance.

Don’t get me wrong.  Companies like Google have a lot of tools to help them make informed decisions.  They have petabytes of metrics.  They have user studies, focus groups, feedback polls, user forums, and feature request tools. And yet, at times these aids still lack the nuance to appreciate the perspective of real consumers.

How do I know this?  Because Google once hired me to make product decisions.

My experience is heavily biased. For example, I live with 4 roommates and every one of us has a smartphone running Ice Cream Sandwich. Two of us are running Jellybean. In the real world, only 10% of Android users are on ICS, and virtually nobody is on Jellybean.  I’m so far removed from typical Android user, it’s shocking when I meet strangers who complain to me about their slow Gingerbread phone.

In the summer of 2011 I was an associate product manager intern on Google Image search.  My team did an IamA on reddit (  One of my projects was leading the refresh of the Google Image Search experience on tablets.  At times, there was disagreement in the team on key issues.  We looked at all the data.  We did user studies.  But, we still had disagreements and I had to make decisions, and those decisions came from my everyday experience as a computer user.  

So I did my best to think like a regular user, while feeling that I couldn’t possibly be qualified to make these decisions.  I’m proud of the product we released, but we were lucky - Image search isn’t exactly a controversial product. What if I had been in charge of Google Buzz? Would I have seen the privacy problems?

A few giants of our industry knew what users wanted better than they did, like Steve Jobs or Jeff Bezos. Yet even visionaries who we thought understood their users, like Reed Hastings, failed to see the pitfalls of splitting the Netflix brand into Netflix and Qwikster.  Mozilla mistakenly assumed that users like software updates and now they’re pissing off their customers every 6 weeks (

Unfortunately every major consumer internet tech company suffers from selection bias. Startups are not immune. Many valley entrepreneurs are building products for themselves and their hipster San Francisco friends.  It’s doubtful that most consumers want to check-in on Foursquare, or upload intentionally damaged photographs on Instagram.  Sure, there may be a sizable market for these products, but it takes a certain kind of lifestyle to see Instagram as anything but silly.

So what can we do to completely avoid selection bias? There are a few things to keep in mind when making decisions and planning product strategy:

- We are not our users. This simple and obvious statement bears repeating.  *We are not our users.*  Unless you’re writing an API or IDE, you are not your user.  If you work for Facebook, just because you are comfortable sharing every aspect of your life, doesn’t mean your users are.

- Our friends are not our users.  Your friend group is not representative of the general public.  They may love measuring their social influence on Klout, but 99% of the population doesn’t give a shit.  So don't think about how your friends use products when trying to decide UI flow or major feature design.

- Our taxi driver is our users.  Take every opportunity to ask regular people what they think about tech products.  Their answers will surprise and shock you.

- Metrics are king.  At the end of the day, it’s always better to settle an argument with metrics. If you don’t have metrics, get them.  If your product hasn’t launched yet, bake in a metrics framework from the start.

There are other, more aggressive strategies - Zappos requires every new employee to serve as a customer service representative.  Google sends its new associate product managers on a two week trip around the world to gain appreciation for other cultures. But, at the end of the day, it’s hard to avoid selection bias in the valley because every moment of every day we’re surrounded by it.  

We may not be able to avoid it completely, but by being aware of our personal selection bias can help us over come it.  Good luck and keep shipping.

Post has attachment
An extremely well-crafted video combining sound, video, photography and design:

Post has attachment
Oooh, a hard deadline for the third installment of "The Great Camera Shootout" - can't wait! If you missed the first two, check em' out here 1) 2)
Wait while more posts are being loaded