Shared publicly  - 
 
The Myth of the "Data Driven" Business


You may have noticed I have been pretty quiet in my blog lately aside from sharing news about our ACCELERATE event in San Francisco in November. It’s partially because honestly I’ve been swamped with new clients, existing work, and the never-ending effort to be a good husband, dad, and friend in the midst of Demystifying web analytics …

But being busy is no excuse to stop sharing ideas and encouraging conversation so let’s dive into something that has increasingly become a pet-peeve of mine: the notion leveraging web analytics to create a “data-driven” business.

I’m sure I have used this phrase in the past in an effort to describe the transformation that companies need to go through in the digital world, relying less on “gut feel” and more on cold, hard data to guide business decision making. Hell, a lot of smart of people have, including Omniture’s Brent Dykes and Google Analytics Evangelist Avinash Kaushik who has gone so far as to describe creating a data-driven culture as the “holiest of holy grails.”

Becoming “data driven” is the way to silence the HIPPO and to more firmly establish the value of our collective investments in digital measurement, analysis, and optimization technology. It sounds great, except for one thing:

A “data-driven business” would be doomed to fail.

I think that perhaps what people mean when they talk about being “data-driven” is the need for a heightened awareness of the numerous source of data and information we have available in the digital world, enough so that we are able to take advantage of these sources to create insights and make recommendations. On this point I agree — better use of available data in the decision making process is an awesome thing indeed.

My concern arises from the idea that any business of even moderate size and complexity can be truly “driven” by data. I think the right word is “informed” and what we are collectively trying to create is “increasingly data-informed and data-aware businesses and business people” who integrate the wide array of knowledge we can generate about digital consumers into the traditional decisioning process. The end-goal of this integration is more agile, responsive, and intelligent businesses that are better able to compete in a rapidly changing business environment.

Perhaps this is mere semantics — you say “potato” I say “tuberous rhizome” — but given the sheer number of consultants, vendors, and practitioners talking about creating, powering, and working in the mythical “data-driven business” I have started to worry that we’re about to shoot ourselves in the collective foot. We (meaning the web analytics industry as a whole) have done this before, first by claiming that web analytics was easy, then by insisting that cookies were harmless … and personally I’d prefer we avoid yet another self-imposed crisis of credibility if possible.

And while this may be semantics, I do disagree with Brent Dykes assertion that in the absence of carrot-and-stick accountability that web analytics breaks down and fails to create any benefit within the business, although I do understand fully where Mr. Dykes is coming from. I simply have not seen nearly enough evidence that eschewing the type of business acumen, experience, and awareness that is the very heart-and-soul of every successful business in favor of a “by the numbers” approach creates the type of result that the “data-driven” school seems to be evangelizing for.

What I do see in our best clients and those rare, transcendent organizations that truly understand the relationship between people, process, and technology — and are able to leverage that knowledge to inform their overarching business strategy — is a very healthy blend of data and business knowledge, each applied judiciously based on the challenge at hand. Smart business leaders leveraging insights and recommendations made by a trusted analytics organization — not automatons pulling levers based on a hit count, p-value, or conversion rate.

Kishore Swaminathan, Accenture’s chief scientist, in his discussion on “What the C-suite should know about analytics” outlines how an over-dependence on data can lead to “analysis-paralysis”, stating:

“Data is a double-edged sword. When properly used, it can lead to sound and well-informed decisions. When improperly used, the same data can lead not only to poor decisions but to poor decisions made with high confidence that, in turn, could lead to actions that could be erroneous and expensive.”

Success with web analytics and optimization requires a balance, and business leaders who will be successful analytical competitors in the future will need to develop a top-down strategy to govern how their businesses will leverage both digitally-generated insights and the collective know-how of their organizations. Conversely, being “driven” implies imbalance and over-correction — going out of your way to devalue experience, ignore process, and eschew established governance in favor of a new, entirely metrics-powered approach towards decision making.

You can do this, but to Swaminathan’s point, what if the numbers you’re using are wrong?

I think that creating a “data informed” business is a huge victory and for most companies a major step in the right direction. What’s more, working to create a “data informed” business shows respect for the hard work, commitment, and passion your employees have for their jobs and your company and products.

Rather than walk in and “embarrass the boss” with your profound and amazing knowledge of customer interactions, you can actively work with your management team by providing insights and recommendations that reflect your knowledge of how the entire business works, not just your amazing talent as web analytics implementer (or analyst, whatever …)

But I digress.

I’m interested in your collective thoughts here people. Am I over-reaching after a blogging hiatus and unnecessarily sniping in hopes of an early Fall dust-up in Google+? Or have you had the same thoughts and/or concerns, that by insisting that everyone needs to do exactly what the data tells them that we risk alienating (again) the very consumers of our efforts? Do you work at a truly “data driven” business and do what the numbers tell you each and every time? Or are you working to create a practice where otherwise smart, hard-working, and passionate marketers, merchandisers, and business leaders can benefit from the type of information and insights you are uniquely able to provide as a digital measurement, analysis, and optimization specialist?

While you consider your response I’ll leave you with a story that has shaped some of my thinking about web analytics over my career. Years ago my good friend Shari Cleary brought me into CBS News in New York to train her editorial team on Hitbox (yeah, Hitbox, I told you it was years ago!) Most of my clients at the time were “new school” but not these guys — they were hardcore news editors from the TV side of the business who had been tasked with making digital news work.

I talked and talked and talked about how powerful Hitbox was and how real-time analytics was going to power the content they put out there in the world. The editors were polite and showed real interest in the training until at one point the oldest and most grizzled of the group stopped me.

“Son, we’re not going to let the data make the decisions for us regarding editorial content,” he said with all sincerity. I was, of course, shocked to hear this — I mean, hell, that is what Hitbox was for! Figuring out which stories generated page views and which needed to be rolled off the page into obscurity.

“Umm, why is that?” I asked, figuring he’d lay into me about the inaccuracy of the system or how painful it was to use Hitbox …

“Because if we let the data drive editorial, all you will read about at CBS News is Paris Hilton’s breasts and Lindsay Lohan’s drinking problem.”

Needless to say, I stopped talking about real-time, data-driven changes to editorial content.

As always I welcome your comments, criticism, and feedback.
8
8
Bryan Cristina's profile photoImmanuel Godwin's profile photoEric Peterson's profile photoCraig Burgess's profile photo
17 comments
 
I believe you're right on, but that it is also a matter of semantics. If one is truly a web analyst then the business goal always has to be top of mind. Making recommendations based purely on numbers is what Website Optimizer is for. Creating a brand and an experience is not.

A brand is something that is cultured slowly over time. A vision to be fulfilled. For instance, Apple would not be what it is today if it was driven by data democracy.

Also, think about the last hotel you were at. There is probably some amenity that it lacked. The door wasn't automatic. The room lacked a fridge. One light was burnt out. Any of these individual things would not make you never use the hotel again, but the combination of all of them over time would alter your view of the hotel and you may decide not to stay there anymore. These are the sort of changes that can't reasonably be tested but must be done because it simply makes sense. Sure, a year long test could be executed, but the hotel must know its brand and live up to it. These are the cases where numbers are not required - you know what to do. The numbers will prove out over time, hence data-informed but the data doesn't need to drive you to action.

Also, so much for that early fall dust-up on G+ ;-)
 
Speaking of companies that are driven by readers that +Ben Gaines mentioned in the comments on your blog. I overheard an interesting discussion on the radio the other day about how data or actually a cost driven focus could risk to undermine the quality of any news sources. This because they focus to much on how much revenue each article is driving in the form of clicks, shares etc instead of looking at quality and how the articles are contributing to the overall focus of the newspaper in question. I firmly believe that must be a healthy approach to data so that organizations don't loose their connection to reality. With that said I also believe that organizations need to look at their own success factors. These factors are not different online than they are offline but they need to be measured (or not measured) differently. Maybe the focus is actually not on how many ads the readers are seeing but rather what articles people actually read and how people interact with the articles.
 
Loved the CBS story :) It drove the point home.
 
+David Schuette agree, data and intuition (or a sense of what consumers want and need) is required. That is more or less what I was saying but thanks for the hotel analogy --- I see a lot of hotels ;-)

As far as "no Fall dust-up" … either everyone agrees with me or folks are used to my cranky-old-man style and know it's better to just ignore me in hopes I will eventually go away. LOL that!
 
I really look forward to hearing more on this topic from you +Eric Peterson. But my "But" is twofold.
First, it's quite late for me as I was nearing closing up I found this post.
Second, I 9-5 for a Market Research company have have an extensive background in consumer behavior and I think there are some great points you make. As much as we want to be data-driven (because it feels good to have our hunches validated in black and white), the more important relationships, and the processes that use that data are often overlooked. I currently have a client who make soul Retail decisions based upon (incomplete) single line item reports (he designed) EVERY SINGLE DAY! It's quite frightening actually. More opinions to come... rest well all.
 
+Jennifer Shaw thanks for the late night commentary. I think we all know someone (some client, some co-worker or friend) who is currently "dramatically underutilizing" the data that can be made available through even the most basic web analytics implementations. But insisting that they are somehow bad people (or bad employees) because they are not led around by the nose by data and reporting is, well, silly.

The key is balance.
 
Eric,

Great post. You asked if you thought you were "over the top" in pondering this question. Two answers: yes and no. Let me explain.

I believe that lots of small businesses don't pay attention to the most basic KPIs that could have a huge impact on their bottom line. After all, companies exist to make the cash register ring. So, I don't think most are making decisions purely from data because it can seem counter-intuitive. The daily grind of employees, overhead, administration, sales, details and networking take the lion's share of the the day-to-day effort. Put this against using data to make business decisions on the level you discuss, and the daily grind wins every time. That's just reality, making a buck. That's the "yes".

That said, I expect to see this type of reflection from you and others in the #measure community. It's our job to ask the difficult questions. To dig deeper, to root out the useless practices that don't lead to the next "Eureka!" point of discovering a huge gold nugget buried deep within either the business or data. It seems obvious (as we are wired this way) to use data to drive some decisions. But it's not to the manager who's just trying to get their next product out the door. That's the "no".

But there is a new breed of manager I see rising up, one that is unafraid of using data - coupled with common sense about their markets, demographics and product offerings - to manage and guide their decisions on a bigger scale. Maybe it's "digital native-ness" (of which I am not) that allows these people to feel comfortable in this arena. I dont know. But this new digital pragmatism is a welcome change.

Re: CBS - great stuff. To me it's where testing breaks down. If we really wanted to get views and downloads, we'd slap lots of skin or violence on our products and make promises that Madison Avenue has been making for years - you know, more sexy, smarter, attractive, richer, more satisfying - the standard fare. Not saying it doesn't sell product. Just not appropriate for most companies selling HVAC ductwork, silicon chips, or high rise lifts.

-Craig.
 
I will always see Analytics as a conversation with someone who is willing to listen. You can have guts backing up your conversation, data, or both (both are good). But data alone isn't the answer. Sure, having good data is important, but I have seen tons of people take good data and make awful interpretations of it. At that point, people made some bad decisions based on data that they knew was good, all because they didn't understand it. A nice conversation (even if that means a report that is set up to explain everything) would have gone a long way.
 
The suggestion of a "Data Informed" Business makes sense, but the wording is too weak for the purpose of many conversations on the topic and would lead to very poor implementation.

In many cases, data should DRIVE the decision process, but not BE the decision process, primarily because short term value is not the only consideration.

Put another way, the "data driven direction" should have a governance model to contain it. The fastest way for me to get to the office this morning would have been to drive full speed the whole way, but of course my drive to work was governed by rules put in place for good reasons. A data driven business should be no different.

The CBS example is a good for this, and it sounds like the older gentleman starting laying out one of the core concerns that an effective governance model could be build around.

Depending on the situation, a governance model itself could be either a structured thing, be people making informal judgement calls, or a combination of the two.
 
+Craig Burgess great point, and thanks for the feedback. I fully agree that there are a new breed of business person emerging out there who are very likely to leverage digitally collected data in an appropriate way. These folks --- I know several --- are already "data informed" but would almost certainly blanche at the suggestion that data was "driving" their decision making. And fully agree, the new "digital pragmatism" is a welcome change, for more reasons than I can possibly count!

+Bryan Cristina I dunno buddy, having a conversation with the business people sounds like a pretty scary thing to me … it would be easier to just ship them a dashboard and be done with it, don't you think? LMAO!

You hit the nail on the head (again) with your "bad decisions from good data" point, and that again is my primary concern. If being data driven is the "holy grail" of businesses --- online and off --- then what if the data or interpretation thereof is wrong? Having seen this --- and not naming names --- the results range from bad to devastating.

Conversely, if businesses simply become more adept at understanding the data we have at our fingertips, primarily by developing over-arching strategy and investing equally in people, process, and technology, the end result in my experience is a new, powerful, and informative source of data that can be incorporated into the broader decision making process. But it starts with getting your arms around web analytics in a more profound way than just building some segments, some custom dashboards, or a fancy Excel spreadsheet.

+Peter Daly Here I disagree, and I got an email this AM from a CEO who said the same thing: "Being data informed is for wimps." Can you explain how a simple semantic difference could lead to a "poor implementation?" In my experience companies are either invested in web analytics or they are not --- all too often the latter --- and at that point my entire argument is somewhat pedantic I admit.

Your analogy is interesting. Could you also just as easily have said that "the knowledge that speed is a factor in your morning commute contributes to your overall ability to get to work on time without breaking any laws or putting anyone else in danger" do you think? If so, you used a combination of multiple inputs --- your years of experience as a good driver (I hope), your knowledge of the relationship between speed and time, and the attention you pay during the entirety of your commute --- to achieve your goal.

Again, my interpretation (and again, we are splitting hairs I suspect) of the "data driven approach" towards your morning commute would be to determine which components of said commute are most likely to impact your key performance indicator --- travel time, percent on-time arrivals, etc. --- and then aggressively work to maximize that component, potentially to the detriment of your license, your health, and the health of your fellow citizen.

Anyway, thanks for your comment … you're the first person who has publicly stood up for the "data driven approach" so I commend you for that. +1 to you!
 
Can you explain how a simple semantic difference could lead to a "poor implementation?" In my experience companies are either invested in web analytics or they are not --- all too often the latter --- and at that point my entire argument is somewhat pedantic I admit.

Sure.

That's exactly where I am coming from. A key aspect of my role is to transition companies between those two stages. For a company that is not committed, "data informed" does not provide the mindset needed to reach the tipping point of becoming really invested...they already think they are "data informed", but they don't know what exactly what that means in our world. "Of course I'm informed! We had X million hits last month!" It is an insult to them to say they are not data informed, because they believe they already are...and it's true. They are already data informed, but with the wrong data at the wrong time with the wrong mindset.

Once a company or group or company has passed the tipping point, the "data informed" mindset makes sense.

All this being said, I think I hardly ever actually use the term "data driven" unless speaking with someone who already understands the topic well, probably for exactly the reasons you describe.

BTW - thank you for your post and reply...you are causing me to think more deeply about the words I use, and I suspect I'll adopt a more strategic vocabulary for how to bring companies to the key tipping point.
 
+Peter Daly that makes more sense, and likely you've captured the nuance that other folks like +Avinash Kaushik and +Brent Dykes were trying to communicate as well (but I'm guessing.) Still, have you ever tried pressing your "data informed" on how well they actually understand the data they have?

What I've seen is that by scratching the surface just a little --- and you say you help companies make this transition but I don't know your work or company (my apologies!) --- what you find are business leaders who are willing to admit they aren't nearly as knowledgable or informed about their digital business as they would like to be. Now, this definitely changes as you move down in the ranks, but that is likely as much a function of people not wanting to appear out-of-touch or somehow lacking expertise on the subject … again, in my humble opinion.

Still, you do make a great point and added great context to the conversation so thank you!
 
I have seen execs/marketers/sales using data to back their decisions meaning even before they settle down with a decision ,they find out if data supports their idea and what insights they could leverage to think through in their decision making - and thats just some of the super smart folks out there and that percentage is always less...on the other hand I have also seen content making data driven decisions and end up creating crap stories(like in your story) and even portals just because that means more pvs, more engagement and more revenue and yes im talking about the big players in the industry doing that..but as that wise gentleman pointed out, wisdom is proved right by all her children...we need data driven decisions and with wisdom?
 
I can appreciate your distinctions between "data driven" and "data informed". I agree that our relationship to data should not become rigid and mechanical. I don't think insight and innovation can ever be reduced to a policy or process flow.

I come at the "data driven" business, not from an analytics perspective, but from a data governance perspective. When I talk about being data driven with my clients, my real hope is to take a little of the gamble out of executives making gut decisions and introduce a little more "management by fact".

At the same time, it is important for businesses to recognize that data driven means that the data has to be of high quality, well understood, and fit-for-use. There has to be a drive to more fully understand and improve the data quality.

I see these 2 characteristics: management by fact and concern for data quality, as being key differentiators for "data driven" businesses.
Add a comment...