Is the "Golden Triangle" Really Gone?
Be careful drawing conclusions from a limited sample of data.
This article talks about an interesting study which, in concept, could be extremely useful to the digital marketing world. But it also demonstrates something I see far too much of these days (and I'm guilty of it myself). Too often we don't dig deep enough into data before we start conversations about it. As a result we can end up talking about a study as if it's fact and rely on its conclusions as if they were rock-solid.
If I were reporting on this study I would have mentioned that the sample size was 53. That's it: 53 people were observed to produce these conclusions. Is this enough to offer validity?
A problem I see in this study is that it does not mention how the sample set was created or what the larger population is. Is this study about Internet users in the USA, North America, or some other geography? Does is have any gender, age, income, educational or other factors? The study itself says the sample group was "mixed age and gender" but that's not that same as "randomly selected". So we don't know about any biases that might exist.
Also I'd add more context about how to rely on this data. What is the confidence level associated with it? Can we be 90% confident the larger population will behave like this group? Or is 50% confidence more reasonable?
Another measure of context I'd want to see would be the confidence interval, or the "plus or minus" measure. If a study says "38% of people clicked here" then it should indicate what that means in light of the larger population. What are the plus or minus ranges we'd likely see in that larger group, based on this study? Just because 38% of this sample group clicked doesn't mean 38% of the larger population would.
I'm not suggesting this study isn't useful. It might be. But the problem is, we have no information that tells us how much we could expect to rely on this data and, therefore, its conclusions. To offer real value, a study should provide this context.
As consumers of data and conclusions from studies like this and many others, we need to be vigilant in making sure we understand how we're using this information. It takes more work and it's more complicated. But it's necessary if we expect to get value from the information we use. #dataanalysis #statistics