Shared publicly  - 
 
Really great post by +Alistair Croll making the case that companies that have massive amounts of data without massive amounts of clue are going to be displaced by startups that have less data but more clue, who will put in place the dynamics to make the most of the data they have and to collect new data in self-reinforcing applications that get better the more people use them.

This is really the premise of our Strata series of conferences - trying to teach companies how to join the data-driven revolution. I think it was about 2003 that I first wrote "Data is the next Intel Inside"; When I wrote What is Web 2.0? in 2005, I really tried to get across the idea that collective intelligence and data driven applications were the key to the future, but it's only now that it's become widely understood.

When +Edd Dumbill and +Alistair Croll proposed the Strata conference to me, it seemed so clear that now, finally, it was time to get off the soapbox, and to put together an event that was practical, and would help developers and executives make sense of how to build cloud data applications.

As Alistair says in this post, "Big data: use it or lose it."
80
64
Danfer Habed Lopez's profile photoCisco Perin's profile photoJeff Ross's profile photoElaine Martin's profile photo
19 comments
 
Sounds like the hedgehog principle from "Good to Great" applied to a different context.
 
Sounds like the premise used by libraries to filter information rather than provide everything available.
 
the thing we learned collectively from the rise of Google Facebook, Amazon etc was how to build big, scalable systems; with the cost reductions through cloud capabilities, startups are in the best position to take advantage by being faster and smater to respond to opportunities
 
About large amounts of data, I'd like to hear an expert view on how data from LHC is handled...
 
The companies with massive data and less clue can attempt to buy clue (and startups).
 
can this apply to facebook groupon example. Facebook possibly has all the data to take on groupon, but groupon seems to be doing better than facebook deals consistently despite having much less data than facebook. Or maybe facebook places to foursquare ?
 
In order to have meaningful statistics, small data seldom is useful. It depends what the data is, of course.
 
Actually, the line is "use it or *lose*"—which IMHO is even more telling. ;-)

Companies that don't figure out how to leverage their data will become irrelevant. There's another factor here: organizations like Infochimps, Azure Data Market, Datamarket, and so on make the barriers to data collection much lower; Mechanical Turk and crowdsourcing help clean it; clouds help crunch it; and machine learning services help find signal in the noise. So it's not just that upstarts ask better questions, it's that what used to look like a barrier to entry is no longer so.

The inertia that large organizations face is often astonishing. I suspect that one of the reasons for this is that broadcast models have lured them into a false sense of efficiency. By which I mean, once, they could prepare a single message and send it to 20 million people—one-way, one-to-many messaging. But today's communications channels are two-way, and many-to-many. They're scrambling to find efficient ways to market at scale. It's telling that many of the reactions to that post were about bad customer service, which is really marketing at the individual level.

The answer, of course, is that companies market at scale efficiently by using the data at their disposal. That's the only way to survive an impatient consumer base and an attention economy.

Fodder for another post, perhaps.
 
+Tim O'Reilly I don't see anything in that post that really refers to big data. The article advocates to make bigger use of the data you have.

Big data is stuff like a major tractor manufacturer that records the operating noise of every single machine they produce in the highest sound quality available - just to hope they don't need it. It is stored to be compared to the sound of the same machine when it comes in for repair, to make an acoustic detection of the cause of defect. Works pretty well.
And as they sell many tractors, the have really big amount of data.

Another example of big data is what +Tuomo Kalliokoski mentions: The LHC, or about any other institution that does measurements at quantum level. Tera- if not petabytes of data, generated within the blink of an eye.
 
We don't really need Big Data... we need what can come from it.. Big (business) Impact and Big (better) Decisions
 
Not a matter of need, +Michael Dell , but just a realization that large amounts of information are all around us. What matters is getting meaningful answers from data in a realistic timescale. This is why the role of algorithms that characterize data (from astronomical data, to personal data, to business data) is key at moving forward. Constantly computing data aggregates (i.e. metadata) that acts as a proxy to your data to understand if you are asking the correct questions in the right context is key.
 
Seems unlikely to me at first blink - usually what happens is that as soon as the startups get interesting the big company buys them. Very rare that people avoid that.
 
I've worked with many large brands, and despite investments in well-known brand analytics platforms, in-house marketers often do not know how to use the platforms. They often feel that the reports they need are either custom, take too long to generate, or cost too much to generate. In addition, I've seen small business owners that spend an awful lot of time with Google Analytics data and are quite frankly focusing on the wrong reports.

We developed Bizwatch to isolate the data that matters most. It is integrated with Adwords & Analytics for that reason, and pulls only what you need to increase leads and lower cost of acquisition.
 
How about addressing real (health) problems that aren't business problems. Alzheimer's, cancer, aging, neuroscience. These are all big data problems (Alzheimer's perhaps to a lesser extent). Make companies based on these big data and science technologies. Then make big businesses and informed decisions.

Obviously, saying "We don't need Big Data, we need what can come from it" is flawed, if you need the result of something you need that thing the result is based on, in this case "big data".
 
+Sebastian 'baboo' Another example are telescope recording every single photons with giga-pixel cameras... Some are already taking data (Pan-STARRS) looking for killer asteroids. Others will become operational in a few years collecting 100+PB of data over their lifetime.
 
heh; exactly what we do for enterprises & government as a start-up...and they like paying us to turn on light bulbs for them.
 
Its also amazing how much data is thrown away every day by companies who fail to see its value or think that disk space is still too expensive. This is especially true in most investment banks I have worked at. Basic question like how are the systems performing compared to 6 months ago can't be answered.
 
I keep thinking of the misuse of data in education. There is a drive to find data but no real thought to quality control. I teach students and know that good data is elusive. This is why teaching has been, for so many years, pure art rather than a mixture of art, science, and personality which is what it needs to be. The shift over the past decade has been toward thinking of teaching as pure science (or worse, as business), but no one in education seems to have good data or a good model for things. Instead, models from other professions and from industry are being put on education like a suit of poor-fitting clothes.
Add a comment...