Shared publicly  - 
 
An addendum to my earlier post on the Cone of Uncertainty. You shouldn't read this, it's too long and boring.

Today the mailman brought me my used copy of Boehm's Software Engineering Economics (1981). This is the "original" source for the CoU, and I'd been feeling silly about not owning a copy, given how often it is cited.

Anyway, I eagerly turned to page 311 which I knew from an earlier quick skim of a borrowed copy was where the "cone" was discussed. (Boehm doesn't give it a name.)

And found a footnote that I missed the first time around: "These ranges have been determined subjectively, and are intended to represent 80% confidence limits, that is 'within a factor of four on either side, 80% of the time'". (Emphasis mine.)

This puzzled me, because I'd always thought that the Cone was drawn from empirical data. But no. It's strictly Boehm's gut feel.

And then I chanced across this bit from a bibliography on estimation from the Construx website: "Laranjeira, Luiz. 'Software Size Estimation of Object-Oriented Systems,' IEEE Transactions on Software Engineering, May 1990. This paper provided a theoretical research foundation for the empirical observation of the Cone of Uncertainty."

What empirical observation?

Curious, I turned to the 1990 paper. Its first three words are "the software crisis" - for a software engineering mythbuster, that's a very inauspicious start; the "crisis" being itself a software engineering myth of epic proportion - possibly the founding myth of software engineering.

The fun part is this bit, on page 5 of the paper:


>> Boehm studied the uncertainty in software project cost estimates as a function of the life cycle phase of a product. The graphic in Fig. 2 shows the result of this study, which was empirically validated [3, Section 21.1] <<
The reference in brackets is to the 1981 book - in fact precisely to the section I'd just read moments before. Laranjeira, too, takes Boehm's "subjective" results to be empirical! (And "validated", even.)

He then proceeds to do something that I found quite amazing: he interprets Boehm's curve mathematically - as a symmetrical exponential decay curve - and, given this interpretation plus some highly dubious assumptions about object-oriented programming, works out a table of how much up-front OO design one needs to do before narrowing down the "cone" to a desired level of certainty about the schedule. Of course this is all castles in the air: no evidence as foundation.

Even funnier is this bit from McConnell's 1996 book "Rapid Software Development":

>> Research by Luiz Laranjeira suggests that the accuracy of the software estimate depends on the level of refinement of the software's definition (Laranjeira 1990) <<

This doesn't come right out and call Laranjeira's paper "empirical", but it is strongly implied if you don't know the details. But that paper "suggests" nothing of the kind; it quite straightforwardly assumes it, and then goes on to attempt to derive something novel and interesting from it. (At least a couple later papers that I've come across tear Laranjeira's apart for "gross" mathematical errors, so it's not even clear that the attempt is at all successful.)

So, to recap: Boehm in 1981 is merely stating an opinion - but he draws a graph to illustrate it. At least two people, McConnell and Laranjeira, fall into the trap of taking Boehm's graph as empirically derived. And someone who came across McConnell's later description of Laranjeira's "research" should be forgiven for assuming it refers to empirical research, i.e. with actual data backing it.

So what do I make of all this?

First, that there is a "telephone game" flavor to this whole thing that is reminiscent of patterns I've seen elsewhere, such as the 10x myth. The technical term for it is "information cascade" (http://en.wikipedia.org/wiki/Information_cascade), where people take as true information that they should be suspicious of, not because they have any good reason to believe it but because they have seen others appear to believe it. This is, for obvious reasons, not conducive to good science.

Second, the distinction between empirical and conceptual science may not be clear enough in software engineering. Mostly that domain has consisted of the latter: conceptual work. There is a recent trend toward demanding a lot more empirical science, but IMHO this is something of a knee-jerk reaction to the vices of old, and may end up doing more harm than good: the problem is that software engineering seems bent on appropriating methods from medicine to cloak itself in an aura of legitimacy, rather than working out for itself methods that will reliably find insight.

Third, people should stop quoting the "Cone of Uncertainty" as if it were something meaningful. It's not. It's just a picture which says no more than "the future is uncertain", which we already know; but saying it with a picture conveys misleading connotations of authority and precision.

If you have things to say about software estimates, think them through for yourself, then say them in your own words. Don't rely on borrowed authority.
Information cascade. From Wikipedia, the free encyclopedia. Jump to: navigation, search. An information (or informational) cascade occurs when people observe the actions of others and then make the sa...
11
5
Alan Dayley's profile photoOmar Fernandez's profile photoDan Mullineux's profile photoAntony Marcano's profile photo
12 comments
 
I'm sad the cone is not empirical. I'm glad you took the steps to find the evidence. I'm glad you shared the reality.

The concept of the Cone of Uncertainty still feels anecdotally correct. I will have to drop the math in future descriptions of it.

 
OK Laurent, you can count me as falling into the trap as well...
 
A cone like shape - perhaps with some power law is a good metaphor for the kind of chaotic influences on a software project. The parameters are closely related to how similar a project is to anything else the team has worked on, how stable the tools and technologies are, how controllable the third party dependancies are etc etc. etc. ad infinitum... but at least its some shared 'metric' we can be hanged with when it go's tits up.
 
The underlying assumption with the cone appears to be that the closer we are to a future event, the more (relevant) data we have about it and hence the more we should be able to quantify or understand it against our current position. The assumption appears reasonable (except for 'Black Swan' events), at least for systems that are not unduly influenced by uncontrollable external factors (markets, weather, people :) ). I wonder if the idea originated from the idea of causality that relativity introduced (http://en.wikipedia.org/wiki/Light_cone)?
 
Hi Gus! What's weird about the Boehm "cone" is that it's inverted: it narrows toward the future instead of widening.

Cones that widen toward a more uncertain future are an intuitive depiction for natural events. There's a "cone of uncertainty" in some writings about real options, which is derived from a binomial lattice (think of a road that keeps forking, and at each fork you turn left or right at random).

Projects are not natural events: they're a result of us "steering" the future in unnatural directions. Perhaps that's where an inverted code might make sense as a metaphor. But it doesn't strike me as intuitive at all.
 
Hmmm... I've not seen the original, but I thought the cone represented the uncertainty over time. As Gus states, when time passes and we're closer to an expected event (or to the original expected date, anyway), then our uncertainty is less. Is that not what Boehm depicts in his book?

Looking at http://en.wikipedia.org/wiki/Cone_of_Uncertainty that's certainly not the depiction used for hurricane tracks.

BTW, that article also refers to Gorey, J.M. (1958). "Estimate Types", AACE Bulletin-November 1958.
and Bauman, H.Carl (1958), "Accuracy Considerations for Capital Cost Estimation", Industrial & Engineering Chemistry, April 1958.
 
Hi George, I'm trying to get my hands on that Bauman article. I'm not convinced that Boehm was in any way inspired by that article though; it's a relatively recent addition to the WP page and as far as I can tell was never cited by Boehm.
 
Hi Laurent!

Thanks for the great research job.

I agree with George. Many subjects/events are somewhat unknown on the project day 1. Some of them can be almost unknown. As the project evolves, these subjects events tends to be somewhat known. Some of them turn to be completely known. Even some new unknowns can show up, the tendency is that uncertainty diminish. On the project last day, it is expected that no unknown persists.

The cone shape only reminds us that on project day zero (planning), with a huge amounts of uncertainty, we should apply large error margin to our estimates. As the project goes on, and the uncertainties come down, we can estimate with more confidence (lesser error margin).

Paraphrasing you, the cone is just a picture which says no more than "the future is uncertain", but the uncertainty decreases during the project life-cycle, and this should be considered for estimations.
 
Hi Laurent,

I was directed to this post by someone and I think it's steered him in the wrong direction, so I wanted to run this by you.

I'd slightly argue against the points raised, but only from the point of view that the numbers attributed to the cone are not universally generalisable, not that the shape is in any way wrong.

The key to the cone is several-fold:

1. The cone represents the uncertainty about the end context at certain points in time. Given the way we deliver stories, every story we pick up that delivers value raises our certainty. Many stories we pick up at the beginning are biggest bang for buck/highest value and combining this with the inter-dependent effect on other stories we pick up (i.e the code is foundation from which the next iterations work, even if that same piece doesn't look like that at the end) we gain the most knowledge from them. Hence the uncertainty does indeed drop most steeply at the beginning.

2. To me, the cone is a retrospective position representing an aggregate of 'all types' of uncertainty. The known-unknowns, unknown-knowns and crucially, the unknown-unknowns. When that uncertainty manifests in some impact, it then illustrates to us how uncertain the context actually is, even if that lowers the known-unknowns. The only constant is the known-knowns, since if they're done, they never change.

3. This model of uncertainty can only ever apply to conditional probabilities i.e. where the gaining of knowledge affects the level of uncertainty - Think of this as the high school maths bag of different coloured balls. Say 3 blue, 5 yellow and 7 red. If you get to know you have a bag and you get to know how many there are in it, those are two pieces of information which you can use to 'zero in' on the the purpose of the exercise. You still don't know what you will pull out when you put your hand in, but you know you have to do it and you know there will be less you 'could do' once it's done. 

The probability of pulling out a yellow ball is 1/5. If you put your hand in, pull a yellow ball out and don't replace it in the bag, you now have a 1/14 chance to pull out another yellow. This is of course, a conditional probability. The same is true of a bag of stories. When you deliver something and it's gaining its value, you know that task is done. The other tasks builds on that knowledge (like the probability of picking out the coloured balls changes both the size of the set AND the contribution that colour makes to the set). This is absolutely true of software systems and applies to anything anyone has ever delivered. Be it a building, TDD software, lean methods etc. the 'thing' under delivery is the amalgamation of all the knowledge and crucially the certainty (i.e. lowered uncertainty) people have gained over time.  
 
Hi Ethar,

Thanks for the thoughtful reply.

Regarding your point 1, consider a startup that does a "pivot". I'd argue that this raises uncertainty, irrespective of how many stories have been delivered prior to the pivot. These stories reduced one kind of uncertainty - if you were doing it right, you have a degree of certainty that you were heading in the wrong direction - but you still cannot be certain that you are now heading in the right.

If you have something which is more adequately as converging on a point in parameter space, by accruing data points, as in your point 3, then yeah - maybe a cone is the right model. I'm not sure this is the most useful way to describe most software projects...
 
+Laurent Bossavit Thanks for the response.

The counter to that is that a startup has less knowledge about the whole sphere of applicability at the beginning. A startup typically THINKS it has enough to form a hypothesis, but that is one hypothesis amongst a potentially infinite set of hypotheses, most of which the startup doesn't know or care about at that moment in time. Hence their knowledge isn't consistent with what they thought and worse, it often becomes clear when they hint at a pivot that it's always been incomplete :) That doesn't change the total uncertainty in the system, even if, as you say, it changes one kind.

However, you're saying what I am saying in the last sentence of your first rebuttal, since you are NEVER sure you are heading in the 'right' direction. You are always heading in a more OPTIMAL direction, since the set of all proverbial points in this space are infinite and you're incrementally building the "it's not in this house" space all the time :) You can even learn something about yourself from inconclusive experiments, and that is that the experiment was useless in determining value/knowledge.

From my personal experience, if you are constraining the software project to the sphere of concern of the stakeholders in their stories, then yes, you are very likely to reduce the uncertainty one story at a time as they go live. So the cone is appropriate. It always happens, otherwise all we ever deliver is wasteful inconclusive functionality and one way or another, we don't. That said, there are many ways to look at problems within team performance. The cone is one which solves one particular problem. Cycle times are another. Throughput is a third. The uncertainty/variation of each stage in a flow affects all of the other two. Determining value, risk, performance, throughput etc. are numerical exercises, usually with some uncertainty. So if we are looking to continually improve, including identifying and reducing waste and increasing our knowledge of the space we work in (and hence, the projects we deliver), we have to know what we are improving 'from'. After all, delivering valuable software has two parts to it. Software, obviously, but also a measure of value, including improving value or reducing value. If we're not conducting that self-reflection, we risk being the wasteful ones ourselves and never knowing it, in some cases taking the business on a dead-end journey with us whilst more optimally pointing competition pass us by. Hence, this has ramifications for our engagement with the wider business environment, who we have for years, asked to trust us but even now still can't always prove that trust was well placed. 
Add a comment...