Shared publicly  - 
 
I'm tempted to half-seriously define a "Software Engineering History Maturity Model". Its stages would be characterized by the sophistication of someone's understanding of where the "waterfall" lifecycle came from.

Level -1: hasn't heard of the term.

Level 0: believes that waterfall is just "the way software is developed". Or, if an Agilist, "the way software used to be develop before Agile".

Level 1: believes that waterfall is a product of software engineering, which itself was invented to solve the software crisis a few decades back. That is why it's the right way to develop software. (Or, if an Agilist, the wrong way, since obviously the crisis is still with us and therefore software engineering doesn't work.)

Level 2: believes that waterfall was invented in 1970 by Winston Royce and enthusiastically adopted by the recently formed software engineering community; it was then and for a long time remained the best way to develop software, and still is except under particular circumstances where you might prefer other models. (If an Agilist, under most circumstances.)

Level 3: believes that waterfall was a misunderstanding of the 1970 Royce paper, due to the author of DoD 2167 taking a look only at the first page of that paper, and was enshrined as standard practice after DoD 2167 mandated a strict sequential process, a caricature of the "reasonable" waterfall which includes feedback loops and prototyping and remains a damn good idea for some software efforts, except all those where spiral, iterative and incremental are more applicable.

(Level 3 can be abbreviated to "has read Larman's book on Agile and iterative development".)

Level 4: has actually read some primary sources, such as the 1968 NATO conference proceedings, and noticed that sequential processes were already well known at that time, but iterative models were openly discussed as well; has come to doubt that a "software crisis" ever existed as more than a marketing device; is deeply puzzled at an "official" history which hinges on someone totally misunderstanding a relatively obscure paper from a few years later and, undetected by the rest of a community of smart people, totally derailing an entire industry for decades. Doesn't even know what to think of waterfall anymore.

Level 5: has actually taken a deep dive into primary and secondary sources contemporary with the various relevant decisions, including the 1996 Dagstuhl workshop where a few major figures of the software engineering movement tried to get historians to work with them to grant legitimacy to their so far nonexistent "official" history of the movement; has noticed how Barry Boehm was the one who dragged the Royce paper out of obscurity back in 1987, at the ICSE conference, just one year before the publication of DoD 2167, and is wondering at this odd coincidence. Despairs at how badly the software profession sucks at making sense of its own aims, insights and history.

(Just as with software maturity models, the higher you go the fewer individuals there are at this particular maturity level. Just as in software, "maturity" here should be understood to mean "how close you are to where I am now", and has nothing to do with actual maturity.)
18
11
Jeff Ratcliff's profile photoLaurent Bossavit's profile photoTed M. Young's profile photoSleepy Fox's profile photo
10 comments
 
I like that the model starts at -1. It strongly suggests that it isn't the ideal starting point.

Nice summary, and I'd say more than a half-serious attempt would be well worth it!
 
I don't have a deep understanding, but I was under the impression that the Waterfall method was really just a straw man used to promote another method.
 
I'm wondering (i.e., musing without knowing the history) if Royce's straw man model took hold because (aside from people doing a TL;DR on his full presentation) it looked like a best practice from other industries (e.g., American Automobile manufactering, at its peak). "They're successful, and that's how (we imagine) that they're doing it" has been a big driver in the adoption of lots of nonsense.
 
Jeff: I'm afraid people took waterfall seriously and still do. Random example turned up by a Google search (Google "waterfall for your project" without quotes): http://www.techrepublic.com/article/waterfall-vs-rad-how-to-pick-the-right-method-for-your-project/1044102

What's horrible here is advice like this: "If your project manager is inexperienced or does not possess strong organization skills, you probably do not want to undertake a RAD project. Stick to more traditional waterfall projects instead."

Whereas my advice would be "If your project manager lacks the experience and skills, don't give them the project. Have them shadow a more experienced PM to learn the ropes. No process, of any description, will save you from incompetence, and if you give the project to someone who's not competent then it's really your responsibility, not theirs - you're the incompetent one."
 
Dave: yes, to the extent that the phrase "software engineering" was an encouragement to everyone in software to think of what they did as the transposition, to their discipline, of what people had done in disciplines that had the "engineering" label.

What's interesting is that right from the start people started disputing that this was such a good idea, but they were silenced, largely because starting a new discipline successfully means you can claim resources, money, prestige while developing the discipline, so the people who stood to benefit from that "runway" situation did not look kindly upon those who were saying "wait a minute".

Case in point: the story about the "Masterpiece Engineering" spoof paper by T.H. Simpson. See here: http://catenary.wordpress.com/2008/05/14/masterpiece-engineering/
 
Oh dear, I appear to be at level 4. I could certainly believe that people looked at the first few pages of Royce's paper and got concerned that the diagrams were getting more complicated and thinking "well, we know what we're doing, we don't need these other feedback loops and such." Overall it's the same way with planning, many people believe that they can create a concrete, detailed plan for a project that lasts for years, just like you can come up with the requirements for a software program that never need to change and will work exactly as desired.
 
@Ted- there's certainly a paradox there; a profession full of frightfully clever folks, many with PhDs and stuff, who epically fail at basic reasoning on fundamental topics. I wouldn't believe it, if I hadn't fallen into the same traps myself.

One possible explanation is that we tend to be intuitive reasoners who latch on to simple overarching explanations - "oh that so totally makes sense of everything" - even when it doesn't. To borrow Phil Tetlock's classification I think we tend to be hedgehogs rather than foxes.

Real history tends to be messy, complicated and highly nonlinear - just like software projects, come to think of it.
 
+Laurent Bossavit Having recently read Kahneman's "Thinking, Fast and Slow" (and now refer to myself as a Kahnemaniac), I find that the cognitive bias of "cognitive ease", where a simple story that seems to make sense is assumed to be true and valid, is why this continues. On the face of it, doing up-front requirements gathering, then being done with that "phase" and moving on to the designing, coding, etc., makes a good story.

The fact that it doesn't work is simply people trying to reduce their cognitive dissonance through the confirmation bias, e.g., "well, even though the projects have been late and not exactly what we wanted, this waterfall thing did work."
 
+Laurent Bossavit  I particularly like your quote: "No process, of any description, will save you from incompetence", it reminds me of Red Adair's quote: "If you think hiring a professional is expensive, wait until you hire an amateur!"
 
+Ted M. Young also enjoying Kahneman's book, and weary of correcting people who think the phrase 'The simplest explanation is the most likely one' is Occam's Razor...
Add a comment...