OK, let's try the "open letter" approach.
4 plus ones
Shared publicly•View activity
View 17 previous comments
- I think you have something hugely valuable to say, but I think it's really important to remember how wildly different the culture of free software is from the culture of a place like Google. Totally different techniques are needed. That's why I said what I did about saying that you're "rusty". It's counterproductive. It shouldn't be, but it is. The only thing that builds cred is working code, and the very best way is always leading by example: "Here's my project, here's how I did it, and here's how you can do it too." After that's done, then you have cred, and once you have cred, you can spend it on things like "see, what I advocate would really have helped you guys".
I'm glad you're pushing the tests you wrote into openssl: that's exactly the right way, I think. I don't think the free software world knows "Google did what Mike says to do, and it's stunningly wonderful."
The free software world is basically allergic to anyone who says, "I'm not writing code for your project, and in fact I'm not writing code for anybody's project, but I have a lot of really valuable advice about how you should write code for your project." What you're saying sounds a little like that, enough like that that it doesn't really accomplish your goal.
However, suppose you spent time architecting a good test framework for openssl, and a list of tests to write on some wiki page, and coordinated people writing the tests and doing a bunch yourself. You'd simultaneously get cred and accomplish the goal.
My thesis, remember, is that openssl lacks tests because writing them is tedious and daunting, and not because people don't believe in testing or devalue it. My thesis is that they say they don't believe in it, and they devalue it because they don't understand how to do it well, and don't want to admit to their lack of understanding and confidence.Apr 23, 2014
- To repeat slightly:
My thesis, remember, is that openssl lacks tests because writing them is tedious and daunting, and not because people don't believe in testing or devalue it. My thesis is that they say they don't believe in it, and they devalue it because they don't understand how to do it well, and don't want to admit to their lack of understanding and confidence.
What that means is that proving that testing is great and valuable won't help, because people's objection is only superficially that they think tests are unnecessary and pointless. That's just a thing they say; their real reasons are that they don't know how to do it. (Isn't this why ToTT exists?)Apr 23, 2014
- OK, maybe we're getting somewhere. :-)
The free software ethic of "working code" is why I wanted to have the tests written before I started making vocal arguments; that was my way of "leading by example" and "making a contribution" in the small way I'm currently able. The problem is, I'm a student now, and as much as I want to do more to contribute in a concrete fashion, writing the proof-of-concept tests to get a discussion started seems like the best I can really hope to do.
But truth be told, writing the tests for "goto fail" and Heartbleed was kinda...fun. I mean, once I figured out the "goto fail" build stubs I needed, it was easy to change the code and test it. With Heartbleed, I sat down and started diving into the code at, like, 9pm or something, maybe 8:30; and once I figured out the library init calls I needed in main(), and the SSL_CTX_new() and other allocators/initializers I needed in SetUp(), and got the first case working nicely around 4:30am, I was back up at 8:30am and had the whole thing done by 1:30pm, with polish and all. And really... I did it for me. I liked it. I was good at it. And I was really...I was alive.
I've been trying to hold my tongue and not make empty promises, but I have flirted with the idea, in the back of my mind, of maybe volunteering to do a bit of testing and code reviews for OpenSSL, if they'd have me. It'd be something constructive for me to do when I want to put down the guitar, besides rewatching episodes of Breaking Bad for the tenth time. The goal would be to have partners-in-crime so I could make myself obsolete eventually, but I wouldn't mind getting the ball rolling a little.
It would be something relatively easy for me to do, I think. Along with my ping for the Heartbleed bug patch, I'm hoping to maybe offer some small degree of my services. Question is, will they have me? I'd be thrilled if the answer is yes. There's still a little Test Mercenary in me. (Obviously.)
I appreciate your well-formed final theses there. I agree with them, and you're 100% right about TotT. And to be honest, I'd sleep a little better--well, at this point, I'd sleep--if I were indeed allowed to contribute my test, and maybe a little more time and help beyond that, to OpenSSL in the near future. It's not my intention to vilify or blindside them, and it'd be great if I had them "on my side" even before this new article goes out, as a means of bolstering the credibility of my arguments--and maybe having a little fun.Apr 23, 2014
- I'm elbowing my way into the discussion here as I've been working as an editor on a long form article on this that Mike's writing for my site (martinfowler.com). I agree that articles like this often feel like monday morning quaterbacking, usually with some sanctimonious exhortations thrown in for good measure. But I think Mike's put together something much better than that, which explores the nuances that is wisely looking for.
I think an article like this is valuable at this point, because bugs like this ought to make our profession sit up and question how we do things. I am rather concerned about how cavalierly our profession seems to accept the notion that "bugs happen" with critical software like this. We ought to aspire to better and incidents like this ought to give us the kick to do so. We all know that a testing culture can't guarantee prevention of these bugs, but many of us do think that such a culture can significantly reduce the risk - and that matters.
We hope the article will be ready for review in a few days. I hope people on this thread will take a look at it before we publish it (email Mike if he hasn't asked you already). I think it makes its point well, without building its case on insight from hindsight. But having worked on editing it I'm as vulnerable to not seeing clearly as anyone, and have learned to value as many skeptical eyeballs as I can find.Apr 23, 2014
- Hi! Thanks for these nuances. I like the "We ought to aspire to better and incidents like this ought to give us the kick to do so." attitude A LOT. That's a much better attitude then "QED, Bitch" (sorry, Mike for using your post as an example).
But two more points:
1. Let's face it, there were other and (in my opinion) much larger issues here at work. E.g. that this piece of software that ensure safe transactions of the majority of internet traffic is maintained by people in their spare time is shocking.
2. What I am missing from these types of conversations is that we (as "we, the people who believe in the value of unit testing") use these incidents to point to the larger community and tell them "don't you see it". What I would love to see is that we use these situations to ask ourselves: "if we have something so good and valuable to offer, what are we doing wrong that people don't get and use it?"Apr 24, 2014
- "QED bitch" was an attempt at a humorous response to Reddit trolls; that particular phrase does not and will not appear in any of my more serious writing. It's also an unfair characterization of the rest of all the work I've done on this subject, in which I've been very careful to outline my reasoning and provide concrete demonstrations of my points. It is fair to question my judgment in making that one post, but don't use that to paint everything else I do with the same broad brush.
I appreciate the additional points you offer. In fact, your point #2 is exactly why I'm compelled to speak up as I have been. I'm hoping that the work I've been doing--the TotT, the article I've submitted to the ACM, and especially the article for Martin's site--will go a very long way towards addressing both points.
I still welcome you (and Bharat) to accept my invitation to review the article. If you can provide such constructive feedback without talking down to me and trying to put me in my place, especially in a public forum, I'd very much appreciate it.Apr 24, 2014