This is a great effort. They're naming names and taking no prisoners. EDIT: S...

Another interesting example: http://reproducibility.cs.arizona.edu/data/tissec15_YavuzNR12_data.txt

It's marked as "theoretical" in the database, with this comment. "VERIFY:COMMENT[string] Since proofs exist for the effectiveness of the system, I'm not sure either if it is necessary to ask for implementaitons."

But there's also: "PI:COMMENT_CC[string] They measure performance. So, we need access to their code. Proof of correctness means nothing, by the way!"

If you look at the paper, it has performance numbers for an implementation. For the overall numbers, though, it still is counted as "theoretical".

I couldn't find the implementation with about 10 minutes of looking.

Mar 20, 2014

Shriram Krishnamurthi+1

The repo is ready; I'm having +Eric Eide +John Regehr +Camil Demetrescu look it over before I publish it.

Mar 20, 2014

Shriram Krishnamurthi+1

All set. Everyone: please contribute! I'll work on improving the UI, etc.

https://plus.google.com/117185293319274359863/posts/X2Gv3DYRaNe

Or go straight to

http://cs.brown.edu/~sk/Memos/Examining-Reproducibility/

Mar 20, 2014

Richard Hendricks+2

I have personal zen engineering koan:
Sometimes the fastest way to the solution is posting the wrong answer.

Apr 8, 2014

Shriram Krishnamurthi

+Richard Hendricks, good point, especially if it involves people on the Web...on the Web.

Apr 9, 2014

Richard Hendricks+1

Many times in internal and customer-facing mailing lists I've seen questions go unanswered for days, and then the minute an incorrect answer shows up, 2 or 3 corrections immediately get sent. I suspect it's a case of "I don't want to get bogged down by question X" but when answer Y comes by people feel compelled to correct it.

Apr 9, 2014

Press question mark to see available shortcut keys