Shared publicly  - 
End-to-end checksum validation is coming to Lucene!  This is big news, because it means Lucene will be able to detect if something in the hardware flipped bits in the index, so when index corruption occurs we can know (with high probability) it was a hardware issue and not e.g. a bug in Lucene.  Be sure to turn on the checking during IndexWriter merging using IndexWriterConfig.setValidateAtMerge(true).
It would be useful for the different files in a Lucene index to include checksums. This would make it easy to spot corruption while copying index files around; the various cloud efforts assume many more data-copying operations than older single-index implementations.
Raffaella Ventaglio's profile photo
Add a comment...