Openbaar
An update about the ext4 corruption report that has spread an unfortunate amount of panic:
I have a theory and a two patch series which might make a difference, but until I get a someone to confirm they are effective, I don't want to push patches to Linus just for take sake of sending a placebo to satisfy the panicked masses. The problem is no one other than one person who is using non-standard mount options (which I should have disabled by default, because they are known to be problematic) has been able to reliably reproduce the problem. Eric and I haven't been able to reproduce the problem at all using the default options, and Eric's reproduction using the scary, non-standard mount options may be caused by a dm-snapshot bug; I'm not sure yet.
Even when I use the problematic mount options which I know in theory could lead to serious file system corruptions if the journal has been corrupted, when I run fsstress under KVM, and then kill -9 the kvm process, afterwards the journal replays just fine and I don't see corruption which Nix has reported. (Eric has reported that when he does this using dm-snapshot, he can repro the problem. I don't know why he and I are seeing different results, and in any case Nix has claimed that the message saying that the journal has been corrupted isn't present --- although we haven't been able to get a full dmesg from one of his reproduction cases yet, so I can't say this with 100% other than for now I have to believe what Nix has told us.)
I've asked Nix to do a series of tests; in particular, do the patch series I posted on Thursday make a difference, and whether he can reproduce the problem at all when removes the highly dangerous combination of mount options: nobarrier,journal_checksum,journal_async_commit. (Basically, you shouldn't use any of these mount options unless you really know what you are doing, and the last two should only be used by developers. I will likely #ifdef them out in the next version, since more development is necessary before they are safe to use.) Nix has said that he will have time to do the request tests over the weekend, so hopefully I'll be able to send patches that are known to actually make a difference to Linus by early next week.
In general, you should use ext4 with the default mount options unless you really are an expert and know what you are doing. We've had plans to create warninigs for the more experimental bits of ext4, but that hasn't happened yet due to everyone being busy. Guess what will probably be getting a higher priority in the very near future? :-/
The evidence at this point points to the bug requiring a combination of issues, perhaps certain hardware, certain mount options, and perhaps needing to win (lose) a race where you crash just as you are trying to unmount the file system.
Unfortunately, a complex, nuanced story like this doesn't drive huge numbers of web hits, so I don't know if all of the web sites that eagerly picked up on this story last week will bother trying to explain what is going on. :-(
I have a theory and a two patch series which might make a difference, but until I get a someone to confirm they are effective, I don't want to push patches to Linus just for take sake of sending a placebo to satisfy the panicked masses. The problem is no one other than one person who is using non-standard mount options (which I should have disabled by default, because they are known to be problematic) has been able to reliably reproduce the problem. Eric and I haven't been able to reproduce the problem at all using the default options, and Eric's reproduction using the scary, non-standard mount options may be caused by a dm-snapshot bug; I'm not sure yet.
Even when I use the problematic mount options which I know in theory could lead to serious file system corruptions if the journal has been corrupted, when I run fsstress under KVM, and then kill -9 the kvm process, afterwards the journal replays just fine and I don't see corruption which Nix has reported. (Eric has reported that when he does this using dm-snapshot, he can repro the problem. I don't know why he and I are seeing different results, and in any case Nix has claimed that the message saying that the journal has been corrupted isn't present --- although we haven't been able to get a full dmesg from one of his reproduction cases yet, so I can't say this with 100% other than for now I have to believe what Nix has told us.)
I've asked Nix to do a series of tests; in particular, do the patch series I posted on Thursday make a difference, and whether he can reproduce the problem at all when removes the highly dangerous combination of mount options: nobarrier,journal_checksum,journal_async_commit. (Basically, you shouldn't use any of these mount options unless you really know what you are doing, and the last two should only be used by developers. I will likely #ifdef them out in the next version, since more development is necessary before they are safe to use.) Nix has said that he will have time to do the request tests over the weekend, so hopefully I'll be able to send patches that are known to actually make a difference to Linus by early next week.
In general, you should use ext4 with the default mount options unless you really are an expert and know what you are doing. We've had plans to create warninigs for the more experimental bits of ext4, but that hasn't happened yet due to everyone being busy. Guess what will probably be getting a higher priority in the very near future? :-/
The evidence at this point points to the bug requiring a combination of issues, perhaps certain hardware, certain mount options, and perhaps needing to win (lose) a race where you crash just as you are trying to unmount the file system.
Unfortunately, a complex, nuanced story like this doesn't drive huge numbers of web hits, so I don't know if all of the web sites that eagerly picked up on this story last week will bother trying to explain what is going on. :-(
9 eerdere reacties bekijken
I've been using ext4 for a couple of years and have not seen any problems on any of the systems I use it with (RH6 systems, mostly). However, some people (pointy-hair management mostly) are nervous about it, so I am now installing ext3 on new systems I am provisioning in the cloud... Since these are mostly systems that are up all the time - reboots are rare, whether they run ext3 or 4 should not matter too much, but I have to respect the concerns of management... :-(28 okt. 2012
+William Boyle if you have management making engineering decisions for you, you have a lot worse problems than ext4 :-P28 okt. 2012
Ted, does the mount-option "data=writeback" qualify as "highly dangerous"?28 okt. 2012
+Lars Bjerregaard "data=writeback" is dangerous if you care about the exposure of stale data after a crash. Some people consider this to be a security exposure. However, if you have a cluster file system which is doing user space checksums (I don't know if hadoopfs does this, but gfs as described in Google's ACM paper does), then this might not be an issue for you. Or if you otherwise trust your userspace daemons on your servers, it might not be a problem. But if you are running a timesharing system where you have mutually suspicious users and you don't want to chance user A getting accidentally seeing an old version of user B's mailbox after a crash, you might not want to use "data=writeback". OTOH, if you are running a single-user laptop or desktop, you might appreciate the slight performance increase of "data=writeback". (Since ext4 has delayed allocation, "data=writeback" is not as much of a huge performance win as it is with ext3.)28 okt. 2012
When I used MS WOS I had some driver problems and dead blue screens reported, that afected thousend of users that where solved in more than a year.
it is amazing all the work and all the effort for one problem at one user.
THANK YOU29 okt. 2012
It's in 3.6.6, fixed by dab43b73a7c7317f941c1314e9a77374ba8999ee (ffb5387e85d528fb6d0d924abfa3fbf0fc484071) upstream.5 nov. 2012
Reactie toevoegen...