Profile cover photo
Profile photo
Moris Urari (morisurari)
My name is Moris Urari.
My name is Moris Urari.


Post has attachment

Post has attachment

Post has attachment

Post has shared content
It's been one of those nights... I just spent hours upgrading MySQL on our production servers from 5.6 to 5.7 (starting with one, of course), then finding a bug because OpenSUSE repos still have 5.7.7 (an RC, released in 2015), and having to downgrade back and restore the MySQL slave I was testing on from another good slave.

Except the restored slave kept crashing with the most cryptic errors, even after I rsynced with --delete, even though all the files were seemingly the same as on the good slave, and I stopped the good slave before rsyncing.

I spent god damn 3 hours on this before finally realizing how tricky the situation was.

You see:

1. I ran rsync without deleting corrupted data because I trusted rsync --delete to fix it and save a ton of time transfering files - the db size is 70GB.

2. Dozens of InnoDB tables looked to rsync like they were the same between the two servers because the file sizes and timestamps were the same. Turns out rsync only trusts those two metrics for speed purposes and does not do a CRC check unless you ask it to with --checksum (which takes a LOT longer, naturally). After I enabled this option and ran rsync (which took about 10 minutes this time instead of being almost instant), it found dozens of files it previously thought were the same to sync. At this point, the slave I was trying to restore finally started without issues.

I feel like a total idiot now but lesson learned and many exclamations can now be found in my rsync notes.

Ugh, it's 3am already? FML.
Animated Photo
Wait while more posts are being loaded