Shared publicly  - 
New ClockworkMod Backup Format

ClockworkMod Recovery now deduplicates files between builds. This results in way smaller backups.
Don't delete /sdcard/clockworkmod/blobs

I've gotten a few questions about how this works, so I figured I'd make a post on it.

Basically, here's what happens:

The files being backed up are hashed (sha256, not that it matters). Then it checks for a file with the name of the hash in

So, if the hash of the file was c5273884b90d490134e7737b29a65405cea0f7bb786ca82c6337ceb24de6f5ed, it looks for /sdcard/clockworkmod/blobs/c5273884b90d490134e7737b29a65405cea0f7bb786ca82c6337ceb24de6f5ed

If the hash file is found, it continues on to the next file. Otherwise, it copies the file to the blobs directory with the file name being the hash.

ROMs and user data, for the most part, do not change too much between builds and backups. So, your APKs, system files, etc, are generally only stored once. This saves a ton of space. Especially between incremental backups.

Some of you may be thinking "well, how do I delete a backup?".
First, never delete the blobs directory. This would actually delete all your backups by rendering them unusable.
Simply delete the usual backup directory, and the next time you run a backup, all the unused hash files will be automatically delete (a process known as garbage collection). The recovery will show "Freeing space..." while this is happening.
Anthony Takata (Tsaukpaetra)'s profile photoMayank Kapil's profile photoGuljit Jani's profile photoIan Sterling's profile photo
Is this implemented in all versions of CWR now, or only certain newer devices? As in, will I see this for the Bionic or (OMG, funny) my DX that I gave to my daughter?
Copy that, over. I'm currently using Safestrap, but it eats up a ton of space on the internal SD and I'm still not 100% convinced that there's not a performance hit when in the second slot, or the "safe side"
I simultaneously love and am somewhat disturbed by the growing usage of the word 'blob'
Interesting. I'll have to Google what blobs mean. Noobing it up!
+Koushik Dutta You did a great thing don't get me wrong. I imagine new threads are opening up as we speak. Why don't my backups restore on cwm6+.?
Backwards compatibility is crucial for ANY sort of backup/restore system. That WOULD be a dickmove if they weren't. And bad business sense. So, since you don't seem (and never have) seemed like a bag of dicks, I had no doubt it would be backwards compatible
I think he's imagining all the XDA peeps deleting the /blobs/ directory just because it doesn't look right or something :D
How does this affect titanium backup's "restore from nandroid backup" feature? I would imagine that function requires a full image...
If all backups now use content under /blobs, what is the procedure of backing up a single full backup to external storage (outside phone) without copying whole /blobs directory?
how would I go about deleting a particular backup to save space? Delete the directory? 

I need to be able to not use up all my space.
Just to make sure I understand- if I delete a backup, I would then need to perform a new backup to have a complete backup? Otherwise any existing backups may be incomplete?
Any way to get CWR 6+ on my DX? It seems that I can only get
How do I view the contents of the blobs directory? Root Explorer show an empty directory and a root terminal tells me I don't have permission to view the contents?
I can't even ls the contents of the clockworkmod directory.

drwxrwxr-x root     sdcard_rw          2012-07-17 06:36 clockworkmod

Looks ok, but

opendir failed, Permission denied

This was from a fresh backup taken by the new clockworkmod (maguro, rooted). The file rights seems broken. This means you can't adb pull the folder either.
So our backups will be encrypted?
+Koushik Dutta It works if I do the adb stuff with clockworkmod booted (after mounting /sdcard), but not when the phone has booted normally.
+Koushik Dutta on my VZW GNexus I can see the contents of the blobs directory as root in /data/media (not as a regular user), and not from /sdcard. This will complicate my usual sdcard backup procedure...
+Koushik Dutta is there a way to have backup skip a directory? I have a few gigs of music synced with google music and it backs up all of that, which i definitely do not need.
So what happens to the contents of /data/media/(blobs) when I format /data during a full wipe?
+Koushik Dutta Ok, so listing /data/media works when the phone is booted, but only as root. listing /sdcard/clockworkmod doesn't work at all unless you've booted clockworkmod. This makes it impossible to extract the /sdcard/clockworkmod folder in the usual way when the phone is booted normally.
Aha!  I see what you mean.  Now that I look at it, /data/media seems to be a mount point that does include items that are mounted elsewhere.

+Koushik Dutta i did a backup and it seems that the space is taken but when i check on blobs and backup folders in cwm folder they are empty .. is that normal?
+Koushik Dutta   aha got it .. but do we need the folder clockworkmod now that on the sdcard i see it put the backups folders normally but they are empty
Here's what I see using Root Browser-

/sdcard/clockworkmod/blobs - empty directory
/sdcard/clockworkmod/backup/<date-of-backup-using-cwm6> - empty directory

/data/media/clockworkmod/blobs - nothing happens other than "ROM toolbox pro has been granted superuser permissions" pop-up

/data/media/clockworkmod/backup/<date-of-backup-using-cwm6> - shows 6 files.

/data/media/clockworkmod/blobs is finally showing files using Solid Explorer beta.

Now size bewilderment. blobs dir size is 2.7 GB while my last backup (taken 2 weeks ago) size is 2.2 GB. Errrr... wasn't it supposed to save space?? I guess I'm missing something or doing something wrong.
+Koushik Dutta I think the way you set permission on backup directory currently will cause problem for user wanting to delete their old backups. It seems to be because of the 0077 umask set in the new ramdisk. The blob directory is good with this umask though because it's not supposed to be modified by users.
+Koushik Dutta Yes, that's true. So, no backward compatibility..? :( Then again, reading what's happening with blobs (hash n stuff) makes sense why first backup with cwm6 won't depend on older backups.

Can I suggest a few fixes?
1. Make backup folder and blobs content accessible at sdcard.
2. When taking backup, the progress log overlaps the progress bar. Please place the bar somewhere else.

Also. ROM Manager is showing "current recovery: cwm" even though touch is installed.
Love all your apps yaar. :)
+Bikram Agarwal Backward compatibility will be really complicated (not hard but take long time to code and debug) to do. I'm sure. New code being clean is good, sometimes making things work with older stuffs just isn't worth it.
Keep getting error when I try to delete a backup
I'm getting an error as well trying to remove a backup through rom manager, my file explorer, and terminal using rm -rf. Any suggestions?
+Justin Beeson The file permissions are broken. As root go to /data/media/clockworkmod and remove the files you want.
I don't mean to break your balls here but as a paying customer shouldn't I expect some level of product testing that doesn't involve my device.
Is this information something I need to know. Or can I continue being a dumb ass and just click back up, our restore as I usually do? In other words, is there anything I NEED to know about this update, so I don't lose/beak anything?
WTF is happening here. I've deleted /data/media/clockworkmod/backups yet My storage is still showing 6 gigs left when it shoudl read 9. 
+Ryan Lestage You'd need to delete blobs too if you want to clear everything out. or run a backup, and let it get garbage collected. The details of how everything work are in the post.
Ok thanks. 

So should I put the previous CWM back on until there is a fix for version 6?
Ok nevermind last comment. I have 9 gigs free again +Koushik Dutta 

So What's the plan moving forward. Install previous CWM?
Any way of triggering "garbage collection" (getting rid of unused files in blob-dir, because backup was deleted) without having to do new backup ?
Backups do seem quite a bit faster.  :o)
now i have three back ups I cant delete
My mistake just can't delete the two old ones
I wonder if you've considered the limit to a FAT32 file system. A FAT32 directory can only contain 65,536 directory entries. A filename that is 64 letters long takes up 6 directory entries, if I'm not mistaken. So the blobs directory in a FAT32 formatted SD card can hold only 10,922 files. As of right now, my blobs directory already has about 6400 files. What happens if the limit is reached?
Ok can someone confirm this thinking:

If I run a backup, it will look for directories in clockworkmod folder. If it doesnt find a folder corresponding to a particular blob, it will get cleaned up, thus saving space. Correct? I just want to make sure I know what's going on.
+koushik dutta my question is why does it want me to pay for touch again? Why didn't it have the same restore feature all the other play apps have?
Do you have additional plans for the blobs beyond backups? If not, I'd suggest moving the directory to be under backups, which will make it less likely someone deletes it before they RTFM.
Nice idea to hash the files! :-) Thank you for all your work!
But now, as there might reside unreferenced blobs until the next garbage collection, an option to delete backups from inside the recovery mode would be very useful.
It's probably impossible to give novice users an understanding of blobs und hashed files and why they shouldn't delete them.
Sweet, I was just wondering when someone would make some sort of dedup for backups. They're such a huge space waster otherwise.
It also might be a neat idea to grat the ability to "blob-ify" an existing old-school backup, freeing up further space.  Although I'm not sure that's worth the effort.
So how do I erase the first back up's from the previous version that are currently undeletable.
Thank you for your continued work, and I look forward to seeing the new version hit the HTC One S soon.
So taking a backup with doesn't fix the old broken permissions. If you delete the clockworkmod directory and take a new backup the folder is set to drwxrwxrwx but still owned by root:root instead of  media_rw:media_rw. Shouldn't the folder be owned by media_rw in order to play nicely when mounted as /sdcard?
A backup taken by will have blobs non-world readable. So if you do the normal adb pull /sdcard/clockworkmod clockworkmod when the phone is booted you will get the backup folder but not the blobs folder. I'd say that's a fail.

failed to copy '/sdcard/clockworkmod/blobs/a7a3a9d11395b782092dc7b442c98ddbe532682fcc3549bf9561dd6bfa321e22' to 'clockworkmod/blobs/a7a3a9d11395b782092dc7b442c98ddbe532682fcc3549bf9561dd6bfa321e22': Permission denied
If I did a factory data reset would that get rid of them?
+Fredrik Duprez Thats only for backups prior to Its just that making a new backup doesnt fix the permissions. I'll release a fix for that.
+Koushik Dutta Great, please make sure that the blobs directory (or actually it is the files there that get 600 and owned by root) This is on without a clockwork folder when I took the backup.
+Koushik Dutta Good stuff! Best would be if the files behaved as if they were put on the sdcard by the normal user. A typical use case would be to extract the files from /sdcard using adb or MTP, wipe the phone for some reason, restore the /sdcard/clockworkmod folder as a normal user, and be ready to restore backups.
Still cant delete original backups can you add a purge option or something.  
+Koushik Dutta I can confirm that rw works on all files (removed the clockworkmod folder via MTP) When I push them back with adb, they will get media_rw:media_rw as owner. Not a problem for clockworkmod I hope?
+Richard Buelt If you run a backup with, it should fix the file permissions on the existing files as well. After that you should be able to remove the old backups.
Nope still cant delete the original backups made from the first version 
+Richard Buelt +Koushik Dutta Then that would be a bug. I only checked that the blobs folder and files got updated permissions, not existing folders under backup. You have to ask Koushik if that's a fixable bug.
Well thank you for trying to help me anyway I appreciate it
+Richard Buelt What you can do to fix it is to boot the phone into Clockworkmod and connect it to your computer. Then use

adb shell rm -rf /data/media/clockworkmod/

This removes all clockworkmod folders and backups and makes it possible to start afresh with or later.
+Richard Buelt Yeah, delete it in a root file explorer or via adb shell in recovery. Future recoveries won't have this issue. Sorry for the trouble.
That's okay just glad your helping me figure it out
+Koushik Dutta unsure if you do this by default (or if it's needed by ROM Manager), but you should chown /sdcard/clockworkmod/blobs/ as root (similar to that bug from a few months back) so at least users without a removable sdcard will be safe.

Edit: reading back through the comments, this will probably cause more issues then solve then, disregard. 
Could you clarify this for me: are the unused hash files only deleted if the corresponding backup directory is deleted?  So if I want to keep the backup files, I simply have to make sure to keep the directory?
+Koushik Dutta I have a nandroid from 3 days ago done with a the previous version of cwm I think it was v5.5.04 I have now updated to 6. Will my previous nandroid work with v6 ?

Edit:   I just flashed my backup. worked like a charm without any issues.   Thank you!
So as we don't delete the blobs folder we can move the backups to our computer and remove from the phone. Am I understanding that correctly? 
+Koushik Dutta Thanks for, it's working great now. There's still something I wanna ask though.

The first boot after restore take a very long time I wonder if that is because file timestamps aren't kept now (I guess odex files need to be rechecked but I haven't confirmed it with logcat to be sure)? I checked the format of dup file, that looks very simple, but it's a simple solution that works.

So if slow first boot problem is because of what I said, adding more fields into the dup files and writing commands into backup and restore scripts should be trivial to fix it.
+Ngoc-Thuan Tran I see the same problem: too much time to boot. How much should i wait? It has already spent 10 minutes..
+Gianmarco Rogo  That's way too long. For me it's long but only around 2 minutes. Did you clean up blobs and everything from previous backup with older CWM, before making the new backup that you're restoring now?

Also a logcat of what is looping inside your phone might also shed light on what failed.
+Ngoc-Thuan Tran Nope. I didn't clean anything. I've just made a nandroid. Then a theme caused bootloop problems. So i decided to /restore. But I'm stiln not able to boot.

logcat, how? Could you help me understanding that?
+Gianmarco Rogo Logcat is a way to get what's going on your phone.

You're gonna need your phone adb driver . This might have your phone cover or you need to find that yourself. Next you'll need adb binary for your PC OS, either install Android SDK or find the binary and accompanied lib on the net. You should be able to search for guide on how to use adb and use it to get logcat.
+Gianmarco Rogo The only thing I can say from that. It's FUBAR. Looks like something critical in Android didn't get restored correctly leading to services unable to load.

It might prove useful to +Koushik Dutta  though.
+Gianmarco Rogo It looks like the permissions in /data may be off, causing the system to be unable to write to it?

Try going into recovery and in adb shell type the following:
mount /data
ls -l /data
ls -l /data/system

then pastebin me that
+Koushik Dutta I've already solved that:
- from recovery, since i was unable to restore, i did factory reset
- then advanced restore -> restore /data
- reboot
everything seems ok.

What was wrong?
+Koushik Dutta I'm not talking about restoring twice (i tried 7 times at least).
The difference is that in "advanced restore" I just selected /data ..
+Koushik Dutta Did you miss my comment regarding slow first boot after restore? Any comment

Partial Quote:
The first boot after restore take a very long time I wonder if that is because file timestamps aren't kept now (I guess odex files need to be rechecked but I haven't confirmed it with logcat to be sure)? I checked the format of dup file, that looks very simple, but it's a simple solution that works...
+Koushik Dutta I'm still confused how the Freeing space (or garbage collector) would work. So I'm running a fully functional ICS ROM and I want to try out the new AOSP JB ROM. I back up the ICS ROM, full wipe, and flash the JB ROM.  While in the JB ROM, I install a few apps. Now I want to restore to the ICS ROM, but don't want to lose the JB ROM, so I back up the JB ROM. Because the JB ROM now has less apps than my original ICS ROM, will CWM garbage collector now delete the apps in the blobs that were backed up with the ICS ROM?  Haven't tried this out yet, but that wouldn't be a good thing for an active flasher like me :).

I also confirm with the long first boot after restore +Ngoc-Thuan Tran reported.  I also experienced tons of FCs, all apps FC after I ran a backup and rebooted the phone. Restore from the backup took care of the FCs but that was strange.  I'm using Touch CWM for Nexus S.
+Koushik Dutta question can you make a backup of a stock ICS ROM ( Verizon ) then go to a Jellybean ROM. Then make a back up of said JellyBean ROM . Then restore the stock ICS ROM with no issues ?
I ask because I don't want to do to much and lose my new stock backup . Also I'm not sure if things will get deleted that are needed. For the stock ROM by making a backup of the JellyBean ROM .
+Ngoc-Thuan Tran I found the cause of the slow boot. It was due to me not properly restoring file times, causing a dex opt that regenerates odex files. Will release a fix soon.
+Koushik Dutta So help me understand this. The old backup folders - which were nice and easy to copy off my card, being tarball'd and all that - have no bearing on the new blobs folder? Is there an easy way to make offline backups of the blobs folder? If I redo my Galaxy Nexus, this folder stays in /path/to/clockworkmod?
I am interested in that too +Ian Sterling. Off device backups are key, particularly with a device with limited real estate like the Nexus 7. I didn't realise things had changed so much when I did my initial, pre rooting backups. I had done a couple more before I realised. My fault for not RTFM but now I am intrigued about the best way to manage backups in multiple locations.
Tried a backup, Installed Jellybean. Then tried another backup.

Then I tried to restore to the old backup. I got an error. I had to factory reset, install cm9, then copy a backup made with the previous version of CWM in order to get my phone back. Ugh. So far, I'm really not liking the new blobs backups.
I got an error restoring my data, then took a look at data.ext4.dup. Noticed that some of the lines containing long file names (e.g., some of the files for the Kindle app, which are long Base64 strings) had been split across multiple lines. I joined the lines back together and the restore went perfectly. I'm on a Nexus S 4G; not sure if that might make a difference, but thought I'd let you know.

I actually like this new backup model. My only question: when you say that garbage collection deletes the unused hash files, is it checking the .dup file for every backup before it does this? I would assume so, but I didn't want to attempt to make backups from different ROMs until I knew for sure (so I wouldn't wipe out backed up data from apps in one ROM but not the other, for example). The previous comment from Ian Sterling makes me wonder, however.
Has the Recovery been updated yet to include the ability to "garbage collect" after deleting a backup yet (without having to do a new backup)?  My Nexus 7 really needs some extra space - but deleting backups isn't as straightforward as it used to be.
I'm a bit concerned about the new backup format. Android seems to be profoundly unhappy with having a folder full of thousands of little files. Even ASTRO file manager takes 3-5 minutes to tell me how big the "blobs" folder is. Such folders take dramatically longer to copy to an off-device backup than a few large files. Moreover on my Galaxy Nexus with Jellybean, the files that show up when mounted to my computer by MTP depend on what has been enumerated successfully by the media scanner. I have had the phone plugged into my computer for over 10 minutes now, about 20 mins after a nandroid backup, and I'm looking inside the blobs directory: 10,208 items and still counting. Windows is very unhappy with this state of affairs.

And after all the hassle, is it saving any space on my backups? My first backup gave me a 2 GB blobs directory. My second backup, only a few days later, with only a few apps having been upgraded on my phone... well that took 30-45 minutes and now my blobs directory is 4 GB. So no evidence of space-saving yet.
Final count: Over 11,000 files in blobs. Now I ask Windows to select the 2,293 items that are most recent (from the newest backup presumably) and I right-click to see if I can count up the file sizes. That window is now "Not Responding".
+Koushik Dutta +Charles Troster I think that can be fixed with storing files under blob like storing keys, values in hash table. Say we have files with sha256 start with xyz then they will be stored in ./blobs/xyz/<files>
This is very funny. As what I have said in my previous comment is already in

Also file timestamps are fixed and users can delete old backups or cleanup old blobs from inside recovery now.

Great work. +Koushik Dutta 
This is an interesting feature... I can't say I like it at all.  The idea of incremental backups is great if you're doing nightlies or updates to ROMs, but frankly, it's a pain in the butt if you're flashing several different roms, and only want to backup ONE of them without running nandroid again.  I love that I can just go into my backup folder and copy the directories I want to keep, rather than the BLOB directory with several backups throughout it.  

Anyway to replicate the old CWM Recovery functionality, +Koushik Dutta ?
Really looks like a matter of speed vs. usability. So far, I'm really worried about the decreased level of usability.

My biggest concern is simply copying a single backup to offline storage. I have no idea how to do that without copying the whole directory made out of thousands of files. This actually fights back the idea of taking a speedy backup, when it actually takes a lot longer to copy it offline.

I made two backups of the same ROM and initiated garbage collection before these from CWR. Then connected my gnex to MTP, and after ~5 minutes loading the content of /blobs, it showed 7949 files and 2521 directories. (~1.7GB)

Finally, I started copying the /blobs directory to my SSD. After a while it started copying files, and the estimation of the copy process was "about an hour" for a few minutes. Then I cancelled.

So I'm just bringing my opinions and findings here to discuss further this new backup model.
+Koushik Dutta I think it's a good approach.

Personally what I'm probably going to do is, keep using incremental backups, until I'm going to make an offline copy. Then I'll use full tar/yaffs backup.
Being able to choose the new method or old method would be great, because as others posted, the new format with many tiny files is very hard for Windows to  effectively copy. While that may be Windows fault, a lot of us are stuck with it, so being able to do the old big backup would be great. Thanks for keeping the app updated and trying new things!  (My blogs directory is copying - according to Windows: About 1 Day Remaining...)
Even with the files structured in subdirectories it's still confusing the heck out of my GNex. Every time I use iSyncr to sync up my podcasts, it triggers the media scanner which then takes a lo-o-o-ng time to complete, and later in the day I find that "sdcard" has taken up 10% of my battery. And now the external.db of android media providers has tens of thousands of extra entries under "files". There is now a multi-gigabyte thumbnails file in my /DCIM/.thumbnails directory and I have a feeling it is related to the massive count of non-media files being tracked by the media scanner.
+Charles Troster The /sdcard/clockworkmod directory should have a .nomedia file that prevents media scanner from checking there. Please verify.
+Koushik Dutta After "freeing space" (Should it even do that if I haven't deleted an older cwm6 backup?) it moves onto backing up and always fails on /data? It leaves me totally unable to backup in cwm6. Is this a bug, or have I really derped something? cwm6.0.0.1 i9100 cm9 nightly
+Koushik Dutta Any idea when the new build that allows us to pick our backup style will be available? The old backup format allowed me to store my backups offline much easier, and gave me a whole lot more piece of mind.
+Ryan Cormack Also, if you can via adb shell
mount /data
ls -l /data/data/
+Koushik Dutta I have ~4.1GB free on my internal sdcard. Yes, CWM came with the CM zip 17/7 nightly that I am on.

That file doesnt seem to exist any more. I pulled the log again, and this time it failed on: 
Error copying blob ./data/

Which doesnt exist either. That closest to it was the file "CAIiEL76U1QuAo7z82vYsXCUGrYqFwgEKg8IACoHCAow8dTjAjCY4Rswu8IF"

So I ls-l it and output: 
# ls -l CAIiEL76U1QuAo7z82vYsXCUGrYqFwgEKg8IACoHCAow8dTjAjCY4Rswu
ls -l CAIiEL76U1QuAo7z82vYsXCUGrYqFwgEKg8IACoHCAow8dTjAjCY4Rswu
-rw-------    1 app_73   app_73       11012 Jul 25 15:03 CAIiEL76U1QuAo7z82vYsXC
PHU4wIwmOEb/pr #"

I am happy to delete my Google Currents app (That looks to be what is causing the problem), or flash a new nightly if it has a new CWM, flash a new CWM that doesnt come with CM, or just delete my blobs and start over. If it is a bug, I am also happy to give you whatever to fix it.

the issue is how do I take the backup off the device?
It has become a bit cluttered. Right now I'm copying the backup and blobs folders to my PC, but what happens when I make another backup, maybe of a different ROM? I'll have another folder in the backups folder, and I'm guessing a dozen more in the blobs folder. To be safe one should probably sync the entire CWM folder.

The old method had one folder per backup. It was bigger, but it was tidy.
I just had some hard time with the new recovery. First, I tried to make a backup (rebooted to recovery). It hanged to "freeing space" before backing up /system, I waited around 20 minutes. Then battery out & back in, tried to do same again - this time it went through freeing space, but while backing up system it rebooted recovery.

Before this, I pretty much did the following:

- At morning took a backup of my current ROM through ROM Manager

- Later on removed couple of older backups (made through ROM Manager) of the same ROM through ROM Manager, leaving only the one created above

Then I rebooted to recovery and did what I described previously.

I've done many backups until now with this new backup format and this is the first time I've had issues. I have to say though, I'm haven't really seen major increase in "incremental" backup process at any point.

So now I would like to manually clear the /blobs dir, but I realized I cannot do that since I need my backup made with new format (switching between two ROMs). I flashed older recovery and made a full backup, and was going to restore the backup of that other ROM without success as it was made with this new format. So now I'll be flashing again to restore the other ROM backup, and re-flash to make a full backup of it. I'll stick with 5 series until there'll be a new update with option to use full backups.

Sorry +Koushik Dutta , I don't know if I have any log for this (any more), at least I cannot find any under /clockworkmod...
This seems to completely hose FolderSync, the app that I use to back sdcard/ up to my local system.  It is taking an average of 20 seconds to scan each folder in blobs/ because, I think, of the enormous number of sub-directories in the blobs/.  I don't think Android file system is real happy with folders containing that many sub-folders and takes a long time to open such a folder.  Just opening it in Astro can take ten seconds or more, again I am sure because of the number of subfolders.

With some 3264 folders in blobs/ it will take FolderSync 10-12 hours to scan all the files in it and decide nothing has changed.  I haven't tried it with Titanium MediaSync but given what Astro shows me I'll bet it's the same.

This is a fatal flaw with the blobs/ change.  Please change it back.  At the cost of 10 hours or more for a backup scan (vs. minutes the way it was before) I don't need the improvement that the change is supposed to provide.

Implementing a hash using the file system is probably not a good idea if that results in an enormous number of files and folders.  If blobs were invisibly implemented all in a single .db file rather than in tens of thousands of smaller ones the impact on the overall system would be minimized.  I strongly urge you to back this out and re-implement it in more conventional ways.  Yes, you will have to implement methods on your database that you now rely on the file system for but that's the price of independence.

I'm really unhappy with being stuck without external backup as they are essential to what I am doing and I eagerly await the option of reverting to the former implementation (including reversion of the data to the former location and methodology.)
+Koushik Dutta What happens if the sha256 of two unique files are same? I know sha256 is very much collision proof. But not completely.
+Koushik Dutta May I suggest another change to the subdirectory layout: Instead of having a single level of subdirs ("000/" to "fff/"), split them into three levels ("0/0/0/" to "f/f/f/"). This avoids the problem that you have up to 4096 subdirs in "blobs/" (which is a bit better than having around 15000 files there, but still causes problems when the system is trying to enumerate them). Instead, you only have 16, which should be very fast.
So just to clarify +Koushik Dutta...  When restoring settings from one device to another... You can copy the backups/ and blobs/ folder to a different device (same model) and restore... Whereas in previous versions you only needed to copy over the backups/ folder or the specific backup in that folder... is that correct?
I think +Nic Romero just helped me figure out why I couldn't restore a backup--every time I make a backup I use Root Explorer to change the backup file name so I know which rom it is.  I'm thinking that changing the name of the backup file caused the restore to fail (I got an "error while restoring /system!" message every time).  This time I didn't change the name and it worked!  Going back and changing the file name to it's original name did not fix the problem with previous backups not restoring, still get the same error message, however.
+Koushik Dutta Still on track with an update to bring support for both backup methods? I've had a habit for a while now to first check Store for updated apps, and then RM for updated recovery :-)
This disaster is even worse than it seemed.  I went back to (or whatever) and deleted blobs.  CWM works ok now but I still can't use anything to sync /sdcard to an off device backup.  Thousands of Permission denied messages accompany any attempt.  I'm afraid that the sdcard file system is hosed and I have no idea how to recover my phone from that.

I can't believe this.  Anybody got a suggestion.  Koush?
+Bobs Dragon This was an issue with an early version of the recovery.

Fix it as so in recovery mode:

mount /data

chown -R 0.0 /sdcard/clockworkmod
chmod g+x,g+w,g+r,o+x,o+w,o+r /sdcard/clockworkmod
Thanks Koush.  Which mode is that?  I don't know of any that allows that form of chmod.  How do I get to the mode you speak of?  Thanks.

Update:  I found that from either ADB or from a console on the device that the chown and chmod commands must have busybox in front of them to get them to the right utilities (I would have thought that all the busybox links would have accomplished that but...)

The mount /data command fails either way.  mount takes two operands if I remember correctly.

At any rate I did the two commands and nothing has changed.  Programs that try to sync the sdcard to an external system don't work and haven't since my ill fated experiment with CWM

If you could create (and test) a script that would fix this file system mess up I'd greatly appreciate it.  The loss of the ability to sync externally is a huge one to me.
I made the problem go away by syncing the sdcard externally using SMB rather than SFTP.  All is well now.  Sorry for the confusion.
someone tried to restore some backup from this new format? Doing this has given me errors telling me that finds no image file
Thanks for explanation. Was wondering why my backup folder was only 17MB...
What if I would like to create a CWM5 all in one folder backup for some purpose? Can it be possible using the new recovery?.
Anybody know why Freeing Space... takes 15 min? Yesterday I made my 3rd backup and the backup stayed in Freeing Space... for 15 min. Not sure why because I have not deleted any backups. Also no .nomedia file on the Nexus S/Nexus S 4G ClockWorkdMod folder. 
I going to twrp. I paid for touch and upgraded phones and I no longer have access to it unless I pay for it again. I believe $7 is a decent amount for any app. Thanks for addressing this issue I have raised a couple times. + note the sarcasm+
well, we only talked about backup ... but how to restore? i backup with that blobs folder in there, i test MIUI .. then I am not happy with it and I restore .. restore finished in seconds... and I am still in MIUI .. please advise how to restore to my old rom, thanks
OK, so I read this whole thread and have a question. Using the new method of backing up, lets say you make a nandroid on Monday, and one on Friday. On Saturday, for whatever reason, you want to restore the backup you made on Monday. Does the garbage collection routine thus make Friday's backup no longer viable?
+Koushik Dutta Any word on when we'll be able to toggle between methods of backing up? The older method was more portable and made for easier offline storage. The new method's backups take a long time to copy off the device.
I normally just keep the last 2-3 backups on my phone to save space and copy the older backups on my laptop.

Two questions:
1. When saving new backups on my laptop, do I also have to copy the blobs everytime?
2. When I want to restore the latest backup do I also need the older backups on my phone?
So I think the best method is: before you make a new backup, got to ROM manager and delete whatever backups you don't need. Then do a backup and it will free up the space that was used by the deleted backups.  Next make a zip of the folders while on your device and then backup that zip file to another device or cloud.  It's a hassle but mostly just a change in procedures that we all hit because there wasn't a huge sign in our faces when we upgraded to 6. 
Well I have 2gb worth of blob files on my 16gb GN no like it , I did a back up and restore hoping it would free some storage but no luck , what gives?
Hey dude, first of thanks for CWM.

I only ever have 2 backups on my phone at a time but if i wanted to format my sdcard i have a problem now. Would it be possible do let us choose which type of backup we want?  I would now have to move the ~700mb + the backups. I know it's sounds picky but i would appreciate the option.

greetz Tobias
+Koushik Dutta Any chance of adding support for ext3 for nandroid backups?

Sorry, fstype 2 option in the recovery.fstab to be exact. We have a phone that got left behind and our best ROM is Froyo, Motorola Triumph. So, we need to be able to have ext4 and ext3 for our nandroid backups. I can rename the backups and the names in the md5 file to ext3 and they work just fine.
After i choose tar as backup method, can i delete the blobs folder now?
+Koushik Dutta I love your work, keep it up. I have the GS3 and I got an error today when completing a BU using Rom Manager. Come to find out that my clockworkmod folder is on the phone and not the SD Card and the folder is 8GB. How did it get onto the phone memory and not the SD Card? What is the proper way to fix this issue? Thanks again.
It looks like this change to the nandroid format/storage method, in 6+ has broken apps like Nandroid Browser and Titanium Backup that were previously able to extract files direct from nandroid's. They don't even see the app data now in those new nandroids.  Still works for old backups...  Anyone know of a way to explore these "new" nandroids?  Or a fix for these apps I've bought and can no longer use for this purpose? Thanks. 
Is there a way to create an old ext-backup with CWM6?
It would be helpful to create backups which can be published in forums etc.
Thank you so much for all the information you have provided.
I recently tried to restore a backup today and it failed.  It was of a backup I had made on my Samsung GS3 (Verizon) of the Stock ICS before I tried out CM10. CM10 isn't ready for me yet (too many bugs), so I tried to switch back. However I have made several backups while running CM10.  Did the "blobs" mess up my older backups from stock ICS?  None of them work. I can only restore the CM10 backups. Can you direct me to some help?
How does one turn this off? I don't backup regularly and move my backups to my PC. I want my backup files to be easy for me to manage and keep track of, not a huge 6GB mess that I can't export.
+Paul Klapperich, it's pretty easy to disable. You reboot to recovery and change the backup mode option from dup to tar.
Now I have a GNex and I'm almost out of internal memory. I've copied my entire clockworkmod folder to my PC. I was wondering if I could delete the contents of the clockworkmod folder off of my phone (since it's all on the PC) to regain some storage space, and then only return the deleted contents to the phone when I need to do a nandroid restore without any problems occurring. I wouldn't want to go do all that only to arrive upon a large problem when I try to go back to stock JB JRO03C or anything like that.
That makes sense now! I was wondering why all these incremental backups had dramatically decreased in size. I remember when all my images used to go up to 1 gig and now the blobs folder holds everything. =o

Thanks for the clarification!
Not knowing the new backup format, I deleted "blobs" folder.  Does it mean my earlier backups are no longer "restore-able"?
I have a similar problem as Don Zhao ...I deleted 'blobs' folder in error.  I now have MD5 errors trying to make new backups. Any kind-hearted advice?
I have something that I need to recover from a previous backup of CM10 on a Galaxy Note (Papyrus db) - is it possible to restore a backup to a virtual machine (on Fedora x86_64 in my case) so I don't have to mess around with the current install (a more recent CM10)?
Thanks, Phil.
May I suggest that you add an option to turn this off.  Case why I might want it off, I do a back up (I do a few backs) and I want to grab a few of them to put on some storage somewhere else.  What blobs do I need to copy to copy the backup?  Do I need to md5 all the blobs?  I would like to move the backup to a more reliable media, while keeping a few on the sd card if I need.
I have only one backup and it's taking up 3GB. Wtf? How is that possible? Can I still restore if I delete blobs. Since storage is so precious because they won't let me store apps on a sd card anymore. 
+Stuart Marks where is that option? I've got recovery on my nexus and "backup mode" isn't an option I can find anywhere. I even tried installing a non-touch recovery. There doesn't seem to be any option to go back to tar backups.
I'm still waiting for the option to switch backup types. I'm not seeing any option that +Stuart Marks suggests, and my clockworkmod folder is approaching some 7gb in size. (and deleting all them damned tiny files takes HOURS.)

BTW, +Koushik Dutta, in case you're wondering (and this isn't the first time I've asked for this answer either), I'm running a maguro with

So where is this option to change backup types? I'd really like to switch back to the old tarball style backups. Or is that something that's only available to premium members? (I'm willing to pay the premium for CWM Recovery, if such a thing exists.)
will a older TAR CWM work with 4.1.1 because this blobs folder is now over 8 gig. i keep 4 backups one for each rom and one pure.
now Koushik i only ask because i want my space back and i liked having the upload all in one container to easily backup. i have been a premium user since the OG and i bought touch as well, but i cant have THE BLOB absorbing my phone's memory whole. as well as the new backup system takes WAY longer. can you bring back the TAR setting? or answer if i can backdate myself with CWM and have it still function normally on 4.1.1.

by mistake I flaseh the bugged v6 and now after flashing back v5 I cannot solve the invisible backup folder problem...

I am trying all possible solution, with no avail...

I have now v5 but when making a backup the clockworkmod folder is visible only under /data/media and not form sdcard

pls help
it seems cwm v5.5.0.2 is still cretaing /clockworkmod folder with root only permissions

/data/media # ls -l
ls -l
drwxrwx---   17 media_rw media_rw      4096 Nov 15 06:36 0
drwxrwxrwx    3 root     root          4096 Nov 15 09:19 clockworkmod
drwxrwxr-x    2 media_rw media_rw      4096 Nov 14 19:10 legacy
drwxrwx---    2 media_rw media_rw      4096 Nov 14 19:07 obb
I managed to cahnge permissions to the following, but clockworkmod still not showing up

/data/media # ls -l
ls -l
drwxrwx---   17 media_rw media_rw      4096 Nov 15 09:33 0
drwxrwxrwx    2 media_rw media_rw      4096 Nov 15 09:38 clockworkmod
drwxrwxr-x    2 media_rw media_rw      4096 Nov 14 19:10 legacy
drwxrwx---    2 media_rw media_rw      4096 Nov 14 19:07 obb

pls pls help cos I am stuck
ok... I had to mount /data first
now I have

/data/media # ls -l
ls -l
drwxrwx---   17 media_rw media_rw      4096 Nov 15 10:07 0
drwxrwxrwx    3 media_rw media_rw      4096 Nov 15 10:17 clockworkmod
drwxrwxr-x    2 media_rw media_rw      4096 Nov 14 19:10 legacy
drwxrwx---    2 media_rw media_rw      4096 Nov 14 19:07 obb

Iike in my last comment... I am rebooting now
nothing... clockwork folder is till not visible...
I can copy folder from /data/media to /sdcard with toot explorer but the copy will end up having root:root owners and still not visible

I am lost...
+Giovanni Stoto Make sure you do not have the file .nomedia in that folder. That'll hide it from various operating systems. I'm not sure this will address the problems you're having, but will at least allow you to see your backup folder from a PC.
the problem must be somewhere else maybe....

I mean, I have deleted with adb the troublesome /data/media/clockworkmod folder created with the buggy cw, v6, I have flashed back the good cwm v5.5 and when creating backups i still cannot see the folder...
If i want to be able to take a copy of a backup off of my phone and store it on my PC, how would i now do this? I assume i can copy the whole clockwork mod directory, is there was to 'export' a single backup to my pc now?
when i try to restore a backup it says that it can't find data.img :/
shouldn't it use the blobs instead of data.img?
+Koushik Dutta update for gnex maguro now has the option to chose your backup method. THANK YOU VERY MUCH for implementing this. =) I'll be purchasing the paid version of ROM Manager shortly. =)
This is really not so good for me, I usually use only one nandroid backup and it had 300-400 MB now It's almost 900 MB! And when I put two, blobs was 1.2 GB and in total it was 1.55 GB! That is not saving space! I could put 4-5 regular backups...
Ted G
its completely ridiculous:
2.8G    blobs
ls -la blobs/|wc
     3978     23868    238680

thats just two backups.  not a well thought out method of doing this at all.  moving on to something reasonable...
+Koushik Dutta Is this option already available? - "I'm going to allow people to switch their preferred backup format. Except a new build soon" - copying/storing "blobs" to PC via USB is still a painful long process :-(
+Stefan Gauntlett thanks, so it is only tar as a zip Format? Will this setting be permanent, even if I start to backup rom from CWM?
+Vincent Vega Yeah that's the one and as far as I'm aware yeah, change the setting and it will use that format regardless of what environment the backup is started from. 
+Stefan Gauntlett I can confirm, this is permanent, even if you save your ROM from CWM recovery :-) Great Stuff, can't stand the "blobs" :-)
I just notice the blobs folder and google to know more about it. Indeed it a nice backup plan. :)
Just want to know, Im using link2sd and link all apps to ext4 partition on my 2nd sdcard. Will it be backup as well? Then, how about restoring the backup later will it restore again to the ext4 partition after I ran the recreate mount script on link2sd before restoring nandroid?
The format of backup and sdcard/ext_sdcard should be read at CW loading despite of "defaults" values, and make it visible what is current set :(
Also ROM Manager is not seeing the ext_sdcard :( like CWM 5.5 used to work.
Hi, I've just ran into some issues with my Galaxy Nexus and Clockworkmod running a backup. I ran out of space and it errored out.
I rebooted, and gone back to the system, but now the system is complaining out of space. I have some old backups, which somehow have reduced in size, and there is a blob directory, which I can't copy locally to my pc. As well, I can't restore it back to the old backups, since they are not showing up. How can I get back space, but keep my restored backups. 

+Koushik Dutta Can I copy the ClockworkMod folder to my computer and delete the blobs folder with my backups and free up all the space taken up by the blobs folder and my backups, not having any backups on my device? And just copy the CWM folder back to my SD card when I want to install a backup?
+Travis Faubel You should be able to, the issue is that there are thousands of files so it takes excessive time to transfer them.
Latest recovery can't be found after upgrading CM10 to newest nightly. Any suggestions? I can verify that the files are there in the clockwordmod/backups/ directory.
Further investigation shows that my /sdcard/ now has a subfolder of /0/ in which everything is residing. Any suggestions to be able to restore my backups?
+Koushik Dutta The backups can't be found by Recovery. CWM Recovery is working fine, or so it appears.
Maguro. I was hoping to roll back to my backup I made just before installing the new CM10.
+Ian Sterling Yeah, the data migration will move all your backups once.

I'm updating the maguro recovery now on ROM Manager. Should show up in 30 min. That will prevent the migration from happening again (and again, and again, every wipe).
Thanks. I noticed that there was now a .nomedia file in my clockworkmod folder. Will that affect whether the cwmRecovery I have on there sees the backup folder?
+Koushik Dutta I've done a complete wipe of the phone (but not the SD card). I can see the old backup from Windows, but can't see the new backup or SD card location. Any suggestion how to fix this? I'd like to make a backup of the current installation, but have no idea how to get the backup to the computer if I can't find it.
okay read through all of this and could not find a clear answer. I had flashed the v6 that created the data/media/clockworkmod/ folder which all backups since then have been saved to. Now I beleive they should be in the sd card folder for clockwork (right?). If so how do we 1) Get the new backups to the sdcard folder 2) do the blobs need to be moved to the sd card folder with them and 3) how do we get rid of and disable the data/media/clockwork file.
+Koushik Dutta I've wiped my SD card and done a backup. But when I plug my phone into my PC to copy the backups off, I can't find the backups folder. "Computer\Galaxy Nexus\Internal storage\clockworkmod" has no backups or downloads folder.  Any suggestions?
I am certain this was a great idea when we started, however, this is what I found:
logged into my galaxy s3:
/storage/extSdCard/clockworkmod # du -sh
12.3G    .
I have a 32G micro sd and I am not able to maintain this data, I would rather take 20-40 minutes for a back up instead of a constantly shrinking available space.  Actually from your assertion above if I did not use my phone, then it would not grow this space.  I am not certain growing the clockworkmod infinitely on a finite micro sd was the best action plan.
I partition my SD card on my galaxy s3, well I'm running black bean ROM bb6 now, I just move all my back up to SD card and cyanogen folder too using es file explorer, that blobs is like 4 GB...takes bout half an hour to move...
Losing the ability to have a self-contained backup seems like a huge con to this new approach. I'd rather spend the time waiting a few more minutes for a complete backup. A major change like this should have been optional/configurable IMHO.
+David Rennalls You can set the backup method in settings while in recovery mode. Set it to TAR and you're good to go. Works for me. =) If your version doesn't have it, either check for a new version, or let +Koushik Dutta know which version and device you have so he can add that functionality into CWR for your device.
+Ian Sterling So, I switched CWM to save to TAR and made a backup. It created a folder "/data/media/clockwork/backup/2013-01-08-1214" with 9 files, size about 1,1GB. Is it sufficient if I just copy this folder to my computer? Also, is it ok that files cache.ext4.tar, data.ext4.tar and system.ext4.tar are 0mb? Thanks!
could you please,please make a stable version of cwm on rom manager or any other program,im just very unsure of using version in test state plus i no experience with the terminal emulator yet.i contacted LG yesterday about the ICS update for this phone and they said they they have pulled it out because it would cost them too much too make ICS for this phone.I have seen tons of your stuff everywhere and you seem to be the only one who can help me at this point.

thankyou so much.

you buddy hopefully in the tech world some day Seango

phone model: LG C800G VIRGIN CANADA
fav root apps: Ram manager,rom toolbox.
+Michal Urban Sorry it took so long to answer your comment. Yes, it's sufficient to copy that folder to your computer. Yes, it's okay that the .tar files are zero bytes in size. If you look, there are similarly-named files ending in .a and .b (for the larger ones) that are considerably larger. If you pony up the few dollars for the premium rom manager, you'll be able to download the backups directly from your phone over your home wifi as an installable .zip file.
something is still broken with blobs, my blobs folder is 4 gigs, with 23,000 files in 4000 folders, with no backups left on the phone. new backup does not clear this folder. I'm deleting it.
+Peter Rippe Use ROM Manager to remove the backups from the phone. That should take care of the space being taken up.
Tired of experimenting ROM on my sprint galaxy s3, when you do a backup, blobs is so big so I just went back to my stock ROM and just root coz mostly I just need the Wi-Fi tethering...
+Koushik Dutta I had 4 backups and over 7000+ blob folders. i deleted 3 of the 4 backups. did another backup. watched and never saw "Freeing space...". rebooted. still 7000+ blob folders. also did a restore of most recent backup i just did. never saw "freeing space". rebooted. still 7000+ blob folders. is there any way to tell if the blobs are being deleted? running CWM
"ClockworkMod Recovery now deduplicates files between builds. This results in way smaller backups."

I'm sorry but my backups are getting bigger not smaller..
I do prefer the old method: 1 backup = 1 folder. I could easily manage them, move them on PC and select from the manager. A huge "blobs" folders with a mess of smaller folders inside is not practical at all.
I only have 1 backup at a time so this system is not so beneficial as the blobs folder is 800mb so is nearly doubling what i would have had without blobs. Although I haven't noticed I do wonder if all those files in blob will eventually slow down the phone.
+Ben Lowe Your blobs folder is only 800Mb?! Consider yourself blessed, I can't get it down under 1.2 Gb on a fresh install, even before booting the ROM! ;)
My blob folder is 4.3 gb. and backup has two backups. how to reduce the total size of backup and blob. 2 backups and total size is 5.7 gb. this suckss. what to delete ? 
Stop backing up the blobs and go back to tar format. Then you can offload your backups once they're made.
Where is this "blobs" folder that i need to copy, because i cant find it :c. I want to transfer my CWM backup to pc. Abd delete from my phone to save space. How can i do this?
If you're not rooted, /mnt/shell/emulated/clockworkmod/backups/ (I think) is where you'll find it. Do adb shell and browse around in there. Even if you are rooted, /mnt/shell/emulated/ is a good place to start looking.
I checked there, i only got a downloads and backup folder in the CWM folder. In backup i only got the backup i made, cant cind blob anywhere. Pls send help ;-; and in download is a custom recovery image that i have downloaded. I've been here for hours trying to find a solution guize.
I also got files like .last_install_path, .nomedia, .recovery_version and .settings in emulated/cwm. But no blobz. ;-; 
And .salt_hash and etcetra, but no blobz
I'd suggest you change your backup format to TAR, do another backup, then just transfer the backup folder in its entirety to your desktop using adb pull. Problem solved.
Wait wut, dude how do i do that? And how do i restore from it? I know how to use the adb commands so im fine with that. But everyone keeps saying its not the backup that matters its the blobs, and im worried because i dont even have that folder :c. But i'll follow this process if you can guide me through please, i'll be grateful. Thanks. 
Sorry, the only thing I know about blobs is from my iPhone days, related to the baseband of the phone radio or something... No idea how or where you'd find them on Android. However, your cwm backup folder is pretty much a complete nandroid backup. You can adb push the backups back to the phone and restore it from within cwm recovery.
Oh okay i understand now, and no need to apologise ^.^ just provides an oppurtunity to learn more about this topic. And i was able to save my backup to pc through a windows app. Thank you very much Ian. I'll hit up this post if i need any further help c: 
You're welcome! Glad I could help.
Add a comment...