Kernel.org has been down for maintenance for a bit, now. This seems, counter-intuitively, like a good time to thank them (and obviously the admins) for years of perhaps unsung service to the community -- and to Android.
The thing many people may not understand is that hosting git is hard
. Or at least, it's very different than hosting a web app -- which means it's hard if only because there are few best practices.
The main reason is fairly simple: if you do a 'git pull', the server side git process has to load the entire revision history of the git project you are pulling from into memory, in order to compute the delta that it needs to send to the client. For large projects, or projects with long histories, this means that each individual client connection has a pretty enormous memory footprint. Android's full repository history is several GB in size. Shared memory and disk caching obviously helps here, but only to a point.
What's more, until relatively recently (and certainly not when AOSP launched), git didn't support HTTP as a transport -- it used a custom protocol, or required SSH. This meant it was incompatible with the huge body of tools and techniques the world has optimized for serving HTTP, requiring git admins to learn, create, and use a whole different bag of tricks.
In the case of the Android Open-Source Project, these two quirks combined meant that kernel.org
was, at the time we launched, almost certainly the only organization in the world that was capable of hosting the Android git repositories on the necessary scale.
github had launched by then, but was still quite new and pretty unlikely to be able to handle the load. I suspect github probably still couldn't -- which I don't mean as a criticism of github. They're optimized for a very different "style" of git hosting, which they're very good at: lots of relatively small projects with high degrees of interactivity. Google's own Project Hosting git support is optimized for that usage model, too.
AOSP, however, is comparatively huge and has different needs. (Android includes an entire OS, after all.) To reduce the massive memory footprint, we split Android into many smaller projects, so that individual 'git pull's only need to compute smaller deltas. We're currently over 300 individual git projects, if I remember rightly. We created the 'repo' tool to pave over the multiple projects and automate the process of syncing a tree. Even with these techniques, the aggregate outgoing AOSP bandwidth just for git pulls
peaks in the range of several dozen gigabits per second.
It could well be that kernel.org
is still currently the only public service in the world that can handle this load, due to their worldwide replication expertise and infrastructure.
Or at least -- once they get back online. :) So I for one say: thanks, kernel.org
, for hosting our stuff, and godspeed with the maintenance work.
(I don't have any details to discuss about when AOSP will be back online, but rest assured we're working hard on getting back up and running as soon as possible. For now I just wanted to say "thanks" to kernel.org
for their great work to date.)