15 minutes, and 8 months

Today is release day in the Android Open-Source Project. I'm nervous. I'm caffeinated. I'm tired too as I've spent long hours in the last few days putting the last touches together and the associated stress prevented me from sleeping well (admittedly, so did the jet lag after returning from a recent ski trip in Europe).

I've got music playing through my headphones to isolate myself from the ambient noise. I've been spending this morning alternating between Silver Swans, The Soft Moon, Scraping for Change, and Ag Silver.

As soon as I got the final green light, I sent an announcement, and I started running the scripts that run the many git commands involved in pushing source code out. It took me about 2 hours overall, counting a few interruptions from coworkers that needed my immediate help. Other parts of the push had been speculatively scheduled for today ahead of the green light and just needed a small nudge.

The public excitement that surrounds such a release is gratifying. I continuously spend a lot of effort preparing such a release, improving my tools, testing things, eliminating dependencies. The way people respond to the release justifies the time spent making sure that the release process is as smooth as possible. I got my 15 minutes of fame again, at least in the community of technology enthusiasts.

Now starts another 8-month waiting game. In over 10 years working on the software side of the cell phone industry, I've learned that it takes about 8 months between getting the software ready and seeing it widely deployed. Edit: Meaning, deployed on new devices.

Why 8 months? It's about the time it takes for the software to get ported to the chips that get used in a new phone, then to get it to run on the actual phone, then to have features added to it by the manufacturer, then to have it customized for the specific phone and for the specific operator, then to have it tested by the device manufacturer, then to have it certified and approved by the operator that will sell and support the device, then to have the actual devices manufactured, distributed to the stores and put on the shelves.

Now, 8 months is just an average, a rule of thumb. The real development schedules can vary by a large amount in both directions. As a rule of thumb, though, it does tell me that 4.0.4 will be quite widely available on new retail phones 8 months from now, i.e. for the 2012 holiday shopping season. 4.0.4 is a solid release, so I already anticipate that 2012 is going to be yet another great year with plenty of exciting new Android devices.

What I do not know, however, is the precise schedule for the release to any individual retail device. I don't even know it for the flagship devices that Google is directly involved in. Thanks for not asking about those, I really can't answer such questions.

Have fun with 4.0.4!

Edited to clarify that the 8-month rule of thumb applies to new devices, not to updates to existing devices that typically require a lot less time.
Android 4.0.4 in AOSP, Jean-Baptiste Queru, 3/28/12 11:08 AM, I'm in the process of pushing the Android 4.0.4 files (IMM76D) into the Android Open-Source Project. This represents an incrementa...
Ryan Lestage's profile photoJean-Baptiste “JBQ” Quéru's profile photoLuke Hutchison's profile photoGert Scholten's profile photo
Enjoy your 15 min. Well deserved!
Thanks for the effort. Waiting to get the update.
Best 15 minutes on the android-building group :)
Now to throw you off: Will it come with the next Google flagship device? :P
As always, thanks for the fantastic work on this! Looking forward to running the new code. Thankfully, I won't be waiting for 8 months :-)
Thank you +Jean-Baptiste Queru, again for all your and your team's hard work.

Very grateful for all that you do for the Android community.
Thanks for all you do. It is appreciated!
+Renaud Lepage - I did joke internally that people made up unfounded rumors about 4.0.5 because each time they looked for something about 4.0.4 they got "not found". Now that 4.0.4 is out, I guess I can make the joke in public :)
Should the galaxy nexus 4.0.4 stock factory images come out soon too? Thanks for your work!
Let's hope the galaxy note will be based on this new release
Thanks for all the hard work, and for the occasional glances at what goes on behind the scenes!
Thanks for all you hard work and sharing these milestones with the community.
Ok...now, about Jellybean...

Haha, just kidding man. Thanks for all the hard work that you do. Your hard work feeds my custom ROM addiction, and I definitely appreciate that.
so now the world has Android 4.0.4 before an official release to the NS4G ~sigh
+Jean-Baptiste Queru How much involvement do you have in what features come to the next version? Are you at all involved in the process or do you just code and maintain the repository?
+Brian Haslip Nexus models (with the exception of maybe verizon galaxy nexus) are all maintained and updated by google. Google seems to have forgotten we exist in favor of their new toy.
+Samudra Sanyal - Good question! Personally, I have essentially no involvement in any user-visible feature. I do have some involvement in shaping the tools themselves, and especially the build system, in a way that makes AOSP easier to manage, but that's about it.
+James Finstrom Ah, yeah, good point. I think it has more to do with it being a CDMA device than them just not remembering to update it though. CDMA seems to be the redheaded stepchild of the Android world.
Great work, would love to know what the improvements are. Are they just tweaks or new features also?

And are you saying we are now at the beginning of the 8 month wait? Hopefully the nexus range will get it tomorrow, not that I'm impatient lol
+James Finstrom If you're savvy enough to follow JBQ then I assume you're no stranger to flashing custom ROMs. Don't worry, I'm sure we'll have a 4.0.4 CM9 kang within the next 24hrs.

Back OT: Thanks JBQ for all of your time and effort, it is much appreciated!
+Mike Crumpler 8montha until it is likely to appear on a device bought from a store. You get it much faster with custom Roms. Sometimes within 24 hours :)
+Mike Crumpler - The list of 698 individual changes since IML76K is too long to list here (but they can be counted with a simple script) ;-) My limited and remote understanding is that this is more on the "tweak and polish" side of things, but I could be wrong.

As for the updates to flagship devices (including the Nexus phones), even though Google tries to get builds ready for all devices at the same time, operator approval varies from one operator to the other and from one device to the other. The 8-month rule of thumb doesn't apply to flagship devices, since all phases are designed to happen at the same time for those, but there's still a bit of operator certification and approval at the end that can't overlap with anything else, and that's what explains the slight device-to-device variance even for flagship devices.
Cool. Thanks for the info +Jean-Baptiste Queru . Well I look forward to seeing that little update symbol in the top corner soon then. Can't believe there are that many tweaks. I can't think of any more than two bugs for my nexus s. But then you guys are perfectionists I guess.
+Niko Roberts I've only once flashed an update to a nexus s so I'm not an expert. I just thought there was one version which worked on all phones and somehow the os just knew how to work on each device. Obviously I knew the likes of htc or samsung can modify the OS to add their own bloatware etc, but otherwise I thought one fits all? Shows how little I know.

So how will I know which unmodified version will work on a nexus s, and which to download. I would want the raw unmodified version how Google intended it.
"I'm staying updated with the release team on the progress of operator approval. Once that clears, the OTA can start, and that's when I can make the factory images available."

+Jean-Baptiste Queru This shattered my mental model, since I was under the impression "started rolling out Android 4.0.4" (Android G+ page) meant it'd already passed the carrier approval stage. Or does starting a roll out always mean unlocked SIM free?

I'm just finding hard to parse that a carriers first contact begins after the OTA is announced, since I always assumed it started before.
+Mike Crumpler what they are releasing is really just a reference base with no hardware drivers, you will need to wait until google sends you an ota update or releases one for your nexus s, or an enterprising developer takes this code and incorporates the needed drivers. In some cases the drivers themselves will need to be updated by the oem as well. safe bet is to wait for googles ota because it will have the updated drivers, then devs can Kang away......IIRC
+David Lawerteh This is being released to the Android Open Source Project. See above post. Carriers are the last in line on the winding trip of android updates. Anyone can download 4.0.4 and it can be used by anyone now, even amazon and barnes and noble......its the open source release nothing more nothing less.
Thanks +Daniel Wilson Only reason I ask is because I bought a sim free replacement nexus s for my wife (she broke her first one ) and it doesn't receive ota updates. Mine does. The carrier says they won't update it as they didn't supply it. But I thought google controlled the nexus ota updates so got confused and instead found a rom to flash. I'll wait for mine to update first then and then go hunting for the wife.
Thanks for all the hard work!
Great Job! thanks for everything! Now lets see what goodies are in store!
+David Lawerteh - Actually, your mental model is correct, and the issue is probably with my earlier explanation. During the entire development, Google works directly with all the companies involved in the flagship devices: silicon vendors, device manufacturers and operators. That happens while Google develops the platform itself and the applications that ship on flagship devices. All those companies test during the entire development.

Deciding that a build is a "final release candidate" (in this case IMM76D) involves feedback from everyone involved. At that point no more changes are made, and final checks are made. That's the phase that we're in right now, with those final checks. Because every company has been involved for a long time, this usually goes quite fast (IMM76D was built on Sunday afternoon, and has already been released for several devices). The final checks don't take nearly as long as a full certification, since everyone has been testing and reporting issues all along. It's primarily a check that the actual final build exactly as it's meant to be released to consumers doesn't have any unexpected issues that hadn't been seen earlier.
+Daniel Wilson - There are actually quite a few drivers included in the AOSP code drop: precisely, all the ones related to the chips that are used in flagship devices. That includes: Exynos 3110, Tegra 2, OMAP 4, Broadcom 4329 and 4330, SGX 540, NXP PN544, plus some GPS chips, some sensors, and probably quite some other stuff that I'm forgetting. Now, it's true that most of those require additional proprietary libraries, but we make many of those available for AOSP purposes, and the companies involved have those ready to be licensed to device manufacturers.

(note: the kernel side of some of those will be available in the next few days, simply because of the complexity of the entire push).
out of all the things I'd like to do LEAST on Earth right now, coding is right near the top of the list, but I can't help being curious ...
+John Bonsall I don't mind using 3rd party roms but that doesn't discount the need/want of a official stock rom.
+Jean-Baptiste Queru do developers involved practice Scrum based stuff such as sprints or is it other methodology that sum up to the 8 months ? The art of surviving intensity and stress is interesting so worthwhile reading.
+Gunnar Forsgren - I've never been quite on that side of the industry, i.e. I've only seen it from the outside, having written some software in shippable quality, and having to wait that many months as it trickled its way out.

Being upstream, I've always seen quite a continuous stream of feedback trickle back, so I'm guessing that there's a lot of agile in there, though I've never been close enough to identify a specific agile methodology (as opposed to seeing the obvious consequences of broad agile principles).

When it comes to operator certification, I'm guessing that flagship devices get a fairly agile treatment too, while follower devices probably follow a more monolithic waterfall model. At least that was still the case 5 years ago, but since Android only got me involved in flagship devices things might have changed over the years without me noticing.
+Jean-Baptiste Queru Seems another nagging question has popped up: Does that same process hold for the non-Yakju variants, or does that introduce a Pandoras box of additional steps?
+David Lawerteh - The same overall principles apply (especially when it comes to working in parallel across all companies involved). There are some minor details that differ (like which company hosts the source tree and runs the build servers), but as far as getting updates out to consumers those details are irrelevant and invisible. Unfortunately those differences have had some impact on AOSP support, and we're learning from this experience in order to avoid those issues in the future.
Thanks for your hard work. But this package can't updage 9020T from 4.03 to 4.04. It's display Status 7 error message...
+Jean-Baptiste Queru I think I see the big picture now (or at the very least the update process seems less like black magic than it did before.) Thanks for the informative responses.
+David Lawerteh - Now, the variants of Galaxy Nexus were handled a lot more tightly than those of Nexus One or Nexus S. For Nexus One, the update schedule for the variants was not even tied to that of the Google version, and for both of those trying to get AOSP working on the variants wasn't even on the horizon. For Galaxy Nexus, that's been improved a lot.

That means that getting AOSP working on the variants went from unrealistically impossible to frustratingly close, which is why there've been some discussions about it. For the older ones, there wouldn't even have been discussions about it, everyone involved knew that it wasn't possible at all.
That's for putting it into perspective. Kudos on the hard work!!
+jisheng guo - Sorry, I can't help you here, I don't know anything about the way the actual consumer updates get sent and installed, I specialize in the Open-Source side of the world.
Thank you for making Android great JBQ! I depend on what you and your team create everyday.
I just wish I hadn't bought the Xoom LTE thinking I was going to get Nexus-speed updates. We're months past the Xoom Wi-Fi getting ICS, and still no sign of Motorola and Verizon providing us with our build.
i would not say only 15 minutes of fame. the help you constantly provide on google groups makes you the everyday go-to person.
Those new git servers seems to holding up nicely. After a code drop this last repo sync was, for me, the smoothest one yet.
+Jean-Baptiste Queru A stupid question - I understand you guys don't want to support ICS on the old Nexus One, but why not release the binary blobs needed to implement the hardware acceleration, and leave the support to the community?
Many of us are using "hacks" such as A2SD+ or XPART, and changing the HBOOT on our devices to be able to install larger firmwares and to have more space for apps and data at the same time; but we are still missing the updated binary libs, and the hacks done until now aren't release quality yet.
+Jacob Nordfalk - I'm hoping that the handling of contributions will get better soon. On my side, I'm back from vacation, and once I'm entirely done with the 4.0.4 push I'm going to have time working on those. On Conley's side, Deckard is coming back to life, and that's a key to being able to scale the way to handle more contributions.
+Jose Bernardo Bandos Rodrigues - The first aspect is that Google doesn't have a license to distribute the graphics libraries for the Nexus One. If we did, I'd gladly release what we have (and, to prove that those aren't empty words, I released the wifi binaries for Nexus One earlier this week, since we finally got a license for those).

The second aspect is that we don't have any updated graphics libraries for Nexus One that work with IceCreamSandwich (or even Hoenycomb for that matter). All I'd be able to release would be the Gingerbread libraries, which is the most recent ones that we have.

Now, if I got updated binaries with an appropriate license, I'd make time to package them and release them, and to put Nexus One files back in the master branch of AOSP. The storage size issue shouldn't be a killer for AOSP, as I think that AOSP IceCreamSandwich builds should just about fit in the 170MB system partition of Nexus One (Edit if you don't include any additional proprietary apps)
Having worked in development myself I understand how it might take around 8 months from source release to devices.

The problem occurs when various websites says something has been released. Many people see that as "oh if I check for update it should be there....wait, what do you mean 8 months!?" So, there ought to be clearer distinction by those web outlets, perhaps by saying something like "released to device manufacturers only" but even then some people wouldn't get it....

The cheeky approach that one could argue is that Google should 'secretly release' a few months earlier to these guys before the general announcement. Then the general public would see that time reduce by some months...
+Jean-Baptiste Queru Thanks for the information Jean-Baptiste. My question would be, why do you guys not release the update.zip files to xda as soon as the OTA rollout begins? (so that the enthousiasts can check them out ASAP!) Is it really a bandwidth issue for Google???

+Patrick Enfedaque - Licensing issues. OTA packages are not licensed to be touched by end-users, only to be installed automatically by the OTA mechanism. With the appropriate license, we distribute the factory images for yakju and mysid, which are meant to be used to restore a phone to its initial state.
+Jean-Baptiste Queru Cool! Did you eat at Tremplin? any night trips to caves des roy? (<-that's how it's spelled, trust me). Any fondue? and i'm guessing 1850 of course. Merci bien!
+Hamid Marc Afsharieh - We didn't go out to eat, because we had 2 small kids in the group. We had a great fondue and a great raclette at home, though. We stayed at 1300, that's the highest we could go with an 11-month-old. It was great, we were minutes from the lifts, probably closer than we'd have been in most places in 1850.
Well, if you put a Samsung Exynos 5250 or Qualcomm Snapdragon S4 in the Google Phone 2012, I bet I will have Android 5.x preinstalled instead of 4.04 :P
I've edited my original post above to clarify that the 8-month period that I have some experience with applies to new devices only.

Updates to existing devices obviously take a lot less time. The requirements places on most chips don't change much from one release to the next, and the quirks related to each chips have been ironed out already. Custom applications written for each device don't need to be re-written, and if they were properly written on top of the SDK they'll need at most a UI refresh, which can be started ahead of source code availability as soon as the preview SDK is released. There's no need to get regulatory approval on the hardware itself, and there's no delay related to manufacturing and logistics.

The flip side is that updates need to pay close attention to regressions, which aren't such an issue on new phones: users don't like to receive an update that damages or removes functionality that worked fine before the update.

I don't have a good rule of thumb here (because in my pre-Android days there weren't ever any updates, and because I'm not close enough to the manufacturers even on the Android side), but I'd guess that the update process might only take about half as much time as new device development.
+Jean-Baptiste Queru So why did it take you and your team five months to "fix" an OS that was supposed to be working when originally released in November last year? Was it that you and your team originally programmed such a horrible mess that it actually took this long to fix or is it because Google just does not care to provide an adequate time frame to fix phone breaking bugs for their customers? I would also like to know why it is that Google feels they do not need to provide adequate customer service for this OS, the Nexus line of phones, and Google apps on their own help forums. There are questions and problems that have not been responded to on that forum for months up to even a year.

Not to mention the complete silence of why was 4.0.3 pulled for the Nexus S? What was wrong with ICS that the OTA was stopped? Also why was the source for 4.0.3 allowed to be used on other OEM phones when even Google deemed it not ready for their own Nexus S. Now there are devices getting 4.0.3 (that is obviously full of bugs) OTA'd onto their devices. Some might not ever get the bug fixes in 4.0.4 and will be stuck with a phone on 4.0.3 and it's bugs forever.

I would also like to know why Google is still falsely advertising the Nexus S 4G? Here is a quote from Google's own page: "With this device, users will be the first to receive software upgrades and new Google mobile apps as soon as they become available." The NS4G has neither been first to receive software, nor has it been first to receive Google mobile apps (Chrome mobile). http://www.android.com/devices/detail/nexus-s-4g-from-google

Does Google expect for developers and consumers to stay with Android when they blatantly have no respect for them?
Thanks and any idea to speed this process? I think that public drivers for the kernel - proprietary or even better open source - will help in this case of Android devices. As any Linux desktop OS if you have the drivers it works. Or if it is not the problem explain why upgrading Android cost that time.
I have just have an idea: Using QEMU at Android devices with a standard arm processor emulator and open source drivers tested, having the last Android working at every device can be almost an instant procedure.

After that probably the only work to be done by brands is to provide an "autoemulator" profile and provide drivers for a not so immediate release, and of course after enjoy it at the emulator provide the "bare metal" version.

Perhaps QEMU will need future ARM64 SoCs, but if this idea is implemented it will arrive on time. And I hope that for this processors Chrome OS desktop will be also included at Android - as Ubuntu for Android - and allowing to install GNU apps to have really pocket personal computers. Or that glasses as the EPSON ones.
+Matthew Sholtz - I work on the AOSP side of things. It took me an my team less than a day and a half from getting confirmation that IMM76D was a final release candidate to getting it staged and ready to get released in AOSP, and it took me and my team less than an hour from getting final green light to pushing the first files out. Everything else happens upstream from me and my team.
+Miguel Mayol - Even Open-Source drivers need to be written. All Android kernel drivers are Open-Source (which is a license requirement in a GPLv2 kernel).

If you want to develop drivers against an emulator, the first step is to have an emulator that emulates the hardware you want to be writing drivers for, and you have a big chicken-and-egg issue here (not to mention a serious performance issue).
+Jean-Baptiste Queru Sorry if i did not explain well the issue:

Qemu has arm emulators profiles, at least my Qemu Launcher has two: ARM little endian and big endian, so make one profile as "Android standard" or "Android qemu nexus-1" - of course it will have less performance - will make this profile as a "Nexus-1" for early adopters and testing. And of course this "android standard (nexus-1) qemu profile" will be better each new version as hardware evolves. This way each new version can be installed at this "default android standard (nexus-1) qemu profile"

And as second step - of course - is to build a "bare metal" emulator for every device public or internal in order to be able to test it at brands labs and or at early adopters phones without putting at risk the bare metal install. This way - if it is available - you can test any new version previously at your emulator of your own machine, Or even - if it is done - at a future one a Nexus+1 profile.

Last but not least KVM and Xen are open source and both able to emulate with a 90 to 95% performance, good enough for early adopters, and of course not for OTA upgrades.

Perhaps what I am saying is a "fool idea" but I cannot find any flaw in the way of speeding new versions implementations.
+Jean-Baptiste Queru You should go into politics for how good you are at dodging questions. That is pretty sad though that you have no clue as to what Google is doing with Android when you work on Google's Android developer team. No wonder it has taken five months to fix ICS, no one knows what anyone else is doing. And apparently no one even cares that this is happening..
+Matthew Sholtz - It's called "playing position". I trust my coworkers for dealing with their domain of expertise, they trust me for dealing with mine. The less time I spend talking to people about aspects that don't actually need cooperation, the more time I have to get actual work done. That's how teams and companies can grow large without falling to the inefficiencies described in "the mythical man-month".
+Jean-Baptiste Queru There seems to be a different version of the GSM Galaxy Nexus in Canada. However I'm reading that this update is pushing out to all International GSM/HSPA+ Galaxy Nexus phones. Any confirmation here, will I get this update in Canada?
+Ryan Lestage - I'm not sure, sorry. From my AOSP seat, I only have visibility over the "yakju" builds (that's one of the variants of GSM Galaxy Nexus, the one I can support in AOSP), and I think that the variant sold in Canada is yakjuux.
+Matthew Sholtz 4.0.1 and 4.0.2 builds were early builds, google themselves said 4.0.3 is the major build or release.

They released 4.0.1 and 4.0.2 ahead of time though the Galaxy Nexus so developers could test their apps on the 4.x platform and ensure a stable transition once 4.x devices became more mainstream. Though I have no idea about the Nexus S

+Ryan Lestage it is entirely possible to change our Nexii devices over to Yakju which are google controlled, ours are samsung controlled because they changed the radios used though the ones that come with yakju builds work just fine
I know it is... but I have my phone set up perfect for my liking. Don't like starting from scratch
+Jean-Baptiste Queru, thanks for your continual hard work on this. I hope this is not a silly question, but can you please clarify why is it not possible to install a newer release of the Android platform (Dalvik, platform code, support libraries) on an older phone without replacing the existing kernel (or, if necessary, such that the existing binary device drivers on the phone can be reused with the newer kernel)? Is the kernel or device driver ABI consumed by the platform not stable enough? Is there insufficient separation between the userspace part of the Android platform and the O/S, as things currently stand? Will there come a time when the underlying O/S layers are stable enough that upgrading to the most recent version of Android could merely consist of getting root and then replacing some libraries and .apks with newer versions?
+Luke Hutchison - The Android code itself isn't deeply tied to the underlying kernel. E.g. the same ICS code works just as well on the emulator (2.6.29), Xoom (2.6.39), and Nexus S and Galaxy Nexus (3.0.something).

What often happens, though, is that hardware-specific user-space libraries can get tied to specific versions of the kernel. That's especially true for Imagination Technologies' graphics libraries, which require a perfect match or the system doesn't start, but I think that other libraries do the same at times. This often happens in situations where the kernel's licensing constraints prevent silicon vendors from architecting their code in a technically optimal way.

So, really, the Android platform itself doesn't care much, but device-specific code that runs under the platform often does.

As for simply upgrading Android without touching the underlying layers, the difficulty is that those layers aren't always sufficiently forward-compatible. E.g. none of the GPU libraries that we had for Gingerbread or Honeycomb would be usable as-is with IceCreamSandwich, because of missing features, bugs, or performance issues.

I hope that answers your questions.
Add a comment...