Shared publicly  - 
 
A brief history of AOSP

(breaking my own rule and writing a work-related post.)

Working on Android for more than 5 years, I got to see first-hand the evolution of the Android Open Source Project. Here is its history, one year at a time:

2008: Release 1.0. That's when the source code and the tools were prepared for the initial code drop. 1.0 was released at the end of that year.

2009: Contributions. That's when the processes were created that made it possible to merge external contributions into official Android releases. 2.0 at the end of the year was the first such release.

2010: Automation. Many release tasks through 2009 were done manually and those got automated throughout 2010. Everything got automated for 2.3 at the end of that year, freeing up time for additional tasks.

2011: Crisis management. That year was spent hitting and handling setbacks. The first rule of 2011 is: you don't talk about 2011.

2012: Hardware support. That year was spent making it practical to run AOSP code unmodified on common retail hardware with 100% support. That was the year of Nexus 7 and Nexus 4, the first devices for which that was the case.

2013: Readiness. At this point, AOSP as a project is where it was originally envisioned, the result of 5 years of hard work. The year is now going to be spent reducing backlogs that have accumulated over the years, so that in 2014 AOSP as a project can run smoothly on an ongoing basis.

As usual, if you have specific questions about the AOSP source code, about building it, or about contributing to it, please head respectively to the android-platform, android-building or android-contrib Google Groups.
581
132
Charles Alva's profile photoDavid Lawerteh's profile photoDaniel Charlton's profile photoPeter Lee's profile photo
89 comments
 
It's gotta be pretty cool to see the evolution of a project like Android grow. Thanks for all your hard work along the way!
 
I'm just getting in to app creation for Android and I'm loving it. I'll have to take a deeper look at this as I continue building. Thanks for the update!
 
Nice read! Thanks!
 
Wow, amazing to think this vision is five years into implementation. Really important pietist too, I'm really glad you've stuck with it to see it to this point
 
+Michael Evans - Yes, that is quite cool. Mostly, it gives me a sense of perspective when something doesn't go quite right: I actually know better than anyone where AOSP is coming from and how it got where it is today, I know many things that could be going wrong (and that occasionally have), and with that experience I can know that a few glitches aren't the end of the world, they're just opportunities to improve the processes even more.
 
Thanks for breaking your rule, this small list is, however, a huge insight into the project! Also, although work-related, thanks for AOSP and your hard work (and many others')
 
+Josh Steiner - Such large projects have so much complexity, so many dependencies and so many unforeseen issues that they can't possibly be planned in detail. Since there can't be a day-to-day plan, if you lose track of the long-term goal, you end up running in circles instead of making actual progress.

Because of the very large size of the code base and since there are many thousands of engineers who work on it at hundreds of companies throughout the world, things can appear slow to the outside. But then when you realize that this code is used in about 1000 new devices every minute, you realize that things are actually happening incredibly quickly.

The goal has always been to have a Open Source mobile platform that is credible for device manufacturers and that the enthusiast community can actually run, modify, port, distribute, and contribute to. I think that I've essentially reached that goal and that we can check that checkbox. Now, the next step is to have fun, to work together and to see where we can take Android from here.
 
when was the project actually started?
 
+Han Honso - The notion that Android was going to be Open Source goes back very far in its history, much before I got personally involved in 2007. AOSP itself was announced as part of the very first Android announcement in November 2007, and the first source release happened in October 2008.
 
Thanks for the post. Interesting read.

I always wondered what was the work flow for devs.

I guess google works on an internal repo and then some of the features get merged in aosp ?

If so how do you decide what gets in and what doesn't? 
 
It's amazing how far it has come in the past 5 years... I will forever be greatful for all you and your team's hard and dedicated work. Keep it up!
 
Great overview indeed. As a customer, too, I am specially happy about the 2012 step. I'm glad we are starting to reap the benefits, broadening the choice of stock Android to "non-Google" (i.e., maguro, mako etc.) devices as well.

Congratulations on your great work.
 
google should make low enad cheap devices for sout asia and middle east...
 
I'm glad that you have always been there to help clarifying answers to people. I have never posted a question in a Android's Google group before, but every time I seek for some answers asked by other folks, I always see your name on it.

Thank you for your hard work, JBQ. 
 
+JF Dionne - Within Google's internal repository, there are typically 1 or 2 active branches in which the only changes are bugfixes, and 1 or 2 active branches in which future features get implemented. Security and CTS fixes tend to go back much further, and some long-term experimental features get developed in dedicated branches to protect the bulk of the team against destabilizing changes.

The most recent bugfix branch gets continuously prepared to be merged into the AOSP master. That's a crossover point where the flow of Google's changes and the flow of AOSP contributions cross and merge. That's the automation I created in 2010.

We are very careful about planning upfront what will be merged to AOSP and what won't, such that the Open Source boundary matches the boundary between git projects. We use our repo tool to manage that boundary. Outside of proprietary device-specific files that come from hardware manufacturers, the basic rule is that everything is Open Source except the apps that talk to Google services: we want to be sure that the Android platform itself remains free of Google-specific code.

As for picking which contributions get in: they need to match the overall goal of the project (e.g. contributions that go against UI designs or security goals won't make it), Google engineers need to feel confident that they can maintain and test that code over time, and the code itself has to pass code reviews and tests.
 
+Jean-Baptiste Quéru sorry if this is a work related question but you mentioned awhile back about CDMA being proprietary, I tried searching for that post and either I'm going blind in my old age or I'm just illiterate, can you point me in the right direction to where I can find that? 
 
well, looks to me that person should have more than one g+ profile ;), thumbs up for AOSP!
 
+Rabid Rotty - There are 3 aspects that have been causing difficulty with CDMA devices in AOSP:

-each device is specific to one network, and instead of using a standard type of SIM card like in GSM networks the way each CDMA device registers with its network is proprietary.

-the CDMA devices that have been supported in AOSP each had an additional network bolted on the side (WiMAX in one case, LTE in the other two), and in each case the interaction between the CDMA network and the other network has been proprietary.

-the proprietary components for these 2 aspects above have typically been less isolated and less portable than the other proprietary hardware-specific files that allow AOSP code to run on actual hardware, and that makes them unsuitable for several processes.
 
+Jean-Baptiste Quéru oh wow I just wanted a link, but yet again you out did yourself. A thousand thankyous and if I could give you more than one plus I would. Thank you again. Oh on a side note tell the wife tomorrow I'm going to start the paleo diet for a month and see how that goes. Take care and happy Saturday.
 
Thanks for your hard work. You guys make the world more engaging.
 
It's enlightening to see how separate the systems are... I.e. ASOP is not Android OS is not gapps etc. 
 
It's always good to hear the human side of Google! 
 
So now I get why there was no new version of Android at I/O and the rumored 4.3 version looks like it'll only be a minor update: now that Android is finally mature and there isn't really all that much left to add to the base OS, you guys are going back through the entire codebase and cleaning up the shortcuts you've taken over the years to rush out releases.

Although this means that we most likely won't see 5.0 until next year, I'm still happy. Keep up the great work!
 
So I guess the 2011 pain was honeycomb tablet and gingerbread phone separate streams and bringing out all together for ics.

It's all worked out brilliantly and a massive achievement to support to many types and configuration of devices.
 
+Brian Haley - There's a lot of breadth and a lot of depth, and as a result it's understandably very hard for anyone on the outside to understand where what they're looking at fits in the big picture.

Within Google, we're usually working on 2 or 3 version at the same time, targeting our flagship devices. When we ship to consumers, we take the source code for that version, remove the proprietary hardware files, remove the Google code, and release that source code to AOSP. That means that Nexus devices aren't technically based on AOSP, but rather than AOSP is based on Nexus devices.

In AOSP, we have source code (of course), but we also have processes (primarily defining how Google releases code and how external contributions are handled) and people running the whole thing (a few engineers dedicated 100% to running the AOSP process, but also a wide range of other Android people from Google).

From there, the source code can be used by different people, to be modified in different way. Specifically, it's possible to target actual devices, in layers:

-it's possible to run the code unmodified by just adding hardware-specific files. That's what we support in AOSP for Nexus 4 and Nexus 7, and we're seeing manufacturers start to do similar things.

-it's then possible to optimize the code for specific devices without changing the functionality.

-it's then possible to add device-specific functionality, to handle the specific peripherals of individual devices, without changing the overall look-and-feel of the system.

-it's then possible to modify the look-and-feel of the system, while still maintaining compatibility with the official programming interfaces defined by Google.

-finally, it's possible to add extra applications on top of that, which might be implemented in a way that runs on any compatible Android device, or might be tied to specific customizations done at any of the previous steps.

Because there are all those steps involved, within Google, within the AOSP project itself, and then in all the situations where the source code is used by other people, the only way to avoid ambiguity when talking about any of those steps is to describe it in a few sentences, as a single word won't be accurate enough.
 
Congratulations! 2012 was a huge leap and I'm really happy you're able to take some time to clean things up instead of being in tread water mode. Great job!
 
All very useful, thanks for your posting.

That means that Nexus devices aren't technically based on AOSP, but rather than AOSP is based on Nexus devices.

Is it a goal for future Nexus devices to be "based on AOSP"?
 
+Henning Hoefer - I hadn't seen that specific post, so this couldn't be a response to it.

That being said, I did spend some time in 2010 curating the issue tracker. What I saw then is that being able to run AOSP code on actual hardware was a critical step toward enabling community contributions: even recently, a quick sampling of the top 30 reports showed that 60% of them could only be worked on on actual hardware. I refocused my efforts on getting hardware support. 2011 got in the way, and once we got 2012 up to speed  it took about a year to work through the dependencies, which brings us right up to right now.

Arguably, the fact that the issue tracker didn't get systematically curated through 2011 and 2012 was actually working as intended: there was no point having a list of potential tasks for the community to work on if most of those tasks couldn't actually be worked on for lack of hardware support, and any time I would have spent in 2011 or 2012 curating the issue tracker is time I wouldn't have been able to spend on getting hardware support. Now that hardware support is here, the issue tracker is going to be looked at more systematically.

Closing that specific report in the issue tracker as "unreproducible" was indeed tongue-in-cheek: by definition, every single time a Googler looked at this issue, the issue tracker wasn't getting ignored. "unreproducible" was a bit like saying "it works on my machine".
 
The way the AOSP codebase is set up, is pretty cool and building android for a device is a matter of having a compatible kernel and the "device configuration" that contains what band how to build and device specific binaries to interface with android subsystems. This without changing little to nothing to the android source code.

Pretty nice... The result you can see in all the android builds for all the devices that do not get support from their manufacturer anymore :-) 
 
+Simon Phipps - Since AOSP is based quite by definition on Nexus devices, we'd have a chicken-and-egg issue if we tried to base Nexus devices on AOSP. As an example, introducing new APIs in Android would be very hard.

That being said, all external contributions that are merged into AOSP get merged into Google's internal code source, which ends up on Nexus devices 1 or 2 releases later.

I realize that my answer has a lot of hand-waving but not a lot of substance, I'm sorry about that. The detailed reality is far too complex to fit in a Google+ comment. It's complex enough that there are some aspects that I know I don't know.
 
Is honeycomb thought of as a complete setback? Or stepping stone? In the least, it had to help realize there must be one code to rule them all. 
 
Well, JBQ, not only you gave us a personal history of AOSP, you are giving us gems about how it works on a daily base before it gets to our devices! That's fantastic!

I am still learning to program for Android (SDK) and to compile Android itself (for my ChaCha and my Mako) but I am well aware of the size of the codebase and the complexity of the dependencies, so this is absoutely fantastic.

BTW, the fact that Mako and Grouper are 100% buildable from sources was really something... Thanks! :-)
 
Thanks for the reply. I'm familiar with this complexity - it was just the same developing Solaris inside Sun, where multiple peer projects had equal claim to being the "master" version and where pulling changes between various DVCS repos was a significant task.  

I think you'll continue to have challenges all the time the public source is the junior partner to the internal development, but I sense you know that :-)  Good luck with your efforts to bring Android development into the open, which remain most welcome.
 
+Robert Alexander - For Android as a whole, Honeycomb was a major step forward. It brought tablet support, it brought many of the UI concepts that are familiar today in Jelly Bean, it brought a lot of new hardware acceleration in the UI.

For AOSP, it was a step back: in an Open Source project, code is king, and, since the Honeycomb source code never got released to the public, AOSP didn't make much progress while Honeycomb was current. Between that and losing our servers in summer, 2011 was not a good year.
 
How hard was it get getting the build automation going? Any lessons learnt while doing it?
 
Thanks for all the hard work! Looking forward for smoother ux experience and better battery life on next AOSP. :-)
 
"What I saw then is that being able to run AOSP code on actual hardware was a critical step toward enabling community contributions" - I agree - I think myself and others haven't contributed as much as we would like upstream because, until recently, it was hard to get (edit: I mean obtain, as opposed to do a bringup) hardware directly supported by AOSP.

I've got a personal policy of "don't submit anything I haven't tested" - so until I could get hardware directly supported by AOSP, I couldn't submit anything without violating that rule.

Unfortunately, since getting these devices, I haven't found a specific issue in them that was fixed by anything other than a CodeAurora cherry-pick, and I assume CAF cherry-picks are probably already in the internal tree.
 
+Jean-Baptiste Quéru "losing our servers in summer, 2011 was not a good year"

Elaborate please! (Also, how can you say that when 2011 was the year of ICS??) :-) 
 
+1 all comments. Very interesting and gives me a lot of info on code I play with on a daily basis.

+Jean-Baptiste Quéru while we're at it, what's the input into aosp coming from oems and general public? Guess it's mainly kernel and hardware related stuff for oems and bug fixes from the general public?


 
+Okung Nyo it may have been a good year for Android, but I could see it being tough on AOSP. There was the whole kerfuffle with Honeycomb staying closed and some of the mess involved with temporarily abandoning some of the flexibility that had always been Android's strength. IIRC that was also the year kernel.org went down.
 
+Alex Bennée - The tricky part, mostly, was to cut the code that we wanted to release as Open Source exactly across git boundaries, including device-specific files.

It would be quite natural to store all the files related to one device in the same location in the source tree, but that would go against the goal of releasing some of those files as Open Source while keeping the others proprietary.

So, we split them, and we've been engineering the code such that it's possible to compile the Open Source files without any dependencies on the proprietary files. After all, what good would Open Source files do if couldn't be compiled because of proprietary dependencies?

With the split, it's easy to maintain a repo manifest that excludes all the proprietary files, and to feed it to the regular automated build system.

Once we learned how to do that split, we applied it to all the devices we've worked on since then. The first split was implemented for Nexus One between Eclair and Froyo, then for ADP1 and ADP2 right before Froyo, and Nexus S was the first device where we implemented the split from day 1.
 
"Now, the next step is to have fun, to work together and to see where we can take Android from here.". Awesome! Love android! Thanks JBQ!
 
+Andrew Dodd - Thanks for your notes, this is helpful.

You're quite right that generally speaking there are few issues left that are both popular, easy to fix, and acceptable into Android.

As for the CAF cherry-picks, we need to be careful that CAF code is only meant to run on Qualcomm chipsets, while AOSP runs on all sorts of processors that aren't even all ARM, so the constraints about portability aren't the same on both sides.
 
+Okung Nyo - In the early years of AOSP, Google didn't have a suitable hosting solution in-house for the source code, so we worked with outside specialists to host the AOSP source files. In 2011, an incident forced us to shut down those servers, and after a lengthy outage we brought things back online on Google-based servers.

The release of ICS in AOSP was a relief for sure (it showed the world that Honeycomb was only a temporary glitch). However because of the incident mentioned above, we didn't have all our services back online yet, and for about 3 months nobody could contribute code on top of ICS. On top of that, the hosting we developed had only been live for a few weeks when ICS was released, and we were still learning how to tune it, so there was a bit of stress there. The good things about ICS is that it was the first release with which Google was able to distribute factory images.
 
+JF Dionne - Surprisingly, we've been getting contributions in many different areas of the system, and not necessarily in the areas you'd expect each company to contribute.

It's both a curse and a blessing. Getting contributions scattered around is good because it spreads the review workload across the whole Android engineering team. At the same time, getting many contributions from the same author in the same module is good, because it allows that author and their reviewer to get used to working together.

I'd rather see the glass half-full, and say that any contributions are good contributions.

The hardest situation is e.g. when someone wanted to make lots of contributions to Bluetooth between 4.0 and 4.2: with all the work to switch from BlueZ to bluedroid, the Bluetooth code in Google's internal branch (that would become 4.2) was very different from the one in AOSP, and it was really hard to accept contributions. At the same time, bluedroid was secret, so I couldn't actually explain why I was rejecting all those contributions.
 
+Jean-Baptiste Quéru Why do you suppose the Chrome project--with a similar sort of structure--has had so much more success with external contributions and a running Chromium?
 
+Jean-Baptiste Quéru The cherry-pick I had in mind is a kernel one, which is already fairly segregated so that it would only impact mako...  Specifically, it's Qualcomm's "kgsl: process all idle checks" patch - without it, the GPU frequency governor has a nasty habit of deciding that it should be at 400 MHz despite load percentages below 20%.  I dislike the GPU tz governor in general due to the fact that decisions are made by a black box, so I'm working on replacing it, but that work isn't entirely complete yet.  (TBH what I have is probably feasibly on mako, but 1080p devices present an interesting challenge...)  Qualcomm's patch is probably in Google's internal trees already though.

Probably this conversation should be continued elsewhere at risk of derailing this thread though.  :)

+Thomas Bushnell My guess would be partly due to the fact that many ChromeOS devices are effectively x86 laptops, and the Chromium project does have support for generic x86 device targets.  This means that it is FAR more likely that someone with an interest in contributing will have hardware that they can test/work with.  Hardware-wise, x86 is far more stable in terms of being a target, so people are also much more likely to have "old beater hacktoy" targets they can work with that aren't significantly different from more current target hardware.
 
First, JBQ broke his rule about not talking about work. Now he's talking about it extensively.

Secondly, "don't talk about 2011."

JBQ subsequently elaborates why we don't talk about 2011.

Really enjoying this, JBQ. thanks for breaking your own rules. It's making a bunch of nerds happy, including this one right here. :-) 
 
+Thomas Bushnell - I'm only going to be able to scratch the surface here.

Chrome lives in the computing world, not in the consumer electronics world, and the rules are different: different support structures, different marketing campaigns, different user expectations.

Also, the browser industry has different constraints. It has a history of Open Source successes (KHTML and Mozilla, both since 1998, and it was possible to view the Mosaic source much earlier than that). It also has a different legal climate.

I think that Chrome is following the right model for what they're doing, and that Android is following the right model for what we're doing.
 
+Andrew Dodd - You're now way above my head in terms of kernel code. I recognize the words you're saying, but the resulting sentences don't make total sense because I'm not as familiar with the code in question as you are. You're right that this discussion should continue somewhere else, and mostly with someone else :)
 
+Jean-Baptiste Quéru I guess from my limited perspective, Chromium always had a build that anyone could build, and it took AOSP much longer to get there. My guess would be that AOSP had the model of working with trusted partners and "releasing" code after it was all done and baked, where Chromium is visible all the time, not just to a small group, and not just after it's released.

That might be connected to the differences you talk about, but it's not clear to me why.
 
+Jean-Baptiste Quéru Sure, that's cool with me. Maybe in another forum sometime; I'd love to hear your thoughts about it more fully.

At any rate, congratulations on a great milestone.
 
+John Rollan - Heh. Speaking generally, when something doesn't go well in AOSP, I'm frustrated about it, and AOSP users are as well. During such times, talking about AOSP is hard: most of the times, I know the deeper source of user frustration, but I can't talk about it in details, and that adds to my own frustration. That tends to turn any discussion into an echo chamber of frustration, which isn't good for anyone.
 
Ah, the 2013 goal explains your recent flurry of activity on the bug tracker.  Really appreciated, though at this point it sort of seems like it would take 5 years in and of itself just to triage everything. 

With the sheer number of irrelevant issues have you given any thought to adopting a more aggressive pre-filtering of issues via an input form like the way the chromium tracker does? That way you could actively direct non-AOSP enquiries to the proper channels (e.g. Google groups/OEM support), and enforcing pre-defined structures would increase the quality of reports which do get through.   

Also while I understand the rift between the private Google issue tracker and the public one is inevitable, isn't there any room for you to refactor the structure/workflow of the public one so that it better conveys state/better aligns with the private one whilst keeping the secrets secret? It's just that at the moment the process seems like a slot machine from the outside looking in. 


. 
 
+David Lawerteh - It is unavoidably going to take a long time to sort through everything, and there will unavoidably be some mistakes made along the way. 31k issues at 1 minute per issue (which is optimistic) will take 3 1/2 hours per workday to complete in 2013, i.e. quite a stretch goal. I still strongly think that's a far better approach than throwing away everything.

We have thought about trying to pre-filter issues indeed, but haven't been able to come up with a satisfactory flow yet. This is still in the back of my mind, though, and maybe we could do something simple but effective.

We're also looking for ways to better link the AOSP issue tracker with Google's. We've actually had different tools at one time or another, but none of those stood the test of time, for a number of reasons. We're not giving up, though, and we're going to try again and see how things work out this time.

Until we have better tools, there's going to be a fair amount of manual work on my plate, and you're right that automation is going to be a critical aspect.
 
I'm still not asking a work-related question, but... Thanks for the information.
 
+Jean-Baptiste Quéru What amount of 'link' of the bug trackers do you envision? When hacking on the platform, I often fins that the code is sufficiently complex to need some history to be understood. When wading through 'git blame', I often find myself looking at commits with a very brief description and an internal bug number. Having access at least to the problem description would often be a help already. Having access to the full discussion would obviously help even more ;)
 
Thanks to all of you involved in this wonderful project, consider myself an Android fan. Some requests +Jean-Baptiste Quéru if you don't mind from an app's developer perspective: Please add more snippets to the documentation and pull out some samples that actually doesn't even compile (Not sure right now of this, if I found them in the samples directory or in Google Code site f.e. tried to learn a bit of renderscript and found that the whole purpose of the thing changed from one version to another, so I end up completely puzzled). Also right now the situation with one of the best things of + 3.0, the ActionBar and at least, Gingerbread's backwards compatibility seems a bit weird for me. (Having to go to a not really full third party library, ActionBarSherlock instead of using a full backwards compatibility library provided by Google as usual).

Over all, as said thanks for this great project and God bless the strongly typed working environments. :-)
 
+Danny Baumann - I hear you on the subject of commit messages, regardless of access to the original bug report. I'll pass the message along.

In terms of linking things better, my perception is that our biggest Google-centric difficulty is to be able to search both the internal and external tracker at the same time, to present unified dashboards.

One major outward-facing difficulty is that there's a gap of time between when an issue is resolved internally in Google's private tree and when the change makes it to AOSP. One feature I'd really like to have would be to automatically close issues in the AOSP issue tracker that gets resolved with a given code drop.
 
+Rubén Gómez - Since maintaining the documentation isn't my domain of expertise, especially for developer-facing documentation, your best bet is to file in the AOSP tracker b.android.com with the Developer Documentation template. Please be descriptive but concise in the title, precise in the description, and file multiple reports if you have multiple issues.
 
Thank you for your insight, very informative! 
 
+Jean-Baptiste Quéru Thanks. In general, I find the vast majority of the commit messages to be very detailed, so please don't pass it along as a complaint :) It's just some commits which are direct bug fixes which aren't well understandable without knowledge about the actual bug (taking a very random example: commit 4d7349bb6df5a01ba451aa1abd4c9f6349a57016 in frameworks/base).
On a semi-related question: I take it's not possible to get history for code that dates back into the donut days, is it? I was wondering about a particular behaviour of the notification LED (the LED being assigned to the first notification that requests it until that notification is removed) was a deliberate design decision or just 'happened' to become like that; but git blame for said code just ended at 'auto import from //depot/cupcake...') :(
 
There was indeed a period in time (late cupcake to early donut) during which we were importing individual perforce changes into git commits but left the change descriptions out. That was deliberate as we didn't have the right practices in places yet to leave confidential information out.

Actually, chances are that the imports weren't exact 1:1 between perforce changes and git commits anyway, so even I might have a hard time getting back to the matching perforce change, especially as the server has been archived.

Newsflash: I just announced that there probably won't be any new releases of cupcake and earlier.
 
Fascinating. More work related posts, please!
 
Thanks for this great post and all the great follow ups JBQ!
 
Merci pour tout, à toi, et à ceux qui sont avec toi sur le projet.
Translate
 
You've gotta love the quick and powerful rise of Android.
 
This remains my favorite post of yours +Jean-Baptiste Quéru Thank you for your dedication and contributions over the years.  I wish you the best for what's ahead of you now.
Add a comment...