Profile cover photo
Profile photo
Haojie Xu
20 followers
20 followers
About
Posts

Post has attachment
Check out this video on YouTube:
Add a comment...

Post has shared content
I drew this Android Views lifecycle a while back for a presentation, seemed useful.  Is there a more formal diagram like this to show View/ViewGroup lifecycle? 

#Androiddev  
Photo
Add a comment...

Post has attachment

Post has attachment
Check out this video on YouTube:
Add a comment...

Post has shared content
How about some Android graphics true facts?

(Edit: there have been a number of comments treating this as being written as an excuse for Android or not mattering to users or such. I'd just like to clarify that I wrote this solely to address a lot of incorrect information that I see posted around the web as truth. This is no attempt to excuse anything, and it is solely for those who already have an interest in writing and reading the often factually incorrect technical information out there.)

I get tired of seeing so much misinformation posted and repeated all over the place about how graphics rendering works on Android. Here is some truth:

• Android has always used some hardware accelerated drawing. Since before 1.0 all window compositing to the display has been done with hardware.

• This means that many of the animations you see have always been hardware accelerated: menus being shown, sliding the notification shade, transitions between activities, pop-ups and dialogs showing and hiding, etc.

• Android did historically use software to render the contents of each window. For example in a UI like http://www.simplemobilereview.com/wp-content/uploads/2010/12/2-home-menu.png there are four windows: the status bar, the wallpaper, the launcher on top of the wallpaper, and the menu. If one of the windows updates its contents, such as highlighting a menu item, then (prior to 3.0) software is used to draw the new contents of that window; however none of the other windows are redrawn at all, and the re-composition of the windows is done in hardware. Likewise, any movement of the windows such as the menu going up and down is all hardware rendering.

• Looking at drawing inside of a window, you don’t necessarily need to do this in hardware to achieve full 60fps rendering. This depends very much on the number of pixels in your display and the speed of your CPU. For example, Nexus S has no trouble doing 60fps rendering of all the normal stuff you see in the Android UI like scrolling lists on its 800x480 screen. The original Droid however struggled with a similar screen resolution.

• "Full" hardware accelerated drawing within a window was added in Android 3.0. The implementation in Android 4.0 is not any more full than in 3.0. Starting with 3.0, if you set the flag in your app saying that hardware accelerated drawing is allowed, then all drawing to the application’s windows will be done with the GPU. The main change in this regard in Android 4.0 is that now apps that are explicitly targeting 4.0 or higher will have acceleration enabled by default rather than having to put android:handwareAccelerated="true" in their manifest. (And the reason this isn’t just turned on for all existing applications is that some types of drawing operations can’t be supported well in hardware and it also impacts the behavior when an application asks to have a part of its UI updated. Forcing hardware accelerated drawing upon existing apps will break a significant number of them, from subtly to significantly.)

• Hardware accelerated drawing is not all full of win. For example on the PVR drivers of devices like the Nexus S and Galaxy Nexus, simply starting to use OpenGL in a process eats about 8MB of RAM. Given that our process overhead is about 2MB, this is pretty huge. That RAM takes away from other things, such as the number of background processes that can be kept running, potentially slowing down things like app switching.

• Because of the overhead of OpenGL, one may very well not want to use it for drawing. For example some of the work we are doing to make Android 4.0 run well on the Nexus S has involved turning off hardware accelerated drawing in parts of the UI so we don’t lose 8MB of RAM in the system process, another 8MB in the phone process, another 8MB in the system UI process, etc. Trust me, you won’t notice -- there is just no benefit on that device in using OpenGL to draw something like the status bar, even with fancy animations going on in there.

• Hardware accelerated drawing is not a magical silver bullet to butter-smooth UI. There are many different efforts that have been going on towards this, such as improved scheduling of foreground vs. background threads in 1.6, rewriting the input system in 2.3, strict mode, concurrent garbage collection, loaders, etc. If you want to achieve 60fps, you have 20 milliseconds to handle each frame. This is not a lot of time. Just touching the flash storage system in the thread that is running the UI can in some cases introduce a delay that puts you out of that timing window, especially if you are writing to storage.

• A recent example of the kinds of interesting things that impact UI smoothness: we noticed that ICS on Nexus S was actually less smooth when scrolling through lists than it was on Gingerbread. It turned out that the reason for this was due to subtle changes in timing, so that sometimes in ICS as the app was retrieving touch events and drawing the screen, it would go to get the next event slightly before it was ready, causing it to visibly miss a frame while tracking the finger even though it was drawing the screen at a solid 60fps. (Edit: for those who need this made clear, yes of course this particular issue is fixed.)

• When people have historically compared web browser scrolling between Android and iOS, most of the differences they are seeing are not due to hardware accelerated drawing. Originally Android went a different route for its web page rendering and made different compromises: the web page is turned in to a display list, which is continually rendered to the screen, instead of using tiles. This has the benefit that scrolling and zooming never have artifacts of tiles that haven’t yet been drawn. Its downside is that as the graphics on the web page get more complicated to draw the frame rate goes down. As of Android 3.0, the browser now uses tiles, so it can maintain a consistent frame rate as you scroll or zoom, with the negative of having artifacts when newly needed tiles can’t be rendered quickly enough. The tiles themselves are rendered in software, which I believe is the case for iOS as well. (And this tile-based approach could be used prior to 3.0 without hardware accelerated drawing; as mentioned previously, the Nexus S CPU can easily draw the tiles to the window at 60fps.)

• Hardware accleration does not magically make drawing performance problems disappear. There is still a limit to how much the GPU can do. A recent interesting example of this is tablets built with Tegra 2 -- that GPU can touch every pixel of a 1280x800 screen about 2.5 times at 60fps. Now consider the Android 3.0 tablet home screen where you are switching to the all apps list: you need to draw the background (1x all pixels), then the layer of shortcuts and widgets (let’s be nice and say this is .5x all pixels), then the black background of all apps (1x all pixels), and the icons and labels of all apps (.5x all pixels). We’ve already blown our per-pixel budget, and we haven’t even composited the separate windows to the final display yet. To get 60fps animation, Android 3.0 and later use a number of tricks. A big one is that it tries to put all windows into overlays instead of having to copy them to the framebuffer with the GPU. In the case here even with that we are still over-budget, but we have another trick: because the wallpaper on Android is in a separate window, we can make this window larger than the screen to hold the entire bitmap. Now, as you scroll, the movement of the background doesn’t require any drawing, just moving its window... and because this window is in an overlay, it doesn’t even need to be composited to the screen with the GPU.

• As device screen resolution goes up, achieving a 60fps UI is closely related to GPU speed and especially the GPU’s memory bus bandwidth. In fact, if you want to get an idea of the performance of a piece of hardware, always pay close attention to the memory bus bandwidth. There are plenty of times where the CPU (especially with those wonderful NEON instructions) can go a lot faster than the memory bus.

EDIT:

Wow this has generated a lot more discussion, and I won't be able to address nearly everything that has been raised. But I'll try to expand on things a bit here to better address what I think are some of the more interesting points.

Some have raised points along the lines of Samsung Galaxy S2 phones already having a smoother UI and indicating that they are doing something different vs. the Galaxy Nexus. When comparing individual devices though you really need to look at all of the factors. For example, the S2's screen is 480x800 vs. the Galaxy Nexus at 720x1280. If the Nexus S could already do 60fps for simple UIs on its 480x800, the CPU in the S2's is even better off.

The real important difference between these two screens is just that the Galaxy Nexus has 2.4x as many pixels that need to be drawn as the S2. This means that to achieve the same efficiency at drawing the screen, you need a CPU that can run a single core at 2.4x the speed (and rendering a UI for a single app is essentially not parallelizable, so multiple cores isn't going to save you).

This is where hardware accelerated rendering really becomes important: as the number of pixels goes up, GPUs can generally scale much better to handle them, since they are more specialized at their task. In fact this was the primary incentive for implementing hardware accelerated drawing in Android -- at 720x1280 we are well beyond the point where current ARM CPUs can provide 60fps. (And this is a reason to be careful about making comparisons between the Galaxy Nexus and other devices like the S2 -- if you are running third party apps, there is a good chance today that the app is not enabling hardware acceleration, so your comparison is doing CPU rendering on the Galaxy Nexus which means you almost certainly aren't going to get 60fps out of it, because it needs to hit 2.4x as many pixels as the S2 does.)

To be complete, there is another big advantage that the GPU gives you -- many more drawing effects become feasible. For example, if you are drawing a bitmap in software, you basically can't do anything to it except apply an offset. Just trying to scale it is going to make rendering significantly slower. On a GPU, applying transformations well beyond simple scales is basically free. This is why in the new default Holo themes in Android we have background images -- with hardware accelerated drawing, we can afford to draw (and scale) them. In fact, if the hardware path is not enabled by the app, these background images will be turned off.
Add a comment...

Post has shared content
A few days ago I wrote a post trying to correct a lot of the inaccurate statements I have seen repeatedly mentioned about how graphics on Android works. This resulted in a lot of nice discussion, but unfortunately has also lead some people to come up with new, novel, and often technically inaccurate complaints about how Android works.

These new topics have been more about some fundamental design decisions in Android, and why they are wrong. I’d like to help people better understand (and judge) these discussions by giving some real background on why Android’s UI was designed the way it is and how it actually works.

One issue that has been raised is that Android doesn’t use thread priorities to reduce how much background work interrupts the user interface. This is outright wrong. It actually uses a number of priorities, which you can even find defined right here http://developer.android.com/reference/android/os/Process.html#THREAD_PRIORITY_AUDIO in the SDK.

The most important of these are the background and default priorities. User interface threads normally run at the default priority; background threads run in the background priority. Application processes that are in the background have all of their threads forced to the background priority.

Android’s background priority is actually pretty interesting. It uses a Linux facility called cgroups to put all background threads into a special scheduling group which, all together, can’t use more than 10% of the CPU. That is, if you have 10 processes in the background all trying to run at the same time, when combined they can't take away more than 10% of the time needed by foreground threads. This is enough to allow background threads to make some forward progress, without having enough of an impact on the foreground threads to be generally visible to the user.

(You may have noticed that a “foreground” priority is also defined. This is not used in current Android; it was in the original implementation, but we found that the Linux scheduler does not give enough preference to threads based on pure priority, so switched to cgroups in Android 1.6.)

I have also seen a number of claims that the basic Android design is fundamentally flawed and archaic because it doesn’t use a rendering thread like iOS. There are certainly some advantages to how iOS work, but this view is too focused on one specific detail to be useful, and glosses over actual similarities in how they behave.

Android had a number of very different original design goals than iOS did. A key goal of Android was to provide an open application platform, using application sandboxes to create a much more secure environment that doesn’t rely on a central authority to verify that applications do what they claim. To achieve this, it uses Linux process isolation and user IDs to prevent each application from being able to access the system or other application in ways that are not controlled and secure.

This is very different from iOS’s original design constraints, which remember didn’t allow any third party applications at all.

An important part of achieving this security is having a way for (EDIT: It has been pointed out to me that iOS does in fact use multiple windows and multiple GL contexts. Lesson to me, just don't talk about anything I haven't directly verified. :) That still doesn't change things for Android, though, where as I mention later we simply did not have hardware and drivers that could do multiple GL contexts until fairly recently.)
individual UI elements to share the screen in a secure way. This is why there are windows on Android. The status bar and its notification shade are windows owned and drawn by the system. These are separate from the application’s window, so the application can not touch anything about the status bar, such as to scrape the text of SMS messages as they are displayed there. Likewise the soft keyboard is a separate window, owned by a separate application, and it and the application can only interact with each other through a well defined and controlled interface. (This is also why Android can safely support third party input methods.)

Another objective of Android was to allow close collaboration between applications, so that for example it is easy to implement a share API that launches a part of another application integrated with the original application’s flow. As part of this, Android applications traditionally are split into pieces (called “Activities”) that handle a single specific part of the UI of the application. For example, the contacts lists is one activity, the details of a contact is another, and editing a contact is a third. Moving between those parts of the contacts UI means switching between these activities, and each of these activities is its own separate window.

Now we can see something interesting: in almost all of the places in the original Android UI where you see animations, you are actually seeing windows animate. Launching Contacts is an animation of the home screen window and the contacts list window. Tapping on a contact to see its details is an animation of the contacts list window and the contacts details window. Displaying the soft keyboard is an animation of the keyboard window. Showing the dialog where you pick an app to share with is an animation of a window displaying that dialog.

When you see a window on screen, what you are seeing is actually something called a “surface”. This is a separate piece of shared memory that the window draws its UI in, and is composited with the other windows to the screen by a separate system service (in a separate thread, running at a higher than normal priority) called the “surface flinger.” Does this sound familiar? In fact this is very much like what iOS is doing with its views being composited by a separate thread, just at a less fine-grained but significantly more secure level. (And this window composition has been hardware accelerated in Android from the beginning.)

The other main interesting interaction in the UI is tracking your finger -- scrolling and flinging a list, swiping a gallery, etc. These interactions involve updating the contents inside of a window, so require re-rendering that window for each movement. However, being able to do this rendering off the main thread probably doesn’t gain you much. These are not simple “move this part of the UI from X to Y, and maybe tell me when you are done” animations -- each movement is based on events received about the finger on the screen, which need to be processed by the application on its main thread.

That said, being able to avoid redrawing all of the contents of the parts of the UI that are moving can help performance. And this is also a technique that Android has employed since before 1.0; UI elements like a ListView that want to scroll their content can call http://developer.android.com/reference/android/view/View.html#setDrawingCacheEnabled(boolean) to have that content rendered into a cache so that only the bitmap needs to be drawn as it moves.

Traditionally on Android, views only have their drawing cache enabled as a transient state, such as while scrolling or tracking a finger. This is because they introduce a fair amount more overhead: extra memory for the bitmap (which can easily total to multiple times larger than the actual frame buffer if there are a number of visual layers), and when the contents inside of a cached view need to be redrawn it is more expensive because there is an additional step required to draw the cached bitmap back to the window.

So, all those things considered, in Android 1.0 having each view drawn into a texture and those textures composited to the window in another thread is just not that much of a gain, with a lot of cost. The cost is also in engineering time -- our time was better spent working on other things like a layout-based view hierarchy (to provide flexibility in adjusting for different screen sizes) and “remote views” for notifications and widgets, which have significantly benefited the platform as it develops.

In fact it was just not feasible to implement hardware accelerated drawing inside windows until recently. Because Android is designed around having multiple windows on the screen, to have the drawing inside each window be hardware accelerated means requiring that the GPU and driver support multiple active GL contexts in different processes running at the same time. The hardware at that time just didn’t support this, even ignoring the additional memory needed for it that was not available. Even today we are in the early stages of this -- most mobile GPUs still have fairly expensive GL context switching.

I hope this helps people better understand how Android works. And just to be clear again from my last point -- I am not writing this to make excuses for whatever things people don’t like about Android, I just get tired of seeing people write egregiously wrong explanations about how Android works and worse present themselves as authorities on the topic.

There are of course many things that can be improved in Android today, just as there are many things that have been improved since 1.0. As other more pressing issues are addressed, and hardware capabilities improve and change, we continue to push the platform forward and make it better.

One final thought. I saw an interesting comment from Brent Royal-Gordon on what developers sometimes need to do to achieve 60fps scrolling in iOS lists: “Getting it up to sixty is more difficult—you may have to simplify the cell's view hierarchy, or delay adding some of the content, or remove text formatting that would otherwise require a more expensive text rendering API, or even rip the subviews out of the cell altogether and draw everything by hand.”

I am no expert on iOS, so I’ll take that as as true. These are the exact same recommendations that we have given to Android’s app developers, and based on this statement I don't see any indication that there is something intrinsically flawed about Android in making lists scroll at 60fps, any more than there is in iOS.
Add a comment...

Post has attachment

Post has attachment
Add a comment...

Post has attachment

Post has attachment
Wait while more posts are being loaded