Shared publicly  - 
The Saga of a CyanogenMod Exynos4 device maintainer - I9300 Jellybean, and things come to a head

Disclaimer:  This series of posts contains the personal opinion/perspective of just one member of the team.  It is not, in any way, an official statement from the project.

This is a continuation of the saga, with the last installment being

Just a warning - I've been sick for a few days which is why this update has taken so long, and in addition, this period was VERY hectic - So things might be a bit disorganized here.

Integrating I9300 Jellybean
This was easier said than done.  To give a bit of history, ARM versions the code drops of their Mali driver to vendors, which vendors then integrate.  Some vendors really hack things up severely (look at all the frequency control cruft in the Mali platform code for pegasus, aka Exynos 4412).

ICS used Mali r2p3 drivers.  The initial I9305 leak and I9305 kernel source used Mali r3p0 drivers.  However, by the time I9300 Jellybean went live, they had reverted to Mali r2p4 drivers.  The Mali r2p4 drivers in the official release, when various framework workarounds are put in place, seem to perform better than the r3p0 drivers that Samsung seems to have dead-ended us with.

The problem is - Mali drivers older than r3p0 need SOME sort of workaround in the frameworks in order to not leak memory like a sieve when used in Jellybean.  Samsung refuses to document this workaround for their broken r2p4 grallocs.  So we were able to finally get rid of the Jellybean gralloc memleaks, but at the cost of being stuck with dead-end blobs that are unsustainable in the long term as Samsung has stopped using them and gone back to r2p4.

The good news is that gralloc, EGL, and the Mali libs work with either Exynos 4210 or 4412.  As a result, the I9300 release was able to fix the memleak on I9100/I777/N7000.  However, we were not able to use the I9300 hwcomposer blobs - there are slight differences between these, including dependence on libfimg3x vs libfimg4x, among other things.

Edit:  I forgot to mention this in the initial post, but this was the third cycle of attempts to get source-based HWC working with our devices, as without it, 4210 devices were stuck using legacy ICS HWC.  (Sorta works, but no vsync support and no way to add it to a blob.)  I think I spent 10-20 hours/week in my evenings/weekends fighting this with absolutely zero progress...

Edit 2 (I warned you guys about this...):  One issue I think I mentioned in the past is that Samsung's kernels tend to have majorly mangled bcmdhd - Things were worse in the I9300 JB drop - Tethering totally failed with the new bcmdhd driver under all circumstances.  No one has yet to figure out why, and since Samsung only does tarball drops with no commit history, it's hard as hell to figure out WHICH thing changed.  It's probably something that would have been fixed if it were one of the only issues, but the problem is, there are so many issues piled on top of each other with these devices that things which might be an attackable problem if they were one of a handful become massive issues when combined with 3247927430423 other things that are broken - When you have that many issues, there just seems to be no end in sight, and far less motivation to keep beating on them.

Hillenbrand (codeworkx) hits his limits
Around this time, the longest-active Exynos4 CM maintainer, +Daniel Hillenbrand (aka codeworkx) hit his limits - - I think he's lasted longer than any other Exynos4 maintainer has, from start to "I've had enough of this crap" - Typical longevity of other maintainers has been on the order of a year or less, see +Atin M as an example.

The reasoning for this was - Exynos4 is a nightmare platform to work with, the only worse platform being Tegra (and we all know about how they are with regards to open source, being the recipient of the middle finger and a big "FUCK YOU" from Linus Torvalds).  Qualcomm has robust and up-to-date reference sources at CodeAurora that actually WORKS with little effort on handsets (HTC requires significant work, Samsung requires some, Sony handsets require almost no work to get CAF code working as Sony is smart and doesn't deviate from the ref source unless they absolutely HAVE to.)

As an example of how up to date CAF tends to be - See;a=blob;f=libhwcomposer/hwc.cpp;h=a2ea3c9a1f98190aae0b7cbdc340d2003d9bf7af;hb=2db42eda10d88d0547bddf419d8eca47f0a8cfbf - Android 4.2 has only been out in AOSP for a week, and CAF already has 4.2-related sources.  Now this is partly due to the Nexus 4 - however also note that CAF has a SINGLE UNIFIED HWC for all recent SoCs - Samsung changes the whole infrastructure with each generation, so Exynos5 stuff (Nexus 10) is of little to no use on Exynos4.

Edit:  One thing about CAF:  Qualcomm does give undocumented "blobs" to their customers.  These consist of "optimized" replacements for parts of the Android system.  HOWEVER, the penalty for not having these blobs is reduced benchmark performance (which is why, as I understand it, the Optimus G outperforms the Nexus 4 in benchmarks), NOT complete loss of some aspects of functionality as is the case with Samsung's nonfunctional reference source.

TI's OmapZoom is similarly up to date and robust.  The end result is:

The closest we had were the Insignal reference sources, and Hardkernel's 2GB tarball drops.  Both of these totally lacked commit history and were vastly out of date - ICS source code months after Jellybean was released.  They also simply didn't work - with the exception of the OMX libraries (hwaccel video) and the FIMG libraries themselves, nearly everything else was broken.  Using source-based gralloc would completely fail to function at all (black screen), libcamera was unable to capture even after multiple attempts to modify it to work with M5MO (Indicative of an underlying flaw in the core source code, as it did not take significant effort to get Exynos3 libcamera to work with M5MO on the Infuse, so it's not a sensor issue.), and hwcomposer was 100% pure placebo - Under any circumstances, its behavior was indistinguishable from having no HWC library present at all.

So between that, other things (see earlier installments of this saga for many examples), and Superbrick, pretty much all of the Exynos4 maintainers were at the limits of exhaustion with Samsung and especially with Exynos4.

I indicated that the Note 10.1 was going to be my last device, and I was not sure how long I'd be staying with the I777 and N7000, as I was exhausted at this point too.  I was tired of being months behind the rest of the Cyanogenmod team because I worked with devices that had more blobs and more interface breaks in the blobs than any other device (Except for Tegra3 devices, but people already knew to avoid these unless they were in a Nexus.)

Sometime around this I emailed my contact with a request for hwcomposer source - no response.  This was not the first time anyone had requested source, but I included pointers to the Omapzoom and CAF reference source as examples of what other companies were doing.  (It makes the request more reasonable if you're asking for source of something that other manufacturers provide source for.)  As to the lack of a response - My guess is Samsung's firewall ate it.  God forbid you even mention the NAME of another company in an email to/from Samsung - the firewall will drop it.  This makes proper discussion of why we're frustrated difficult when any meaningful email will get blackholed.

Not long after, a bunch of users on XDA unhappy with the situation started a social media shitstorm.

Samsung asks for help
Our contact reached out to us to try and calm the storm.  Mind you, this isn't the first time Samsung has had a social media storm/flood, and this isn't the first time we've been asked to try and calm it down.  In the past, the storm contained unreasonable requests, like "Give us ICS for the Galaxy SL - we know you have it!"  (Even though they didn't have a shred of evidence to support that claim) - We would help try to calm these down.

However, this time, Samsung had driven the community and a bunch of developers past the breaking point.  +Daniel Hillenbrand  +Guillaume Lesniak , myself, and others (such as +Gökhan Moral ) were at the point where if Samsung didn't make significant changes, we were going elsewhere (+Atin M  had already retired long ago...).  Thanks to Sony's great efforts over the past year AND Nexus devices being far easier to obtain, we did have places to go.
My response:  "Give us what I asked for in my email last week to which you did not respond"
Daniel's response included numerous CAF links as examples of what Qualcomm was providing that Samsung wasn't
Atin echoed our feelings - We wanted reference source comparable in quality to CAF and omapzoom, not the broken crap that is out there now.

My role in this calmed down for a while, as I began packing for the Big Android BBQ.

My next post will be outside of the timeline of this saga - Rather than be the next installment in the history of this whole mess, it will be a technical summary of where we are.  It will contain some minor spoilers, although I don't expect any of it to be surprising to anyone who has followed things thus far.  The next installment in the saga will be coverage of the BABBQ.

Next installment up at:
Zach Jenkins's profile photoIbrahim Awwal's profile photoAndrew Dodd's profile photoScott Remick's profile photo
I wish I could tell my side of the story with that level of detail, but that's (obviously) confidential.
Again aan great post in this incredible saga. I like my N7000 and tend to keep it for a while at least, but you opened my eyes for sure.
My next device will not be a Samsung, unless.... ;-) 
+Jean-Baptiste Quéru Yeah, it seems like your experiences with various companies have been significantly different from mine.  (Obviously, just a guess since you can't really comment)  Although, admittedly, there are some aspects of certain manufacturers that can be extremely frustrating for AOSP (to the point of being showstoppers) but merely annoying to those willing to be in a "greyer" area with respects to binary distribution.

Also, Exynos5 seems to be in much better state than Exynos4 was in terms of opensource support.  Sadly, it's not too useful on 4, just like exynos3 code from crespo wasn't too useful for exynos4 devices.

Of course there's also a difference in perspectives - It's your job to work with this stuff, for many of us, we do this in our spare time as a mental exercise/challenge.  There are certain things one tolerates less when one is not getting paid to tolerate them.
nice writing style :) and a great article 
I think my next devices will be +Nexus  .  I have an S2 (i777) running CM10 and thank you so much for your work but if you don't have the documentation you need for custom components then we continue to run the risk of another superbrick.  Thank you for bringing the story out and I hope it can better direct customers to other device manufactures who support their devices (and follow GPL).
I own an i777 and have lost almost all hope to have an stable CM 10.x release.

From the @SamsungExynos(Samsung LSI) and Insignal point of view they probably believe that they have released what's needed to make their reference board work. Then you have @samsungmobile that releases the bare minimum that they feel they need to release to meet their open source obligations, which don't include their production binary drivers as those are proprietary, or at least they believe they are.

Now my educated guess is that Samsung LSI is under contractual obligation with Samsung Mobile and cannot release anything beyond what has already been released. I think the community need to rattle Samsung Mobile's cage as they are the ones that produce the final distribution of whatever version of android your phone gets.

The gray area is how to interface with those binary objects that Samsung is so eager to protect, and that the "secret sauce" that the Exynos CM maintainers want to get a hold of. That's where documented API's would be of great benefit in order to know what the binaries expect and what you should get in return.

Now looking at the Google v. Oracle trial where API's where at the center of the dispute, and how Samsung has been targeted in copyright and patent litigation over the past year I'm sure they will very hesitant to release any of the "secret sauce" that might later be of use to them against the fruity death star.

So we are pretty much F-ed.
I did another edit that does cover one "quirk" of Qualcomm's source releases vs. binaries.
I wish I could go Nexus, but the lack of removable SD card is a deal breaker. That and removable battery are requirements for any phone I buy. And I really like OLED screens
So CAF essentially gives full git history for the code they release? Why doesn't Samsung do this? It feels like they're trying to hide some mythical competitive advantage by withholding information, but from whom they're trying to hide it is unclear, and it frankly makes no sense. Especially as a SoC manufacturer themselves I would think that they would want others to use Exynos, but to date pretty much the only other phones using Exynos are the Meizu phones (which I hear are incredibly locked down). Samsung's behavior seems irrational at best, although I guess as a large corporation with many divisions sometimes internal factors can drive decisions that seem insane to those looking from the outside in.
Didnt knew they were worried about social storm, now i will join too until my i9100 go away. Great postos btw, learning a lot about Android.
+Alex sm the problem is perhaps that the person behind @SamsungExynos is not in a position to influence or implement changes to the T's and C's present on the original 4210 contract or other Exynos4 SoC's contracts.

One thing is for sure, they are concerned of the dev backlash now that Exynos5 is being sold as a more than mobile player.

In any case things are unquestionably f-ed up at Samsung, or they have just grown to big to care or to know what goes on with other divisions. Corporations that big tend to have divisions like LSI be completely autonomous and only care about revenue generation. Until somebody makes a compelling business case in favor of the benefit of keeping the devs on their side there will be little buy-in from top management to get what the community is asking for.

Yes, it is an f-ed up situation but that's large corporations tend to work.
So Samsung has released source code, it's just that you're not finding it very useful because it's not well documented or out of date (ICS code instead of JB)? It does look like they're making an effort then, but they're either slow or there are internal reasons holding them back from just dumping whatever they have whenever you want it, is my guess.

I hope you guys are asking nicely though, I know it's frustrating to keep asking and not get much of anything useful back, but remember, if you're too pushy they might just say "screw you". After all, you're asking them for a favor, they're under no obligation to release anything. But I do hope they come around.
So you say that you'll be leaving  the i777 soon. What are you going to move to? Is there a different company or phone that you do suggest moving to?
+Bill Puckering, the problem would seem to be that the source they release does not compile to the binaries that they provide on the phones, which makes it useless and possibly a GPL violation if the code in question is GPL licensed, but as Andrew said it's hard to prove that's the case. In some cases the provided source doesn't even compile at all and in most cases it doesn't work which I don't think constitutes making an effort.
+Ibrahim Awwal Unfortunately, all of the stuff in question is Apache licensed.  Legally, they don't have to release it.  However, it's still fair to hold them to the standards of quality and timeliness set by CodeAurora and Omapzoom, and recommend that people don't purchase their devices until they do if AOSP support is desired.

There's maybe a 25% chance that we'd be able to get their stuff to work by totally reworking all of the memory allocation stuff in the kernel and using their blobs...  But that would just get us back to the memleak pain we dealt with for months since their repos only cover ICS on Exynos4 and are missing:
1)  Any documentation on what the new gralloc architecture is like (It's COMPLETELY different in Jellybean)
2)  Any information on their new custom ion implementation (libsecion), needed by newer gralloc implementations
3)  Any documentation on how to make Mali r2p4 not leak memory like a sieve
+Andrew Dodd I actually have an OTG cable but it's just for reading off my camera's full-size SD card. Leaving it permanently connected to my phone at all times as a substitute for an internal MicroSD card just isn't a practical solution. :(

I totally appreciate your frustrations with Samsung and each post you make just makes me angrier with them. But I just can't be on the same ship with those who cry out "Never again, Samsung! I'm getting a Nexus next time!" since the Nexus, while fully-supported software-wise, is crippled hardware wise. I NEED removable battery AND expandable internal storage. These things have been de-facto standards in cell phones for many, many years. Just as Samsung's lack of software support infuriates me, so too does Google's lack of requiring removable batteries and expandable storage in the Nexus models. There's no legitimate excuse for this either. Their hypocrisy in "pulling an Apple" by sealing the battery inside AND preventing you from the accepted standard of adding on storage w/ a MicroSD (things I've been able to do since my earliest Motorola flip phones) riles me up to no end. I would ditch Samsung and get a Nexus or something else if there was a suitable, equal or better option. But there just isn't.

I need a removable battery because I've never had an Android phone that could last me through a whole full day of use, so I keep spare charged batteries. I need removable memory because I not long store tons of files on my phone, but I back up to the removable storage automatically and daily. I can't tell you how many times failed phone hardware has been saved by me being able to just pop out the MicroSD, move to a replacement phone, and restore right back.
Add a comment...