Shared publicly  - 
An example to go with my previous post on root..

Let's say that I wanted to write an application that would let me block or rate limit network access for other applications. Seems easy, just run "iptables" as root and add some firewall rules. Calling "su iptables ...." and managing the list is easy. The harder, but much better way would be to extend the framework. This also has the side effect of opening this up for other developers to use.

To do this, you need two things:

1. A way to add the rules (which requires root)
2. An API to add the rules
3. Access control to this API

All Android systems run a daemon, "netd", which runs as root and manages various aspects of the network such as tethering and traffic shaping. The framework has a service, appropriately named "NetworkManagementService" which communicates with netd using a simple protocol over a socket. Applications with the right permissions can get a handle to this service using Binder, and control the network without actually needing root.

So to build a firewall API, it's really easy. You put the pieces that require elevated privileges into netd, then add a few methods to the NMS such as "addRule", "deleteRule", and "listRules".  You can create and enforce a new permission, "android.permission.MODIFY_FIREWALL_STATE" that applications would require. You can even pop up a "scary" dialog similar to the newish VPNService when something needs it.

Then of course you upload your patches to the CM Gerrit, we iterate a bit, and ship it. If it turns out to be insanely useful, maybe it will go to Android proper.

Now you can write your app and a whole new class of applications that you couldn't do without using the root sledgehammer before. Yeah, it's harder, and you need to learn the system architecture a bit, but the result is much better and more importantly it's not a gaping security hole.

Of course it's possible to write malware that mirrors all of your packets to a remote site without your knowledge using this API, but Android's VPNService is actually more suited to this and it's already part of the framework :)

I might be exploiting this as an opportunity to sell the ideas behind CM, but I think it's a powerful concept. If your app needs to do something that normally can't be done, you can easily bend the system to your will and do it right.
Brian Lube's profile photoJon Stanford's profile photoJordan Pitts's profile photoblue presto's profile photo
Still doesn't take into account those that prefer to run rooted, debloated stock...
You are Genius. Should be working for NSA.
+Dan Rhodes  I'm interested in doing it better than stock. CM is about removing limitations.
If android can be bent and compromised so easily then why even stop the current avenue that has always been approached. I think I'm missing something
+Steve Kondik Right there with you...but most Android users are not and limiting "root" style functions to only CM seems to be against the spirit of Open Source. Sometimes I like a nice, stock ROM with full hardware camera functions.
I think it's important to focus on the end goals, not the means. Define use cases, implement proper mechanisms for those, without leaky security implications.


-Not everything should be an API. In some cases things need to be in direct user control to be settings that apps can't change, to remove cases where apps can programmatically damage one another's operation.

-Multi-user matters. When multiple users have apps running on the same device, it's really important that apps from one user don't get in the way of the apps of another user or access the data of another user.

-APIs that control global state need to be carefully crafted to avoid conflicts. In the example above, the system shouldn't end up in a situation where one app adds global filtering rules while another one removes the same rules.
+Dan Rhodes But why CM needs to provide a root solution for stock ROMs?

Also, what hardware function not present on CM yet?
I honestly use root to eff around in the /system partition and that's it.. oh and update my rom using rom manager.. but I'm sure +Koushik Dutta will get it working without root somehow..
The thing is +Dan Rhodes I don't think they would limit it to CM. I'm sure they'd open source it so PA etc. can use.
+Pedro Carneiro +Brian Johnson I love CM and have run since my Magic, but the day the CM or any AOSP ROM camera can start and burst capture my toddler is the day I agree with this stance.
+Dan Rhodes yes I really do miss the camera features of TW. But I think CMs Nememis project phase one will change that.
+Jean-Baptiste Quéru - "Not everything should be an API. In some cases things need to be in direct user control..."   -- But what if I, as a user, want to delegate that control to an app?

My uses cases for this involve having Tasker put my device in an "open" mode at home (no lockscreen password, adb on) and a secure mode when away (lockscreen password, adb off). I have tasker also turn gps off when I know it won't be any use because I'm indoors.  These are things that Android has decided "need to be in direct user control", but I as a user want to delegate that control to an app that runs rules that I have written. There needs to be an escape valve to allow power users to invent their own use cases.  The escape valve can be root, or it could be some "special" APIs that have to be enabled in Developer Settings...
+Dan Rhodes doing the same features on open source code? Anyway, we are a lot off topic here..
I've been toying with this idea for ages. I want a little snitch for android so badly
+Steve Kondik I love this idea but how do we convince OEMs to either (a) allow us to [legitimately] choose CM as our OS and/or (b) get Google and those same OEMs to support APIs that may only exist in CM?
+James Mason - The kind of use case you have in mind is exactly why there's a UI to control VPN or input methods. Those are too dangerous to only control by permissions.

Adding UI is a very slippery slope, since that tends to make the system harder to use, so for every additional setting there's a tricky negotiation with UX.

Developer settings should only control behaviors that are relevant to people developing applications. This is not a place to hide advanced user settings.
Do you need root to use custom boot? Or Flash ROM?
+richard holland I don't know about custom boot but flashing ROMs is a function of the recovery and does not require root in the least although for devices with no legitimate way of unlocking their bootloaders, root may be required to get to the point where one can install a custom recovery.
Abolishing root requirement meaning no more su UX (I meant UI)? That sounds awesome. Going down the Linux route?
SO CM currently doesn't need root access for everything in it to function?
+Steve Kondik While I agree with the sentiment I suspect it's gonna be a long hard sell. You're asking app developers to put in more work to limit their market.

If the framework changes are good enough they might make it into CM. If they're really good enough they might make it into AOSP. And then in a few years they may even be running on a majority of handsets.
+Grant Smith thought so. Everything has become so easy since the g1 I forgot why root was needed.
Edit: thank you
+Jean-Baptiste Quéru Yeah, the devil is in the details of course. I just want to see these kinds of powerful applications working without creating gigantic security holes. 
Am I the only person here who read "root sledgehammer" and smiled? 
I myself haven't rooted in a couple of years, but I'm learning a bit from tour posts.

+Steve Kondik - Yup. Bypassing all security is a great way to experiment and explore the possibilities, but that's not an end goal, it's a means to better understand the needs and desires, and to use as a starting point to properly implement the actual functionality with enough safeguards to make it suitable for the general population.

It's important to recognize the possibility of writing the OS-level code (order of magnitude: single person), compiling existing OS-level code (order of magnitude: thousand), using power-user features (order of magnitude: million) and exposing functionality to the general public (order of magnitude: billion).

Today, we're still in a situation where the vast majority of humans don't use a computing and communication device on a regular basis and many have never directly used one. It's very hard to be the one person writing code that'll be used by the 1 or 2 billion people who use that type of software at all, but it's ever harder to do so for the 5 or 6 billion people who don't even use such software ever.
+Jean-Baptiste Quéru - "The kind of use case you have in mind is exactly why there's a UI to control VPN or input methods."

Putting the UI there however breaks my use case, since I want my Tasker script to do it without intervention. However, I do not want some random APK to do it without intervention, even if there were some "take it or leave it" permissions in the Manifest.

Android has decided that a UI must be present, because of the second case, to protect the "billions" from your earlier post. But that denies me (and the "millions") the ability to do the first case, unless I/we use root (or a custom recovery) to push an APK to /system.

I agree that "Developer Options" is not the place to put "Advanced User" options. But then what is the place for advanced user options that are too dangerous for the "billions", but desired by the "millions"?  Because that's what a lot of us use root for, and it would close a big hole if there was a supported way to do things like this.

What about a UX like Accessibility or Device Administrators use?  This way, in addition to installing the APK with whatever permissions, I also have to explicitly grant the it permission to make changes behind my back?  Some pretty scary stuff lives under Device Administrators, why not add more scary stuff?
+James Mason - At my level, this is why Android is Open Source, this is why Nexus devices have unlockable bootloaders, this is why the proprietary binaries that are necessary to run Android on actual hardware are available: it opens the possibility of tuning the system deeper than what retail products allow at any point in time.

Now, beyond that, you're asking UX question to a software engineer, and that's a slippery slope. I know that I know nothing about UX, which means that I know that I need to defer to specialists when there are UX questions.

I still don't think that an API is necessarily an appropriate substitute for a setting. I don't think that APIs are end goals, I think they're just means to achieve specific goals, but they're not necessarily the only or the best ways to get there.
+Jean-Baptiste Quéru - I agree with you there, and that is why I buy only Nexus devices.

But it also means that I need to use the sledgehammer of root to do a lot of things that might be done more cleanly.  By tuning Android for the "billions", the "millions" of power users need to use the hacks developed by the "thousands" to customize the system to their liking.

I'm not so much wondering about a specific UX, as whether there could/should a way for the "millions" to flip a switch (or lots of switches) in stock Android to let them do things that are too dangerous for the "billions"... so that they don't have to use ugly hacks.
+Bob Igo - Tasker/Secure Settings are the bulk of my use cases. They only require root because they are doing "dangerous" things...
I still wish I could be using a Nexus 4. As nice as this HTC One is, it's still locked down and proprietary....
+James Mason - Yes, there's a horrible reality in software, which is that as soon as you have 2 users you won't be able to satisfy everyone. What's worse, the more powerful software becomes, the fewer users become satisfied.

Don't get me wrong, I fully understand that you're hitting a legitimate use case where Android as it exists today doesn't satisfy your needs. I'm not doubting that.

I do question whether an API is the right way to achieve what you want. APIs might not be huge sledgehammers, but they're still hammers, and not everything is a nail.

I expect that a UX person will claim that solving the problem of having too many settings by adding an extra setting goes in the wrong direction. I expect that they'll say "why can't the system just learn my habits, and adapt?" and this is where things do become more interesting.
+Jean-Baptiste Quéru - +1000000000 on "as soon as you have 2 users you won't be able to satisfy everyone". 

I really like "why can't the system just learn my habits, and adapt?", but I'm a programmer at heart and would like to have a hand in teaching the system what I want, and that's why I use a scripting language like Tasker.  I expect that even non-programmers may want a (smaller) hand in teaching the system, because it is so complex.  But maybe not: the Nest has been remarkably successful at learning people's habits.

I would still use root on Android anyways, for the same reason that I would use it on Linux, ChromeOS, and MacOS: to explore and tweak. But ideally I would only need to use it for exploration and fun (in Terminal/adb) rather than to implement operations that are part of my day-to-day use of the device.
+Jean-Baptiste Quéru I sort of see your see your reasoning here, but at the same time I don't really understand it. Judging by the responses in the other post, 90% of what people use root for is either networking or backup related. In the grand scheme of things, creating APIs for both of those would be fairly trivial, and would remove just about every reason for requiring root access as a power user.

Now, I understand that these APIs would be very powerful and potentially dangerous. But by refusing to provide them, you're forcing millions of users to resort to root instead, which is far less secure. At the risk of derailing the discussion, but because I feel this is a fair analogy that many people would agree with me on, I think that this is a situation like the War on Drugs: "We think that having a marijuana plant in your backyard is too dangerous, so you must buy your marijuana from Columbian drug cartels."

If you're really worried about security across the entire Android ecosystem, create these APIs. As they make it to the stock ROMs of the various manufacturers, very few people will have any reason to search for security holes to gain root privileges. At that point you will reduce the pool of people who know how to completely compromise the security of any given Android device from anyone who can type 'xda' into a Google search to only hackers who know what they're doing.
+Steve Kondik but, by doing this, devs need to choose between all devices and only cm &co custom rom devices. And the odds are not very good.

This would be extreemly usefull if manufacturers or google itself merges it in no time, but this kind of "black magic" wont happen, one thing is allowing to take a screenshot... Another one is to allow such deep lvl stuff...

Anyway, wont this kind of things take away from us the freedom android always gave us?

I mean... I dont want to be sandboxed on a bigger box than the one apple tries to keep me...
How about posting some guide on how devs can put up that kind of APIs? I'm no expert on C. I think that's what xposed framework has been doing. But unfortunately it still requires root. So to have that feature, I think it should be made at the low level. 
+Jordan Pitts - As far as I know the network cases are already mostly covered by the built-in tethering or by the VPN API, and if that existing functionality doesn't cover all the use cases it provides a solid starting point onto which to add some refinement.

Backups are already handled through adb backup, which mostly needs some UI love at this point, but the underlying functionality has been exposed for years.
Don't worry about camera. Nemesis will take care of that.
+Jean-Baptiste Quéru, speaking of (unfinished?) things that need some more polishing - whats the case with bulit-it (hidden...) privacy manager in Android 4.3?
Are we gonna see it soon, fully working, in next Android revision?
+Jean-Baptiste Quéru But is adding a hidden "Advanced User Settings" menu actually "solving the problem of having too many settings"?  From following the context of this conversation (and personally agreeing with +James Mason), it seems to be about addressing the use cases of the "millions" within the secure operation of an OS designed to retail to the "billions".  

Presumably an "Advanced Settings" menu would be hidden from the average retail user, much as "Developer Options" is currently, thereby leaving their UX completely unaffected.  Surely we don't doubt the power user's ability to navigate an additional settings menu in place of messing around with root?

Is it possible that this is simply less of a priority for commercial reasons?  Power users don't have much of an alternative at this point, and rooting provides a functional if less than ideal option.  From a purely economic perspective, it doesn't make a lot of sense for Google to prioritize a suite of "Advanced Settings" at this point.  Which is of course, why we're all grateful for +Steve Kondik's initiative here.
+Jean-Baptiste Quéru Couldn't APIs fix all of this as long as they are disabled by default? Under Security, you could add a section called APIs, which lists each category of "advanced" API and have them disabled by default. That way, the user could enable these apps to have access to whatever they want, as long as 1) the user enables the API and 2) the user enables access to the app.

Of course, this is all under Security, so it wouldn't have too much potential to confuse the user by having too many options in the main menu, as the Security section isn't opened very often.

Do you know any UX guys who would be willing to join this conversation? It would provide some telling insight into how something like this would take place.
Man, I don't know. The android framework itself resides in the system partition, which in turn needs to have an elevated privileges. Modifying the service is seamless, but placing it as part of the system might arrive at some issues. I understand that having root means having it all, and having it all means having the problems altogether. This might come in handy for devs and become a feature for KLP. I still hope for the benefit of everyone, both consumer and corporations.
Ultimately, It's my device. Its not Steve Kondik's, Google's, or Koush's. Its an open source platform and its my device. If I want to have root access, I should be able to have root access. You can warn me of the risk's of having root access and then its my own fault if something happens. Despite this being a good idea, we should be given the choice of whether or not we have root access.
+Jean-Baptiste Quéru on the point of backups, at least, it seems like a huge oversight to expect everyone to use adb when there's over ten million downloads of Titanium Backup and it's one of the top five paid apps. Clearly there are a lot of people that feel strongly enough about having this functionality on their device that they're willing to compromise their security to get it.
Bob Igo
+Jordan Pitts Titanium Backup + rsync + Tasker = unattended backups with sync.
Got an idea rather than root, add username permissions and groups that have access to different devices and things like Linux.
Genius man...actually I really don't know what or talking about but sound sophisticated for
My problem is if I want to write an app to access your new APIs my app will only run on CM. This is fine if I have a place to market the product.  However it has to be a place that users can expect an app to only work on CM.

So your stuck writing apps that work for rooted devices or apps that work for CyanogenMod devices.  I don't see Google making Firmware Vendor a search filter anytime on the Play Market anytime soon so how should app developers approach these APIs?
small sidepoint +Jon Stanford:  In this example you have to have root for the product to work, thus limiting your customer base.  Without direct access to root you cannot change the IPTables structure.  Thus you have a choice of rooted stock or a ASOP architecture.  

What +Steve Kondik is suggesting takes longer, and requires more effort, but in the long run you have a product that can be marketed to all devices, not just rooted stock or ASOP.  You also remove back doors (in the need for root) and make the system overall more stable and usable.   Yes, in the short run you are making a product that is for specific operating systems, but is the option to make hack software everyone that's rooted can use, or make a good product that folks would pay for?
+Brian Lube yes we can and do lead by example, at a firmware level however the ROM community is smaller than the stock rooted community and CyanogenMod is a subset of the ROM community as a whole.
+Jon Stanford the implication of what I'm reading here is that if an app developer and a ROM developer were to work together to create an API that removes the need to root a device in order to perform one of the three main root functions (adblock, firewall, etc; kernel settings control; and native backups) in a secure manner, that API would have a good chance of making it into AOSP in some form and then that app would be compatible with all ROMs.

This also doesn't require you to give up making root apps all together; you could create a root version and a CM version of the same app, or possibly some sort of toggle in settings or automatic detection feature.
Add a comment...