Profile cover photo
Profile photo
Dustin M
Dustin's posts

Post has shared content

Imagine that you're in your early teens.

Imagine that you're named after a major religious figure living centuries ago in the Middle East.

Imagine that you're interested in Science and Technology, so much that you try to Engineer things on your own, which then forces you to learn the Mathematics that help you along the way, all that while you're still in your early teens.

At this point, two things can happen.

In one case, your name is associated with Christianity, your skin is white, you live in France. You get encouraged, doors open for you, you get guided toward top education, you'll eventually get amazing jobs, and you'll end up living a comfortable life. That's my story.

In another case, your name is associated with Islam, your skin is brown, you live in Texas. You get shamed, you get arrested, schools close their doors on you. That's Ahmed Mohammed's story. I don't know how that story ends, but I'm really hoping it ends well.

That's a pretty extreme case of privilege. But it is privilege nonetheless. We have to recognize such extreme cases of privilege if we want to be able to fight all forms of privilege. That's the only way we can eventually reach a point where all men are created equal, where we all have certain unalienable Rights, including Life, Liberty and the pursuit of Happiness, because clearly we're not there yet.

Ahmed's Liberty has already been seriously infringed, and from this point his pursuit of Happiness is in jeopardy, possibly for the rest of his Life.


Post has shared content
Irresponsible or unavoidable borrowing?

Growing up in Europe, I didn't pay much attention to the construction of the Euro, and whatever little I remember has nothing to do with the economics of it. Now, older, having lived in the US for a while, with a Greek wife, I'm looking at the way the Euro is unraveling and I've been using the opportunity to try to figure out how it works (or, rather, why it doesn't).

The core mechanism that allows multiple states to share the same currency is pretty simple: since the weaker states can't devalue their currency to compensate for their trade deficit with the stronger ones, money has to flow from the stronger economies to the weaker ones in order to maintain the balance.

We see that in the US: as measured in GDP per capita, there's about a 2:1 ratio between the strongest states and the weakest ones. To compensate for that, a lot of money flows between states, through the federal government. Most taxes in the US are federal taxes, i.e. about 75%, and the federal government doesn't necessarily spend the money it collects in the exact states where it collects them. As an example, every year about 130 billion dollars paid by California in federal taxes don't make it back into California. Texas and New York are the two other states that have a negative balance of more than 100 billion each. For those 3 states, that outflow on money represents 5.7%, 7.2% and 7.4% of their respective GDPs. California is literally sending money to other states so that those states can buy California stuff. The same is true for Texas, New York, and about 20 of the 50 states that are sending money to the other 30.

Looking back in history, the Marshall Plan followed a somewhat similar logic: the US sent aid to Europe, to allow Europeans to buy US goods, which was both a stabilizing mechanism for European currencies that otherwise were in a devaluation spiral, and an outlet for the huge US industrial production. For reference, the Marshall Plan amounted to 120 billion dollars (in today's dollars) over 4 years, which is tiny compared to the amount of money that the federal government now redistributes across state lines.

We can compare that to the situation in the Eurozone/EU, where the GDP per capita varies by a factor of about 2.3:1. Germany's balance in the EU budget is negative by less than 9 billion Euros. France's and Italy's follow at approximately 6.5 billion and 6 billion. Germany's 9 billion Euros is tiny compared to California's 130 billion dollars, especially since Germany's GDP is 60% larger than that of California. Since the US and EU economies have approximately the same size, that's a reasonably apples-to-apples comparison. The biggest negative balance that a Eurozone country has with the EU is about 0.41% of its GDP. The biggest positive balance is 1.3%. Within the US, only 4 states out of 50 fall within that range.

That's the problem right there: Germany is not flowing enough money out to other Eurozone countries to compensate for its own very strong economy. That's true of other rich European countries as well, e.g. Netherlands, Austria, France.

From the Greek point of view, the only way to get that money to flow in order to maintain balance had been for the government to borrow. That wasn't irresponsible borrowing. That was mechanical, predictable. Greece's poor historical discipline around government finances only accelerated an unavoidable process, but it's not a root cause.

In fact, predictably, pushing Greece into austerity made things worse, much worse: with the root cause being Greece's relatively weak economy compared to the rest of the Eurozone, an austerity approach can only put Greece in a position where it needs even more money to flow in in order to maintain balance.

Even if we assume that all of Greece's debts get somehow forgiven with no further constraints and that Greece manages to run a balanced government budget, it would still be in an unsustainable position in the current Eurozone as its weaker economy would force additional money to flow in. Unless the Eurozone very significantly increases the amount of money that it redistributes across borders, Greece should get out of the Euro at the first opportunity, i.e. literally Monday morning, July 6.

Worse, with Greece out, it's only a matter of time for another weak country to find itself in the same position: that might be Portugal, Spain, Italy, or if Bulgaria, Romania or even Hungary join quickly enough that might go through that same death spiral quickly enough to see the Eurozone as a revolving door, with barely enough time to come in before being back out.

Once that first batch of weak countries is out, there'll always be more that'll be at the bottom of the scale and will find themselves in the same position. France is comfortably in the middle of the pack within Europe today, but attrition will eventually push it toward the bottom, and France having to leave the Euro is a true nightmare scenario for everyone.

In order for the Eurozone to survive, its rich members will need to send a lot more money to the poorer ones: the rich ones literally can't continue reaping benefits from a currency based on the European average without sharing those benefits with the poorer ones that bring that European average down. Otherwise, the Euro will consume country after country until it hits a country that is literally too big to fail.

Post has attachment
The line for Pliny the Younger on Friday. Turns out I make a cameo around 1:08.

Post has shared content
"Emo mode" FTW!
The merge window being over, and things being calm made me think I should try upgrading to F21..

Not the smoothest upgrade ever, but it's not too painful either. Here are my quick notes to make it come out right:

The yum upgrade instructions suggested finishing off the install by installing "system-release-workstation", but that did nothing but conflict with the firewall rules.

Doing a "groupupdate" to "Fedora Workstation" seemed to work better.

Post-upgrade woes:

 - as usual, gnome extensions don't work, since the gnome shell "versioning" is a joke. But at least the Frippery panel favorites updated fine from And Gnome3 remains quite usable with that extension in place. 

 - The "Lock Screen Icon" extension by +Sriram Ramkrishna (highly recommended - yes, I know about cmd-L, but I'm just not a magic keysequence kind of person) apparently isn't up-to-date on, but editing the version information by hand to 3.14 makes it work again.

 - the new gnome-terminal seems to default into a new "Emo mode" (aka "Dark Theme"). I don't know who thought it was a good idea to make a terminal application have its own depressed theme different from all other applications, but I'm guessing they spend their days cutting themselves and listening to death metal, and thinking they are "cool". 

But after fixing that, F21 seems to be fine. Knock wood.

Twitter is cool but wanted to bring my short rant over there to G+ so I can elaborate.

SSU has had a +Red Hat​ site license for at least 5 years. It's made life easier for both us in IT and a few other departments on campus in terms of spinning up a new server, be it physical or virtual. That said, we've never had more than perhaps two-to-three hundred instances running at any given time. Most of those are VMs and of those, most are dev/test instances.

Since day one, RH has required the licensing of their Proxy Server product. That's added about $2.5k to our annual license cost. The first few years we had all of IT's instances running through Proxy. Then the product started turning into a pile of crap, so we simply pointed instances directly to RH.

Over the past two or so years, RH has been trying to move us to their Satellite product. We had looked at Satellite but the cost didn't justify the features. It makes even less sense because we're running Puppet.

RH's new requirement of Satellite adds about another $6.5k to our annual cost. For a small campus like ours (using RH's formula our FTE is only 1279) I can't see the point paying another six grand a year for a product we'll never use.

Looks like this year we'll change to per-instance licenses, which will actually save us money but be only slightly less flexible. After the holidays I expect we will start looking at moving off of RH entirely.

As a side note, this is particularly sucky for me since I contributed to Red Hat's first release (Halloween). Like they say, all good things ....

Post has attachment
More data on the Nexus 6 debacle. So glad I stuck to my original intention to never own this device. /cc +Brian McDaniel 

Post has shared content
On LPX13D, SELinux, and root

As promised, here are some more details about the current situation.

Why it breaks

Google has really put some effort into better securing Android, and we've seen a lot of SELinux related commits to the AOSP tree over the past months. There is some disconnect between the AOSP tree and actual L preview builds, some things from AOSP are not in the L preview build, and vice versa. Ultimately, it's a pretty good bet these things will mostly align, though.

On most devices and firmwares, SuperSU's daemon is started by the service script that runs at system boot time, as user root with the init context. This is what the daemon needs to function.

Recently, they've started requiring all started services to run in their own SELinux context, instead of init. Developers and security guys following AOSP have known this was coming; AOSP builds have been logging complaints about this specific service not having its own context for a while now.

Now this script runs as root, but as the install_recovery context, which breaks SuperSU's operation, as it is a very restrictive context.

In the last AOSP build I have tried (a few weeks old), there were a fair number of other holes that we could use to launch the daemon. At first glance(!), it seems those have all been closed. An impressive feat by the guys working on this, if it proves true.

How to fix it

To fix root, all that really had to be done was ensure the daemon's startup script is run at boot as the root user with the init context.

There are multiple ways to do this, but unfortunately for now it seems that it does require a modified kernel package (changing the ramdisk).

In the modified kernel packages I've posted for the Nexus 5 and Nexus 7, the daemon's startup is fixed by commenting out the line in init.rc that forces the script to run as the install_recovery context, so now it runs as init again, and all is well.


As stated above, it seems for now that modifications to the kernel package are required to have root, we cannot attain it with only modifications to the system partition.

Combine that with a locked bootloader (and optionally dm-verity) and a device becomes nigh unrootable - exactly as intended by the security guys.

Exploit-based roots are already harder to do thanks to SELinux, and now because of the kernel requirements for persistent root, these exploits will need to be run at every boot. Exploits that make the system unstable (as many do) are thus out as well.

Of course, this is all dependent on OEMs implementing everything exactly right. If a certain OEM doesn't protect one of their services correctly, then we can leverage that to launch the daemon without kernel modifications. While I'm fairly certain this will be the case for a bunch of devices and firmwares, especially the earlier L firmwares, this is not something you should expect or base decisions on. It is now thus more important than ever to buy unlocked devices if you want root.

It might also mean that every firmware update will require re-rooting, and OTA survival mode will be broken. For many (but far from all) devices we can probably automate patching the kernel package right in the SuperSU installer ZIP. We can try to keep it relatively easy, but updating stock firmwares while maintaining root is probably not going to work as easy and fast as it did until now.

Apps need updates

Unsurprisingly, with a new major Android release, apps will need updates. None more so than apps that go beyond the Android API, as root apps do, but even some non-root apps will be affected by the security changes.

As one example, someone posted in the SuperSU thread of a kernel flashing app that didn't work. From the logcat you could see that it was looking for partitions in /dev/block from its normal non-root user and non-init context. That used to be possible, but now it is restricted: normal apps no longer have read access there. 

The solution for that app is actually quite simple: list the /dev/block contents using root instead. But simple solution or not, the app will still need to be updated.

By far most root apps should be updateable for L without too much issue. There are indeed exceptions that will need some special care, but those are rare.

Permissive vs enforcing

The kernel packages I posted for the Nexus 5 and 7 LPX13D  firmware keep SELinux mostly set to enforcing. I say mostly, because SuperSU actually switches a small part of the system to permissive, so apps calling su can do most things without much interference. The details on this are lengthy (yes, your apps will be able to modify policies as well if needed, which should be rare), and I will document these for other developers after L retail release, assuming it will all still work at that time.

Alternatively, you can set the whole system to permissive or otherwise disable SELinux. There are other kernel packages released that indeed do this. The advantage here is that it instantly fixes some apps' issues, as the SELinux based restrictions have all gone the way of the dodo. The disadvantage here is that you've just shut down a major part of the security system of the device.

Some would argue that a device with an unlocked bootloader, root, encrypted modem firmwares of which nobody really knows what they're doing, etc, is inherently insecure, and thus disabling SELinux doesn't make much difference.

I personally disagree with this. While I do agree that these things weaken security down from the ideal level, I would still not disable more security features than I absolutely need to. Just because you cannot eliminate all attack vectors, is no reason to just completely give up on defending against them.

It is of course your own choice if you want to run a permissive system or not. I will strive to keep everything working in enforcing mode though, and I hope other root app developers will do the same - as stated earlier in the post, I believe this is still possible.

(everything in this post is subject to change for retail L release, obviously)

Post has shared content
This. And from what I've seen so far from the new version of OS X and from their watch, I'm beginning to wonder if Apple's design ability is starting to falter.
Android Lollipop has me convinced that Google is getting better at design faster than Apple is getting better at services.

Post has shared content
"What has the law enforcement community up in arms is the prospect of losing access to the data on these smartphones in cases where they have a valid, court-approved search warrant."

A search warrant is a court order issued by a magistrate, judge or Supreme Court official that authorizes law enforcement officers to conduct a search of a person, location, or vehicle for evidence of a crime and to confiscate evidence if it is found.
So, from this, we can see that a warrant doesn't give the government the right to the actual information or whatever is covered by a search warrant. It merely gives them the power to look for it.

A warrant authorizes search, it doesn't guarantee evidence.
This is an important legal difference. Otherwise, the government could compel you to make all your private information and property accessible to them at all times so that, if a warrant were to issue, they could sweep in and take it.
Just 10 years ago, this notion would be considered an absurd hypothetical to propose in the service of some argument. "No one's going to require you to preregister all your private info with the government! You're nuts! Jeez, this guy's taking crazy pills!" Except this isn't a hypothetical, is it? This is nothing less than exactly what Comey is asking for, point blank.
"Comey said he could not understand why the tech companies would 'market something expressly to allow people to place themselves beyond the law.'"
The onus is not automatically on the people to prove that what they're doing is somehow "within the law." It is very much the burden of the government to show in individual cases that what the citizen did was beyond the law. This is the crux of "innocent until proven guilty." Why does a major figure in law / law enforcement need this explained?!
"This is not about mass surveillance. Law enforcement authorities are not asking for the ability to surveil everyone’s smartphone..."
No, they're not asking. They went ahead and mounted a huge, costly, and concerted effort to create that capability, and then they did it without asking. What the Washington Post Editorial Board seems to want us to believe here is that we shouldn't even bother about the fact that a necessary side effect of what they're supporting would allow this practice to go on uninterrupted.
Apple and Google understand that by not encrypting everything by default, they're giving tacit approval to this ongoing rights violation.
"After all, the government in many other situations has a right — and responsibility — to set standards for products so that laws are followed. Why not smartphones?"
You better check yourself, WashPo EdBoard. The government doesn't have the "right" to do squat. The government doesn't have any "rights" — the government has explicitly granted powers (also known as "enumerated powers"), which are strategic, limited infringements of the rights of the people aimed at establishing a "more perfect union" and all that. These are infringements we accept as part of the social contract laid out by the Declaration of Independence, the Constitution, and other founding documents.
These documents are very clear that these powers have limits. The very purpose of the Bill of Rights is to limit the possible interpretations of these powers such that they may never authorize infringement of certain rights. Innocent before proven guilty, being free of unreasonable search and seizure, etc. How is it possible that educated people writing for a major newspaper editorial board do not know this?
"But smartphone users must accept that they cannot be above the law if there is a valid search warrant."
Smartphone users are not putting themselves above the law, even in the situation where there is a valid search warrant and the government cannot discover their data. It's perfectly legal to have private data encrypted. You can encrypt data with a pad and a pencil and a little knowledge. Shall we outlaw that?
You might argue that this is fundamentally changed when encryption is automated and can be applied in bulk. Ok, have it your way — I don't want to be accused of attacking too soft a target.

PKE allows encrypted messages to be exchanged; it is automated by a technology called secure shell (SSH) and other secure protocols. Shall we outlaw these?
"A police 'back door' for all smartphones is undesirable — a back door can and will be exploited by bad guys, too. However, with all their wizardry, perhaps Apple and Google could invent a kind of secure golden key they would retain and use only when a court has approved a search warrant."
Can the EdBoard please run this stuff by their lowest level IT person before publishing it?
Exactly what might the difference be between this mythical "secure golden key" and a back door? Surprise, surprise, there is none, and these people are bad at their jobs of knowing things and then writing about those things.
"Ultimately, Congress could act and force the issue..."
Yes, I suppose Congress could abolish several of the Amendments. Why are we talking about that as if it's a good thing?
What is wrong with you people? I feel like I am taking crazy pills.


Post has shared content
Was scanning through old posts and came upon this cheat sheet I wrote a couple of years ago. As I cogitate on the future and introspect on the past, this post feels so relevant.

I had a few people ping me on my cheat sheet for team building and organization. Posting my late night top 10 list on building and running great teams. Some are even a bit contradictory (isn't that just life!). 

- Have a clear mission (statement). Always. Every team needs something to stand for

- Organize the team for (over)communication and execution. Whether it is a small startup or a large company, having some basic cadence in place is key (standups, Thurs dogfood builds, weekly launches, OKRs etc find your fit). It provides a framework to operate

- Process is sometimes thought of as a bad thing. If process helps execution, then so be it.  If process is used to substitute for weak leadership, lack of trust, or worse to micromanage, then push back. Hard.

- Be honest with your team. Trust them. "Buffering' the team from critical decisions/information is a recipe for disaster. I think some variation of this applies to your product's users too

- A wise colleague once said, "A leader always explains or fixes". This is amazingly true. Need to explain your decisions to your team or go fix them.

- Hang out with your team. Sit with them and talk about their life/ambitions. Make sure its a group of friends and not just a team of colleagues. There is a lot of stress in what we do, its best to go through it with friends

- The best thing you can do for your team is to back those who have potential. Work hard on their behalf, expose them to huge opportunities, open doors for them, they will fly and hopefully take the team with them

- Its ok to be very very detailed and micromanage while you build up trust. But you better build it up fast.. and then let go. Let your leads make decisions, let them own, and see how they rise to the challenge
- Make decisions. Be decisive. But thoughtfully. Sitting on the fence is painful (literally). Making random decisions without context is way worse

and finally...

Be Brave. Question everything. Disagree if you think something is a mistake (but then commit if that is the plan outlined by your leader). Don't be overawed by positions/titles, ask hard questions. If someone fires you because you questioned them, its not the right org anyways.
However don't just find issues. Help solve them!

Wait while more posts are being loaded