Mike Hearn
2 years agoPublic
The packet capture shown in these new NSA slides shows internal database replication traffic for the anti-hacking system I worked on for over two years. Specifically, it shows a database recording a user login as part of this system:

Recently +Brandon Downey, a colleague of mine on the Google security team, said (after the usual disclaimers about being personal opinions and not speaking for the firm which I repeat here) - "fuck these guys":

I now join him in issuing a giant Fuck You to the people who made these slides. I am not American, I am a Brit, but it's no different - GCHQ turns out to be even worse than the NSA.

We designed this system to keep criminals out. There's no ambiguity here. The warrant system with skeptical judges, paths for appeal, and rules of evidence was built from centuries of hard won experience. When it works, it represents as good a balance as we've got between the need to restrain the state and the need to keep crime in check. Bypassing that system is illegal for a good reason.

Unfortunately we live in a world where all too often, laws are for the little people. Nobody at GCHQ or the NSA will ever stand before a judge and answer for this industrial-scale subversion of the judicial process. In the absence of working law enforcement,  we therefore do what internet engineers have always done - build more secure software. The traffic shown in the slides below is now all encrypted and the work the NSA/GCHQ staff did on understanding it, ruined.

Thank you Edward Snowden. For me personally, this is the most interesting revelation all summer.

How we know the NSA had access to internal Google and Yahoo cloud data
Jeff Weiss2 years ago
Until this article no one had mentioned that the intercepted traffic was on leased fiber, not on the public internet.  That makes the cleartext transmission seem like a less glaring error, I suppose I can see how it wouldn't seem necessary.   In fact, anyone claiming in was necessary probably would have been seen as paranoid until now.

Still, encrypting data sent over the wire is not difficult.  Considering the value of the data in question, and the number of parties who could access it (at least two - the fiber owners and the government), it seems like a worthwhile investment.   Lesson learned, I suppose.
Mike Hearn2 years ago
I think the fact that Google uses private fiber has been well known for quite a while actually. Just search for [google dark fiber] and you will find many news stories discussing that, and it was mentioned off-hand in previous stories as well (I think).

Yes, that's pretty much it. Encryption was being worked on prior to Snowden but it didn't seem like a high priority because there was no evidence it would achieve anything useful, and it cost a lot of resources. Once it became clear how badly compromised the fiber paths were, there was a crash effort to encrypt everything.

Re: "not difficult". I disagree. Doing end to end on the scale of Google is a lot harder than it looks. Ignoring CPU capacity constraints, the entire thing requires a large and complex key distribution and management infrastructure (fortunately already present). Also lots of different protocols flow over our wires, each one of which has to be handled.
Jeff Weiss2 years ago
At Google's scale, everything is difficult.  I meant "not difficult" relative to all the other feats they've pulled off.

I can't say I blame them, really.  They haven't historically come across as careless with their users' data - just the opposite in fact.

If only the NSA manages to steal Google users' personal info, they're doing relatively well.  There's always room for improvement.
Mike Hearn2 years ago
Right, sure. Compared to some other initiatives encrypting cross-dc links wasn't a moonshot. Self driving cars definitely rate as harder :)+46
John A. Tamplin2 years ago
I don't know in Google's case, but when I was at an ISP before, dark fiber meant we owned the fiber in the ground and were responsible for terminating it with our repeaters/routers/etc.  So, to tap it would require either compromising the equipment we owned or someone physically digging up the fiber, inserting a tap, and putting it back.  You could conceivably detect such a tap with TDR, but especially if this happened under the cover of a cable cut you might just assume the tap was an imperfect splice to repair the cut.  So, I don't think it was unreasonable to assume that dark fiber was "safe".

When I came to Square, it seemed over the top that even connections between services in the same datacenter were secured with mutual auth SSL -- it doesn't seem so excessive now.
Jeffrey Yunes2 years ago
I don't think this is a "big people vs. little people" thing. I'm little, Google's big, and we're both on the same side of this. Rather, this is a "government vs everyone else" thing.+49
Apples Flibbr2 years ago
Shocking, disgusting stuff. As you said, laws only matter to common folk, not the state. Good work Mike. +15
Laurent Gaffié2 years ago
Well said Mike, well said.+1
Jason Braddy2 years ago
+John A. Tamplin It's possible to tap an optical cable without breaking it, by bending it far enough that some of the light leaks out through a gap in the jacket. This can be detected by looking for unexpected drops in the light level via DOM, etc., and I believe that critical defense/intelligence network paths do this already.+14
Jeff Weiss2 years ago
+Trevor Loucks that would dramatically change Google's business model, since they generally offer free services in exchange for collecting people's data.

Most of the offerings where privacy is a concern, the best private service wouldn't be a service at all.  It would just be software you run yourself.  Or maybe it would run on some cloud service, but the important point is that the provider doesn't see your data or care about it, they just lease you some computing capacity.
John A. Tamplin2 years ago
+Trevor Loucks In many cases that would mean the service couldn't operate - for example, GMail couldn't communicate with other SMTP servers if Google never had the decrypted message on its servers.  If you have a doc for collaborative editing, how do multiple people get the same key?  You certainly wouldn't have the ease of saying "share this document with the following" and it just work.

Besides that, decrypting in the web browser using JS is slow and prone to other vulnerabilities.  It currently isn't practical for web-based services to operate as you suggest.
Ian Batterbee2 years ago
+John A. Tamplin  Also worked at an ISP, for us, dark fibre was simply something where bits you shoved in one end came out the other untouched. The dark part referred more to the fact that the packet wouldn't congest with other people's traffic on the way, and would typically be a leased service on someone else's CWDM or DWDM network, though it could also be simple patching with no equipment in the way.+1
Stephan Beal2 years ago
Okay, now admit it: how many of you had to google "Sisyphus" at the end of the article? ;) (i did!)+1
These assholes think they're above the law. Fuck them!

Let's use bitcoin to defund them.
Amber Yust2 years ago
+John Mardlin Yes, the project was started well before Snowden. The pace just got ramped up immensely in the past year or so.+6
Jeff Weaver2 years ago
quite a post for bonfire day
Guy would be proud :-)
Mike Hearn2 years ago
Nobody knows how to build services like the ones Google provides but where the service provider is blind to the traffic. It's not as simple as "Google makes money off ads so they don't want to change things".

Ad-funded services are actually a good thing from a privacy perspective! A lot of people and media commentators don't realise this. The alternative to advertising is direct payment by end users for a service. Unfortunately, there are no ways to pay someone for something online anonymously. The closest is Bitcoin but it's still very young and immature, most people can't use it. So if the dominant paradigm for Google services was that you paid for them, you'd pay for them with a credit card and then anonymous accounts would be impossible, we'd always know exactly who you were. Even if we couldn't read the message traffic, we would still know immediately and automatically that Edward Snowden of BAH Hawaii had sent a message to Glenn Greenwald of the Guardian, which is obviously of great interest to the NSA. You don't need to be a genius to guess what a message from a high-clearance member of NSA staff to a journalist might say. All you need to know is that they're communicating at all.

Currently though, anyone can sign up for a webmail account and provide any bogus name they like (and people do). Neither the webmail provider nor the advertisers know who a user really is, nor do they care. That's a pretty decent situation to be in.

Anyway, although I'd love it to be different, there aren't any viable alternatives to the way cloud services operate today. These things either get fixed legislatively, or they don't. Cryptographers are inventing lots of really amazing technology that might one day make a truly private p2p cloud possible, but those techniques are probably decades away from being competitive (if they ever are).
Can we please stop downplaying the risk of privately owned fibers being tapped? This risk was very well known more than TEN years ago. Banks defended against it more than TEN years ago by deploying bump-in-the-fiber encryption devices on their dark fibers.

In case you want to read up on this, try this 2003 Blackhat presentation:

When you have a dataset as valuable as Google's you KNOW this will be a big target.

Additionally, fiber encryption is handled below or at layer 2 and so there aren't "lots of different protocols" to be handled. CPU constraints are not relevant because this is not something you would run on the server CPUs. You buy or make line-rate hardware encryption devices.

The only valid statement is "the entire thing requires a large and complex key distribution and management infrastructure". But that's par for the course if you promise in your privacy policy that "[w]e restrict access to personal information to Google employees, contractors and agents who need to know that information in order to process it for us, and who are subject to strict contractual confidentiality obligations and may be disciplined or terminated if they fail to meet these obligations."

This was without a doubt gross negligence on Google's part, plain and simple. Google left the back door open despite a decade of very, very public warning signs.
+Mike Hearn "These things either get fixed legislatively, or they don't."

Then they won't. The TLAs will ignore the laws and lie about it until caught. Then do it again.
Lauren Weinstein2 years ago
I've had quite  a few people ask why I haven't seemed to be more upset publicly about all the recent NSA and other surveillance disclosures. It's real simple -- there has been nothing disclosed that I (and many others) haven't assumed was going on at the hand of every capable surveillance org around the world for decades, one way or another. This sort of behavior goes back to the dawn of written communications -- tech has just made it easier in some ways (by concentrating flows and making storage so cheap) and harder in other ways (by providing means for robust encryption, when it's used).

And recent history (that is, some decades) speaks clearly to the following. No matter what the politicians tell you, not matter what any countries' agencies say, there will be no substantive long-term change in these practices. Opportunistic gathering of all data they can get their hands on will continue, especially if it can be declared to be foreign or international in nature and so not subject to purely domestic restrictions (when those exist).

At the best, we can hope for a bit more oversight (at least in the short term) and push hard for more transparency so that companies like Google can explain to the public what really happens in terms of data demands from government, and not be faced with when did you stop beating your wife? no-win situations where hyperbolic, false accusations can't even be legally refuted.

Above all from a practical standpoint, it has to be encrypt, encrypt, encrypt -- as Google is now engaged in big time. Contrary to the assertions of the spooks, encryption to block or degrade opportunistic mass data collection is unlikely to significantly damage major targeted anti-terrorism efforts.  When the agencies have something they really want to target, they can use warrants and even user endpoint attacks to deal with most kinds of common encryption. 

But what's so important about encryption at Google scale is that by making it significantly harder to do the mass, vacuum cleaner type surveillance, the opportunity for governments building up enormous databases of such material composed almost entirely of innocent parties' data can likely be curtailed in meaningful ways.  And it's that sort of data collection that is most potentially subject to abuse, especially retrospective abuse under some future government quite possibly with a very different set of motives entirely.
Jeff Weiss2 years ago
+Mike Hearn in many cases it is simply not necessary to ship your personal info to someone else to organize.  If you have a good internet connection at home, you can just put a modest box alongside your cablemodem that is capable of storing all your data and organizing it (with the right software, of course).  And you would control your own privacy. 

The big missing piece is the software.  Unfortunately all the effort has been poured into services like Google's because quite frankly, people don't know any better.  Nobody knows the value of their data so they give it away.  I would much rather contribute to a kickstarter for open source software to do the same thing.  That way I know the application is working for me, not someone else.
Ed Burnette2 years ago
+ 1000000 from the US+3
Michael Schwartz2 years ago
Way to go Google for stopping all the hackers (even the ones my tax dollars are funding to spy on me!).  Thanks to the crypto geniuses of the 90's, encryption is no longer a monopoly of the NSA. We have the power to have privacy, and if we work on open standards and open source software, we will also have the economies to make it ubiquitously supported by all domains on the Internet. Thank you for your transparency on the issue! +4
Watson Ladd2 years ago
"Fuck You" is nice. Service and a court date is a lot nicer.+11
I know that Googlers work for the security to help people in all countries for the right to privacy and spying governments, not only when it concerns free speach. I'm confident that you at Google are working on even things like these. That's why i like what google, and also employees like you Mike, are doing for everyone in the world. Thanks. +6
Chuck McManis2 years ago
Knowing what Brandon and the opsec team is capable of, I am sure that given your new found understanding of the risks involved you will be up to the task mitigating this latest attack against the infrastructure. Kind of defines APT in a whole new way.+2
Marco Tedaldi2 years ago
The spy services are clearly criminal organizations. And the law is quite clear on how to handle criminal organizations and the members. +4
Jay Perkins2 years ago
Kudos to Mike for speaking out as Brit.

Americans suddenly have a fear that speaking out against the actions of its government will have real world  consequences. Like investigations of your internet use and your finances.
I  heard a rumor as if in USSR some copper cables were going in pipes with dry air, or an inert gas, under certain pressure that was constantly monitored.+1
Jerome Chan2 years ago
What is there to prevent the NSA from demanding the private keys for the encrypting software?+3
Kazimierz Kurz2 years ago
encryption, hower complicated, is the only sollution. And even if encryption would be not very strong, it has to be done! Because even if it is easy to breake, it rise the cost for the atacker. It is a trade game.
If You do something, even simple, - it may work for some time...
Justin Lolofie2 years ago
brit outrage is classy outrage +++2
Brandon Downey2 years ago
While I can't comment about our employer's use of encryption, but if your concern is the compromise of current private keys revealing things about past encrypted sessions, technologies like this can help:
Danny Sullivan2 years ago
As an American, I'm finding the spying revelations disgusting. We do have laws against such warrantless tapping, and it shouldn't be allowed.

But Google's "big people" that can go after the big people you feel will escape these illegalities. Right now, aside from whatever private lobbying is almost certainly going on, all Google is doing publicly is arguing for a right to reveal number of requests you give out.

It would be nice if Google stepped up the game. File a lawsuit against the US government. Go after the big people.

When Google felt China violated its security, Google pulled out of the entire country. It seems to have a much higher tolerance for US spying -- or bottom line, while it doesn't like it, it'll hurt the bottom line too much to do a similar withdrawal.

But a big fat huge legal action would be the F-You you really want, and that many would feel only a company of Google's size could pull off. It'll be interesting to see if the higher-ups act on your call.
Larry Gritz2 years ago
I really hope this is making Google (and other companies) think hard about their "free services in exchange for mining data and selling ads" strategy. It's that business model that made all this surveillance easy. I'd like to see Google offer to make us the customers rather than the advertisers, allow us to pay for legitimately private and secure services, and as a bonus finally stop being bombarded with ads.+3
Simon Zerafa2 years ago
I wonder of having the traffic on leased fibre was really the smart move in the end? All of the traffic on that fiber was Google's. If it had been on a public link it would have been encrypted and far safer.+2
Kevin Lyda2 years ago
+Larry Gritz Did you not read +Mike Hearn comment above? There is no widespread, easy to use anonymous payment system. If you pay for your Google services, you're traceable. If advertisers pay for your account, you are not. This is not rocket science.+4
Colin Tulloch2 years ago
It's nice to hear from people inside Google. Interesting tidbit regarding "internal database replication" traffic.

Sure it would be great if the the group of people we gave a monopoly on courts didn't utilize same said monopoly to abuse the courts... ;D
What would be better of course is not granting any one group a monopoly on courts ;D.

Keep up the good work!
This is the sign of transition (and recognition) from APT to NPT: National Persistent Threat :)+1
Karl Koscher2 years ago
I think it's time to do some, er, fuzz testing on the protobuf library.+1
Matt Lutze2 years ago
+Jeff Weiss "If you have a good internet connection at home, you can just put a modest box alongside your cablemodem that is capable of storing all your data and organizing it (with the right software, of course).  And you would control your own privacy. "

There's lots of really great ideas that will never have traction with the people most at risk of being compromised, because the general public doesn't understand how any of this works. Most never, ever will.

Ask your neighbor (exception of course if you're surrounded by fellow nerds) how WiFi works. They'll tell you you plug a modem/router into your cable line and connect your computer to the signal. They'll have no idea about the actual negotiation of  channel negotiation or collision avoidance, or even why their password used to be one length and changed when their cable company upgraded them.

That's what's missing. Not the next best bit of software. Internet tech is magic at this point.

Also, question, how do you get to all of your private info from a friend's house, or from work, or from your mobile?

"I would much rather contribute to a kickstarter for open source software to do the same thing.  That way I know the application is working for me, not someone else."

Bootstrapping an OS project is cool, but how does it continue to run and provide a service? The money has to come from somewhere, and either they're following an add supported model, corporate ownership/stewardship or selling directly (with all of the downsides to that last one +Mike Hearn mentions above).

The OS project is working for you until your $50 runs out, then they're working for whoever gives them the next check.
Kevin Lyda2 years ago
+Luke Ryan For what it's worth I'm a former Googler and others I've worked with there have expressed similar outrage and annoyance privately to me. I do still hold stock but Google doesn't pay me to say good things about them. But in my experience the folks I worked with there were some of the most ethical and user-protective people I've ever had the pleasure of working with.

IIRC +Mike Hearn is in SecOps and while I can't recall if I ever worked on anything directly with him, my general experience with that group is that they thought about security in great detail. They focus on internal and external threats and would push for policies that were painful to other teams in order to achieve greater security. I still miss my hands-free bluetooth Google Authenticator for instance (meanies).

And they got pushback on those. So while people may sneer at them for not getting leased fibre links encrypted, do realise that encrypting links has latency costs which are very real in a global network. Latency has actual, measurable monetary costs in Google - and any team that can provide such measurements has likely fought against WAN encryption. Meanwhile the SecOps guys cannot provide a monetary metric for not encrypting. Even now they can't do so - and this is on the slope of worst outcomes for not encrypting.

Huge amounts of info from Google's networks are in the hands of the NSA - and Snowden has already proven that the NSA has shit access controls on this data. How much Google source code (which also goes over these links) has made it into the hands of their competitors? How much sales data? How much business data? How much personal data has reached their competitors? And how much has this cost Google and how much will it?

People who work at Google should be mad - and from what I've seen they are. This has not just hurt their reputation to users, this has been an unknown cost to them internally. How much of the work they've done internally has been stolen and used by others?

It's my hope that people in SecOps use this event wisely to build more security into Google's products and to Google's internal infrastructure. I hope they will engage actively with efforts to update open protocols (or create new ones) to defend against invasions of privacy as Bruce Schneier has called for.
Kevin Lyda2 years ago
Perhaps it seems contrived to you. It does not to me.

And I'm pretty certain Googlers can win an electronic arms race with the US gov't and every other one if they put their minds to it.

And it would appear that they are now that they realise how much of a threat those entities really are.
The people who do business as "government", whatever "government" they claim to work for, are criminals.

Some of them don't know they are (the authoritarians who obey orders, just doin' m'job types, these are the kinds of people who executed Jews on command in Nazi  Germany), and some of them are fully aware of the criminality of their behavior (the psychopaths a.k.a. social dominants according to Bob Altemeyer's taxonomy, who give the orders and come up with the lies to make the orders appear virtuous).

We know they are deliberately malevolent because they routinely do things that would be called "crimes" (implying deliberate malevolence) if any one of us did them.  We know they are acting either under the delusion that when they perpetrate evil it's virtuous, or under the full knowledge that when they conspire to perpetrate evil they are guilty but they just don't give a shit.  "I'm really good at killing people", said the president lately... this does not strike me as an innocent moral confusion, but rather as psychopathy.

We should therefore not be  surprised when they perpetrate abominable acts such as violating private property to gain access to information that would have never been volunteered by their victims.  In fact, this is one of the most minor transgressions they could perpetrate.  We're talking about people who have conspired to mass murder brown people abroad, and cage millions of non-violent black and white people at home.  They will cage or murder you if you resist or disobey them.  And sometimes they will murder you for sport, as Al-Awlaki's nephew found out, and the thirteen year old boy discovered, both seconds before their deaths.

The question that you must ask yourself is: at which point do you stop believing in the virtue and authority of obviously malevolent and delusional people?  Does it even matter whether what they do is "illegal" or "legal", when they get to define that, through writing their magical pieces of paper, and through their own trampling of their own promises in said papers?

Ask yourself that question today, because tomorrow it might be too late -- you might be dead.  You need to remember that these people are willing to murder you if you disobey or resist them.  That is what they do, they do it righteously, they will do whatever the fuck they want to  increase their power over you, you must never forget that.  Start asking yourself how that could possibly be justified, so you can discover it cannot.
+Kevin Lyda When you mention encryption latency as an obstacle I suspect you are thinking of server-based crypto.

I memory serves me right, a line-rate layer 2 fiber encryptor for 10G or OC-192 would add between 10-20 microseconds to a 20-100 millisecond inter-DC link. This is negligible.

Of concern is that Google cannot rely on the big vendors to procure these things since the vendors are either owned or in bed with the spooks of their own volition.

Fortunately Google has enough engineering prowess to build its own though. The IP blocks can be gotten COTS. And it seems that Google engineers are now sufficiently angry that they may just get it done.

It is good to see that those in the trenches do not approve of this kind of thing.

As for the cost... there is violating one's own privacy policy despite decade-old, well-documented, comparatively low-cost threats. The data on the fibers simply was NOT protected. Not sure if that qualifies for a class-action suit and what kind of financial cost that would entail. The reputational cost is significant.
Mike Hearn2 years ago
FWIW, I do not do security related work at Google  any more, I moved on to other things.

Encryption is subtle. Link-level hardware crypto was considered and discarded for various reasons. Not sure how much detail I should get into here, but suffice it to say, server-to-server encryption done in software running in our datacenters is more complex but is believed to give better (more robust) outcomes.
William Payne2 years ago
+Andree Christaldi Isn't this what the Magna Carta was supposed to stop? King John going back on his word, perhaps?
Waqas Shah2 years ago
I see where this is going. First, Schmidt curses the NSA, now we see Google employees cursing NSA. Afterall, Google is the biggest source of information in the world and obviously for NSA, and for it to pretend as the White Knight in front of everyone is one way to get the trust back of the people which affected Google's reputation along with NSA's. Nice strategy though.
Mark Enriquez2 years ago
never forget...

"all your base is belong to us"

truer words have hardly ever been spoken.
Nathan Cho2 years ago
+Waqas Shah Really?
On it's face, this post is very cool.  However. the encryption between the datacenters is meaningless.  They're not just listening from outside the buildings.  They have the capability to bug the servers as well as the fibre.  It's just more of a heavy lift without the fibre taps.  This is simply a layered reveal by Snowden.

Recall that Eric and Greenwald have both hinted that there may be more revelations to come.  Guess what? There are.

The problem for Snowden & Greenwald is that the bigger revelations will have a huge impact on the internet. There are select entities who think they're coms are safe due to encryption.  A change in this thinking would actually do serious damage.
Devin Jeanpierre2 years ago
+Waqas Shah an entirely reasonable theory, but I think it'd be hard to keep tens of thousands of engineers repeating the party line, whether they be anonymous or no.
Scott Nutting2 years ago
High level officials on both sides of the pond should go to jail over this.+7
Owen Iverson2 years ago
i wish there was something we could do.  but it seems that any actions will get us labels as terrorists or traitors to our country.

i wouldn't be surprised if violence ends up being the last resort as it always is.
Randy Terbush2 years ago
+Jonathan Langdale Just 15 years ago, the US government regulated who could sell cryptography software in the US. It should come as no surprise that along with that regulation, you were only allowed to use one vendor's crypto stack. RSA

It would be naive to think that an encrypted network would solve the problems that we are facing today.
Spencer Chastain2 years ago
I would be willing to pay a service to store my data encrypted and not generate/host my private key.
Owen Iverson2 years ago
+Mark Enriquez "all your base ARE belong to us". Sheesh!+3
+Randy Terbush Ironically, it's hampering efforts to solve global problems.  We're not capable of compromising or thinking long-term.  Therefore, the only way you're going to achieve change on a global scale is through catalysts whereby change as a result of a catalyst cannot be easily predicted by the environment, like the butterfly effect.  We need to fool ourselves, in affect.

Dealing with pollution in China would be a prime example.

It's my position that having enough geopolitical global information and pattern recognition can allow us to calculate (modeling) realistic catalysts for global change that cannot be consciously achieved by so called "individuals."

Of course, for you to accept my argument you also have to admit to yourselves that there is no free will.  This is a counter-intuitive argument that would suggest accelerating what the NSA is doing even though I might have a natural resistance to it out of fear for abuse.

And in a way, while the NSA might be motivated by other intentions they may not even realize that they're their own catalyst for changes that would see a redefining of what it means to be secret.
John A. Tamplin2 years ago
+Gabor Gabesz The sad thing is it is a gray area whether they broke the law.  The Patriot Act (and others) gave sweeping powers of data collection all in the name of national security.  Obama campaigned on removing the Patriot Act, and then supported it his first week in office - presumably he saw something that made him believe such suspension of privacy is required, and if everybody in the government believes it they think they are doing the right thing.

The problem of course is if you have an event that occurs with very low probability (terrorist attacks), to detect it you have to be ready to act at the slightest hint, and you will get far more false positives than false negatives.  This will encourage those who aren't statistically minded that the program is necessary (look at how many we caught [even if many of those are false positives]) and effective (there hasn't been another 9/11 so we have prevented attacks).  Thus, they will justify expansions of the program in a death spiral to privacy.
Tim Scanlon2 years ago
+Jason Braddy
 Using a splitter is a heck of a lot easier than all that. All you have to do is buy and install it. They're nearly undetectable depending on where they're put in, it's difficult to find one even with an OTDR.
Keith Milner2 years ago
If I were Google, I would be tempted to use any spare capacity on fibres to pollute the NSA data set with false information.
I would pump data over those lines which created  a whole new parallel world of users, accounts, and address books, all artificial of course.

I would then engineer data which indicated mass activism in specific locations, or terrorist activities which didn't exist. and see how many resources could be wasted on the Government chasing up false leads. If these could be exposed publicly, so much the better.

And, of course, Google wouldn't be doing anything wrong by creating this "test data".
Matt Walker2 years ago
What if Google and the NSA are one and the same! What if the NSA is an arm of Google or vise versa! Maybe you didn't get the memo!+1
Joanne Leon2 years ago
Time for Google to use its power to start a grassroots campaign.  I forget where I read it, maybe The Guardian, but there was a suggestion that Google and other mega tech create an NRA-like grassroots movement.  Time to side with the people.  Too bad you didn't use your power to do that in the first place and too bad it didn't become a huge deal for you when the NSA was just obliterating privacy rights of the little people. Now that they've targeted you, you rise up.  But that's water under the bridge.  Time to stand with the people and use the tremendous power that you have.  These guys use terrorism as an excuse to exploit every damned thing they can.  What are we going to do now?  Google could make a difference. Will they?+3
Fallon Best2 years ago
Mike stick to IT, You point focus on security but lack ideological integrity and depht. and your profanity lased sentences reflects a level of maturity and integrity you don't have. I belief in FOI of most govt activities, except for spying/espionage work. The security agency of NSA & GCHQ have a duty and a right to store data, and mine credible (only credible) threat profiles to the law and order of society. If you don't understand, respect that there are others who do, those who realize that world is in a state of economic, and ideological wars and government have a need to remove individual rights of privacy from persons who are threatening other individuals rights to safety and security. +1
William Dowell2 years ago
The statement level 3 put out the other day all but confirmed that they help the UK - probably at Poole and elsewhere. +1
Keith Milner2 years ago
So, +Fallon Best  you are saying that all of the principles on which our democracies and models of government were built on, are wrong?

More than that, you are saying they do not, and should not apply?

You are saying that we have been lied to for years about the very nature of our governments, and that is alright?

You are saying that we are kept in the dark about what our elected officials spend our tax money on as long as they keeping us nominally safe?

You are saying we should just roll over and accept it? That we should trust these unknown people to do the best for us, to only use this power for good?

Is that really what you are saying?

Well, Fuck that!
The United States was founded on the idea that all men are created equal and have equal rights, so long as they are white men. Are those the correct principles?

People have always been lied to, have they not? This seems especially true before there was the internet, or radio. We lie to ourselves too.

Maybe these foundations are not what we think they are? There are no more Kings searching for revolutionary weapons.
Hang Cheng2 years ago
+Fallon Best your statement goes against a fundamental principle of a democracy: the government is held accountable to the governed.  How can we hold it accountable when we cannot see what it's doing?  You ask Mike to "respect that there are others who do".  It sounds like statement from an authoritarian government that asks its populace to hand over their rights and give the government freedom to do what it pleases.  "The security agency of NSA & GCHQ have a duty and a right to store data, and mine credible (only credible) threat profiles to the law and order of society." is patently false when applied to American law.  Lastly, "Mike stick to IT, You point focus on security but lack ideological integrity and depht. and your profanity lased sentences reflects a level of maturity and integrity you don't have." is very ironic.  Ad hominem attacks are not reflections of maturity.+6
Brian Russell2 years ago
+Jerome Chan In the UK, nothing. Withholding keys if demanded to by law enforcement agencies is punishable by incarceration.
JF Gunter2 years ago
The biz model of Google, 2-faced book and the rest is based upon spying. Is private spying OK then?
JF Gunter2 years ago
Correction of spellchecker: I call your company Ooogle ....
Loïc Hélias2 years ago
Thanks Dude !
From France.
Brian Dirk2 years ago
+Jeff Weiss It seems fairly clear now that claims that it should have been encrypted  being "paranoid"  are in retrospect  reasonable.  Though I'm not trying to evoke the cliche  "Just because you are paranoid, doesn't mean they aren't out to get you." +1
Don Johnson2 years ago
Hey, I just wanted to thank you guys for attempting to keep our data safe.

Oh - and I agree - Fuck these guys, if we have to, we'll build a new Internet, one with decentralization and security models built into the core protocols.
赵亚雄2 years ago
+Mike Hearn Anyone slightly exposed to any serious production security issue will know that any security issue is non-trivial, and none of them is simple. If it is simple or even not-difficult, it will be done...
Al Clark2 years ago
+Kevin Lyda I'm sorry to say it, but +Mike Hearn may be downplaying the importance of Google's business model in its inability to move to a user-pays system.  I thank +Mike Hearn  for commenting here, because I think his comments are a great service to the community, I do, however disagree about the viability of user-pays.

It's possible to use a 3rd-party payment processor to handle transactions so that Google never holds credit card data associated with a particular person.  All Google would know is that you logged in, sent some CC details to a processor, and the processor confirmed the transaction.  Then Google increments the amount of time you're entitled to their service.  If there are logs, they can be disabled.  The incrementing can take place a non-determinate amount of time after payment. Etc., etc.

In the US at least, you can go into a convenience store, buy an AMEX gift card in cash, and use it online.  Boom.  Problem solved.  And bitcoin isn't as hard as it is made out to be.

It's not clear why this hasn't been implemented.  Maybe in the company's estimation the above just isn't simple enough for the average person.  That's fine!  They're a business entitled to exercise their business judgment, but it's not entirely accurate to say that it's payment systems make the model unworkable.

Again, I say this within the context of thinking that Mike's comment, and the discussion it generated, are very good things.
Paul Dickson2 years ago
Chrome needs the encryption functions built in its JS engine so JavaScript code can just call it, with less overhead than doing the bit-twiddling in JS.

I use Google Drive for backups, but everything is AES256 encrypted at my end first. +Duplicati
Anasha Cummings2 years ago
For everyone else prompted by recent revelations to want  their own end-to-end security, check out this startup trying to get off the ground right now with a game-changing technology based on tokenization rather than encryption:
David Enzel2 years ago
Thank you for having the courage to express your point of view.+1
Jason Weill2 years ago
Congrats on getting Reuters to print the word "fuck" for the first time I can remember.+6
David Gilbertson2 years ago
Google vs. the NSA. Awesome.+1
Angelo Cuissi2 years ago
Any chance we get some PGP on Gmail interface?
Diego Amorocho2 years ago
Keep up the resistance to all criminals, no matter what colors they fly.
Lauren Weinstein2 years ago
+Tony s I am not an apologist for anything. I merely have expressed my opinion of the reality of the situation, in contemporary and historical contexts, based on decades of observing the technical and political aspects of global surveillance operations by nations around the planet. And I have suggested that technical actions, rather than political approaches, have the highest probability of most effectively mitigating mass surveillance abuses. No, I am not an apologist. But it's fairly clear what you are. However, decorum prevents me from using the appropriate terms for that here.  Do have a nice day.+1
Nik H2 years ago
Thank you. I am happy to hear that Google is waking up to that shit, and doing something about it. I am also happy to hear that employees at Google are people like you and me, outraged about the large scale - and completely illegal - spying efforts, and not willfully cooperating or looking away. Anyway thanks for doing the right thing - I believe in people more than corporate talk so this goes a long way restoring my faith in Google keeping my stuff safe.
Babak Rezai2 years ago
+Michael Schwartz Thats what pisses me off the most, besides the fact that is clearly against the law. Taking our money and using it against the very people who are funding them. 
Babak Rezai2 years ago
I would rather like to see Google spending is Lobbying money and lawsuits with other company's and groups such as the EFF to put the breaks on this kind of stuff.
Lauren Weinstein2 years ago
+Babak Rezai That's the real irony. It's doubtful that it will ever ultimately be found to be against the law -- even if it ever worked its way through the court system (very doubtful that will ever happen general form) -- especially since this particular activity apparently took place outside the U.S. The national security directives that we know about pretty well exempt all of this kind of activity, and have for, well, at least since around 1940 or so.
Babak Rezai2 years ago
I know that, but it still doesn't make it right. Governments should work for their people, this entire thing stinks. Its also only through immense political pressure that will have things changed. As it stands there probably isn't enough. Partly I think that a lot of us are not surprised by these revelations, as stomach turning as they may be. Guess the big lesson here is encrypt everything.
Lauren Weinstein2 years ago
Of course it's wrong. But a look at national security history shows the pendulum has swung back and forth a bunch of times since the days before NSA ... was NSA. Politicians can never stay focused on this, because by the time the next election rolls around most people are far more concerned ... as usual ... with economic matters. Understandable, in a slowly recovering economy. And the very nature of the spook biz makes it always possible for them to trot out examples that we can't see, but are still convincing for getting the next black line items passed in the budget. And of course, if there is another major terrorist attack, well, you can probably kiss remaining civil liberties goodbye in short order. PATRIOT and HSA, pretty clearly authorized this stuff (we warned at the time! We were called unpatriotic for questioning!) regardless what dissembling politicians say now. As far as unconstitutional is concerned. Well ... you can think these activities are unconstitutional, and I can think they're unconstitutional, but we don't get votes on that. Unconstitutional is what the Supreme Court says it is. Period. And anyone who thinks this court is going to take any significant steps away from deference to the executive branch on natsec matters doesn't know the court very well. That's just the way it is. I don't like it. You don't like it. Our not liking it plus a $1 will buy a bad cup of coffee. What we can do is take technical measures. That's where we can have real influence -- given that we don't have billions of dollars to buy politicians as an alternative.
the real problem about separation of power is lack of control!
who controls the NSA? the same laws that govern the camorra. dog eat dog. 

who controls a corporation? stock owners? CEO? who exactly? and who is really responsible? at banks?
being a jerk and being a failure is not a crime.

and being a vigilante is great for being a president. those who can hardly spell PATRIOTISM will always back you up.

all ppl need is the illusion of an enemy and they'll give up any freedom. or as BF would have said: Y'ALL DON'T DESERVE NEITHER: FREEDOM NOR SECURITY.
Obama is more of a gullible puppet  than Tony B** war. Yea that's right, I used the B-name.

When will these ppl understand that even should the NSA win, we all lose 'cuz there will be no freedom left to separate us from Cuba/North Korea/Iran and all those other countries which are relatively NSA-free ^^
Mark Metson2 years ago
Fibre level hardware encryption doesn't really provide end to end encryption at the application level, it would just be end of a fibre to another end of that fibre, presumably? So you'd get all kinds of spots where fibres begin and end where something and thus someone has keys for that next span of fibre. Proper end to end you need your browser to encrypt data to your chosen set of recipients, with none of the fibres being able to decrypt any of it...
Mark Metson2 years ago
Presumably something like this social app (google+) would have to involve each user who wants to read something you posted contacting your machine at your home or where-ever you keep your data and convince it that it is a user whom you don't mind granting a read-key to that data... Of course one bad apple in the lot could then publish the data elsewhere...
Jeff Ferron2 years ago
Thanks Mike!
Poor Google Inc., being spied by the NSA
mathew2 years ago
Good luck. You know what would be awesome? S/MIME support in Gmail and end-to-end encryption in Hangouts.+1
Well said and thanks for the honest sharing.
Lauren Weinstein2 years ago
+Tony s Under normal conditions I wouldn't keep talking to someone who has clearly pushed into the troll zone, but this is important stuff so I'll try one more time. If anyone is raging it's you -- I suggest anyone who wants to understand more about the nuances of all this not depend on Twitter and G+ comments and dig through the long-form items on my blog ( Or not.

You seem to have a problem separating analysis from wishful thinking, and from recommendations. You will not find anywhere that I have suggested not complaining, not suing, not taking any and all legal actions that seem appropriate -- not just in the U.S. context, but against foreign intelligence agencies as well, who are equally complicit in these abuses.

But the history -- which it appears you have not studied -- of such legal/political efforts suggests that they will be of very limited effectiveness in the long run.  This is not the first intelligence blowup of this sort -- we've been through them before with telephony and even telegraph before that -- and the spooks/politicians behaved in exactly the same dissembling manner.

As I noted, when PATRIOT and HSA were being passed, I and others argued that they were enabling exactly these kinds of abuses. We were ignored, told to shut up for being unpatriotic -- by the same politicians who for political motives now have taken a 180. And they'll take another 180 when this quiets down. (It is instructive to read the entirety of PATRIOT and HSA by the way -- everyone needs a hobby and it's very illuminating).

The main legal areas where efforts have the best chance of being useful are in terms of oversight and transparency -- these are critical and sectors where there is some decent chance of positively moving the ball, though resistance is still very high, even to letting companies make substantive reports about natsec demands made of them.  I continue (as I have all along) urged strong actions to try get real reforms in these areas.

But don't confuse that sector with actually getting surveillance to significantly stop. Like I said earlier, no matter what you're told, history is clear that the surveillance orgs always drift back to their old ways. And since so much of it is black budget, from the outside it's impossible to effectively police.  I hate that. It's awful. It's also the reality.

Now, that doesn't mean it's wrong to try change it. I support that. I just don't expect substantive real world cutbacks in actual surveillance.

So ... it becomes super important not only to push for the legal changes where they can do the most good, but most of all to use technical means like encryption to try mitigate mass surveillance abuses -- by making them as painful and expensive as possible.

Google (and other firms) are working both the legal and technical angles on this, as they should.  My personal belief is that in the long run the technical side will have the most benefits, given the historical precedents of the agencies involved around the world.

I'll add that while I am a consultant to Google, I am speaking for myself, of course.
It's like with sex. Everyone's doing it but when someone else finds out it is embarassing...... ;)+2
Phil Karn2 years ago
+Mike Hearn I know how difficult it can be to integrate encryption into applications and operating systems, but link encryption is pretty simple: you put a box at each end and load them with the same authentication key. Presto, everything on the link is encrypted, and the computers at the ends don't see anything different.
Phil Karn2 years ago
+Tony s I don't understand why you have a problem with what Lauren is saying. I've known him for a long time. We disagree on a few things, but by and large you, he and I all come from the same place and agree on the same values, and that can't be said for the many in government who seem to think the mere concept of personal privacy is a national security threat -- or a threat to their own power, which they often conflate with the national security.

I don't see Lauren opposing legal and political challenges to the NSA. He simply doesn't think they will do much good, and that technological defenses are our best bet. As an engineer, I naturally agree. But I am also painfully aware of the limitations of purely technological defenses and I think we should focus our political and legal efforts in those areas.

For example, I think we can do a pretty good job protecting the confidentiality of a communication from third parties with encryption, assuming we can improve endpoint security -- the difficulty of which is not to be underestimated. But I simply don't see a technological solution to the problem of location privacy. A mobile network needs to know where you are so it can deliver a call or text message; that's just how it works. And while it doesn't need to know where you were yesterday, there isn't much we can do to keep that information from being recorded. Sure, you can turn your phone off and remove the battery but that kinda defeats the point of having one. So I think this is where we need to focus our legal and political efforts.
T Mcgrane2 years ago
i have computers that are older than most of you; so here is my advice.
fuck obama and the NSA
throw that shit right back at them
hack away
those obama girls on the net are fair game
michelle and company too
don't tell me you don't know where their fucking node is
Paul Racko2 years ago
Frankly, I've never completely trusted Google. In the back of my gut, I've always sensed that Google was a well-disguised spy op and why I've never shared my personal info via their channels. Decentralization is the way to go, and I think KEYHOTEE will play a major role in securing our transactions in the near future.
Stephen Gardner2 years ago
Glenn Greenwald used to have a google administrated blog at which has been removed.  Does anyone here know whether Glenn asked that it be removed or whether Google removed it for other reasons.  I suppose we can all guess what "other reasons" might mean at this point.
Andreas Triller2 years ago
But what if the NSA or some other agency waits until the dust in the media has settled and then issues a court order to Google and other companies to hand over the used private keys, so everything is in the clear again? I really don't think this "problem" has a technical solution. Politics must regain control over the "state inside the state" that the secret intelligence services have become. They must define and enforce what is illegal. Anyway, in the mean time, +1 for the F*** you.
John Smith2 years ago
When Google says "Fuck You Obama", then I'll start trusting Google again. No technology company should be in the business of electing politicians and favoring one party over another, using data collected from users who did not sign off on it. Google has to play CYA now, because they were knee deep in the Obama campaign and everybody knows it. Calling out the NSA is fine, but as long as Google remains a partisan propaganda machine, it will always be suspect and I will never trust them with my data.+1
Keith Milner2 years ago
The trouble is it's not just Obama, it's endemic in the political system. Obama has certainly been complicit in allowing this to continue, and supporting the continued funding of it, but he didn't start the fire.

If Romney had won the presidency last time around, we would be in exactly the same position.

The problem is the political system allows you to select between two parties who, fundamentally, are 95% the same. If the people want big changes, there is no way for that to happen within the current system.

In any case, political lobbyists have far more influence than the President.
Thomas Hart2 years ago
This has been great ready - There have been so many sayings like  'History repeats itself' , 'Absolute power corrupts absolutly' .  There is nothing new here.  We all just got shaken out of a deep sleep and were woken up to it.  As someone new to the field here is my question to everyone.    If you encrypt data today - How long before it will be able to be unencrypted?    Is our data just virtually secure at this time even if it is encrypted?    I am of the belief that once we started putting things on the cloud, it was out there and available potentially forever, secure or unsecured, encrypted or not.
Derek Watson2 years ago
It's good that Google seems to be as mad with the NSA/GCHQ as its IT-savvy end-users are. I have already stopped using Google Drive in favour of BTSync. Neither is open source but temporarily I am happier with BitTorrent's brand values. There may be costs for Google in providing end-to-end encryption but there are greater costs in being perceived as a pipeline to a user's local unauthorised data-mining authority.

I think Mike has hit the nail on the head - this is a fight between users and establishment authorities which have found the perfect tool for maintaining their historical privilege indefinitely under the guise of preventing 'terrorism'. In the UK you must divulge any key by law so the only systems that are any use must have plausible deniability.

It will end with the revelation by Snowden that assymetric-key encryption has been broken because the NSA/GCHQ can manipulate large primes. Impossible? The 'impossible' seems to be happening every day now. What is plan B when that happens?
Randy Balbuena2 years ago
I'm glad to hear the traffic is now encrypted, but the efforts to protect the security and privacy of Google users must not stop there. Google should be more proactive and not wait for another Snowden.

It's hard to believe they had access to everything in Google's cloud, including users' passwords. It's even harder to believe that Google was not encrypting data traveling between its data centers.

Encryption is good, but is just one aspect of security. We all want to see a Google that is continuously working to protect all its users and their data, not only from illegal government intrusion but from any kind of intrusion.

Remember, they'll be watching, looking and working on "new access opportunities."
Eugene Seagraves2 years ago
I thank you guys for standing up for what is right.  I once worked at NSA and know realize I know some horrifying stuff.  It is easy to go with the flow when it seems the way of life.  God has corrected me of that behavior.  But now I am behind the 8ball of the system.  Every time I try to speak out I get the hammer dropped on Twitter and FB.  Our leaders are blind due to God.  As our constitution alienates Him.  And we act like children wrt world.  We bully.   +1
William Payne2 years ago
It looks like your Tier 1 provider fucked you over by collaborating with the "ancien regime".+1
August Zajonc2 years ago
I'm surprised by the anger at google. It seems that this was government effort at a pretty large scale, they managed to get folks in charge of the physical fiber to go along with very sophisticated intercept of private traffic.

We must remember that the NSA has denied all of this repeatedly, claiming no backdoor access to google data. And Yahoo etc have confirmed that they use the "highest standards" of security and there is no backdoor access. Google has backed off their claims that there is no backdoor access. Given this post I can see why! 
Eugene Seagraves2 years ago
The Zionist extremists are the Synagogue of Satan in Rev 3:9.  Israel was headed for peace in the mid 1990s as PM Yitzhak Rabin was in favor of sharing the land with the Palestinians, i.e. the Oslo Accords.  But he was assassinated by a zealot Zionist who claimed to be directed by God.  But if you look at the actions by the "Israeli" government they are cursed by God.  In Deut 27 curses are defined for 1) killing in secret, 2) denying justice, 3) moving a boundary stone, 4) leading the blind astray.  The actions of the "Israeli" government I will give the name Israel48, as I can find no historical precedent and associated scripture that supports the 1948 entry into the land.  

Given the use of force to remove Palestinians (Bulldozers, Settlers with Guns, Assassinations, Misleading Propaganda), it does not matter how you look at it, 10 Commandments, Deut 27, or Love your fellow man, it is wrong.  My God would not have His people do evil things to accomplish their goals. So I declare the Israel48 Government is acting as  a Synagogue of Satan. Revelations 2:9 I "...know the blasphemy of them which say they are Jews, and are not, but are the synagogue of Satan." Revelations 3:9 "Behold, I will make them of the synagogue of Satan, which say they are Jews, and are not, but do lie; behold, I will make them to come and worship before thy feet, and to know that I have loved thee."

Further I have been shown that the secret society network that exists in our nation (US) is a part of this terror creation machine.  It could even be said that we are the two feet of the statue of Daniel 2. The US is a divided nation, as are the feet of iron and clay of the statue.  The US makes profit from bloodshed, over 80% of the worlds weapons are US companies.  Habakkuk 2 says woe to he who builds a city on bloodshed.  We tithe heavily to death, in 2012 the 4 richest counties by household income in the US are Arlington VA (Pentagon), Howard MD (NSA), Fairfax VA (CIA), Loudon VA (Defense Contractors).  So we clearly reward the development of death technology.  We sacrifice our youth to war/terror/death.  Name one positive thing that has come out of any war action in our lifetime.

Secrecy and fear are the tools of evil.  Secrecy promotes division and fear.  Fear is the opposite of trust, therefore not of Love.  Love is ofter meant as a cover term that means to give fundamentally but much more in terms of received joy and implied trust, generosity/openness, and acceptance/understanding. So none of Fear/Control/Anxiety, Withholding, Judgment are present in what is often referred to as Love but it can all be termed Liberty.

God has shown us with the concept of the trinity as a model for communication: Truth(Give Honesty/Receive Trust), Love(Give Charity, Receive Openness), Humility(Give Listening, Receive Understanding).  So it can easily be seen that Fear blocks Truth, Secrecy blocks Love as well Understanding if the other side keeps secrets.  Additionally Fear is an autonomic response that when engaged makes the response to an input a full scale output so either fight or flight.  So is it can be seen that two people talking that if both are in fear it is an unstable system, that ends in a battle, or one or both retreat.  The same applies for two peoples. Take for examples North Korea and the US, we fear terror, ad they fear the same if not starvation.  So they luckily are wiser than us or perhaps not afraid.  Because we rattled our sabre but made no attempt to communicate.  Now consider Iraq. In that case Saddam was a seasoned leader who knew He could not allow the bully to push Him around so he stood his ground.  But we in grand style took him down only to find that the justification we uses was a lie. So we by not communicating, destroyed a man and leader who was acting in righteousness.

There is so much more.  We have forsaken God and lost our way.  The Church is Leavened.  The wheat growth not there.  If we do not open our eyes and repent we are done.
Eugene Seagraves2 years ago
NSA was bending Fiber since the 1990s
I personally helped develop tools to receive cell phone calls.
Many local, state, and federal agencies have these radios.
DRTI a Boeing company sells them, we had sold many hundreds if not thousands.
NSA participated in an assassination in Jan 1996, causing Palestine to retaliate just before the election to elect the replacement for Rabin following his assassination.
NSA routinely spied on Americans prior to 9/11, they simply worked with the FBI.
We continually spied on foreign nations leaders with enormous resources 24/7.
DNR Dial Number Recognition) filtered phone collection has been standard practice for decades. 
Cell tower records are kept indefinitely.  Your phone registers even when not in use with every tower it nears.  That info ID's your location every moment for the last 20 years.
Beto Mendez2 years ago
I just want to say, thanks for posting.
Eugene Seagraves2 years ago
I also know my accounts have been monitored for years. They fear me as they have been trying to frame and/or harass me into oblivion.  But God says no. They will soon feel His power.  Just wait and watch.
Aku Ankka2 years ago
Hi, I would appreciate an other opinion in a case on something we all working on IT should be ware of (politics, police and intelligence committing crimes). Please do have a look at Facebook under 'Finland Unrevealed'. Should be seen outside Finland (blocked), if not then use proxy. Sincerely, Åke Tyvi Ps. These people 'fuck up' differently!