Profile cover photo
Profile photo
John Watkinson
Software developer in NYC, Co-Founder of Docracy, Inc.
Software developer in NYC, Co-Founder of Docracy, Inc.


Post has attachment

Post has attachment
Our intern +Nicholas Matthews wrote an awesome, minimalist chat app for Android Wear, check it out!
Add a comment...

Post has shared content
Add a comment...

It's Not Metadata, It's a Social Graph

[A note about me: I was a Ph. D candidate in the Department of Electrical Engineering at Columbia University from 2006 to 2010. While there, I spent some time with colleagues from the Department of Statistics. They were working on problems of analyzing graph data, including phone records such as those revealed to be collected by the NSA. Much of the following is based on my discussions with them during that time.]

In this past week, the astonishing scope of NSA surveillance of both American citizens and others around the world was revealed. The details of if/how the Prism program is actually implemented at major Internet companies is still being investigated. However, it is absolutely confirmed that the NSA captures all of Verizon's call records (and very probably the call records of the other major carriers). This is being justified by using a loophole that categorizes the records as metadata, since the actual audio of the call (or a transcript thereof) is not being collected. "We are not listening in on your calls," is the response from government officials. However, the data being collected are absolutely not metadata. Rather, it is very powerful data, from which an amazing amount of information can be inferred.

Firstly, a quick inspection of the word metadata. The term is frequently used by computer professionals and it means data about data. What would qualify as the metadata for phone call data? They would be mundane details such as the length of the data files, the format it is stored in, and the character encoding used to store the information. The actual data are the phone numbers of the caller/receiver, the start time and length of the call, the locations of the caller/receiver and the audio of the call itself. So, the NSA is not collecting metadata, they are collecting a subset of the data (everything but the audio).

The power of this data is not realized by an NSA agent manually perusing your call records ("Hmm, looks like he called a tow truck from a highway, I guess his car broke down"). Instead, it's much more akin to the social graphs of Facebook, Twitter, Google+ and others. They don't just have the calls that you made, but they have the calls of everybody you called, who they called in turn, etc. This allows them to infer an amazing amount of details about you. What are your eating/sleeping habits? Where and whom do you visit within the US and abroad? What are your political affiliations? Are you religious, and if so, what religion? Are you looking for a job, are you having an affair, are you sick or healthy, are you pregnant, are you likely to be involved in a strike or political protest? All of this and more can be determined with astonishing ease and high accuracy from these data alone.

Of course, companies like Google and Facebook can determine all of this information about you as well, and likely much more. This kind of inference is what allows them to help advertisers target customers so effectively. But, the radical difference between the NSA collecting phone record data and Google, Facebook and others collecting data is that there is no choice for people to opt out.

It remains to be seen whether the citizens of the US will allow the government to continue to binge on their private data. But make no mistake: this is not harmless metadata. They are collecting real and vital data, and it represents a government intrusion into the private lives of citizens the likes of which would have made the Stasi profoundly jealous.
Add a comment...

How Bitcoin Works - Explained as Simply as Possible

Bitcoin has been getting a lot of attention in the press lately. It is an intriguing phenomenon, but difficult for the lay person to understand. This article attempts to demystify the currency with the very minimum use of technical jargon. It turns out that Bitcoin works in a very fascinating and counter-intuitive way, so this should be a worthwhile read.

The Basics: How Bitcoin is Used
Bitcoin is a crypto-currency. This means that it is entirely virtual (no physical coins or bills), and its security is derived from cryptography. Users of the currency will typically run software called a wallet. They may also opt for their wallet to be hosted by a service on their behalf (such as Users can create any number of addresses with their wallet. These addresses operate similar to bank accounts; they can manage and transact Bitcions. Users can conduct transactions virtually between their addresses. There are also exchanges (the largest being where Bitcoins can be exchanged to/from conventional currencies.

However, here is the really unusual thing about Bitcoin: there is absolutely no central authority or "bank" that tracks balances and mediates Bitcoin transactions. The way that the Bitcoin system gets around this is really interesting, but it will take a little bit of explaining.

Proof of Work
How can we trust a loosely-confederated system of computers with no central authority to transact money safely and securely? There seems to be too much opportunity for an attacker to take advantage of the system. It is not an easy thing to overcome, but the idea is that if the majority of these systems are honest (or at least not all colluding together), then it can be done, using a concept called proof of work.

Consider the following analogy. Most people are familiar with "Where's Waldo?" (originally titled "Where's Wally?" in the UK). It is a series of puzzles consisting of very detailed drawings. Somewhere in that drawing is the Waldo character, and it is the goal of the reader to find Waldo in each puzzle. Imagine a really large "Where's Waldo" puzzle, the size of a billboard. It may take hours and hours to find Waldo in that puzzle. However, once you find him, you can measure his co-ordinates and relay them to somebody else. This person can immediately confirm that you did indeed find Waldo. This is the crucial requirement for a "proof of work" problem; that it be difficult to solve, but very easy to verify.

Now of course, it is possible to get lucky and very quickly find Waldo in a puzzle. Everybody has probably had this experience once in a while. But it is not possible to consistently get lucky this way, and any one Waldo-seeker will eventually have to put in the time to solve a lot of puzzles.

Bitcoin runs on puzzles that are essentially very similar to this. The puzzles are called blocks. Each block is very hard to solve, but easily verified once solved. Furthermore, each new block is linked to the previous block, forming a chain. It is as if when you find Waldo in a puzzle, he is holding a tiny little sign that tells you where in the world the next puzzle is. Unless somebody finds Waldo in this puzzle, nobody can work on the next puzzle. Also, these blocks are computer-generated, and so no person knows where Waldo is at the onset. They only way to find him is for everybody to start looking.

If there are a great many computers working to solve these blocks, then the chain of blocks will grow pretty quickly, relative to a few computers toiling away in obscurity. And if these computers are running the honest and unadulterated Bitcoin software, then the chain of blocks they solve will be the "right" ones (they will contain valid and honest balances and transactions). If an attacker (or coalition of attackers) are trying to include falsified ("wrong") blocks in the chain, they will not be able to keep up with the rate of block-solving that the majority of honest workers produce. The result is that the longest chain of blocks can be relied upon to be correct and honest. Only if the attackers could somehow collectively control more computing power than the honest workers could they falsify the system. As long as the Bitcoin community is thriving (as it is today), this hijacking scenario is seen as being nearly impossible.

It remains now to understand who is doing this work on the blocks and why. The computers that run the software that solves these blocks are called nodes, and they are operated by miners. Miners are incentivized to solve the blocks, because every time they "find Waldo" they are awarded a small prize of Bitcoins, an award that is encoded right in to the newly-solved block. The award amount per block slowly decreases over time, and will eventually drop to zero by the year 2140, at which time 21 million Bitcoins will have been created. There are just over 11 million Bitcoins in existence today.

What ties this all together is that the Bitcoin transactions between addresses are broadcast to all the nodes, and they get included in the blocks. So, if two people agree to transact some coins, this intent is broadcast to the miners' nodes. Public Key Cryptography is used to ensure that the user spending the Bitcoins controls the indicated address, the same technology that is used for other secure web transactions. The miners will include the transaction into the next block if it is valid (the sending address has the required amount of Bitcoins, addresses and cryptographic signatures are valid, etc.) and then the transaction will become fixed in the block chain. For the parties involved in the transaction, they should not yet assume the transaction is completed once it is included in a solved block, but rather should wait until several more blocks have been appended to the chain, as this will ensure that their transaction has definitely made its way onto the longest and therefore official (consensus) chain. Each additional block that gets added after the one that includes the transaction reduces the chance that the transaction block was somehow hijacked by an attacker. The convention is to wait for six blocks to be added before considering the transaction to be fully validated, with the first of those six containing the transaction. This takes approximately an hour. However, many Bitcoin merchants will accept a transaction as valid much earlier than this (even instantaneously), and absorb the risk that a small fraction of them may be fraudulent. This is not unlike the risk merchants take with chargebacks on credit cards.

From the point of view of the user, much of the above is done automatically, including doing the cryptographic verification and waiting for six blocks to formally confirm a transaction. Use of Bitcoin online doesn't feel all that different from paying online with other methods, such as credit card or PayPal. However, unlike regular cash, Bitcoin is arguably not well-suited to in-person transactions. If two people exchange conventional cash in person, they can visually verify that they received legitimate-looking legal tender. With Bitcoin, what would be exhanged instead would be the private key of a Bitcoin address. The recipient could verify with a mobile device that the address had the correct amount of cash at that moment, but there would be no way to ensure that the payer wouldn't immediately double-spend the cash, since they could still have a copy of the address's private key. To guard against this, the recipient would have to immediately (probably with a mobile device) transfer the funds out of the address to a new address under the recipient's exclusive control. Only after that transaction was verified could payment be assured.

Transaction Fees
The health and security of the entire Bitcoin ecosystem depends on the miners, as they collectively push the consensus block chain along, process transactions and mint new coins. But since the Bitcoin awards per block will eventually run out, it begs the question of why anybody would continue to bother spending computing resources on mining. The answer is that users can set aside an optional fee when conducting a transaction. Whoever mines (solves) the block containing that transaction can claim this fee. Currently, it is possible to conduct most Bitcoin transactions without paying any kind of transaction fee, as the mining community is primarily focused on the mined Bitcoins and not the fees. However, as the currency becomes more popular and the Bitcoin mining rewards decline further, there will need to be an incentive for miners to include transactions into a block. This should create a market for transaction fees. If a transaction has a very low or zero fee, then it may take a very long time to be confirmed into a block, if ever. However, a transaction with a large fee will very likely be confirmed quickly.

A Few Other Details
Individual Bitcoins have become quite valuable, but they can be subdivided into very small denominations of up to eight decimal places. There have been some physical "Bitcoins" minted, but they are really just clever ways of hiding a private key so that it can only be accessed once, after which the "coin" is destroyed.

Also, there is a common but erroneous belief that Bitcoin is completely anonymous. While the addresses that are exchanged over Bitcoin are not personally-identifiable, Bitcoin was not designed with anonymity in mind. The full log of all Bitcoin transactions is generally available, although older records may be harder to find. Could governments track Bitcoin transactions back to their owners by network activity, possibly correlating this with other data sources? The answer is yes, they probably can.

Hopefully this explanation of Bitcoin has proven useful to those curious about this futuristic currency. This description is by no means complete and there are many sources-- some listed below-- that go in to far more detail. Also, it was not the goal to evaluate the merits or legitimacy of Bitcoin here, but merely to discuss how it works. Please alert me to any errors or important omissions. Full disclosure: I own 1 Bitcoin (that's right; one single, lonely Bitcoin). Thanks for reading!

Original Bitcoin paper:
Wikipedia Page:
Bitcoin Wiki:
Public Key Cryptography:
Add a comment...

Post has attachment

Post has shared content
A shot at legal comedy:
Here is an important document for those who plan to hold a really fun Independence Day BBQ: an air-tight and comprehensive party waiver!
Add a comment...

Post has attachment
Legal for the people, y'all.
Add a comment...

Post has attachment

Post has attachment
Wait while more posts are being loaded