Good bye, Google Maps… thanks for all the fishTL;DR: We at StreetEasy decided to build our own maps using, among other tools, OpenStreetMap, TileMill, MapBox and Leaflet, instead of paying hundreds of thousands of dollars per year to Google. And yes, the money pushed us into doing it, but we're happier with the result because we now control the contents of our maps.
(and now there is a follow up post too https://plus.google.com/118383351194421484817/posts/8x6xSQYypwt
)We were all happy...
Our site, StreetEasy (http://streeteasy.com/
), has been using Google Maps embedded in our pages for the last 6 years. We're a real estate portal, so most of our pages have maps in them. So when Google announced they new usage limits (see http://www.dailymail.co.uk/sciencetech/article-2056128/Google-Maps-start-charging--thousands-sites-apps-hit-fees.html
) , we were a little worried.
25,000 free map views per day, and $4 per CPM (1,000 views) beyond that. On Christmas day, when everybody was opening their presents, we did ten times that. On a good day, we do 600K-700K pageviews (http://www.quantcast.com/streeteasy.com
We did the math and came up with numbers that reminded me of Oracle licensing in 1999. Six, seven, eight hundred thousand dollars. We met with Google salespeople, expecting to negotiate better terms, and they were nice, and they offered us discounts, but only to about half of what we've calculated.
In our opinion, their price was off by an order of magnitude. It's very, very hard to work out a $2 CPM cost in any site's business model, when most of the time, if you're lucky, you're making $1 CPM off your pages. And yes, StreetEasy does much better than that, and it would not have bankrupted us, but it would have also meant giving away a significant chunk of our profits.It was not just the money!
$200,000 to $300,000 a year is, at the very least, the same as hiring a very good engineer for a year (and paying all the taxes and benefits and costs and still having a lot of money left). It was enough money to finally push us into doing our own maps.
Because despite Google Maps being such an awesome product, it had it's downsides. One is that your site looks just like every other site with maps on the Internet (and I know you can customize their colors now, but that costs even more!). Another is that you have no control over your maps, so when you're trying to point out the location of this wonderful apartment, Google might thing it's a good idea to cutter the map with random local businesses (and yes, they've gotten better at it, but often it's just noise). Or they might have bad data, and there's very little you can do about it except report it and wait. (I've always been annoyed at "Classon Pointe" being shown in the middle of Harlem, probably a mistake by some mapping data company decades ago, again, something that has been corrected, but that highlights the problem)
I've always wanted to have our own maps, but thought it would be impossible, or at the very least, a huge amount of work. Something not worth considering, given the rest of a long list of things we also wanted to build on StreetEasy. But with a potential invoice for a third of a million dollars hanging over our heads, we had enough "carrot" (or is it "stick"?) to revisit our priorities. At the very least, we should do more research and see what our options were.Looking beyond GMaps
Our first option was, of course, Bing Maps. I'm sure Microsoft is having a great time helping all the Google Maps Refugees, and I have no doubt they would have offered us a very cheap licensing deal, but it still meant using someone else's maps, and leave us with license renegotiation risks a year or two down the road. But it was an option.
Then, my coworker +Jordan Anderson
, sitting quietly across my desk, pointed out that his "other job", the site he had built with a friend before joining StreetEasy, the fabulous Ride The City (http://ridethecity.com/
), did not use Google Maps, but their own tiles, and an open source JS library to display them.
A couple of days later, at a NYC Big Apps hackathon where we were showing off our public APIs, I met +Javier de la Torre
) and he showed me his awesome product, CartoDB (http://cartodb.com
) and gave me a few more pointers. And I saw what +Alastair Coote
was doing for his taxi app and got excited with the possibilities.
I spent the next week reading and browsing and searching, discovering the wonderful world of digital cartography, and being amazed at how far the open source tools had advanced in the last few years. The world of Open Source Cartography
We now had a great tile renderer, Mapnik (http://mapnik.org/
), that was at the core of pretty much every mapping tool out there. Great "geo" and "gis" functionality for Postgres, in the form of PostGIS (http://postgis.refractions.net/
), Open Layers (http://openlayers.org/
) and Modest Maps (http://modestmaps.com/
), and other libraries to abstract your mapping backend behind a common API, such as Wax (http://mapbox.com/wax/
) or Mapstraction (http://mapstraction.com/
But then I discovered the "second generation" of tools, built on top of what I just listed on the previous paragraph, and it blew my mind. Things like CartoDB or TileMill (http://mapbox.com/tilemill/
) or Web Map Studio (http://cloudmade.com/products/web-maps-studio
TileMill, in particular, was just amazing, and Carto CSS (http://developmentseed.org/blog/2011/feb/09/introducing-carto-css-map-styling-language/
) made map design look like something I could actually do!
And of course, OpenStreetMap (http://www.openstreetmap.org/
), the Wikipedia of mapping. An open source (well, technically, Creative Commons) data set, covering the entire globe, with lots of details (sometimes too much detail, like the voltage and gauge of a subway line!). It has a few errors here and there, but you can go and fix them yourself (as I've done http://www.openstreetmap.org/user/sdelmont/edits
).The path we took
I settled on Leaflet for the front end, mostly because it was small, fast, clean code with a good API that resembled Google Maps v2. It's a good thing that when we first implemented maps on StreetEasy, we did it through ruby that generated the JS code, so all I had to do was "implement an new backend". If I were to do it today, I might use Wax or Mapstraction instead, to ensure I could change map APIs if I had to.
It was fairly easy to implement most basic features. Showing a map, adding markers, adding polygons, info popups (we had our own code for that, just had to hook it on the right events). I spent a couple of days getting our "polygon editor" to work (something I plan to contribute back to Leaflet as soon as I have time to clean up the code). And of course, the dreaded "does it run on IE?" time (I ran into some issues with onload events on script tags, but that was all).
I installed Postgres and PostGIS, downloaded OSM extracts from http://download.geofabrik.de/osm/north-america/
, because there is no point in downloading gigs and gigs of worldwide data when all I care about is the area around NYC. Imported it using osm2pgsql (http://wiki.openstreetmap.org/wiki/Osm2pgsql
) and started playing with TileMill.
I discovered the work of Mike at Stamen (for example http://mike.teczno.com/notes/osm-us-terrain-layer.html
) and was inspired by it. Found a couple of TileMill projects (https://github.com/mapbox/open-streets-style
) to better understand how to organize them. Ran into High Road (http://mike.teczno.com/notes/high-road.html
), a set of queries that makes OSM roads much more manageable and stylable.
And I spent days and days tweaking maps. Just to get to a point where we were not unhappy with our maps. Something that was good enough to flip the switch.
We added building outlines from NYC Open Data (http://nycopendata.socrata.com/
), and our own neighborhood boundaries to decide where to put the labels (and trust me, we have the best boundaries for NYC).
As soon as I had something that didn't cause my coworkers to vomit, I uploaded the tiles to S3 and started testing it on our site. A few days later, and a lot more map tweaks, we started using the new maps for some of our users. And as of Jan 10th, we flipped the switch for all pageviews on our site.
We decided to host our tileset with MapBox (http://mapbox.com
), from the great guys at Development Seed.. We could have unpack the mbtiles file produced by TileMill and just upload them to S3 (see http://karchner.com/2011/02/21/extract-images-from-an-mbtiles-file-or-getting-actual/
), but we went ahead and paid for MapBox, in part because it means less servers to worry about, in part because we want to support the guys that brought us TileMill, and in part because of the promise of more cool features down the road. And most importantly, because they promised to help us make our maps look nicer, and they know about nice maps.
Take a look at the results: http://streeteasy.com/nyc/sales/midtown-all-manhattan/status:open%7Cbeds:2?map_all=1Where to now?
If I haven't made it clear, we're not completely happy with how our maps look, but we were happy enough to go ahead. We want to make them look great, with more data (such as subway stations) and better labels and lots of other little things. Development Seed will help us with that, and we've been learning a lot ourselves.
We'd also like to have a "live mapnik server", producing tiles on demand (and caching the results, duh) to make it easier to tweak our maps. Right now it takes a couple of days to go from OSM import to tile rendering to uploading multi-gigabyte files and finally showing them on the site. A live server would let us change a stylesheet and see the results right away.
We will try to contribute back to all these open source projects as much as we can. I already have some code for Leaflet for polygon editing and encoding, for example, and we've started doing edits on OSM.What about geocoding?
You'd probably noticed I didn't talk about geocoding (the "art" of converting a street address into a set of coordinates in a map, in case you didn't know). That's part of what Google offers as part of their Maps APIs.
Well, at StreetEasy we built our own geocoder for NYC, using the City's database of streets and buildings. So it's not something we had to worry about as part of this transition.
But in case you need to do some geocoding, there are plenty of tools (for example http://highearthorbit.com/geocommons-open-sourced-geocoder/
) that use OSM data.The Year of the Open Map
I think that someone at Google got their pricing wrong by an order of magnitude. Large companies might be willing to pay that kind of licenses, but this is not the CMS market in 1998, where people would pay half a million for a Vignette license and another million for Oracle. There are so many open source options out there that the value of proprietary solutions has come down dramatically.
And if Google keeps pushing companies into experimenting with these open source solutions, it's only going to get better. I think 2012 is going to be the year of the Open Map. And I'm happy to be part of the front lines.UPDATE
I've posted a follow up with some clarifications at https://plus.google.com/118383351194421484817/posts/8x6xSQYypwt