Profile

Cover photo
A Darian
Lives in London
125 followers|3,381 views
AboutPostsPhotosVideosReviews

Stream

A Darian

Shared publicly  - 
 
Sebastian Delmont originally shared:
 
Good bye, Google Maps… thanks for all the fish

TL;DR: We at StreetEasy decided to build our own maps using, among other tools, OpenStreetMap, TileMill, MapBox and Leaflet, instead of paying hundreds of thousands of dollars per year to Google. And yes, the money pushed us into doing it, but we're happier with the result because we now control the contents of our maps.

We were all happy...
Our site, StreetEasy (http://streeteasy.com/), has been using Google Maps embedded in our pages for the last 6 years. We're a real estate portal, so most of our pages have maps in them. So when Google announced they new usage limits (see http://www.dailymail.co.uk/sciencetech/article-2056128/Google-Maps-start-charging--thousands-sites-apps-hit-fees.html) , we were a little worried.

25,000 free map views per day, and $4 per CPM (1,000 views) beyond that. On Christmas day, when everybody was opening their presents, we did ten times that. On a good day, we do 600K-700K pageviews (http://www.quantcast.com/streeteasy.com).

We did the math and came up with numbers that reminded me of Oracle licensing in 1999. Six, seven, eight hundred thousand dollars. We met with Google salespeople, expecting to negotiate better terms, and they were nice, and they offered us discounts, but only to about half of what we've calculated.

In our opinion, their price was off by an order of magnitude. It's very, very hard to work out a $2 CPM cost in any site's business model, when most of the time, if you're lucky, you're making $1 CPM off your pages. And yes, StreetEasy does much better than that, and it would not have bankrupted us, but it would have also meant giving away a significant chunk of our profits.

It was not just the money!
$200,000 to $300,000 a year is, at the very least, the same as hiring a very good engineer for a year (and paying all the taxes and benefits and costs and still having a lot of money left). It was enough money to finally push us into doing our own maps.

Because despite Google Maps being such an awesome product, it had it's downsides. One is that your site looks just like every other site with maps on the Internet (and I know you can customize their colors now, but that costs even more!). Another is that you have no control over your maps, so when you're trying to point out the location of this wonderful apartment, Google might thing it's a good idea to cutter the map with random local businesses (and yes, they've gotten better at it, but often it's just noise). Or they might have bad data, and there's very little you can do about it except report it and wait. (I've always been annoyed at "Classon Pointe" being shown in the middle of Harlem, probably a mistake by some mapping data company decades ago, again, something that has been corrected, but that highlights the problem)

I've always wanted to have our own maps, but thought it would be impossible, or at the very least, a huge amount of work. Something not worth considering, given the rest of a long list of things we also wanted to build on StreetEasy. But with a potential invoice for a third of a million dollars hanging over our heads, we had enough "carrot" (or is it "stick"?) to revisit our priorities. At the very least, we should do more research and see what our options were.

Looking beyond GMaps
Our first option was, of course, Bing Maps. I'm sure Microsoft is having a great time helping all the Google Maps Refugees, and I have no doubt they would have offered us a very cheap licensing deal, but it still meant using someone else's maps, and leave us with license renegotiation risks a year or two down the road. But it was an option.

Then, my coworker +Jordan Anderson, sitting quietly across my desk, pointed out that his "other job", the site he had built with a friend before joining StreetEasy, the fabulous Ride The City (http://ridethecity.com/), did not use Google Maps, but their own tiles, and an open source JS library to display them.

A couple of days later, at a NYC Big Apps hackathon where we were showing off our public APIs, I met +Javier de la Torre (from http://vizzuality.com) and he showed me his awesome product, CartoDB (http://cartodb.com) and gave me a few more pointers. And I saw what +Alastair Coote was doing for his taxi app and got excited with the possibilities.

I spent the next week reading and browsing and searching, discovering the wonderful world of digital cartography, and being amazed at how far the open source tools had advanced in the last few years.

The world of Open Source Cartography
We now had a great tile renderer, Mapnik (http://mapnik.org/), that was at the core of pretty much every mapping tool out there. Great "geo" and "gis" functionality for Postgres, in the form of PostGIS (http://postgis.refractions.net/). A few javascript libraries to present the results inside web browsers, such as Leaflet (http://leaflet.cloudmade.com/), Open Layers (http://openlayers.org/) and Modest Maps (http://modestmaps.com/), and other libraries to abstract your mapping backend behind a common API, such as Wax (http://mapbox.com/wax/) or Mapstraction (http://mapstraction.com/).

But then I discovered the "second generation" of tools, built on top of what I just listed on the previous paragraph, and it blew my mind. Things like CartoDB or TileMill (http://mapbox.com/tilemill/) or Web Map Studio (http://cloudmade.com/products/web-maps-studio).

TileMill, in particular, was just amazing, and Carto CSS (http://developmentseed.org/blog/2011/feb/09/introducing-carto-css-map-styling-language/) made map design look like something I could actually do!

And of course, OpenStreetMap (http://www.openstreetmap.org/), the Wikipedia of mapping. An open source (well, technically, Creative Commons) data set, covering the entire globe, with lots of details (sometimes too much detail, like the voltage and gauge of a subway line!). It has a few errors here and there, but you can go and fix them yourself (as I've done http://www.openstreetmap.org/user/sdelmont/edits).

The path we took
I settled on Leaflet for the front end, mostly because it was small, fast, clean code with a good API that resembled Google Maps v2. It's a good thing that when we first implemented maps on StreetEasy, we did it through ruby that generated the JS code, so all I had to do was "implement an new backend". If I were to do it today, I might use Wax or Mapstraction instead, to ensure I could change map APIs if I had to.

It was fairly easy to implement most basic features. Showing a map, adding markers, adding polygons, info popups (we had our own code for that, just had to hook it on the right events). I spent a couple of days getting our "polygon editor" to work (something I plan to contribute back to Leaflet as soon as I have time to clean up the code). And of course, the dreaded "does it run on IE?" time (I ran into some issues with onload events on script tags, but that was all).

I installed Postgres and PostGIS, downloaded OSM extracts from http://download.geofabrik.de/osm/north-america/, because there is no point in downloading gigs and gigs of worldwide data when all I care about is the area around NYC. Imported it using osm2pgsql (http://wiki.openstreetmap.org/wiki/Osm2pgsql) and started playing with TileMill.

I discovered the work of Mike at Stamen (for example http://mike.teczno.com/notes/osm-us-terrain-layer.html) and was inspired by it. Found a couple of TileMill projects (https://github.com/mapbox/open-streets-style and https://github.com/mapbox/osm-bright) to better understand how to organize them. Ran into High Road (http://mike.teczno.com/notes/high-road.html), a set of queries that makes OSM roads much more manageable and stylable.

And I spent days and days tweaking maps. Just to get to a point where we were not unhappy with our maps. Something that was good enough to flip the switch.

We added building outlines from NYC Open Data (http://nycopendata.socrata.com/), and our own neighborhood boundaries to decide where to put the labels (and trust me, we have the best boundaries for NYC).

As soon as I had something that didn't cause my coworkers to vomit, I uploaded the tiles to S3 and started testing it on our site. A few days later, and a lot more map tweaks, we started using the new maps for some of our users. And as of Jan 10th, we flipped the switch for all pageviews on our site.

We decided to host our tileset with MapBox (http://mapbox.com), from the great guys at Development Seed.. We could have unpack the mbtiles file produced by TileMill and just upload them to S3 (see http://karchner.com/2011/02/21/extract-images-from-an-mbtiles-file-or-getting-actual/), but we went ahead and paid for MapBox, in part because it means less servers to worry about, in part because we want to support the guys that brought us TileMill, and in part because of the promise of more cool features down the road. And most importantly, because they promised to help us make our maps look nicer, and they know about nice maps.

Take a look at the results: http://streeteasy.com/nyc/sales/midtown-all-manhattan/status:open%7Cbeds:2?map_all=1

Where to now?
If I haven't made it clear, we're not completely happy with how our maps look, but we were happy enough to go ahead. We want to make them look great, with more data (such as subway stations) and better labels and lots of other little things. Development Seed will help us with that, and we've been learning a lot ourselves.

We'd also like to have a "live mapnik server", producing tiles on demand (and caching the results, duh) to make it easier to tweak our maps. Right now it takes a couple of days to go from OSM import to tile rendering to uploading multi-gigabyte files and finally showing them on the site. A live server would let us change a stylesheet and see the results right away.

We will try to contribute back to all these open source projects as much as we can. I already have some code for Leaflet for polygon editing and encoding, for example, and we've started doing edits on OSM.

What about geocoding?
You'd probably noticed I didn't talk about geocoding (the "art" of converting a street address into a set of coordinates in a map, in case you didn't know). That's part of what Google offers as part of their Maps APIs.

Well, at StreetEasy we built our own geocoder for NYC, using the City's database of streets and buildings. So it's not something we had to worry about as part of this transition.

But in case you need to do some geocoding, there are plenty of tools (for example http://highearthorbit.com/geocommons-open-sourced-geocoder/) that use OSM data.

The Year of the Open Map
I think that someone at Google got their pricing wrong by an order of magnitude. Large companies might be willing to pay that kind of licenses, but this is not the CMS market in 1998, where people would pay half a million for a Vignette license and another million for Oracle. There are so many open source options out there that the value of proprietary solutions has come down dramatically.

And if Google keeps pushing companies into experimenting with these open source solutions, it's only going to get better. I think 2012 is going to be the year of the Open Map. And I'm happy to be part of the front lines.
3
Add a comment...

A Darian

Shared publicly  - 
 
I am so confused! He's a dog. But he has human arms!
1
Add a comment...

A Darian

Shared publicly  - 
3
Add a comment...

A Darian

Shared publicly  - 
 
This is why it's always a mistake to buy DRM'd content. Always. No exceptions.
1
Eshan Shah Jahan's profile photoA Darian's profile photoJonathan Rivera's profile photo
7 comments
 
The DRM on Blu-Rays and DVD's is about as intense as when you used to have to put tape over that little square on a VHS tape. I love Steam....best example of basic unintrusive copy protection mixed with a wonderful service. Artists should be paid for their work, and consumers shouldn't be punished by DRM for paying them.
Add a comment...

A Darian

Shared publicly  - 
 
Great blog post about how our parents' generation failed us.
1
Add a comment...

A Darian

Shared publicly  - 
 
Some people get invited to the best parties.
1
Add a comment...
Have him in circles
125 people
Meg Barthelme's profile photo
John Pershing's profile photo
Joel Micalizzi's profile photo
Rosalyn Luna's profile photo
Michael Stillwell's profile photo
S. A. Hunt's profile photo
Jason Shah-Jahan's profile photo
Stephanie Hope's profile photo

A Darian

Shared publicly  - 
 
Amazing. This guy tried pretty much every pizza sold by the slice in Manhattan.
1
Eshan Shah Jahan's profile photoA Darian's profile photo
2 comments
 
Well, I was wondering about this. A slice of pizza per day seems like it might be something you could tolerate without gaining weight. But I'm not sure.
Add a comment...

A Darian

Shared publicly  - 
 
Parasitic tech companies ought to be treated like other parasites.
1
Add a comment...

A Darian

Shared publicly  - 
1
Add a comment...

A Darian

Shared publicly  - 
 
There was no note. I'm sure whoever did this got away with it. Ugh.
1
Alan Hsu's profile photoAndy Mui's profile photoCarey G's profile photoJorge Alexander Casanova's profile photo
4 comments
 
wow, that's a nice lookin setup too :\
Add a comment...

A Darian

Shared publicly  - 
 
Too many carbs.
1
Eshan Shah Jahan's profile photoAlan Hsu's profile photoA Darian's profile photo
4 comments
 
Reminds me of mashed potato sandwiches at Rutgers. I wonder what these cheap meals to the NHS budget...
Add a comment...

A Darian

Shared publicly  - 
 
S. A. Hunt originally shared:
 
Aaaaaaand now you know what to get me for Christmas. #Adorable
1
Carey G's profile photo
Carey G
+
1
2
1
 
We always joke about how our dog has an imaginary friend that she wrestles her toys from. We call him Bad Ronald. I think this is who he is.
Add a comment...
People
Have him in circles
125 people
Meg Barthelme's profile photo
John Pershing's profile photo
Joel Micalizzi's profile photo
Rosalyn Luna's profile photo
Michael Stillwell's profile photo
S. A. Hunt's profile photo
Jason Shah-Jahan's profile photo
Stephanie Hope's profile photo
Places
Map of the places this user has livedMap of the places this user has livedMap of the places this user has lived
Currently
London
Previously
New York - Berkeley
Links
Contributor to
Story
Introduction
I was born in the second millennium, common era.
Basic Information
Gender
Male
Stuart really knows his stuff. He managed to diagnose what was wrong with my bike from just photos I sent before he even checked it out in person. And he's really approachable and friendly, not intimidating in the way bike mechanics can be sometimes if you're not an expert yourself. I'd highly recommend him!
Public - 11 months ago
reviewed 11 months ago
1 review
Map
Map
Map