Profile cover photo
Profile photo
Ólafur Arason
Ólafur's interests
View all
Ólafur's posts

Post has shared content

Post has attachment
I've been doing a little bit of experimentation with GO, I even did proxy that simulates slow connections:

Here is my latest piece of experimentation, using channels as a way of getting information back from C. It seems to work, now I just have to experiment more with it.

package main
import "fmt"

extern void go_callback_int(void* icp, int i);

static inline void myprint(void* ic) {
        int j;
        for(j=0; j < 3; j++) {
            go_callback_int(ic, j);
import "C"

import "unsafe"

//export go_callback_int
func go_callback_int(icp unsafe.Pointer, i {
    ic := *(*chan int)(icp)
    ic <- (int)(i)

func main() {
    ic := make(chan int)
    go C.myprint(unsafe.Pointer(&ic))
    for {
        select {
            case i := <-ic:


If You Are Designing Your Own REST backend You're Doing It Wrong

The only reason why I know this is because I've been guilty of that my self. This is meant to start a dialogue so please comment.

Let me walk though first what you are probably doing. You pick your favourite programming language and framework and get going. Whether that's node.js and restify, python and django or ruby and rails.

Then you are going to pick your database. Whether that it's tried and tested of MySQL or shiny and new of MongoDB. This choice is probably going to affect how you scale. Also how well you know some of the different databases and approaches.

Then you start coding. You hopefully care about how the URL schema looks like so you make a really nice interface for developers to work from like this:
GET          /users                  - will get you all of your users
GET          /users/olafur      - will get you one user
POST       /users                  - will make a new user
PUT         /users/olafur      - will update the user
DELETE /user/olafur         - will delete the user

You will go through all of your objects mapping to REST like this and hopefully you will end up with something sane. This is really nice to hook up with jQuery and mobile interfaces.

Now you have to scale. You hope that writing data to the server isn't going to kill it, so you hope you don't get too much traffic like that. You know how to handle reads at least. You have something like Nginx and Varnish, with Memcached, then you try finding bottlenecks and seeing if some more caching doesn't solve that. It's truly amazing to see the difference it makes.

Now you hit an API that has to do some async behaviour and now your screwed. There are some solutions for that but they make the code really complex, even Node.js.

But every one of these steps I've described are incorrect, now let me tell you why. Let's work our way back.

The first problem is that you have a lot of moving parts going on before the data you're trying to put into the database ends up there. There are problems with your APIs losing data because of errors or downtime. A lot of the Internet is going over unreliable wireless technologies. So your beautiful REST calls are now riddled with exception handling, because there are so many ways of things going wrong.

So what do you do. Of course you stick a REST database in front of your API. What does that accomplish, let's first talk about speed, we are talking about 4x writing speed improvements. You don't lose data when writing to the APIs. Databases are probably more solid than code you write. CouchDB is truly a speed freak when it's dealing with REST, it has security built in and validation. When you need to scale you have a multi master database so you stick one closest to your users and they all sync with each others. So we have covered scaling and dealing with the speed of writes and the reads.

How do you deal with writing to the REST APIs if the clients have bad Internet connections? You don't. You write to the native implementation in your browser[1][2] or mobile [3][4]. Then you sync with them when you have a connection. This also cuts down on traffic you have to get from the server. Trust me: it's an order of magnitude difference. You might say, "Why not implement syncing in your framework of choice?" If you did this, then you probably have to rewrite them because syncing works in CouchDB by keeping track of revisions and what has changed. This is hard to retrofit to a framework.

So then you don't have beautiful URLs right. It might be a valid use case for some simple API you have to maintain to have nice looking URLs but not for anything that has to scale. And it's possible in CouchDB with rewrite rules.

I personally like the model of a staging database and main database. You can create a rewrite rule so all writes go to staging and all reads come from main. What this gives you is a record of all the incorrect API calls without polluting your main database. What makes sense is to have types of documents you putting into the database and using views with map functions to sort them out. You don't have to do it like that but if you gain a lot from that.

So this is all fine and dandy for stuff that doesn't require processing but what if you actually want to do something more than just to store data?

You can do what I did and write a service that watches for changes in a database and then put those changes through plugins, in a flow like structure. Or you can use my implementation[5], currently only in python. This abstract the CouchDB from you code some your receiving information and sending back a response. I've written it in Tornado, but who knows asyncio looks pretty good.

With that kind of architecture you don't need to handle as much load as is being written to your servers you just handle as much load as you want. Balancing responsiveness with cost of the machines. But the reads are still going to be fast.

So why isn't everybody doing this then? We are still learning how to structure things well so I'm only able to write about this because of the awesome work of databases like CouchDB that are not afraid of being misunderstood and people that have formed the best practices from all the mistake they have done. I've done plenty of mistakes and will do plenty of mistakes in the future. The important thing is to learn from them.

I have to say that I really love REST and I love beautiful URLs but life is about doing the right thing, as often as you can get away with.


#Python #CouchDB #NodeJS #Programming #RubyOnRails #REST
Wait while more posts are being loaded