Profile

Cover photo
Ólafur Arason
Attends University of Iceland
Lived in Álftanes
173 followers|157,816 views
AboutPostsPhotosYouTube

Stream

Ólafur Arason

Shared publicly  - 
 
 
This is some brilliantly written, thoughtful, funny, and spot on commentary on gamergate: https://medium.com/the-cauldron/why-gamergaters-piss-me-the-f-off-a7e4c7f6d8a6
Chris Kluwe played in the NFL for eight years, but he’s been a gamer for 26 — and he’s tired of the misogyny in today’s …
4 comments on original post
1
Add a comment...
 
If You Are Designing Your Own REST backend You're Doing It Wrong

The only reason why I know this is because I've been guilty of that my self. This is meant to start a dialogue so please comment.

Let me walk though first what you are probably doing. You pick your favourite programming language and framework and get going. Whether that's node.js and restify, python and django or ruby and rails.

Then you are going to pick your database. Whether that it's tried and tested of MySQL or shiny and new of MongoDB. This choice is probably going to affect how you scale. Also how well you know some of the different databases and approaches.

Then you start coding. You hopefully care about how the URL schema looks like so you make a really nice interface for developers to work from like this:
GET          /users                  - will get you all of your users
GET          /users/olafur      - will get you one user
POST       /users                  - will make a new user
PUT         /users/olafur      - will update the user
DELETE /user/olafur         - will delete the user

You will go through all of your objects mapping to REST like this and hopefully you will end up with something sane. This is really nice to hook up with jQuery and mobile interfaces.

Now you have to scale. You hope that writing data to the server isn't going to kill it, so you hope you don't get too much traffic like that. You know how to handle reads at least. You have something like Nginx and Varnish, with Memcached, then you try finding bottlenecks and seeing if some more caching doesn't solve that. It's truly amazing to see the difference it makes.

Now you hit an API that has to do some async behaviour and now your screwed. There are some solutions for that but they make the code really complex, even Node.js.

But every one of these steps I've described are incorrect, now let me tell you why. Let's work our way back.

The first problem is that you have a lot of moving parts going on before the data you're trying to put into the database ends up there. There are problems with your APIs losing data because of errors or downtime. A lot of the Internet is going over unreliable wireless technologies. So your beautiful REST calls are now riddled with exception handling, because there are so many ways of things going wrong.

So what do you do. Of course you stick a REST database in front of your API. What does that accomplish, let's first talk about speed, we are talking about 4x writing speed improvements. You don't lose data when writing to the APIs. Databases are probably more solid than code you write. CouchDB is truly a speed freak when it's dealing with REST, it has security built in and validation. When you need to scale you have a multi master database so you stick one closest to your users and they all sync with each others. So we have covered scaling and dealing with the speed of writes and the reads.

How do you deal with writing to the REST APIs if the clients have bad Internet connections? You don't. You write to the native implementation in your browser[1][2] or mobile [3][4]. Then you sync with them when you have a connection. This also cuts down on traffic you have to get from the server. Trust me: it's an order of magnitude difference. You might say, "Why not implement syncing in your framework of choice?" If you did this, then you probably have to rewrite them because syncing works in CouchDB by keeping track of revisions and what has changed. This is hard to retrofit to a framework.

So then you don't have beautiful URLs right. It might be a valid use case for some simple API you have to maintain to have nice looking URLs but not for anything that has to scale. And it's possible in CouchDB with rewrite rules.

I personally like the model of a staging database and main database. You can create a rewrite rule so all writes go to staging and all reads come from main. What this gives you is a record of all the incorrect API calls without polluting your main database. What makes sense is to have types of documents you putting into the database and using views with map functions to sort them out. You don't have to do it like that but if you gain a lot from that.

So this is all fine and dandy for stuff that doesn't require processing but what if you actually want to do something more than just to store data?

You can do what I did and write a service that watches for changes in a database and then put those changes through plugins, in a flow like structure. Or you can use my implementation[5], currently only in python. This abstract the CouchDB from you code some your receiving information and sending back a response. I've written it in Tornado, but who knows asyncio looks pretty good.

With that kind of architecture you don't need to handle as much load as is being written to your servers you just handle as much load as you want. Balancing responsiveness with cost of the machines. But the reads are still going to be fast.

So why isn't everybody doing this then? We are still learning how to structure things well so I'm only able to write about this because of the awesome work of databases like CouchDB that are not afraid of being misunderstood and people that have formed the best practices from all the mistake they have done. I've done plenty of mistakes and will do plenty of mistakes in the future. The important thing is to learn from them.

I have to say that I really love REST and I love beautiful URLs but life is about doing the right thing, as often as you can get away with.

[1] https://github.com/olafura/sundaydata
[2] https://github.com/daleharvey/pouchdb
[3] http://www.couchbase.com/mobile
[4] https://cloudant.com/product/cloudant-features/sync/
[5] https://github.com/olafura/sundaytasks-py

#Python #CouchDB #NodeJS #Programming #RubyOnRails #REST
104
38
Robbie Jack's profile photoSimon Cornelius P Umacob's profile photoDimitris Karteris's profile photo黃建璋's profile photo
20 comments
 
+Ólafur Arason oh, you were talking about performance. I was talking about validation, role-based permissions, column-filtering etc. Yep, sqlalchemy on its own can be pretty slow, but combining it with Gevent  (or any non-blocking library) gives it a huge performance boost. 
Add a comment...
In his circles
182 people
Have him in circles
173 people
Sai Pinapati's profile photo
Dave Cottlehuber's profile photo
Margrét Arnardóttir's profile photo
Sooraj E's profile photo
Lijun Guo's profile photo
Henrý Þór Baldursson's profile photo
Frenzy Hunter's profile photo
Carlos Raimundo's profile photo
OpenPhoto's profile photo

Ólafur Arason

Shared publicly  - 
 
I've been doing a little bit of experimentation with GO, I even did proxy that simulates slow connections:
https://github.com/olafura/turtleproxy

Here is my latest piece of experimentation, using channels as a way of getting information back from C. It seems to work, now I just have to experiment more with it.

package main
import "fmt"

/*
extern void go_callback_int(void* icp, int i);

static inline void myprint(void* ic) {
        int j;
        for(j=0; j < 3; j++) {
            go_callback_int(ic, j);
        }
}
*/
import "C"

import "unsafe"

//export go_callback_int
func go_callback_int(icp unsafe.Pointer, i C.int) {
    fmt.Println("gci",i)
    ic := *(*chan int)(icp)
    ic <- (int)(i)
}

func main() {
    ic := make(chan int)
    go C.myprint(unsafe.Pointer(&ic))
    for {
        select {
            case i := <-ic:
                fmt.Println(i)
        }
    }
}

#golang  
1
Add a comment...
People
In his circles
182 people
Have him in circles
173 people
Sai Pinapati's profile photo
Dave Cottlehuber's profile photo
Margrét Arnardóttir's profile photo
Sooraj E's profile photo
Lijun Guo's profile photo
Henrý Þór Baldursson's profile photo
Frenzy Hunter's profile photo
Carlos Raimundo's profile photo
OpenPhoto's profile photo
Places
Map of the places this user has livedMap of the places this user has livedMap of the places this user has lived
Previously
Álftanes
Links
Education
  • University of Iceland
    present
Basic Information
Gender
Male