Profile

Cover photo
Ólafur Arason
Attends University of Iceland
Lived in Álftanes
172 followers|167,097 views
AboutPostsPhotosYouTube

Stream

Ólafur Arason

Shared publicly  - 
 
 
This is some brilliantly written, thoughtful, funny, and spot on commentary on gamergate: https://medium.com/the-cauldron/why-gamergaters-piss-me-the-f-off-a7e4c7f6d8a6
Chris Kluwe played in the NFL for eight years, but he’s been a gamer for 26 — and he’s tired of the misogyny in today’s …
1
Add a comment...
 
If You Are Designing Your Own REST backend You're Doing It Wrong

The only reason why I know this is because I've been guilty of that my self. This is meant to start a dialogue so please comment.

Let me walk though first what you are probably doing. You pick your favourite programming language and framework and get going. Whether that's node.js and restify, python and django or ruby and rails.

Then you are going to pick your database. Whether that it's tried and tested of MySQL or shiny and new of MongoDB. This choice is probably going to affect how you scale. Also how well you know some of the different databases and approaches.

Then you start coding. You hopefully care about how the URL schema looks like so you make a really nice interface for developers to work from like this:
GET          /users                  - will get you all of your users
GET          /users/olafur      - will get you one user
POST       /users                  - will make a new user
PUT         /users/olafur      - will update the user
DELETE /user/olafur         - will delete the user

You will go through all of your objects mapping to REST like this and hopefully you will end up with something sane. This is really nice to hook up with jQuery and mobile interfaces.

Now you have to scale. You hope that writing data to the server isn't going to kill it, so you hope you don't get too much traffic like that. You know how to handle reads at least. You have something like Nginx and Varnish, with Memcached, then you try finding bottlenecks and seeing if some more caching doesn't solve that. It's truly amazing to see the difference it makes.

Now you hit an API that has to do some async behaviour and now your screwed. There are some solutions for that but they make the code really complex, even Node.js.

But every one of these steps I've described are incorrect, now let me tell you why. Let's work our way back.

The first problem is that you have a lot of moving parts going on before the data you're trying to put into the database ends up there. There are problems with your APIs losing data because of errors or downtime. A lot of the Internet is going over unreliable wireless technologies. So your beautiful REST calls are now riddled with exception handling, because there are so many ways of things going wrong.

So what do you do. Of course you stick a REST database in front of your API. What does that accomplish, let's first talk about speed, we are talking about 4x writing speed improvements. You don't lose data when writing to the APIs. Databases are probably more solid than code you write. CouchDB is truly a speed freak when it's dealing with REST, it has security built in and validation. When you need to scale you have a multi master database so you stick one closest to your users and they all sync with each others. So we have covered scaling and dealing with the speed of writes and the reads.

How do you deal with writing to the REST APIs if the clients have bad Internet connections? You don't. You write to the native implementation in your browser[1][2] or mobile [3][4]. Then you sync with them when you have a connection. This also cuts down on traffic you have to get from the server. Trust me: it's an order of magnitude difference. You might say, "Why not implement syncing in your framework of choice?" If you did this, then you probably have to rewrite them because syncing works in CouchDB by keeping track of revisions and what has changed. This is hard to retrofit to a framework.

So then you don't have beautiful URLs right. It might be a valid use case for some simple API you have to maintain to have nice looking URLs but not for anything that has to scale. And it's possible in CouchDB with rewrite rules.

I personally like the model of a staging database and main database. You can create a rewrite rule so all writes go to staging and all reads come from main. What this gives you is a record of all the incorrect API calls without polluting your main database. What makes sense is to have types of documents you putting into the database and using views with map functions to sort them out. You don't have to do it like that but if you gain a lot from that.

So this is all fine and dandy for stuff that doesn't require processing but what if you actually want to do something more than just to store data?

You can do what I did and write a service that watches for changes in a database and then put those changes through plugins, in a flow like structure. Or you can use my implementation[5], currently only in python. This abstract the CouchDB from you code some your receiving information and sending back a response. I've written it in Tornado, but who knows asyncio looks pretty good.

With that kind of architecture you don't need to handle as much load as is being written to your servers you just handle as much load as you want. Balancing responsiveness with cost of the machines. But the reads are still going to be fast.

So why isn't everybody doing this then? We are still learning how to structure things well so I'm only able to write about this because of the awesome work of databases like CouchDB that are not afraid of being misunderstood and people that have formed the best practices from all the mistake they have done. I've done plenty of mistakes and will do plenty of mistakes in the future. The important thing is to learn from them.

I have to say that I really love REST and I love beautiful URLs but life is about doing the right thing, as often as you can get away with.

[1] https://github.com/olafura/sundaydata
[2] https://github.com/daleharvey/pouchdb
[3] http://www.couchbase.com/mobile
[4] https://cloudant.com/product/cloudant-features/sync/
[5] https://github.com/olafura/sundaytasks-py

#Python #CouchDB #NodeJS #Programming #RubyOnRails #REST
100
38
Kurt Williams's profile photoÓlafur Arason's profile photo
17 comments
 
+Kurt Williams Here are some good links:
http://guide.couchdb.org/draft/security.html
http://wiki.apache.org/couchdb/Security_Features_Overview
http://couchdb.readthedocs.org/en/latest/api/server/authn.html

But basically you have a couple of options of registering user, one is with the built in system and then using either Facebook or Twitter:
https://github.com/ocastalabs/CouchDB-XO_Auth

After you have registered then you can use the built in system which provides a couple of ways of authenticating like cookies, http basic and oauth. Or you can use the Facebook and Twitter.

The facebook and twitter login is provided by a plugin and CouchDB can support what ever scheme you want to login with if you are so inclined, but it "has" to be in erlang.

That's authentication and registration, then there is security of the databases, they are governed by roles and each database can use a mix of usernames and roles to control, read and write access.

For example if you have a group working on something than you create a role for that and assign it to the members that need to be in it and the database that stores the data.

But that isn't enough you also have to make sure that some people can't mess with certain fields so you have write validation:
http://guide.couchdb.org/draft/validation.html

Hopefully soon, depends on how the merge is going, we will have read validations, but they might impact speed so use sparsely like with the write validation:
https://github.com/refuge/rcouch/wiki/Validate-documents-on-read

The best way is to spit things up into different databases to enforce security.
Add a comment...
Have him in circles
172 people
Fannar Freyr Ívarsson's profile photo
Arngrímur Vídalín's profile photo
Dallas Gutauckis's profile photo
Sigurlín Atladóttir's profile photo
Viðar Másson's profile photo
AngularJS's profile photo
Margrét Arnardóttir's profile photo
Lunatics! (Animated Series)'s profile photo
Halla Tryggvadóttir's profile photo

Ólafur Arason

Shared publicly  - 
 
I've been doing a little bit of experimentation with GO, I even did proxy that simulates slow connections:
https://github.com/olafura/turtleproxy

Here is my latest piece of experimentation, using channels as a way of getting information back from C. It seems to work, now I just have to experiment more with it.

package main
import "fmt"

/*
extern void go_callback_int(void* icp, int i);

static inline void myprint(void* ic) {
        int j;
        for(j=0; j < 3; j++) {
            go_callback_int(ic, j);
        }
}
*/
import "C"

import "unsafe"

//export go_callback_int
func go_callback_int(icp unsafe.Pointer, i C.int) {
    fmt.Println("gci",i)
    ic := *(*chan int)(icp)
    ic <- (int)(i)
}

func main() {
    ic := make(chan int)
    go C.myprint(unsafe.Pointer(&ic))
    for {
        select {
            case i := <-ic:
                fmt.Println(i)
        }
    }
}

#golang  
1
Add a comment...
People
Have him in circles
172 people
Fannar Freyr Ívarsson's profile photo
Arngrímur Vídalín's profile photo
Dallas Gutauckis's profile photo
Sigurlín Atladóttir's profile photo
Viðar Másson's profile photo
AngularJS's profile photo
Margrét Arnardóttir's profile photo
Lunatics! (Animated Series)'s profile photo
Halla Tryggvadóttir's profile photo
Collections Ólafur is following
Basic Information
Gender
Male
Education
  • University of Iceland
    present
Places
Map of the places this user has livedMap of the places this user has livedMap of the places this user has lived
Previously
Álftanes
Links