Shared publicly  - 
 
I'm slowly working toward the next update of The Great Web Framework Shootout, and with Fabric's help I have been able to add a great deal of modularity to the testing structure.

A few of the latest commits add support for alternative servers to some of the control tests. You can also now specify ApacheBench args for the Fabric script directly from the command line.

Here's an updated look at the control test results, with the addition of Python uWSGI, Ruby Thin, and Perl mod_perl. I've also included a graph with higher concurrency results.

We could still use help implementing Fabric scripts for the rest of the frameworks. Check out the GitHub repository's dev branch here: https://github.com/seedifferently/the-great-web-framework-shootout/tree/dev
3
2
Howard Lewis Ship's profile photoSeth Davis's profile photoNaoki INADA's profile photoDmitry Borodaenko's profile photo
8 comments
 
The huge difference (2x+) between Passenger and Thin tells a different story than earlier benchmarks:
http://izumi.plan99.net/blog/index.php/2008/03/31/benchmark-passenger-mod_rails-vs-mongrel-vs-thin/
http://ariekanarie.nl/archives/51/mod_rails-vs-thin-vs-ebb-vs-mongrel

I suspect the reason is that your simplistic test is much better at bringing out the difference between the servers than a full-blown Rails application used in those other tests.

Any chance you could add Unicorn to the mix? I'm really curious how it would compare to Thin. I was trying to package Passenger for Debian, but I'm beginning to think that my time would be better spent improving the Unicorn package...
 
Curious if you are going to pick up my pull request for Apache Tapestry? Thanks! Be in touch if you need any additional help getting it set up and configured.
 
+Dmitry Borodaenko That may be the case, but I am still not convinced that the Passenger numbers are correct (or at least, are the best apples-to-apples configuration against the likes of mod_wsgi). Do you know anybody who could check my Passenger setup and let me know if I should be doing anything differently? My experience with Passenger is limited, so I have only been testing with the default config.

I just pushed a Unicorn Ruby control test task to the GitHub repo. Feel free to give it a shot yourself, or give me some time and eventually I'll get around to posting some updated charts here.
 
+Howard Lewis Ship Thanks!--It's on my radar but I just haven't had the time to get to it yet. I'll let you know if I need any help.
 
+Dmitry Borodaenko I did some messing around with Passenger config directives today and found that setting PassengerHighPerformance to "on" and RackAutoDetect to "off" brought the Passenger numbers up by about 20%.

While I think "RackAutoDetect" is probably safe to disable and still keep it apples-to-apples with the mod_wsgi config (as much as such a thing is possible), I'm hesitant to give "PassengerHighPerformance" the green light as it seems to be considered experimental since it has issues with other core Apache modules (mod_rewrite in particular).

What do you think?
 
Since the cavalry didn't arrive yet, I'll try my best shot, though I'm by no means a Passenger expert, never used it in production. There are different ways to set up a site with Passenger. It's primary selling point is that it's very simple to add as a module into a proper front-end Apache with all its bells and whistles, if that's what you're doing you certainly wouldn't turn PassengerHighPerformance on.

On the other hand, that approach turns your front-end web server into an application server, which is generally not a good idea from security perspective alone, and also takes away some performance optimization options. That's why many people recommend to run a Rack server (be it Passenger, Thin, Unicorn or whatnot) behind Nginx, and sometimes Varnish, too. If you put Passenger behind Nginx, you can have Nginx do all your rewrites et al., so there's nothing to lose by turning PassengerHighPerformance on.

On the third hand, Passenger in back-end role is often set up in standalone mode, which is based on Nginx and not Apache. I suspect the performance story with that setup will be completely different, too.

All that said, I think the best way forward is to see how the performance of Passenger with second and third options compares to other Rack servers, most prominently Thin and Unicorn. If it's still consistently lower, that would mean that it doesn't make sense to use Passenger in the back-end mode, and with only the simplicity as its remaining selling point, it's reasonable to use the first option for benchmarking. If Rack in back-end mode (either as Apache module or standalone) is on par with other servers, you'll have to keep all three results, or at least two (front-end mode and the best of the two back-end modes), to show how much performance you have to sacrifice to keep your setup simple.

Does that help at all, or did I manage to muddy the water even further?
 
+Dmitry Borodaenko Great points, thank you for your input.

With thin and unicorn apparently being so much faster than Apache+Passenger no matter what configuration I try to throw at it, I think it probably makes more sense to test Passenger with PassengerHighPerformance turned off and assume that people will use Thin/Unicorn if they are given the option.

It would be interesting to see what Nginx+Passenger looks like. Perhaps I'll take a look at that configuration in a future update.
Add a comment...