In the last two weeks I've been working on moving our autopgkgtests (for gating the landing into the Ubuntuu development series) into the cloud. We've already had had that for a few weeks for amd64 until ProdStack 4 broke down, but there were several shortcomings.https://code.launchpad.net/~pitti/+git/autopkgtest-cloud
now has a simple worker which accepts test requests over AMQP, runs them in ephemeral VMs in ScalingStack, and puts the results into swift. I modified debci to get along with artifacts being in swift instead of the local file system (all changes are upstream) so that we get a nice web browser on http://autopkgtest.ubuntu.com
similar to http://ci.debian.net
. The data you can see there is "real", but there are still a lot of "tmpfail" errors from development.
This does away with Jenkins and the hideous pile of rsync/state files mess that we've piled up over the years, and radically simplifies and robustifies the entire thing. The entire cloud worker is some 250 lines of Python, which is less than a single Jenkins job :-) The rest of the code is just a shim for debci to download the test result metadata from swift, and of course the Juju charms. There is a single shell script for deploying the whole thing now.
TODO before this can go into production:
- We need more power! </TimTaylor> → Our ScalingStack tenant for this purpose only has 10 instances ATM which is too little to keep up.
- Debugging/fixing a nasty race condition with rebooting tests (https://launchpad.net/bugs/1470209
- Teaching britney to issue requests through AMQP.