Had a great (but exhausting) weekend running the infrastructure at +NorthSec
As usual, this was a great stress test for the Linux kernel, LXC, Ubuntu and our network equipment. We were running just shy of 10000 LXC containers, about a hundred VMs co-located with those, several hundred VLANs including nested VLANs (QinQ + jumbo frames) and an insanely complex IPv6-only network.
Remember that we simulate a whole IPv6-only Internet FOR EVERY ONE OF THE 30 PARTICIPATING TEAMS. And that's only the base on which we build the actually competition!
Oh and did I mention we're running all that stuff on only a handful of 10 years old server (HP DL380 G5)?
Overall things went relatively smoothly, we only found two kernel bugs (on 3.19) which we're still debugging and even though the load average on our servers regularly spiked above 3000, which on a 8 thread system is rather high, things didn't fail horribly.
Getting rid of those pesky spinning drives and moving to SSDs is now pretty much at the top of our list for next year as 300iops shared with 2000 containers and 20 VMs was a recipe for disastrous failure.
SSDs should get us to about 200000iops so really hoping we'll get enough sponsors to make this happen! :)
Now back to work, trying to figure out some of the LXC and kernel issues we ran into and starting planning for NorthSec 2016!
Hoping to see you there next year!