Adventures with systemd
I have just had my first real production interaction with systemd. I should warn partisans in advance that the conclusion is not clearly pro- or anti-systemd, the upshot is, there's a surprise interaction, and the means to manage it. Also the Debian community is awesome.
My issue was, for several systems, mostly VMs and cluster nodes in a test cluster that I'm setting up, the default config on Debian Jessie upgraded from Wheezy, SSH sessions do not cleanly exit when the system reboots. This is bad because my cluster-management tools, which push commands out to all the systems via ssh, will fail to exit or generate error codes which are false positives in this circumstance. Rebooting is rare, but important.
There's a Debian bug thread about this, linked below, and there are two solutions, both of which work, and I have used each in different circumstances.
The first solution is to enable "UsePAM" in your sshd config. Apparently, with UsePAM turned on and libpam-systemd installed, SSH sessions get registered in some global set of connections, and can be found and cleaned up at shut-down time. To my mind, this is a pretty strange interaction, and represents the kind new, poorly-managed global state that we were all freaking out about when we first learned about systemd. I have implemented this solution on the head node of the cluster in question, and it works.
For the client nodes of the cluster, I didn't really like this solution, it has the bad side effect of making all SSH connections display the message of the day and other status info (last log-in, etc.) that comes from working the PAM mechanism. Because client nodes do daemon-driven MPI connections, it's important that these connections be clean, there should be no status messages or other system traffic, once the SSH handshake is finished, it should go straight to the MPI traffic, otherwise MPI can get confused and behave poorly. One can in principle edit the PAM config to eliminate the superfluous messages, but you might still want them for console log-ins, and even so, you can still do it, just have the pam-ssh files not use any of the common-auth or common-session stuff, but then there's a lot of near-redundancy, it feels like the wrong scope.
Fortunately, there's the second solution -- if you disable "ssh.service" and enable "ssh.socket", you get clean session termination without requiring PAM in the sshd_config. This is what I have done on the client nodes of my cluster.
So, as I say, my first real operational issue connected with systemd, and so far so good.