Shared publicly  - 
Various people have asked me for a few words on Canonical's decision not to adopt systemd. I am not going to say much. But a few things:

From what I am hearing behind the scenes this appears to be very much about control. They control Upstart (both by being maintainer and even enforcing copyright assignment), and they think they don't control systemd. Of course, free software has never been about control, and whenever something deviates too much from where you would like it to go you have the freedom to fork. [In addition in times of git it is dead-easy to maintain your own patch set.] Similar to companies like Sun (who was too afraid to give up control about Java) it's all about control for them, and that is sad, and never good for the health of a project.

I think this decision is not good for the Linux ecosystem. Ubuntu has now become an island that is growing more or more apart from any other bigger commercial Linux. Because they have not adopted systemd they will have to continue to develop and support infrastructure (such as ConsoleKit, independent udev) that is officially orphaned by its developers and maintainers. They are stuck with a half-obsolete stack that receives no new development. Of course, Canonical could step up and invest major work in the development of their platform, but that would definitely be a first for them, and I seriously doubt they have enough knowledgeable engineers for that. Canonical contributes barely anything to the Linux plumbing layer, much the same way they stay away from the kernel. There are now two options for them: a) stay stuck forever with a half-obsolete stack or b) invest a lot of work to develop their stack entirely on their own in order to stay competitive with us.

Anyway, summary is: I wish them good luck, they are now going their own way, stuck in a legacy stack with little new engineering, becoming an island of their own. In the short time frame they might save money with this decision, in the long run this is going to be quite expensive for them, since as our stacks start to diverge more and more adopting systemd later on will get more and more expensive.

That all said: I am sure that "cloud" and "focus" (Mark's favourite buzz words) are totally going to make Upstart a piece of software of the future... ;-)

It's a sad day for the Linux ecosystem. But it makes things a lot easier for us, since we can give up on any considerations of making it easy to them and keeping their way into the systemd world open.
Colin Guthrie's profile photoBernie Innocenti's profile photoBrian White's profile photoBruno Medeiros's profile photo
I like how Shuttleworth can't use proper capitalization when writing 'systemd'.
/me wonders whether all distributions that are not going to adopt systemd and console/policy/whatever-kit are also going to use obsolete stuff and are going to end up in hell as well, or whether this was just about upstart...
Egon Rath
Why do you think that everyone has to adopt systemd as their init/cron/syslog replacement? Linux lives from the huge amount of different flavors available. If someone (Canonical this time) wants to stay with Upstart, it's their choice. Other distributions and even other unixoid systems like the BSD's are using other init types as well. IMHO it's by way more dangerous to build dependencies to systemd (i venture to remind that GNOME eventually will need systemd as a dependency in the future - correct me if i am wrong). Doing so will degrade other systems than systemd-enabled linux to second-class citizens at best.
+Egon Rath there's very few relevant distros not going the systemd route so ultimately that's where the interest lies and hence where people tend to move their development interests because it makes their lives easier
+Egon Rath It's their choice, sure, but does that mean that it cannot be criticized?
+Egon Rath also it seems that many other distros "focus" on systemd. And it's totally okay if some do not.
And the sad thing is that Mark's stated reasons aren't technical, they're all bold, bland, broad, blase bumpf. (Couldn't resist alliteration.)
+Egon Rath It's their choice, just like forking is always a choice. But who else uses upstart? Not many. Who uses systemd? Fedora, OpenSUSE, Mandriva, and Arch, Debian and Gentoo package it. More testers means more bug fixes. I know who I'd bet on. Lennart's whole point was that if they go it alone they can expect to diverge more and more from everyone else - and that doesn't help them.
I wonder if there is some form of tie up with Google to keep using Upstart. At the end of the day they are now the two main users of Upstart - Ubuntu and its derivatives including Google's own derivative, as well as ChromeOS.

The thing is that it will be exceptionally difficult for other distros to use Upstart even if they wanted to, also the converse is true as Lennart pointed out - Ubuntu users who want to use systemd wont be able to unless the re-package the plumbing components which is a shed load of work.
I don't mind if ubuntu want to keep upstart, but it does definitely not help debian adopt systemd, and that's sad.
+Jeff Waugh I know, and he publicly said that ChromeOS was going to be using Upstart. Which is why I said that it could be one of the reasons for Ubuntu to stick to using it instead of moving to systemd. Google would benefit from the Ubuntu user base using Upstart for ChromeOS.
Hasn't Ubuntu been helping with the python3 transition in Debian?
I agree Mark's systemd announcement was lousy because it didn't fully explain the reasoning. Canonical has done and may still be doing consulting for Google for ChromeOS so that may be a pressure too. And of course Debian won't be switching to systemd any time soon.

Canonical badly needs some PR help to answer these questions and make controversial positions clear. The systemd thing will definitely be discussed at UDS Oakland in two weeks so maybe that will help.
Alex Bennée
I don't get why the "Linux" community has this obsession with everybody using the same plumbing. I mean there are certainly a lot of commonalities as various distributions have settled on mature components but it's generally things that have organically grown into that place. Both upstart and systemd are pretty new in comparison to what they intend to replace and I personally see no harm if having multiple options for system init available.

For me the primary concern I have when using distributions is that I can get access to the code of those components to furtle around with them. This is true for all the distros I run now be it Ubuntu, Fedora or Gentoo.
I've heard rumors that upstart provides a nicer code base than systemd. More inline comments, better tested, better structured. Ignoring the other technical aspects, I can understand why one would not easily give up a well-maintained/well-understood code base so easily.
Are you saying that standalone udev (as in - without systemd) is becoming unsupported? because this is how it sounds like from third paragraph.
I baled on Ubuntu when they adopted upstart. I prefer to start and stop X manually, so that I have the command prompt before and after rebooting. I spent days trying to do this before giving up and switching to Arch. Arch respects the years I've invested in Linux, and I've been a happy user for the last 3 years.
+Michael Hasselmann While I haven't looked at upstart at all, the systemd code is fairly easy to understand. I can't compare the two, but I can't imagine systemd being significantly harder to understand.
See...It's stuff like this that made me want to switch from Ubuntu. Glad I finally made the switch to Fedora - I'd much rather use, and be a part of a distro that innovates, contributes upstream and is part of the larger Linux ecosystem than a distro that simply does it's own thing, and contributes little back. Ubuntu has served me well since 2005, but I started off with Red Hat 7.1 and used Fedora through Core 2 - glad I came back around - been running 17 and loving it.
As someone interested in this: thanks for stating your case.
+Jeremy Bicha I fully agree about Canonical needing someone who communicates such things instead of Mark (or at least supervises the controversial bits). The bit regarding systemd was worded in a surprisingly agressive manner and this reflects really poorly on Ubuntu. Not only in the community but also in the press...
+Andrew Wyatt ahh I forgot to check but now I look at myself I'm obviously an Ubuntu supporter so can be ignored? What makes you jump to that conclusion?

I'm not attempting to justify anybodies screw-ups, I was just pointing out that linking to a random bug comment wank-fest proves nothing. There are countless examples of arm-chair coders decrying the failure to fix bug #x on many FLOSS bug trackers across all the communities. I'm sure if I could be bothered I could find similar examples in the Fedora bug tracker where some random users sense of entitlement isn't being addressed quite fast enough for their liking.
+Alex Bennée Don't listen to the angry anti-Ubuntu folk, they are mad that Canonical is moving further away from the community. Linux, Distros, and users have all had a very close relationship in the early years of GNU/Linux. It was a friendly, to a point (vi v. emacs), warm environment and to a point it still is.

However, I would venture to say that some in the community feel forsaken by Ubuntu becoming so popular but doing what it can in earnest to uniquely identify itself in the sea of Linux distros. Especially the go it alone stance that they have taken time after time. It makes many feel betrayed that someone comes into the community and wants to take but not give back.

As you pointed out, homogeneity is not ideal. It does make things easy, and +Lennart Poettering pointed that out. As #ubuntu continues to rely on upstart, they will need to supply their own resources to upkeep that difference. That is something the Lennart feels may not be possible by #canonical given their history of leaning very heavily on the community contributions.

I use #slackware where I can and the distro still uses BSD style init scripts. I wish Canonical well in their continuance of upstart, it will allow a little more diversity to exist out there in the world of Linux. It will increase the difficulty for Ubuntu to be maintained inline with the rest of the community, but I don't see that as a problem since they are pretty much dead set on building their own stack.

Lennart's post is basically a vent about all of the "go it alones" that Canonical has given over the years, and I think (of course all of this paragraph is just conjecture) that the diatribe that Mark gave about upstart was just the final straw. Lennart's post has merit in that Canonical will find it continually difficult to keep their go it alone posture. However, the reason for going with systemd should not simply be, everyone else is doing it.

I for one thank you for your post, one solution should not rule them all.
The only two linux distributions that have the potential to make it big on the desktop scene use upstart.

Chrome OS and Ubuntu
+Scott James Remnant uh? Increasing requirements? Systemd does not require an initrd, and never did. In fact with the adoption of systemd we reintroduced support for initrd-less boots in fedora.
This is great news. Having a major distribution not drinking the systemd kool-aid will hopefully stave off its cancer-like growth long enough for a viable udev replacement to be developed. I'm no fan of Ubuntu, but the enemy of my enemy...
From Canonicals point of view their island helps them greatly. If they continue to use technology that their competitors don't then it will become more difficult for them to compete if Ubuntus standards become the default by... Default. Lets say that they do breach some market strongly and become a truly dominant player, what then? Most of their products use tech that anyone can assemble themselves and sell legally. What's to stop Oracle from slapping their name on Ubuntu, adding their own services and then undercutting Canonical on price while increasing support? All of their recent moves appear to be trying to cover up this weakness while also increasing their monetization of the software that they use.

Any strength can be or become a weakness from the right perspective. I respect and love oss and use it daily. I don't agree with many of Canonicals choices either but I am starting to see what they're trying to do.
Hmmm... expecting lemming-like conformity from anyone involved in Linux is extremely baffling to say the least. That train left the station marked OSX and Windows...
Furthermore, it seems to me like the only people who want to control what others do are the the ones pushing systemd. One man's freedom (Mark) is another man's annoyance at a lack of conformity.
+Scott James Remnant the usr merge requires an initrd iff you have /usr on a split off disk. Otherwise an initrd is not necessary. Also, systemd does not require the /usr merge, hence it doesn't require an initrd in any way, not even indirectly.
Hm. I thought it's still possible to have rest of stack working fine without systemd. Debian for example still defaults to original init. If that's not the case, we're at start of very dark times for GNU/Linux and free software in general. Especially considering growing threat from OSX.
+Aliaksey Kandratsenka I'm running udev-182 and sysvinit on a Gentoo machine with separate /usr and no initrd/initramfs, so it's still possible. With the my-way-or-the-highway attitude expressed by +Lennart Poettering I'm afraid it won't last long though.
What do you mean when you say that Canonical will have to maintain an independent udev if they choose to stay with upstart ?
Will udev have some kind of dependency on systemd ?
+Lennart Poettering and to think that +Kay Sievers wrote “After udev is merged into the systemd tree you can still build it for usage outside of systemd systems, and we will support these builds officially. In fact, we will be supporting this for a long time since it is a necessity to make initrds (which lack systemd) work properly. Distributions not wishing to adopt systemd can build udev pretty much the same way as before, however should then use the systemd tarball instead of the udev tarball and package only what is necessary of the resulting build” only three weeks ago.

How quickly do the things change.
I just love how the triple U distro is slowly forking itself into a corner...
+Scott James Remnant you are mistaken, we support /var split off just fine, without initrd. Same for /home. These splits make a lot of sense and are not cumbersome to implement, hence we support them.
I don't believe you should be all that negative about this.

Linux ecosystem has done quite well without systemd for years. Our servers and work stations (mostly running el6) do quite fine without systemd, despite I'm really looking forward to benefiting from improvements I see in Fedora now. Systemd is a fairly young project and if it proves really as valuable important as you've expressed I'm sure Ubuntu will adopt it for some future release.

Also, though I don't use Ubuntu regularly, I've tried some later releases and I'm rather pleased with the user experience they deliver and I seriously doubt they lack knowledgeable engineers. Plus their platform is already solid enough to survive a couple of releases without touching the plumbing at all and just concentrating on desktop and ux (though I'm pretty sure it won't happen).
Matthias Urlichs I think Mirosław Baran was asking essentially the same thing as me and marcin kowalski about the support of stand alone udev.
As was already said, the third paragraph sounds like udev without systemd will need to be maintained separately which is surprising considering what was said during the udev/systemd tree merge.
+Lennart Poettering Die Sonne scheint, und die Blüten blühen. Wir sollten einen Cuba Libre darauf trinken und drüber lachen.
Scott, can you point to a whitepaper or any documentation on the filesystem layout ChromeOS use. I think the configuration you are referring to is somewhat novel and I'm not sure I have a good understanding of it. I think I know the more traditional /usr is its own partition situation that I believe Lennart refers to when talking about split /usr. I want to make sure we all have a good grasp on the more complicated situation you seem to be hinting at on ChromeOS.
+Mirko Boehm don't you think that a common courtesy requires to use a language that's common to all of the thread participants?

I personally wouldn't mind having a cuba libre and good laugh too, especially when the sun flares and the flowers blossom, but why talk over the heads of other people who may not speak German?
Scott, Actually i think some more detailed specifics would indeed help understand if systemd can cover the specific case in question. Since its a unique and extremely novel arrangement, the likelihood of misconceptions about what can and cannot be achieve increases. the complications associated with /usr as a partition and what that means for access to executables in /usr/bin/ are well understood. But the situation you are describing quite obtusely is I fear not as well understood. The devils in the details and I'm not sure your providing enough details about what you are doing for anyone to make a reasonable statement about whether systemd without an initrd will work in your case. The exact details matter a weebit.
+Scott James Remnant Out of (pure!) interest, is the desire/requirement to go without an initrd/initramfs security related? Directly related to signing issues or...?
Not saying anything necesarily pro or con.. but one of the thing that systemd does is remove the requirement that a "shell" be present on a system. With that said... I'm sort of wondering if that will result in the increase of very closed "black boxes"... all running the <cough> "open" Linux kernel. Maybe it won't be any worse than it already is though... thoughts? (I mean there ARE benefits to not having to have a shell... but it may lead to... well.. you can fill in the blank...).
Thanks for making Linux into exactly the same "island" apart from every other unix that you are claiming about Ubuntu vs every other linux.
Choice quote from you on a blog by Robbie Williamson at

"Really, whatever Ubuntu does is up to Ubuntu. I will provide those who care with the right arguments, but I don’t really care enough about Ubuntu to actually try to influence your decision making process."

So, I really have to ask why you need to be unprofessional about Canonical/Ubuntu's decision to stick with Upstart, when it was already outlined here.

Believe me, I am far from being an advocate of Canonical/Ubuntu right now, but I am also not going to promote your opinions either. You wrote a tool, they wrote a tool. VI/Emacs, Gnome/Unity/KDE, MySQL/PostgreSQL, WHO CARES!!!

Multiple developers make similar tools all the time. It is ultimately up to the users to determine who's tool is better.
+Scott James Remnant Note that systemd does work when /usr is split off exactly to the same level as any other init system. We are not special there. We do not make stronger requirements than Upstart does, or sysvinit does. The makefile carefully makes sure to distribute the tools we need between / and /usr, to support that, even though we don't use that on Fedora or so.
+Måns Rullgård So you have a separate /usr and no initram fs? That's great. This also means that you must have installed "mount" and various other tools (perhaps lvm2, raid etc) into your root filesystem. Therefore if you wanted to provide your /usr on a server and share it over NFS to a data centre, each separate machine would have to have a root filesystem that included such utilities. This approach, while technically possible, has many disadvantages. The /usr merge is a move to ensure that all packaged binaries and libs live in a filesystem that can and should be kept separate and (if security is a concern) mounted read only.

The point +Lennart Poettering, +Harald Hoyer and +Kay Sievers have all been making about the /usr merge is primarily one of practicalities and usefulness, not of impossibilities. It makes sense to try and get all binaries into the /usr tree. It makes sense to use an initrd that can be easily regenerated and deployed over PXE for datacenters. This doesn't mean that +systemd will not work with other setups, and comments like yours (and I only use it as an example) seems suggest people getting caught up in rhetoric rather than actual facts.

I can see why +Scott James Remnant might want /usr to be kept separate and in some way trusted/signed and why he wants to avoid an initramfs, but that still requires certain things to remain outside of /usr in order to mount it. This is the same fundamental reasons for using an initramfs. The tools needed to mount /usr need to be somewhere that is already mounted. How are these tools trusted - is there some separate trust mechanism for them? If there is then fine, if not, then the whole concept is build upon foundations of sand. (I've genuinely no knowledge of this, so this is not a flame comment, just an observation)

But really, the difference between an initramfs containing all the necessary foo to mount /usr and / containing all the necessary foo to mount /usr is not all that different. People seem to get stuck in the headlines here rather than actually understanding the problems and the different tradeoffs of both solutions.
+Colin Guthrie FOR ME using an initramfs is a hell of a lot more work than keeping the tools required to mount /usr in /sbin. For deploying to hundreds of machines, an initramfs might be simpler, but I only have a handful, most of which couldn't share an initramfs anyway.

Linux systems used to pride themselves over offering the user choice. Now these choices are being removed one by one, seemingly for no other purpose than to enlarge +Lennart Poettering's personal playground.

It is already possible to use initramfs/initrd. It is already possible to put everything in /usr. Those who want/need these features already have the option. There is no need to force it on everybody else.
+Måns Rullgård you still seem to be missing the point. As +Lennart Poettering has stated several time it is is still just as possible as before to have /usr on a separate partition and mount it without an initramfs. You are no less supported than before! Please read the comments. You can still have a separate /usr if you like. You just have to make sure you have all the prerequisite boxes ticked fro that to work - the same as always. I guess +Lennart Poettering's mistake is being vocal about the problems actually are - people were maybe just blissfully ignorant before?

So I'm not sure where you get the feeling that your choices are being removed. Sure there are comments and articles that state that a separate /usr not mounted by an initramfs is "broken by design", but this is really somewhat subjective. The reasoning is basically due to the fact that tools and libs that should, by rights, normally be kept in /usr have to move to / to support a separate /usr. If this is something you and the team who packages your distro are happy to jump through hoops to achieve that goal, then that's fine. If they no longer want to support a myriad of complicated setups (and as someone involved at a distro level, I can very much see the desire to not support all combinations of RAID, LVM, and encryption and disk configurations from a purely practical point of view), then that's their decision.

I, for one, welcome the fact that several distros are helping each other out (I've received a lot of help from Suse, Arch and Fedora folk and I like to think I've given at least something back to them too) when working on the plumbing layers. Declaring that their distro packaging will not support certain crazy combinations is perfectly reasonable to me. Users are, as always, welcome to deviate from that path, but they have to understand that they'll have to solve problems on their own. There is nothing wrong with that and I really don't see why anyone has a problem with it or sees it as removing choice. Just don't expect me, as a volunteer distro packager and developer to spend days of my life developing and testing something for your specific setup/layout that I have no personal interest in ever using. Do that yourself and if it's a sensible, mergable patch, I'll merge it. That's choice and it is, as always, your choice.
You are missing the fact the user being just as free to deviate from a distribution does not address the distribution changing in a way that makes deviation harder. I prefer a design that lends itself to flexibility, over one that merely technically allows it at a great cost.

I also think systemd violates seperation of concerns. It shouldn't do crons job any more than any other process that just happens to run all the time should. I don't want a half baked cron-lite built in to something else as my cron.

This assembly of somewhat related tasks into one program that does the various tasks in a more run-time efficient way (modules in one daemon vs multible daemons) and more structured and manageable way (config files that can only define a finite number of things instead of scripts that can do literally anything) is well suited to appliances (phones, set top boxes, game consoles, etc...) NOT the default way to operate a general purpose server platform.

Still no ExecStatus? I understand why not, because you can't. It doesn't make any sense in a system that's designed to react to events. You can't have an ExecStatus=myscript because you'd have to run it every few seconds forever, polling to emulate the event triggering the rest of systemd lives by. But, I need init scripts that are readable, editable, and written in the ubiquitous and easy language that's already known by everyone and fully flexible, not in yet another special config file that's only used by one program and only does a finite number of possible things. You can't manage a shell script fully because it might do anything, but, too bad, I need to be able to do anything.
Some folks commenting here are so confused... and it's not me...
^^ Maybe me ^^ I didnt follow the whole discussion, but is the decicion of Mr. Shuttleworth and Canonical really so bad!?
+Colin Guthrie I'm afraid I'm missing why it makes so much sense for everybody. Having / as just mount point works if you have an early boot system taking care of what the current / does, so you are back to square one for most purposes...

So basically shoveling "/usr contains everything" down everybody that do not care is just spite.
+Scott James Remnant :-D Meanwhile, do you feel that ChromeOS can forego the initrd because it's shipped on known hardware, or is a whole stack of hardware compat stuff built into the kernel image itself?
+Luca Barbato OK, so I'm afraid I don't really buy the "spite" argument. I mean spite is a strong driving force but I'm pretty certain it's not the driver here :p

Let's backup for a second and think about this logically. You, as a user, sysadmin, whatever, decide that you need a separate /usr. Why is this in the first place? I can think of two reasonably good arguments for doing this: 1) Because I'm building a datacentre and I want all my machines to share the same /usr saving me sysadmin overheader, or 2) I'm a user who cares about security and I want my /usr mounted readonly unless I am doing my upgrades (maybe with a few hash checks on top for good measure). There may be other reasons too of course, but in most cases on a typical laptop or single server a separate /usr is not needed.

Now let's consider the case of not using an initrd. You might have an ext4 /usr. That's fairly easy, all you need is mount and maybe a handful of other binaries (fsck) and libs in / for that. OK, so we move those that should, by rights, be in /usr into / to deal with that and you're happy. But now another user has an LVM /usr, so we have to move all the LVM stuff into / too, (lvm2, vg*, lv* binaries) but because these tools link against ncurses, we also have to move libncurses and libncursesw into / too. OK, that's now done. But wait! Someone has RAID /usr, so let's move mdadm, let's move cryptsetup, let's move all the other stuff that's needed for the plethora of filesystem setups there too!

OK, so now everyone can get their boot, but what about the two previous scenarios that were outlined for having a separate /usr in the first place? 1) Server install sharing /usr. OK, so now this is basically completely broken. It just doens't work any more, I have no mount, no libncurses, etc. so I have to share the root FS too. This is certainly not the neat an manageable system I'd hoped for. 2) The security concious user now has a much harder job as he cannot simply mount his encrypted /usr readonly and avoid certain problems, he has to now implement systems to verify the mount and cryptsetup binaries and all the libs in / too if he wants to audit his security.

So, fine, believe it is spite if you like. I'd rather make such decisions based on technical considerations.

So, if a distro says "we'll only support a separate /usr when the initramfs deals with the job of mounting it before init" I think, "that's great" because it actually allows me to do even more crazy things with my setup because I can tweak the initramfs a lot more easily than I can the running system to move binaries and libraries around as I need them for my zany ways.

That same distro might quite happily support a initramfs free boot if /usr is not split off. The two are not mutually exclusive.

And, just to bring this back to a topic of systemd, you can still do a separate /usr with systemd without and initramfs is that floats your boat. In ChromeOS, the example du jour, this would be fine. They manage their split of binaries between / and /usr and they control it as they see fit. That, in itself, is not an argument against systemd.

So, no, it's not spite.
Where's the -1 button when one needs it?

Lennart, you make it sound like systemd is the de-facto standard in the Linux world, while in fact it's been adopted only by Fedora, followed by an unbelievable amount of criticism from unhappy users.
Suse is in the process of adopting it, but it's not a user-driven change at all. Just a few developers/maintainers are pushing the change, and a very few fan-boy users who just cheer whatever the devs do without really having an opinion of their own that's based on any technical arguments, and most users who say anything on the topic at all, hate it. For now systemd works ok on suse for the utterly typical desktop most of the time, and server admins know how to manually revert to sysv since both packages are still packaged and supported for this release, and regular users that have problems are generally told by other users how to switch back to sysv and that always fixes the problem. The real problems will only come when they actually remove the sysv-init package from the install sources. In any event, what suse does should not be a supporting argument for systemd's viability since suse has gone down the crapper getting worse and worse the last few releases. It's no longer a distro to hold up as a shining example of excellence. systemd should not exactly be proud that suse is adopting systemd.

Also minor addition to the list of os's that use upstart, webOS uses upstart.
Add a comment...