Shared publicly  - 
To clarify a few things regarding that "debug" mess:

I will change systemd to not log to kmsg if journald is around when "debug" is specified. We will continue to log to kmsg during the early bootphase however als long as journald is not running yet. At that point there is simply no other option for that, because persistent storage is not available yet at that point and we need som way for people to get their logs out of the system, and early kmsg is the only way really, since it is hooked up with netconsole, serial tty, crashkernels, ... and all that other stuff.

I also believe that it is the right thing for the kernel to ratelimit whatever userspace sends it. For the same reason as we have to ratelimit all log streams in journald too, regardless where they come from. If you get a stream of logs from a 'differently priviliged' component then you need to rate limit it, that should be common sense. Of course I wish the ratelimiting applied is always configurable for the user, so that he can control how much and how soon things get dropped.

For me it is out of question though that systemd and other core os components should continue to parse the 'debug' kernel cmdline option, and increase their debug levels then. Generic options like that are supposed to be useful for real people, and there's a long history of options like that which influence both kernel and userspace (quiet, root=, ...). We are putting together an OS here after all, not just a kernel, and a kernel is just one component of the OS among many, and ultimately an implementation detail. We are writing an OS here for the general purpose, not just a toy for a clique of kernel developers. Moreover, there are individual kernel cmdline options for both the kernel itself and systemd to control just their log levels, and nothing else. so if you want finegrained control, you already have it, 'debug' is just the simple option that groups them all under a single oneshot option. It's the option an admin can specify which tells him why the system doesnt boot, regardless whether the kernel or systemd is at fault or any other part of the core os involved in boot. Thats simply userfriendly.

I do not appreciate all that talk from certain parts of the community that the systemd folks need to 'get tought a lesson'. That turns this into some kind of power game, which I am totally not interested in. Comparing us to Putin, or trying to bind merging of completely unrelated code to us doing whatever some kernel devs want (which is blackmail at worst, and childish at best, and childish we can be on our own enough, thanks) will just make us ignore you, and that's it. I prefer to deal with technical questions, and intimidation is not a technical question.

what the various news outlets wrote about all this is quite ridiculous. It's sad how badly these sites report, quite a shame.

And that is all.
Florian Schmidt's profile photoLuckis K Pac's profile photoCameron Wood's profile photoSteve Brown's profile photo
Wow. What is the big deal with systemd, anyway? Did somebody sign a contract that commits them to using systemd? Personally, I would drop it like a hot potato given the attitude of "I don't care what you think, I'm going to do whatever the hell I want."
Seems like having a systemd.debug option that takes precedence over the debug option would be a simple solution. You could then have one easy flag, but with a way to override it. 
Ok, so exactly what was the problem with "systemd.debug" again?

I guess this does mean that we have to apply my patch to rate-limit messages into the kernel. Fine, I don't hate that patch, but I do hate the fact that systemd seems to think "we're being ass-holes, and it's not our problem, others should protect themselves against our f*cked up aways".

So this really really doesn't make me want to ever work with Kay Sievers, because this all just reinforces the fact that he just doesn't care if his changes cause other projects pain.

In the kernel, we have that "no regressions" rule for a damn good reason. For example, it is not a valid excuse to say "well, user space shouldn't have done that to begin with, so now we can break it".

But Kay seems to think that breaking other peoples workflow and uses is fine. And is totally unapologetic about it, and closes bug reports when it happens, and dismisses the obvious and reasonable fix.

+Greg Kroah-Hartman: I really hoped that you could push the trivial and obvious one-liner through. What's going on here?  Instead you are spreading Kay's unapologetic crap.
+Linus Torvalds The bug that caused the repeated assertions in Borislav's case was fixed in systemd -git for some time (per  David Herrmann in systemd ml).  Borislav was provided the flags to disable the logging for his case.

Please re-read just the first three comments of , before the opinions started being discussed.
+Ken MacLeod .. and I repeat: what's wrong with "systemd.debug"?

Seriously, what about next time somebody has a problem? WHAT THE HELL IS THE ADVANTAGE OF SYSTEMD STEPPING ON KERNEL OPTIONS?

The very bugzilla you point to, Kay blithely just says ""debug" can cause "too many" messages to be useful anymore if things are broken". And leaves it at that. He doesn't care about other projects, which was my very point.

Would you want to work with a person who makes it very clear (over and over again) that he doesn't care if he causes you pain? 

From reading this it seems like lennart thinks that the kernel is just an implementation detail. Very well, I think systemd should go find another kernel to run over.
+Linus Torvalds why systemd.debug? Make it generic and name it init.debug! Systemd is just one init daemon among others. In five years year or in the embedded world another daemon will implement init daemon functionality.
addendum: if I want to debug the kernel "debug" is fine. For the init process - because something strange happened in the init phase "init.debug" is fine. For all other daemons/processes the particular .config must be edited (or whatever). This sounds like the way to go - it is consistent and clear. This is what the user expect. 
+Ken MacLeod +Hagen Paul Pfeifer If this sort of productive refinement of detail discussion was how things went on the bug, I doubt there would have been an explosion over the matter.  Instead it went more along the lines of "put up with it..."

Nobody is going to be upset about the exact naming of the config field.
Tim Bird
Sighh.  I thought Lennart would "get it", but apparently not.  He hasn't backed off one bit.  I have been avoiding systemd in my embedded work, for mostly technical reasons.  Now I have non-technical reasons (distrust of the systemd maintainers judgement) as a reason to avoid it as well.
+Ken MacLeod I also tried to get Borislav to compile and test systemd-git. (edit: got an email from Borislav - they are working ot it.) Anyway, the problem should be gone in systemd-git, there is a known workaround, and if we really fear about the issue returning (from systemd or elsewhere) then ratelimiting in the kernel seems like the right solution. This is a storm in a teacup.
Throwing in my two cents from user perspective I like systemd. So I'm interested in having an opportunity to configure things the way like Lennart told in his post here. And I don't see any reason to discuss this issue at this personal level like it happened in the last days. This confuses me and I think others as well and I'm thinking than it might better stepping back from thoughts about 'which inin system I should use in future' as there seems to be a big personal mess between all this absolutely brilliant minds working on kernel and stuff nearby. IMHO
No, +Thomas Andersen , the proper solution is for Lennart and Kay to start LISTENING to Linus, and STOP using the kernel's "debug" as their own debug switch.

You know how you can get a chick mad at you for doing something wrong because you didn't listen to her? The solution isn't to do the right thing -- that won't make her happy -- the solution is to listen to her!
As an enduser I'd expect that debug turns on all debugging there is. If I already know where the problem is, I could use systemd.debug or kernel.debug.

As a developer I understand that it's a pain if you're  used to just set the debug flag and you were fine.
+Linus Torvalds it's not just a kernel option. The fact that it is being handed to userland is indication enough. It's a generic signal if it's being handed to userland. Deal with it ;-). If you want kernel-only, why not call it kernel.debug?
Well Greg...I understand Linux frustration with Kay and a decision needs to be made because if He keeps making a mess for many that shouldn't be allowed.  Now, you all guys willing to help fix the issue come together and do something about it...and don't need Kay anymore.  It's difficult to deal with people like that. Instead of helping they only creates more trouble. Peace. Thanks for everything you do for Open Source Community. Thanks
+Linus Torvalds
Other projects are using the "debug" from /proc/cmdline as well, e.g. Debian live-helper/live-boot. The only difference is that it is not using /dev/kmsg at the same time. It would be good to get a clear statement if projects are allowed to use "debug" as a system wide flag or if it should be interpreted as a kernel-only argument.
+Jan Wildeboer It's an option in the kernel command line, that is very well documented in kernel docs to be the kernel-only parameter to enable kernel-only debug messages. Also, it's widely used and described by anyone working with kernel.

The fact that it's exported to the userspace only indicates that kernel devs are kind enough to export the user as much information as possible. There are tens of really kernel-specific options handed to userspace too, but they still remain kernel-only options.

So you're wrong, debug is just a kernel option. Deal with it :).

Also, there are already perfect example of what people do to avoid spamming the kmsg - see fsck's logsave(8), or dracut's rd.debug instead of debug, which are all perfectly fine and used at depth for years.

logsave(8) gives the possibility to save logs for some time without spamming the kmsg (and thus breaking people's scripts and habbits), so, if systemd decides to use it - it's free to parse the debug option - and journalctl/dmesg will show different logs (systemd and kernel, respectively).

"rd.debug" approach (systemd.debug, which Linus is talking about), OTOH, also makes everyone happy - "debug" remains what it used to be, without introducing regressions in workflows and scripts, and if someone wants to debug systemd too - one can add "systemd.debug debug".

So perfectly sane alternatives exist, and these alternatives exist for a long time and had been proven to work flawlessly.

Till the end, if someone indeed wants a "one parameter to do it all", there should be introduced a new one. Say, "all_debug_on". A small patch can be submitted upstream to make it an alias of "debug", and all the userspace can start parsing it and output its logs to kmsg.
+Thomas Andersen Correct. I don't mind people piggy-backing on some fairly obvious generic term like "debug" per se. I don't know if the old init scripts did that, but I do know they did it for "quiet", which is basically the reverse of "debug".

What I mind is people closing bugs and not admitting mistakes. If Kay had even said "sorry, the excessive output was a bug in systemd, it's already fixed in current -git", that would have been a valid reason to close the bug.

And for the people thinking this is a storm in a teacup: this is not the first time Kay has done this, which is why I personally get so frustrated.

Kay has done the exact same thing with major bugs that were not fixed anywhere else, and that caused machines to fail at boot time, and Kay happily pointed the finger elsewhere for months at a time and closed bugzilla entries. 
Side explanatory note: and it's because of that known history of abusive behavior that I would prefer systemd now use "systemd.debug".

The old init scripts may or may not have parsed "debug", but we never had any reason to care. Now we do, and people are (I think) understandably upset that systemd not only screwed up, but then the people involved weren't even willing to say "sorry" about it but instead go "uh, it wasn't our bug, deal with it".
My most used "kernel" option is 1 to force my system to go to run level 1. From my user/SysAdmin perspective I don't care what part of the commands are kernel reserved or not. So a clean solution would be to have a system option and a kernel option. ATM that difference is kind of obfuscated because the mechanism is the same.

Now the fact that systemd when it sees debug starts sending stuff to kmsg is an interesting way to work with generic options. It shows that kernel options can be overloaded for non-kernel use and that is normally OK because the kernel isn't being (ab)used.

The way Kay handled this "bug" is one thing. The way Boris went nuclear is another. Lennart tries to find an acceptable solution and we should all cool down and work on exactly that. 
I agree with you on the options part, though I'd reword the "interesting way to work" to something more negative, as it breaks a long-present workflow scenarios and scripts. Basically, it's introducing a regression in people's workflow.

On the acceptable solution - Lennart proposes a way to fix something, but not the original problem (I'm not referring to how it was handled) - the original one being that it's a regression in lots of people's workflows. He even states it openly in his message - "We are putting together an OS here after all, not just a kernel, and a kernel is just one component of the OS among many, and ultimately an implementation detail. We are writing an OS here for the general purpose, not just a toy for a clique of kernel developers." - so, basically, "we will break your workflow because some other people want it".
Boris went "nuclear" because Kay wasn't handling the bug. Kay basically said, 'not my problem, wont fix, closed!". That pisses people off. It shows what Linus has been stating. He doesn't give a shit about you.

I even mentioned in other threads that if Kay had simply said, "Oh, systemd is spamming kmsg, because of a bug in systemd, go download the latest", we all would have been happy and done what he asked. But no, instead of saying there was a bug in systemd, he just said Boris's system is broken, he needs to deal with it.

And yes, this isn't a technical issue. But non-technical issues do affect technical ones. When you can not trust the developer of code you depend on, the technical solution is to find someone else you can trust.
I'm not trying to "take sides" here at all, only forward on what people are saying about it (i.e. this was +Lennart Poettering 's statement on the issue, not +Kay Sievers 's statement) so that everyone can talk about it in a semi-rational manner.

The facts are:
  - the original bug that was causing the syslog spew to stop the box has been fixed in systemd.  Why that wasn't mentioned in the bug report, I really don't know.  If that hasn't been backported to your favorite distro, then go poke the maintainer of the systemd package for that distro, the upstream developers can't do much about that anymore.
  - I've posted a patch, with some later fixes by +Hannes Reinecke to move the systemd "debug" flag to the "systemd" namespace instead of relying on the "generic" "debug" flag name.  That should, in my opinion, make it easier for the two different groups to do development without being bothered by log messages in areas they don't care about.  If that patch is accepted or not, that's up to the systemd developers.
  - The rate-limiting kernel log patch is good, it prevents userspace programs from DoSing the kernel, always a good thing to prevent, and have.

So we have 1 bug that was already fixed, and 2 patches created out of all of this.  A rather productive result in my opinion in the end.  We've had worse outcomes for much crazier arguments in the past.  We're growing up!

As for different people's personalities, I'm friends with everyone involved here and don't really appreciate being put in the middle of it for something that I (for once) had nothing to do with.  Everyone involved here should be adult enough to handle it on their own without needing to drag "Dad" (i.e me) into it.

Here's a nice note for all the reporters who are emailing me, go suck it. I'm deleting all of your emails, as trolling for stuff like this just to write sensational stories about it, is just as bad as chasing after actors trying to enjoy a vacation and taking pictures of them on the beach. Yes it's titillating, and drives page views, but really, is that the best use of an Liberal Arts degree?
+Linus Torvalds Calling other peoples behavior "abusive" seems to be just a tiny bit hypocritic. It seems you have engaged in such behavior quite a bit.
+Andreas Tunek I've worked several years with +Linus Torvalds and although people call his yelling at kernel developers "abusive", that's not what Linus himself is talking about. Note, I've been on the other end of Linus's "abusive" talk, and honestly, I was in the wrong. Some might say Linus goes off a bit too much, but it really didn't bother me. He let me know exactly where I screwed up and I worked to fix it.

What Linus is calling abusive behavior is the way systemd doesn't listen to its users. In fact, where you see Linus act with abusive behavior is whenever you see a kernel developer acting like the systemd developers. That is, they ignore a bug report or say its not their bug. The famous "SHUT THE F*CK UP, Mauro" that was the poster child of another public blow up was exactly because Mauro wasn't owning up to the bug in his code. Ironically, the user app that broke was PulseAudio (Lennart's creation).

The reason the Linux kernel is such a high quality project compared to its size and speed of development is because Linus keeps a tight rein on issues. Any bug is a big deal. If the Linux kernel started treating bug reports like the systemd folks have, then the quality of the kernel would fall like a whale that suddenly appeared in the sky.
+Kurt von Finck lol, the famous "Whale in the Sky" chorus:

 "Whale in the sky keeps on turnin'
  I don't know where I'll be tomorrow
  Whale in the sky keeps on turnin'"
+Steven Rostedt
If calling someone, who has not coded for the kernel Linux for a year, "a f*cking primadonna" is not abusive, I do not know what is. Maybe you know?

And is systemd not allowed to use the debug variable? What exactly is the "primadonna" doing wrong? Wasn't the bug that systemd logged to many messages for Linux to handle fixed in systemd?
So as strictly an end user of all things kernel related, it seems like the outcome is a bug got fixed and some patches were generated. Probably time to move on? I'm not trying to silence anyone ... but, you know ...


And I don't know what bugs land tomorrow.
+Andreas Tunek did you even read what I wrote?

Yeah, Linus swears and calls people names. That's not the type of abuse he was talking about. He swears for a reason, and that's when people ignore fixing their shit that breaks other peoples projects.

I rather be called a f*cking primadonna than to depend on code that causes me to keep writing work-arounds for my code. That's an abuse at a totally different level.
+Steven Rostedt
I read everything you wrote.

The bug (systemd logged to much to a Linux logging system) was fixed. All the other drama IS about Torvalds being abusive. If you like that kind of environment, good for you! (The kernel) Linux is Torvald's project and he can decide what goes there.

Calling systemd developers names is abusive behavior. But complaining that developers on other projects are abusive is hypocricy.

"I rather be called a f*cking primadonna than to depend on code that causes me to keep writing work-arounds for my code. That's an abuse at a totally different level."

Linux does not depend on systemd. Systemd depends on Linux. There does not have to be anything in Linux to workaround a systemd problem. Systemd might expose strange behavior in Linux (through a bug in systemd in this particular case) but if Torvalds does not think they are bugs he has every right to not do anything about them.
Very simple point here - regardless of what happens elsewhere, the core has to be robust. You can't just blame others for causing a denial of service, when you can and should prevent the system from being broken.

Yes, everyone has to do the right thing. And if the kernel throws away excessive messages that should never have been sent to it, then if others want all their messages, they have to change their ways.
andreas Tunak
Some one has to give reality check and KS doesn't seem to get the message in any reasonable way and Linus has to  step in multiple times to get the bugs fixed. If linus is being abusive why are u  finding fault in linus instead of finding fault in KS since he is the one making Linus angry. Bugs are always going to be there for most of the software written and not recognizing the bugs after being told multiple times is the worst one developer can do.

Don't do the crime if you don't want to do the time.
+Linus Torvalds +Steven Rostedt I'm following up on +Greg Kroah-Hartman's patch from the systemd side. So far it is not getting applied as it is still not clear what problem it is solving. As you seem to know what is going on, I'd appreciate some feedback.

My understanding is that there was at some point a bug in systemd causing too much debug output (and there may also be a kernel issue with it not handling unlimited amount of debug output). Let me stress that this problem was (of course) taken seriously and no excuses were made for it. It has been fixed upstream for a long time. It also seems like the kernel side is being worked on, so that's good too.

There may have been a misunderstanding in the recent bug report, where Kay was defending the usage of "debug", not the existence of past bugs. I'm sure this could have been communicated more clearly, but now that it has been cleared up repeatedly hopefully we can get past that.

It seems also clear to me that the usage of "debug" in itself, to give reasonable debug output from systemd, is not a problem, and we are using the kernel interface as intended. If our debug messages could be improved, I am very happy to work on that, or to take patches.

There is of course the possibility that bugs in systemd may cause excessive debug output in the future. If so, we will of course take such issues seriously and solve them as a matter of urgency. However, the mere possibility of this happening, does not seem like justification for not using "debug" at all. I assume this is what Kay was referring to when saying that you can end up with too much debug output if something goes wrong.

So in short, I really don't see what the problem is here, and I'm so far putting it all down to some miscommunication.
Why not mount a tmpfs at the very beginning, dump the log there, and use pivot_root to get rid of it later when journald is ready to run?
+Piruthiviraj Natarajan

So Torvalds goes to all kinds of different projects that runs on Linux and steps in to get bugs fixed? KS has no way to make any impact on Linux unless Torvalds let him impact Linux.

Once again, systemd is a program that runs on Linux. Nobody has to run it if they want to run Linux. Calling a systemd developer names because systemd does something that Torvalds does not like is abusive.

Also, there is nothing systemd can to that is abusive to Linux because systemd depends on Linux, not the other way around. Systemd has used Linux interfaces in a bad way (logged to much), but that bug has been fixed in systemd. What else do you want the systemd developers to fix?
+Tom Gundersen I think the concern is more with attitude than behavior.  Stuff like, "That is the expected current behaviour, 'debug' can cause 'too many' messages to be useful anymore if things are broken," and then closing the bug.  Little effort was made to understand the concerns of the reporter - the assumption was just made that he must be wrong.

Linus doesn't want the maintainers of kernel code treating bug reports in this manner, so he doesn't want Kay to maintain any kernel code until the pattern changes.  He doesn't want to go merging in kdbus and then have to deal with drama when Kay explains to 14 other projects that they've all been doing things wrong all these years with userspace dbus.

I do respect that Kay/Lennart have done a great deal to advance the whole Linux experience.  Maybe they just need somebody to volunteer to do PR for them and give all the bugs a first-pass before they look at them.

And there will be disagreements from time to time.  The whole ext4 ordered data debate comes to mind, and the reason people use Linus's kernel is because for the most part he tends to make pretty sane decisions.  I'd just view him as more of a "filter" than a "manager" in the typical sense, but that might be unfair since I'm looking more at git/lkml/etc and I have no idea what other communications happen between the senior kernel maintainers/etc.
+Tom Gundersen I think the main issue is attitude to bug reports. I've mentioned previously if Kay had simply had said, "Oh sorry, but that excessive output was caused by a bug in systemd, please upgrade" nothing would have escalated. But instead he said:

" That is the expected current behaviour, "debug" can cause "too many" messages to be useful anymore if things are broken."

And basically defended systemd's abuse of debug.

I'm fine, and I believe that +Linus Torvalds is fine if systemd does a little more if "debug" is found on the kernel command line. The problem we have is that it can do a lot more, and when someone complained about it, they said tough. That shows that systemd can not be trusted with the use of "debug".

Note, other "Base OS" tools also use "debug", but they do not use that option to write back into the kernel. systemd is unique in that aspect, and that can cause issues when you are trying to debug the kernel.

How's this for a compromise? Perhaps when a user just types "debug" systemd can write as much as it wants, but it buffers its writes until journald comes online and then it dumps to that. But if you want to use kmsg, then the user must use "systemd.debug" on the kernel command line. That way dmesg doesn't get abused by systemd just because "debug" was added to the command line?

+Andreas Tunek nobody forces us to run systemd, and if the systemd developers wants wide adaption of their code, then they had better start listening to complaints of their code. Otherwise people will find something else to use.
+Tim Bird Why do you mention Lennart in this context? His name doesn't even show up in the bug report. Linus is criticizing Kay's behavior, don't always assume that everything systemd is Lennart. Lennart is actually trying to resolve the situation and I think his proposal sounds reasonable.
+Steven Rostedt thanks for getting back to me. I think the issue in the bug report was a simple miscommunication. Would be nice to avoid those of course, but I guess that will always happen. In the future, I guess the best way to solve a bug which one feels is not given the appropriate attention is to take it to the mailing list, as that has a lot more eyes on it.

My understanding is that Lennart is working on passing less data through kmsg, so maybe that will already solve all concerns. I'll pass on your suggestion though.
+Linus Torvalds is free to curse out Kay, give him the finger, whatever. That doesn't bother me at all (and it doesn't seem to bother Kay either, since last heard from he was downtown slurping noodles, not home sobbing into his pillow).

The thing that does bother me is that kdbus is now barred from the kernel on non-technical grounds. As I understand it, kdbus is an improvement that the guys on the automotive side really want. So this hurts users, not Kay. And not just any users, but a class of users where improvements could, literally, save lives.

Even if Kay is all Linus says he is, wouldn't it make more sense to appoint a minder to review any "WONTFIX" or aged bug reports, or bug reports with more than x entries instead?
+Chris Atkinson Linus didn't ban kdbus, he banned contributions from Kay.  If somebody else steps up and maintains kdbus I'm sure he'd be happy to accept their commits.  He wasn't banned for life either - he was told to get his act together.

Linus isn't really paid to manage individual kernel contributors - he is more of a gatekeeper.  Since Kay apparently works for Red Hat there is presumably somebody who is paid to manage him full time - an enviable post to be sure.
Can we have the flag for init system debug verbosity be init.debug? As it has been mentioned already, more than just systemd has reacted to the debug flag. It would be useful to get the verbosity regardless of whether you have assumed systemd is the init on the box.
Does this mean we are fixing something that was never broke . Tired of reinventing the wheel. KISS Tired of systemd
+Rich Freeman I'm not sure that's how I read what Linus wrote to Greg K-H: "This is relevant to you [i.e., Greg] because I have seen you talk about the kdbus patches, and this is a heads-up that you need to keep them separate from other work."

To me that means that even if Greg (and presumably anyone else) were willing to commit and maintain kdbus, LInus won't merge it. That seems more like banning code than people.
Just forget about all this nonsense. Use Slackware or Gentoo.
+Kenneth Harrison "systemd isn't being taken seriously" is certainly a stretch.  Virtually every major distro supports it, and many are intending to make it the default.  Certainly it is fairly well-supported on Gentoo, though nobody is going to stop anybody from maintaining eudev.
+Linus Torvalds should a) thank the systemd developers for exposing this weakness in his kernel and b) rate-limit. At the same time, systemd developers should have closed the bug as 'obsolete' or 'already fixed' rather than 'won't fix' or 'not a bug'.
Watching kernel developers vs. systemd developers is like celebrity deathmatch, not very smart but quite entertaining. :)
After the success of Git, Linus Torvalds should start his own init system project. Something in the line of "system initialization done right".
I don't know how valid my comments are, and I'm pretty sure i've misssed some serious fundimentals of this 'arguement'/'misunderstanding'  -- But I'm going to add my 2 pence anyway

Where I work, if Project B, relies on functionality from Project A, and Project B builds some new stuff that 'confuses' Project A, that is forces Project A into a work around, I got a kick Project B's ass! We don't care that it's works/is easier/is cool/awesome/can blame someone else ... it's quite simply against 'good etiquette' because now, Project B is pushing 'technical debt' onto Project A -- and Project  A is relied on by Project B - Z.

In a nut shell, if the Kernel says 'NO' then screw you Project B, don't cause extra pain -- submit a patch to Project A, or request assistance via 'back channels' not in the public space - In my eyes, the kernel will 'always' take precedence, simply because more systems rely on it. Having said all that - I'm not a developer/coder for anything distro related, so it's very possible, i've completely misunderstood this issue - but in my world, if your app breaks my infrastructure, your app gets ripped out by it's hair, until it behaves properly, i don't see why this is a problem here :)
Nobody is forcing anybody to run systemd.  Granted, on most distros you probably won't be given a choice about the matter either way, but nobody said you had to run that distro.  Other distros are likely to give you a choice, and certainly Gentoo will be one of them.  However, I doubt eudev will ever become mainstream.  It will probably be a while before systemd is entertained as the recommended default, but at this point you can set up a Gentoo install using systemd with very little effort (just a matter of picking the right kernel options, setting a USE flag, and pointing the boot line at it).  Hard dependencies on OpenRC are being removed, and there will probably be a profile that drops it and sysvinit entirely in the fairly near future and stage3s which eliminate the need to set up anything to use systemd.  Most likely both will exist in parallel for a while then.

The biggest issue I have with systemd is that the interfaces/etc are all unstable making it hard for anybody to keep up.  But, sooner or later the project will mature (I can't see RedHat sustaining that rate of change once it ships in RHEL).  They really have done a lot to advance the state of the art and the whole socket-level dependency management concept was a paradigm shift.  That is what makes some of the attitude issues frustrating.  There is a lot to be said for getting kdbus into the kernel - imagine being able to control what processes are allowed to talk to other processes via SELinux roles, or better still to allow them to send some types of messages but not others!  In some ways it might do for userspace APIs what capabilities are doing for the system call interface.

Nobody is going to force anybody to use any of this stuff.  However, I think that just mocking the folks doing the work is foolish.  I think Linus is right to call them out on attitude (and to be honest more than a few have called him out on his own demeanor).  However, you don't really see him knocking the basic design of systemd, and there is a reason that just about everybody is starting to move in that direction.  Sure, if you're just running one desktop plugged into a LAN whose hardware rarely changes you can get away without most of this stuff, but then again setting IRQ/DMAs with jumpers on ISA cards wasn't all that hard in such a situation either.
time to clean up this thread by deleting all of the stupid responses, and just closing it for comments, you all suck.