Shared publicly  - 
 
Writing a "what do you think of this proposal" type email to linux-kernel or on my blog on a Friday evening, isn't the best way to get good feedback. But, let's try an experiment here, how about I post my first thoughts here, see if I get any feedback, and then on Monday post it to a larger audience based on the response?

I'm giving a talk about this topic at LinuxCon in Vancouver next week, and have had a number of different conversations about this with different groups lately, so that is why I am thinking about it at the moment:

Latest version of this text can be found at
https://raw.github.com/gregkh/stable-presentation/master/longterm-future.txt

---------------------------------------
The future of the -longterm Linux kernel releases:

History:

2.6.16 became a "longterm" kernel because my day job (at SUSE) picked the
2.6.16 kernel for its "enterprise" release and it made things a lot
easier for me to keep working at applying bugfixes and other stable
patches to it to make my job simpler (applying a known-good bunch of
patches in one stable update was easier than a set of smaller patches
that were only tested by a smaller group of people.)

Seeing that this worked well, a cabal of developers got together at a
few different Linux conferences and determined that based on their
future distro release cycles, we could all aim for standardizing on the
2.6.32 kernel, saving us all time and energy in the long run. We turned
around and planted the proper seeds within the different organizations
and low-and-behold, project managers figured that this was their idea
and sold it to the rest of the groups and made it happen. Right now all
of the major "enterprise" and "stable" distro releases are based on the
2.6.32 kernel, making this trial a huge success.

Last year, two different community members (Andi and Paul) asked me
if they could maintain the 2.6.34 and 2.6.35 kernels as -longterm kernel
releases as their companies needed this type of support. I agreed, and
they have done a great job at this, but it doesn't seem that any other
groups other than their individual companies are using these kernels.

Today:

Now that 2.6.32 is over a year and a half, and the enterprise distros are
off doing their thing with their multi-year upgrade cycles, there's no
real need from the distros for a new longterm kernel release. But it
turns out that the distros are not the only user of the kernel, other
groups and companies have been approaching me over the past year, asking
how they could pick the next longterm kernel, or what the process is in
determining this.

To keep this all out in the open, let's figure out what to do here.
Consumer devices have a 1-2 year lifespan, and want and need the
experience of the kernel community maintaining their "base" kernel for
them. There is no real "enterprise" embedded distro out there from what
I can see. montaVista and WindRiver have some offerings in this area, but
they are not that widely used and are usually more "deep embedded".
There's also talk that the CELF group and Linaro are wanting to do
something on a "longterm" basis, and are fishing around for how to
properly handle this with the community to share the workload. Android
also is another huge player here, upgrading their kernel every major
release, and they could use the support of a longterm kernel as well.

Proposal:

Here's a first cut at a proposal, let me know if you like it, hate it,
would work for you and your company, or not at all:

- a new -longterm kernel is picked every year.
- a -longterm kernel is maintained for 2 years and then dropped.
- -stable kernels keep the same schedule that they have been (dropping
the last one after a new release happens.) These releases are best
for products that require new hardware updates (desktop distros,
community distros, fast-moving embedded distros (like Yocto)).
- the normal -stable kernel rules apply, as documented in
Documentation/stable_kernel_rules.txt for the -longterm kernels.

This means that there are 2 -longterm kernels being maintained at the
same time, and one -stable kernel. I'm volunteering to do this work, as
it's pretty much what I'm doing today anyway, and I have all of the
scripts and workflow down.

Public Notifications:

The current kernel.org site doesn't properly show what is and is not
being maintained as a -stable and -longterm kernel. I have a proposal
for how to fix this involving 'git notes', I just need to sit down and
do the work with the kernel.org admins to get this running properly.

Thoughts?
54
21
Dmitry Eremin-Solenikov's profile photoLinus Walleij's profile photoLeoš Bitto's profile photoMark Brown's profile photo
45 comments
 
For consumer devices with hw enablement requirements there will always be tension between product release schedules, -longterm schedules, and -longterm hw enablement backports. In -stable, hw enablement has been minimally intrusive. What are reasonable -longterm patches aside of the normal -stable-style rules?
 
I'm not sure, just from reading this, why 2 years would be considered longterm. That is certainly pretty short for an enterprise OS support timeframe. Otherwise it sounds like a solid plan as long as the feedback you have currently covers a wide enough audience of distributions and use cases.

From an embedded perspective I know some usages which are in the 4-year-old version range and they either manage their own packports for bugs, or live with them. While there may not currently be a -longterm for embedded this would be a great opportunity to provide a base version which can be reasonably promoted as stable enough for embedded use.
 
+Chris Wright it would be the same rules as -stable has for -longterm. That is what I've been doing for a while now and it works out quite well. For new hardware enablement, or major backport stuff, that's up to the individual companies to do, I'm not going to handle that at all. I'll change the text to say the normal -stable rules apply, thanks.
 
+Ted Johnson after 2 years, the chance that bugs and other fixes actually apply to a kernel is very infrequent, so the need to maintain them in this manner drops of quite quickly. Also 2 years is a long time for me to have to have hardware around (and a distro) that still runs the old kernel, I'm having problems today keeping a system running for the 2.6.32-stable tree at the moment.
Oh, and thanks for the review, also +Chris Wright thanks as well.
 
I like the idea of 2 year cycle, but am not sure about having 2 longterm versions at the same time as that can cause some confusion and it would require more work from you.

Then again, am not a kernel developer, just a user so I don't know how much weight my opinion has. Ideally everyone should use current version. Even if that does break more things it also means more testers.
 
+Ted Johnson also, "embedded" means a lot of different things, I'm talking to companies that use "embedded" Linux, yet throw everything away and start over every 6 months with new versions of everything (think Android development) so 4 years is laughable to them, 1 year is more than enough.
 
I like the idea of having a new long term kernel every 1 year. Maintaining it for 2 years is quite reasonable. That way I can stick with what works and don't have to upgrade my laptop distro everytime a new release is made. Recently, I updated to 2.6.40 from 2.6.38 and found out that bridging doesn't work well with MLD packet forwarding. As it turns out, the problem was reproducible with 2.6.39.4. I'm not sure of the exact cause. It ruined my day because I definitely need a kernel newer than 2.6.38 on my laptop. Otherwise, the graphic get distorted up when I use browser.
 
+Way Chuang Ang I still recommend updating your distro every release, otherwise you will not be able to have new devices work properly with your system.
 
Ah, I was not thinking Android as firmware. I was thinking more like stable hardware platforms getting new features (like, say, Tomato firmware on the Linksys WRT54G) and lasting a lot longer than 2 years. Gotcha. I'm new to embedded linux these last 2 years so I don't have much to pull from there.

However, in 10 years of Linux support for enterprise customers (Fortune 25 clients), I simply never saw companies doing 2 year platform refreshes on any enterprise projects. What I would typically see is a 6 month planning phase, 2 months implementation, and 4+ years of production. At best, we would see some hardware growth planned for the lease refreshes every 3 years, with every-other lease refresh being new hardware of the same architecture (6 year production lifespan for a given OS image).

So, are you suggesting having features & new hardware support backported to -longterm, or only simple bug/performance fixes? I suppose I'm confused by the "Consumer devices have a 1-2 year lifespan" text in the middle of what I perceived as a discussion of Enterprise Linux supportability.
 
+Ted Johnson Enterprise customers know how to support kernels for longer than 2 years, that's what they get paid for and they do it well. It's just that the ability for the community to do something like that starts to taper off after 2 years given the massive rate of change in the kernel.

You can see that today in how those "Enterprise" distros handle new releases, oracle and suse both switched kernel versions on their last releases because of the need for new features, where backporting would not work given how fast kernel development happens now, vs how it used to work 5 years ago.

And the rules for -longterm would be identical to what we do today in -stable and -longterm, they are described in the file, Documentation/stable_kernel_rules.txt. This is not a change in how things work other than the way we pick those -longterm kernels.
 
The primary worry I have is that the current, already laggard distro upgrade latency to the latest stable kernel will slow down even more and they will settle on the yearly long-term kernel.

Since the maintenance overhead of a stable kernel drops down to close to zero after 2 years, it would not be totally unthinkable to have 7-8 long-term kernels maintained in parallel, instead of the current planned 4 (the two stable kernels plus the two long-term kernels).

With that we'd remove all constraints and asymmetry from version picking: a stable kernel is maintained through 2 years and a Linux distribution can easily continue doing it from any specific stable kernel version they ended up picking as a base, after that period of time.

But yes, 7-8 kernels is certainly more maintenance overhead than 4 - but it's a very robust and vendor-neutral model to just maintain all stable kernels for 2 years.

You could even try to spread out some of your maintenance overhead: for example maintainers who mark commits as -stable back-ports could be expected to do a tested commit against all of your current stable trees, in well-defined places on kernel.org.

Your tooling could pick those commits up without having to guess whether a patch really applies and without the overhead of asking developers to backport, etc.

Since more than half of the -stable commits come via that angle, this would have some positive effect already.

(There might be other tricks as well to reduce your workload: such as using Git to do -stable backports: you could apply a fix to the oldest stable kernel - if it applies cleanly then it can be applied automatically to all future kernels, via a git merge of the old base. This avoids the duplications.)

I'm sure we could think of other tricks as well to make this more manageable.

Anyway, I don't think you will go this way initially due to the maintenance overhead involved, but you should be aware that this is a really good possibility from a user and distro POV. The current -stable maintenance period is seen as a bit short even for users - let alone distros.

A good topic we could discuss at the Kernel Summit perhaps? :-)
 
I think a longterm kernel release is a great idea because it can standardize the distros and make the life of the kernel hackers much easier. Good luck with the proposal.
 
We are using your longterm (2.6.32.y ATM) for our close to 2000 machines cluster with very few additional patches and it works very well with Debian (Lenny & Squeeze right now). As we need extra patches and the distribution's kernel usually turns out to be too old for us after some new machines arrive, this is a very viable way for us. So, I think 2yrs would be splendid for us and would give us choice to upgrade to the next LT kernel when we want and not start chasing the latest kernels until a new LT tree emerges.

Regarding the possible confusion, why not tag the longterm kernel with the year it was made LT? Thus 2.6.32 might have been longterm-2010, 3.0 might become longterm-2011, ... that way one would have ways to distinguish these easily.

Also, if distros were able to converge once a year on a kernel, I think that would be a major success.
 
I'd prefer if the real code stream was of such quality that -stable etc backports were unnecessary.
 
It's not just quality, for us the influx of new kernels and its updates is just too fast :)
 
I like your proposal! I also think it is really important that you communicate the changes between the longterm kernels in a well fashioned manner. Communicating properly what is and is not being maintained as a -stable and -longterm kernel is key. Once Google+ for business is out, I recommend you communicate the changes via that.
 
a 2 year long term would be more than enough for companies to decide what kernel to base off. gives the general dev cycle an idea where to start planning; if they like latest updates that are not in the latest long term kernel, then they'll probably just need to wait til the next one and base their dev cycle on that. if not, having knowing that a longterm stable kernel is going to be in use for the next 2 years, dev cycle has more than enough time for a stable distro (or whatever else) and a bunch of minor releases and rc's in the middle.
what would be the "no longer supported" expiration date for longterm kernels? i can see some, not wanting to move off of an older stable kernel just because... well, "it ain't broke, don't fix it".
 
I think the one year schedule is a good balance. It just takes a while for the enterprise customers to grab onto a new kernel, and more frequent updates wouldn't really get used much.
 
Definately like, two years would be.plenty for meego. I don't see more as being useful or needed - where this is really needed the long term maintenance could be handed over to a responsable osv as well. I do like the planned cadence, but I wonder if osv's aren't going to pick a version right in between the long term kernels you plan to maintain :]
 
+Ingo Molnar heh, all kernels are "stable" for 4 years? yeah, I could do that, but I'm not crazy, nor do I have tie time for doing that.

I don't want to inflict any extra work on any developers for the -longterm kernels, unless they want to do the backport work. That wastes time that they could be using for mainline work, and I don't want to impose on them for that at all.

And yes, git might be able to help me out here, but I've found that quilt actually works better for the backports instead.
 
+Carsten Aulbert most "community" distros release more than once a year, so that wouldn't help them, and "enterprise" distros release once every few years, so that wouldn't change them either. This proposal is to help the other "non-traditional" distros out, or the end-users who roll their own releases, like yourself.

+Lars Marowsky-Brée the upstream is stable enough, it's just hard to get people to update their massive collection of hardware-specific patchsets for every kernel in order to be able to move to the new one (embedded people, I'm looking at you.) This should help those people out, as they are asking for help right now.
 
+Zeno Davatz We use the linux-kernel mailing list for communication of "normal" things, and I will use the main kernel.org page as well for this (as described). Trying this out on google+ was just an experiment, as we aren't a "business" it doesn't really make sense if/when they add businesses to g+ to do anything different.
 
The Linaro kernel team is now doing monthly releases based on the latest stable kernel, and I don't think we will be using the longterm kernels for any official Linaro release in the foreseeable future. However, I think it's good to formalize the process of picking one and the Linaro member companies will certainly be interested in using that for their products.
I'm also trying to push harder for the Linaro members and the other contributors to the ARM platform to actually mark bug fixes as required for the stable/longterm backports. My feeling is that right now they see little value in that because most of their products are not based one of the longterm kernels anyway.
 
+Arnd Bergmann yeah, the lack of stable patches for the "embedded" drivers and arches has been quite disappointing over the years, which shows to me that no one in that area is even using the -stable or -longterm kernels for their systems (or if they are, they are not pushing anything upstream from them). I've talked to Linaro about this and it is good to hear they are using the stable kernels, so hopefully this lack of contribution will change in the future.
 
sounds like some kind of midterm target might be wanted for the 'community' distros
 
I'm sure they can I was just suggesting as a way to get them closer to the same page so they can reduce duplicated effort
 
Speaking at the moment with my sysadmin hat on (and probably not the audience +Greg Kroah-Hartman is trying to reach with this thread), I really don't want to see any of the distros encouraged to to fall further behind mainline. The reason the "enterprise" distro folks want a stable KABI is largely for things like storage device drivers during the O/S install processes. More often then not, these drivers are already included in mainstream and the binary drivers wouldn't be needed if it was "easier" to make fresh installs with a newer kernel. I think a lot of large shops deviate from their "enterprise" distros supported kernel because of this issue.

What I really care about is when the kernel interaction with user space changes in a way that the kernel can't be upgraded independent of all other packages. The breakage is usually something like udev no longer properly setting up devices or device-mapper not behaving quite right. The "enterprise" distros are trying to solve this problem in addition to the KABI issue by backporting some new features but they never keep up fast enough. Eg. There's no good reason RHEL5.x couldn't have had cgroups added on if the distro wasn't stuck in time with 2.6.18.

Solely for my needs, my desire for -longterm branches isn't based on a time interval but when these subtle behaviour changes with userspace happen. I want it to be lower risk to jump off the stale kernel release of my "enterprise" distro and jump forward to the next -longterm release.
 
I've witnessed the momentum of Android baseline kernels and from what I see ARM-based chipset and handset vendors will align the majority of their work around what Android choose. They all also have more or less deeply forked kernels due to a lot of out-of-tree code much of it requiring constant patching, and if there are nice stable updates they will be folded into the stream of fixes, no doubt.
 
But your communication for longterm and stable versions of the Kernel are aimed at businesses I suppose. Because those are the ones that care about 'stable' and 'longterm'. In any case, your Initiative is more then good and welcome! I just think that for businesses to take notice a new communcations channel could help.
 
I'd have to second what +Linus Walleij is saying about Android - a very large chunk of the embedded market is just going to completely ignore anything upstream does and use whatever kernel Google picked for the most recent Android release, even people who aren't using Android themselves as lots of the chip vendors have a big focus on Android. This probably points to maintaining a -stable for those kernels.
 
An update about the 2.6.25 kernel from +Andi Kleen:

Andi reports that the 2.6.35 kernel is being used by a number of
different distros, but they will be phased out as their support lifetime
expires. There are also a number of embedded users of the kernel as
well as some individual ones. So that -longterm kernel is having a lot
of benefit for a wide range of users.
 
So how's the -longterm candidate gonna be chosen each year? Every 4th release... or the Nth each year... or...?
I could imagine, that submits for the -stable release are pushed for certain releases (last before stable to get it in for sure, or first after stable to get enough testing) and in the end, the designated -longtime releases become what was the stable branch (in the 2.0, 2.2, 2.4 era) and the releases in between are seen as developer branch (as 2.1, 2.3, 2.5). I'm not sure if it would be good or bad, I just missed this POV in the discussion so far.
 
Greg, the current proposal text reads quite sound; having read the comments, I can also note that enterprise/longterm releases seem to be rather preferred as the resync points by non-embedded but still out-of-tree projects either (openvz and lustre come to mind) but they might be more comfortable with not being expected to sync every year, some discussion might make -j sense.

For me (community distros and minor handmade FLOSS-directed hosting), the timing looks good -- I actually curse Shuttleworth hard for inflicting braindamaged 6-month release cycles on his folks and attempts to do so on others as well, I've seen people worn out of that.

Just in case, our Clustrx HPC distribution is currently using 2.6.32, so thank you a lot for the efforts!

Ingo, two longterm kernels means 3 years of support, not 4 :) (mind the overlap)
 
Just my 2c:
What about prolonging -stable not till the next kernel release, but till 2 kernel releases happen? This will help people/distributions as then one can decide to use "previous" stable kernel if the last one isn't good enough for him.
 
+Georg Wassen as discussed on the linux-kernel mailing list today, I'll pick the next -longterm kernel based on how well a specific -stable kernel is working out, and if there are going to be a lot of users for a specific one (that indicates resources and willingness to help with testing and maintenance.)

+Michael Shigorin glad that this proposal makes sense to you, thanks for reviewing it.

+Dmitry Eremin-Solenikov I don't want to be maintaining that many different kernels all at once, sorry. While it is easy for me to handle this type of work, each kernel added is a non-trivial addition to the load, and to the "now which kernel did this patch not work for?" type of mental confusion. So no, I'm not going to be supporting -stable kernels for that long of a timeframe.
 
It seems to me that maintaining a stable kernel for a long time is exactly what is being offered by various "enterprise Linux" providers - SUSE, RedHat, etc. Is it really needed to duplicate their effort?
 
Just a currently-end-user perspective, but seems to make sense. Obviously vendors will pick whatever they want, and hopefully they'll notice that long term kernels save resources they can spend on something else.

Probably could be a bit more specific on how the long term ones actually get chosen. Even if it's on the level of "The .stable gets used in quite a few places, and it's been quite a while since we did the last one, and nothing major seems to be broken either". Gut feeling is probably how it works best. I'd guess that just after a major rewrite or addition of core functionality is probably not the place to start one, but just before a change that requires changes to lots of drivers is probably the worst place a .longterm could be... Same goes for userspace-visible changes. If users get stuck on some version for several years, basing it on the "new" behaviour probably best for everyone, since there's at least the chance of "just get the latest upstream one" making things better.

Maybe just some criteria that exclude a kernel from being long term, even though it's been a long time since the last one, and leave it at that?
 
I think this is a good idea. If I understand this correctly it would mean that if a company is using distro X LTS version this can use the long term kernel that is being patched and updated throughout that LTS cycle.
Would this also mean that in my situation where it was found that a bug related to screen drivers was fixed in a later kernel would also be patched to the long term kernel, that would be nice.

In a small company this would probably also mean that you can set up e.g. 10 laptops using the same distribution which again will be using the same kernel throughout the laptop lifetime (+/- three years).

All in all I think this is a good idea.

Trond
 
From our point of view, it would be perfect. This will help us to prioritize resources and plan ahead when to upgrade. I think the issue is more on how a kernel version is selected for long term support. I would dare to suggest that the kernel version should be marked for long term support even before the stable version is out. This would allow us to start upgrading our distro to that version. Any comments?
 
It's interesting how tech has changed lives.
Add a comment...