Jonas Schäfer

Stable vs. Rolling Release—An Argument

Since I have integrated myself into the XMPP/Jabber community more, I came into contact with more people whose opinion I respect a lot, but whose views also differ a lot from my own.

One of the recurring topics in the programming chat room (which, by the way, isn’t just about programming, but also DevOps and other stuff, which explains why the topic even comes up there) is the frustration caused by so-called "stable" Linux distributions.

With that, folks typically mean a distribution like Debian, Ubuntu, Red Hat Enterprise Linux (RHEL), SuSE Linux where releases happen every few years and then the software does not get major upgrades until the next release. In contrast to that, there exist distributions such as ArchLinux where there are no defined releases; very recent software is always available for each ArchLinux installation, as soon as the package repositories get the update (which is typically very soon after the upstream release).

Definitions: What Does "Stable" Mean (to Me)?

Stable Linux Distributions

When I think of a "stable" (I’ll be leaving out the quotation marks from now on) Linux distribution, there are a few key properties for me:

  • It has well defined releases. A release, for me, consists of a set of specific package versions which are shipped (distributed) together. When installing for example the initial "Stretch" Debian release (Debian 9.0), you will always get the very same packages on each install. It is reproducible.

    This is not 100% true, because there are point-releases in Debian; For example, there exists the Debian 9.1 release. So if you made an install of Debian 9 before and after a point release, you’d get different setups. But as long as no additional point release happens, you will get identical stuff.

  • There has been some type of quality assurance (QA) before a release is made. The QA does not necessarily need to be explicit or paid or whatever; the QA may simply consist of distributing a release candidate for a certain amount of time and waiting for bug reports (and then triaging and possibly fixing those before the release).

  • There exists a non-trivial (two or more years) time window where support is more or less guaranteed for (at least a well-defined non-empty and non-trivial subset of) the packages included in the release. With support, I mean that grave issues (leading to the package being unusable) are fixed with updates. This most notably includes security issues.

  • There exists security support. This means, to me, that security issues are addressed in a very timely fashion (for example, within less than 48 hours) after a fix becomes available, if at all possible. If it is not possible to fix the issue, I expect a workaround to be announced (even if the "workaround" is "Turn off all your machines running X and hide in the woods").

Note that none of the above says "it must not contain any bugs". Also, it says nowhere that I expect all bugs to be fixed during the lifetime of a release.

It also doesn’t say that software should be kept up-to-date (in fact, I consider up-to-date software an anti-feature sometimes; see below). In fact, it is rare to have a non-rolling-release distribution which has releases more often than once per year.

Ubuntu and Fedora do have release cycles more often than that, but they also don’t offer long-term support (longer than one year) on every of those releases (or any, in case of Fedora), making them a weird hybrid of rolling release and stable.

To summarise it in one paragraph: Stability, in this context, means to me that I can predict the behaviour of the operating system and that some effort is put into being able to being able to continue to operate it just like I did yesterday without having to expect breakage by security issues or unexpected updates.

Examples of stable distributions, in my mind, are Ubuntu (Long-Term-Support only), Debian, and RHEL (and derivatives).

Rolling Release

What I wrote above about stable distributions means a lot of management overhead for those. In contrast to this, there exist rolling release distributions such as ArchLinux (I know that others exist, but unfortunately, this is the only one I ever used, so I can only speak for it; likewise, the folks I argue with also only use Arch or a derivative of it as far as I know).

A rolling release (RR) distribution is essentially a single package repository where updates are pushed. When you install an RR distribution from scratch, you always get a different installation (installing it without internet access is also typically rather futile or at least frowned upon) because it is likely that an update to the package repositories has happened since the last time.

The Argument: Rolling Release vs. Stable

Now for the main argument.

As I mentioned in the introduction, this post came into being because I see people frustrated with stable software distributions. And I also see people "forcing" (no wrenches involved!) me to justify my personal choice of using a stable distribution on all of my servers.

I try to make this argument as balanced as possible. In the end, it all comes down to personal choice.

I will grant rolling release distributions a massive advantage: they provide recent/up-to-date software. This is a major win for many use-cases. I, for example, like to work with modern Python features. I can fully understand folks who love to work with Rust or Golang or other rather fast-moving languages ("fast" is here to be taken with respect to the typical two-year-or-more release cycle of stable distributions).

In those environments, you really and absolutely do not want to be slowed down by a distribution which releases once every other year. And this is in fact the reason why I’m using Debian unstable on my development machines.

However, on my servers, I prefer stable a lot. And here’s why.

Why I Prefer Stable Distributions

Quality Assurance

A stable distribution provides me with a combination of packages which are very likely known to work in that specific combination. Most packages are provided by somebody who actually uses them. This combined with the typically significant user base of well-known stable distributions means that it is likely that all of the typical configuration combinations are actually in use somewhere.

When a new release is in the making for a stable distribution, folks will try the various things in the testing channel and report issues when they don’t work as expected. The issues are then fixed. This is a somewhat crowdsourced non-automated integration test for the packages in the distribution. From software development, I know that unit tests (in this analogy and from perspective of the packaging architecture, the tests run after/during the package build are the unit tests) are good and all, but without integration tests, they are very, very useless.

However, this is the world of Free Software. I fully know that I cannot rely on somebody having tested a specific combination; if I care about it, I’ll have to do it myself (which I do). When something goes wrong, I can downgrade my systems to the previous stable release (which I did when dovecot-ldap broke our use-cases in the 2.4 release).

Mix & Match

As I mentioned, I myself like to use modern features of the languages I work with. This means that sometimes, I’ll have to go off the beaten stable path. For those cases, there exist (with Debian) the backports repository. You can (using apt pinning) select specific packages (or even versions) which shall be sourced from the backports repository. Package metadata makes sure that you don’t do something blatantly stupid (like upgrading the perl version to something which isn’t compatible with the apt release you’re using), so there’s that.

But aside from that, you are now kind of on your own. There is no official security support for those packages (although the maintainer might still make an effort on their own to provide that), and they have not been "integration tested" during the release candidate phase.

However, a backport package doesn’t come out of nowhere. There has been a need for it, and somebody will likely be running it already in production somewhere. So a few backport packages might not hurt (you still have to be careful, obviously, especially with security upgrades).

I generally try to avoid using backports (or, shudder testing) on production systems.

Now, in response to this, I often hear something along the lines of "if you’re using backports, why aren’t you using rolling release altogether?", which is a fair question.

My reply to that is: The stable distribution provides me with a base system I can rely on. It will behave predictably.

Update 2018-12-08: I reworded the part starting from there until the end of this subsection after some useful feedback.

Now, with backports in the mix, I have to take care that they don’t mess up the system. I will have to re-do this whenever the backport gets an update. In contrast to the rest of the system, I lose the security support; this means, most notably, when a security fix comes out, I am faced with the choice of upgrading the package ASAP (the fix will be based on the next upstream release, so it will most likely also carry new features and bugs), delay the upgrade further by doing additional tests (e.g. on a dedicated test setup) and/or (temporarily) remove/disable the package altogether.

Of course, this is not ideal (and this is the reason why I’m not using backports in general); but having to do this for one (or a few) packages is still much better than having to do it for all the packages.

Time Is A Factor

To be clear: I do most of my admin work in my free time. I run communication services for friends & family (think instant messaging, email, voice chat); I run various static websites. And all of that with the full infrastructure stack: virtualisation, OSPF-based routing, own DNS servers, etc. etc.

I do this because I value my privacy and believe that self-hosting is in most cases preferable to using a hosted service. And also because building and maintaining infrastructure can be a hell lot of fun.

However, I also have a life besides this. You know, somewhere I need to get those friends & family I can provide the services for!

As much as I like the feeling of being important when doing sysadmin work and being appreciated, the appreciation typically ends when you have to cancel plans when something breaks and you need to fix it to ensure that nobody loses their email for that day.

And this is where all of this ties together. A stable distribution, and some self-restrain from always having to use the most recent fancy stuff, makes such events extremely rare. New features most of the time also mean new bugs. Stable releases do not get new features. They only get very specific security updates (which may of course also introduce new (potentially grave) bugs, like in the famous Debian SSL disaster of 2008), and they get those updates only when absolutely necessary.

I can thus, with good conscience, allocate rather slim time slots to run the daily upgrades on the servers. I’ll have an eye on the monitoring (and I of course use apt-listbugs and apt-listchanges and read through the notes while upgrading), but I’ve never had the issue that an upgrade broke something in Debian.

To Wrap It All Up

A stable distribution is no magic bullet. It merely provides me with a reliable and predictable basis. It can’t provide me with the most recent software all the time. It provides me with security upgrades. The packages are known-ish to work in the combinations. And it is unlikely that an upgrade will break all the things.

It means I have to live with fewer/less modern features. It means I have to work around bugs which might have been fixed three years ago in the upstream source. And you know what? That’s fine with me. What’s not fine with me is having to cancel plans because a required security fix also pulled in a bug which stops users from being able to log into their email accounts.