Recently, my server had a major outage caused partially by my own inexperience as well as some key failures in some key pieces of software I use. But like any good developer, I’m going to place blame upon the OS, specifically apt (not really, but I have to for the bit).
What happened?
Well, Debian 13 Trixie just released to mass acclaim (I should know, because I’m a big fan of Debian on servers), and I was in the midst of making a transition plan to upgrade my install. However, what had started out as a simple learning experience in a VM quickly escalated into the solution to a five-alarm fire of my own making.
By default, Debian does not allow users to install packages outside of the version of Debian that they have installed. I needed to get around this issue due to a compiler version mismatch on my main build server I use to speed up compiling C (see my post on distcc and ccache for details on getting that setup). So I decided to modify apt’s pin configuration to only install select packages from unstable (specifically GCC version 14 at the time). However, when Debian made the packages from unstable available as Trixie packages, something in my apt configuration caused a cascade of failures that resulted in something akin to Debian having a stroke.
Firstly, the postfix server running on my server failed. I still do not know what fully caused this outage, but without this key piece of infrastructure, issues would compound before any notification on my end. Once the email server was down, the email I would usually get notifying me of automatic upgrades would never come, and therefore, apt did not send the email detailing the critical errors quickly stacking up as the packages in unstable were shifted out into Trixie. I believe this is where the apt pin preferences was updated, causing the rest of Trixie to be pulled into my Debian Bookworm install. Apt failed soon after, as the apt sources file no longer matched the currently running OS.
Once I noticed that my email server wasn’t responding, I was able to login and see the damage. The entire OS had to be reinstalled from scratch, and just two days before starting a new job, meaning I had no email, no calendar, and no VPN back to the house.
So…new Debian version?
Not quite. You see, while Debian was quietly lighting itself on fire, I was playing with a FreeBSD virtual machine instance. I had zero intention of moving anything to it just yet (had that experience with my laptop, and, oh boy, were there some show-stopping bugs). However, I quickly fell in love with the level of polish and support FreeBSD has, and I was seriously considering moving to it instead of upgrading to Trixie just for the learning experience and to see if it was up to snuff to power the services I need.
It had it all: sane defaults, ports packages that I can customize to the third degree, a ZFS version that actual works; and I was enamoured after only about an hour of using the VM. However, the graphics drivers aren’t quite up to par for my standards, though they are extremely high quality. Fortunately, I wasn’t planning on using it to play games and render graphics, I was going to use it completely for work.
The server I have at home now rocks FreeBSD 14.3, and I could not be happier about it.
Final thoughts
If you’re any kind of sysadmin or Unix developer, you need to give FreeBSD a try if you haven’t already. It has changed my personal views on what a Unix system should be in very positive ways, and I will definitely be staying with it on my server. However, you might want to check the hardware compatibility on the FreeBSD wiki or check with sites like this one before taking the plunge on a personal system.