Recently, I’ve been experimenting with cloud storage solutions I can host myself. Nextcloud, ownCloud, FileCloud, and insertCloudOfTheWeekHere were all solid recommendations from friends, but I wanted something a bit less bloated. Eagle-eyed readers will see that I was using Syncthing for this task. Syncthing is a great option for syncing files between mixed machine types, but it suffers from a quirk in which certain files will be forever stuck in transit. Plus, it requires a web-portal to function effectively and I want something I can call up on the CLI via a cron job or manually as needed without resorting to webapps.
Why not Nextcloud?
Nextcloud is…fine, for some people, but I’m not its target user-base, and that becomes apparent when I need to sync git repositories with half finished commits between machines. Nextcloud (and by extension its cousins, FileCloud and ownCloud) offer a solution for the generic user that makes zero assumptions about how the sync data is being used. It has plugins, webapps, office suite integrations and other bloat that I do not need bundled with my file sync utility.
So I began looking around the web for a lightweight, preferably CLI-based, application that could fit my specific needs without a ton of bloat. I already have a FreeBSD server running ZFS, capable of sharing over NFS, so why not NFS? Well…I soon discovered NFS has a few quirks that make my specific situation fundamentally incompatible with it. Firstly, I need something that works over the web, and NFSv4 with its Kerberos authentication is woefully ill-equipped for that job. Secondly, I need it to work over WiFi because although my primary machine is a reasonably powerful workstation running Arch Linux, I also need to be fairly mobile, so I have a cheap Arch laptop I can use as an SSH surfboard. I quickly discovered that NFS is rather touchy about network inconsistencies, and WiFi isn’t exactly the most stable connection type.
Rsync then?
rsync was a very compelling option, but it too had some quirks that made
it ill-equipped for this task. It does work if one needs to transfer a lot of
files in one direction, but it became clear after running it every time I
needed to sync some git repos over the wire I needed something bi-directional.
Enter Unison
I found out about unison via a Reddit post somewhere, thinking it to be just
another clone of rsync. I couldn’t have been more wrong. It had everything I
needed a file sync utility to do: calculate checksums, only send file deltas,
sending files compressed with a sane algorithm, bi-directional, etc. Now I only
use unison and a simple cron job to sync everything between my two
workstations and the server in the basement. It even has a very nice batch mode
to sync multiple directories under a defined root.
How do we get it working?
If you’re on a Linux or FreeBSD, you’re in luck because most distros will have
it packaged. If you’re on Windows, the program can be downloaded from
Github and installed manually. To my
knowledge, there’s no installer provided, but I haven’t extensively checked.
For my specific setup, I used the following config file located at
~/.unison/Default.prf:
# Unison preferences
root = /home/dholman/repositories
root = ssh://dholman@www.gymli.org/repositories
path = xenithos
path = rune-engine
path = blog
sshargs = -C
Along with the following crontab entry:
*/5 * * * * unison Default -batch 1> /dev/null 2> /dev/null
And that’s all it needs to fully sync files between my local machine and my server. Since it utilizes SSH for transport, I can use it over any kind of connection, including on public networks. Now, if you require these files on an Android device…I’ll leave that as an exercise to the reader to figure out, as that piece of the puzzle I still haven’t solved myself. In fairness however, I typically don’t need access to git repositories and text files on my phone. For media files I can always install immich or Jellyfin on the home server, and for general backups I have some other apps and self-hosted stuff for those. For now, I think I’ll keep this configuration around and maybe link a friend or two’s server for a proper 3-2-1 configuration, all using simple shell tools and a cron job!