Considerations on how to run services in 2025

This article is part of the How to run services in 2025 series.

Table of Contents

Intro

I've been running services on (virtual) servers on the internet for over 20 years and every five to seven years, I think about how to operate them. To be honest, at some point, you have an uptime of almost two years and don't dare carry out an update that includes a restart. This presents a good opportunity to migrate the services one by one to improve the whole setup.

Update: 2025-07-08

I did it anyway. I received an email from my hoster, that a reboot of the host system is required. After over 670 days a reboot was inevitable! I took the opportunity to create a snapshot of the system and gave it a try:

# pacman -Sy
# pacman -S archlinux-keyring
# pacman -Su

And some other things mentioned in the Arch Linux News Archives, which are also mentioned by pacman while running the update.

In short, I did the following steps:

  • Shutdown the system
  • Make a snapshot
  • Boot up and install updates 🤞
  • Run required manual tasks for some packages (See Arch Linux News Archives)
  • Run Ansible Playbook
  • Fix Ansible Playbook
  • Adjust Ansible Playbook (because stupid me did changes on the system without adjusting the Playbook, don't do that!)
  • Run Ansible Playbook agian (and repeat fixing, adjusting, run...)

Everything works as before. The main issues I had were some deprecated Ansible Tasks and changes on the system that I hadn't included in my Ansible Playbook and had done manually in the past. Had I adjusted my Ansible Playbook, as I should have, none of this would have taken much time.

I currently use Arch Linux as the operating system for my virtual servers, which have full disk encryption. The services run in Docker containers and the systems are managed using Ansible. Backups are created daily with BorgBackup. In my opinion, it's a good foundation that can be used as is for another couple of years. I wish I had been more disciplined in keeping the system and containers up to date.

What do I want:

  • Full disk encryption
  • Automatic operating system updates
  • Automatic updates for containers
  • Manual intervention in rare cases
  • No direct dependencies to services provider (vendor lock-in)

Since all services are already running in containers, the choice for the operating system doesn't make a huge difference. The only interesting part is how well full disk encryption is supported. Why bother to encrypting? The main reason is actually data theft. For example, if my hoster "lost" the disk image for whatever reason or a data breach. Automatic operating system updates and full disk encryption require manual intervention to enter the password on a reboot. This also works with ssh.

Operating system

At this point, I want to discuss the operating systems I have experience with or that seem interesting for the intended use.

Arch Linux

Arch Linux does not officially support automatic updates, which is why it may not be an option. However, if I really want it, one option would be to send an email every day after n days of uptime to remind me to run pacman and reboot. At least with Arch Linux you are always up to date.

Debian / Debian derivatives

I do not have any bad experiences with Debian and Ubuntu. I used Debian/testing as my daily driver for years before switching to Arch Linux. At my last workplace, we used Ubuntu for the servers and it did its job great. You get automatic updates with UnattendedUpgrades, which work after some configuration. Such a setup runs without much manual intervention when combined with a cron job that sends an email if a reboot is required to enter the password for the full disk encryption.

The one thing that has bothered me about Debian/Ubuntu so far is how up-to-date the packages are. The Docker and Podman versions are relatively outdated. This is especially annoying for Podman because many improvements have been added in the latest versions.

A distribution upgrade would be due every two years, which would require a manageable amount of time. This also worked out well in my professional career.

Fedora Core OS

Fedora Core OS is a minimal operating system optimized for containers. It comes with automatic updates by default, as well as the option to run a rollback if something goes wrong or doesn't work as expected. I use it for my OctoPrint server at home. It runs only the OctoPrint container, which is pulled freshly (updated) every time the device boots up. In this case, I disabled the automatic updates because an update-triggered reboot is inconvenient during printing. Since I don't print every day, that's okay for me. Maybe I spent some time making the update service printing-aware.

A system is set up using an Ignition configuration, which describes how the system should be set up. It is possible to define services directly in the configuration file. As a result, I see two options for managing the systems.

  1. Everything is described in the Ignition configuration. This means that with every change (add/edit/remove a service), the system would be setup as new, but preserving the data (/var). The _ Ignition_ configuration must be available somewhere, ideally on the internet. Secrets (passwords, keys etc.) shouldn't be visible in plain text. Therefore, additional services are required:
  1. Basic setup with Ignition configuration and the final setup by Ansible or other tools which describe the state of the systems and optionally of the services.

The second approach seems to be the most sensible. Only a tang server is required for the LUKS encryption. The other secrets, such as passwords, keys etc can be managed by Ansible Secrets, as is already the case in the current setup.

NixOS

I don't have any experience with NixOS. In my opinion, the declarative approach is very intriguing. This raises the questions: How far can it be used, are automatic updates supported and how to handle secrets?

Others

A few more that might be worth a look:

Summarizing

So far, I prefer the combination of Fedora Core OS and Ansible. At first glance, it could even restart completely without manual intervention when used with tang. In this setup, the tang service would be the Achilles heel and would have to run somewhere secure.

Ubuntu/Debian is a reliable option and UnattendedUpgrades keeps the system up to date. The past showed, that security vulnerabilities where fixed immediately with updates, which where installed flawlessly by UnattendedUpgrades.

Other operating systems are certainly worth a look, especially NixOS. I'm curious to see how it behaves here.

Container Engine

Here I would like to make the switch from Docker to Podman. I have been using Podman locally for years and I love it. By default, it is rootless, runs without root privileges and doesn't need a service (daemonless). IPv6 works seamlessly with Podman, which was difficult with Docker a few years ago. Perhaps the situation is better today.

Container orchestration

Do you even need this on a small scale or is it only worthwhile for more than three servers? So far I've only used it, but not yet run it myself. Here are a few options I should look at:

Deployment

Oh, jeez! Depending on the final setup, there are many possibilities. Especially if you use K8s:

I actually find the GitOps appealing. But what do you do with the tiresome topic of secrets (passwords, keys, etc.)?

Backups

So far I have been using BorgBackup in combination with borgmatic. It works reliably once set up and restoring data has also worked without any problems recently. You always should look out for alternatives and new developments and I recently came across restic in combination with resticprofile, which looks interesting. But so far I didn't encounter any issues with Borg.

Next steps

  • Have a look at NixOS
  • Setup a system with Fedora Core OS or NixOS
  • Use plain Podman, K8s or …?
  • Try out different ways for the deployment