What and Why
Hetzner offers cheap cloud servers and additional block volumes that can be mounted at runtime. With bcachefs, one can use one filesystem that can be expanded with block volumes in case the local storage fills up. Here I describe how I set it up and tested the concept. It likely works just like that on other providers like OVH.
The setup
As a long-term happy user of NixOS, I want to run NixOS on the server. The NixOS wiki has an article about how to install on Hetzner Cloud, with one option using nixos-anywhere. I can not recommend the traditional ISO method, as that will require working with a clumsy remote KVM shell.
So the installation starts wiih a distribution provisioned by hetzner. I chose Debian 13 as a starting distro, as Ubuntu 26.05 errored complaining about unsigned PE executables or something on kexec. I did not investigete further, as kexec is all we need from the distribution and there are plenty distros to choose.
I already had a server running, that I was rebuilding. The flow for a fresh server maybe slightly different, but not too different. I also have dns records set up for filts.net with A and AAAA records pointing to the vps. Hetzner sends an email with the root password for the rebuilt machine. You have to login once with ssh, to set a new root password. After that we can copy an ssh key:
ssh-copy-id -i ~/.ssh/id_ed25519.pub root@filts.netlogin again to install kexec-tools
apt update
apt-get install kexec-toolsThe machine is now ready for nixos-anywhere. Get yourself a new directory to store the configuration and initialize git. hardware-configuration.nix is the same as on the nixos wiki:
{ config, lib, pkgs, modulesPath, ... }:
{
imports = [
(modulesPath + "/profiles/qemu-guest.nix")
];
networking.useDHCP = lib.mkDefault true;
nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
}as is flake.nix with updated nixpkgs:
{
inputs = {
nixpkgs = {
url = "github:NixOS/nixpkgs/nixos-26.05";
};
disko = {
url = "github:nix-community/disko";
inputs = {
nixpkgs = {
follows = "nixpkgs";
};
};
};
};
outputs =
inputs@{ self, nixpkgs, ... }:
{
nixosConfigurations = {
madalena = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
./configuration.nix
inputs.disko.nixosModules.disko
];
};
};
};
}configuration.nix is similar, but I decided to setup IPv6 right away. Otherwise my ssh client will wait for a timeout due to the set AAAA record:
{
config,
lib,
pkgs,
...
}:
{
imports = [
./hardware-configuration.nix
./disko-config.nix
];
boot.loader.grub.enable = true;
networking.hostName = "madalena";
networking.useNetworkd = true;
systemd.network.enable = true;
systemd.network.networks."30-wan" = {
matchConfig.Name = "ens3"; # either ens3 or enp1s0, check 'ip addr'
networkConfig.DHCP = "ipv4";
address = [
# replace this subnet with the one assigned to your instance
"2a01:4f8:1c1c:5630::1/64"
];
routes = [
{ Gateway = "fe80::1"; }
];
};
services.openssh.enable = true;
nix.settings.experimental-features = "nix-command flakes";
users.users.m = {
isNormalUser = true;
extraGroups = [ "wheel" ];
initialHashedPassword = "$y$j9T$2DyEjQxPoIjTkt8zCoWl.0$3mHxH.fqkCgu53xa0vannyu4Cue3Q7xL4CrUhMxREKC"; # Password.123
};
environment.systemPackages = [
pkgs.btop
pkgs.net-tools
];
programs.neovim = {
enable = true;
defaultEditor = true;
};
system.stateVersion = "26.05";
}I don’t know about the details of the password salt and rounds produced by mkpasswd. It may be wise to not set a valuable password here, if you are going to publish your configuration some day. Set up something better after first login. Finally the interesting file, disko-config.nix:
{
disko.devices = {
disk = {
main = {
type = "disk";
device = "/dev/sda";
content = {
type = "gpt";
partitions = {
boot = {
size = "1M";
type = "EF02";
priority = 1;
};
ESP = {
size = "512M";
type = "EF00";
content = {
type = "filesystem";
format = "vfat";
mountpoint = "/boot";
};
};
swap = {
size = "2G";
content = {
type = "swap";
discardPolicy = "both";
};
};
root = {
size = "100%";
content = {
type = "bcachefs";
filesystem = "main";
label = "local.sda";
extraFormatArgs = [
"--discard"
# later we can label new volumes as 'remote' and
# set-fs-option --background_target=remote
"--foreground_target=local"
"--promote_target=local"
];
};
};
};
};
};
};
bcachefs_filesystems = {
main = {
type = "bcachefs_filesystem";
extraFormatArgs = [
"--background_compression=lz4"
];
mountpoint = "/";
# default is relatime
# mountOptions = [
# "noatime"
# ];
subvolumes = {
"subvolumes/home" = {
mountpoint = "/home";
};
# not sure what this will be good for
"subvolumes/nix" = {
mountpoint = "/nix";
};
# state lives here
"subvolumes/var" = {
mountpoint = "/var";
};
};
};
};
};
}The vps has 4GB of ram, so I decided to configure 2GB of swap, for good measure. That allows some unused stuff to get paged out. Much more memory pressure than that can handle will likely have severe performance impacts anyway. Take note of the extraFormatArgs. For a single device system, those won’t make a difference, but we set it in advance for later, when block volumes will be added as background targets. This basically sets up the local storage as a cache for the slower remote block volumes.
Right now, I don’t have a use for subvolumes either, but sooner or later, bcachefs will hopefully will gain support for send/receive. With subvolumes set up like this, it may be possible to backup var easily and incrementally. Everything in nix should be reproducible from the configuration.
Finally it is time to run nixos-anywhere to install NixOS on the vps:
nix run github:nix-community/nixos-anywhere -- --flake .#madalena --target-host root@filts.netTesting
Now for the fun part. In the Hetzner management interface, I added a 10GB volume with the manual mount option. The device immediately showed up as sdb in dmesg. Nice. Let’s not use this name though, but a more stable id. Let’s find out the stable id:
# ls /dev/disk/by-id/ -l
total 0
lrwxrwxrwx 1 root root 9 Jul 26 15:19 ata-QEMU_DVD-ROM_QM00003 -> ../../sr0
lrwxrwxrwx 1 root root 9 Jul 26 15:27 scsi-0HC_Volume_106465919 -> ../../sdb
lrwxrwxrwx 1 root root 9 Jul 26 15:19 scsi-0QEMU_QEMU_HARDDISK_9572256 -> ../../sda
lrwxrwxrwx 1 root root 10 Jul 26 15:19 scsi-0QEMU_QEMU_HARDDISK_9572256-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul 26 15:19 scsi-0QEMU_QEMU_HARDDISK_9572256-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jul 26 15:19 scsi-0QEMU_QEMU_HARDDISK_9572256-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jul 26 15:19 scsi-0QEMU_QEMU_HARDDISK_9572256-part4 -> ../../sda4Add it to our bcachefs:
bcachefs device add / /dev/disk/by-id/scsi-0HC_Volume_106465919 --label=remote.106465919
bcachefs set-fs-option /dev/sdb --background_target=remote --data_allowed=userThis sets up the volume as a background_target. Kent Overstreet, author of bcachefs, sugessted to add data_allowed=user, which disallows journal and btree writes, thus keeping those local. To test it, I let ChatGPT write a scipt that fills up a directory with 40 * 1GB files of random data:
#!/usr/bin/env bash
set -euo pipefail
# Directory to store the files
OUTDIR="${1:-random_data}"
# Total amount to write (GB)
TOTAL_GB=40
# Size of each file (GB)
FILE_SIZE_GB=1
mkdir -p "$OUTDIR"
for ((i=1; i<=TOTAL_GB; i++)); do
filename=$(printf "%s/random_%02d.bin" "$OUTDIR" "$i")
echo "Writing $filename..."
dd if=/dev/random \
of="$filename" \
bs=1M \
count=$((FILE_SIZE_GB * 1024)) \
status=progress
sync
done
echo "Done."Since my server has 40GB of local storage, that filled the remote volume nicely. After a reboot, bcachefs found the volume automatically without any further configuration needed. I continued to add another volume bcachefs swiftly rebalanced to fill the new volume. I ihen evacuted and removed first device:
bcachefs device evacuate /dev/sdb
bcachefs device remove /dev/sdbAfter that, the volume can be unmounted in the Hetzner console and deleted. The system still rebooted, so I got rid of the random data and the second volume, which concluded my test.
Conclusion
I am not aware of any other local filesystem, that can be expanded and shrunk so easily at runtime. As far as I know, btrfs and zfs are oriented towards raid-like setups with evenly sized disks. As far as I know, the same functionality can not be achieved by layering any shrinkable filesystem on top of block layer solutions, like lvm. With resize2fs ext4 can be shrunk from the end of the underlying block storage, but what I’ve done here, anounts to evacuating a middle section of the block device and it can even be done online.
Not only can bcachefs handle differently sized block devices, but also devices with different IO characteristics. While the default block device on the Hetzner vps sits on local nvme storage, the additional block volumes are backed by ceph, and bcachefs can prioritize writes to the fast local storage without much configuration.