Incus Cluster with Ceph and OVN on Hetzner

Your own private cloud, without a data center of your own. Public cloud is out when the data is too sensitive, an own data center is too expensive for a small team. Rented bare metal sits in between: the hardware belongs to Hetzner, the platform on it is yours.

Incus is lean, quick to learn, and now part of the Linux Foundation, which counts for a platform you plan to run for years.

Three bare-metal servers, Hetzner vSwitches and the Ansible roles of incus-deploy. At the end there is an Incus cluster with Ceph as shared storage and OVN for the instance networks, and a VM that reaches the internet. Every command below was run on a real setup.

Stack: Incus on top of OVN and Ceph, a Hetzner vSwitch below, three dedicated nodes on Hetzner bare metal

What you need

In Hetzner Robot, each vSwitch gets exactly one VLAN ID, and a server can be attached to at most five vSwitches. Five VLANs is therefore the maximum per server, and this setup uses all of them. You create them under Server → vSwitches, then attach the servers with Add server. The Hetzner vSwitch documentation describes the steps and the limits.

For SSH access there is nothing extra to do: the Hetzner installer places the SSH public key you selected in Robot into authorized_keys of root on every node. Ansible logs in with the matching private key. If Ansible runs on one of the nodes, connect to it with ssh -A, so the key from your machine travels along.

The nodes are called node01, node02 and node03. The names appear in /etc/hosts, in ~/.ssh/config and in the inventory, so if you pick other names, change them everywhere.

The network plan

Every kind of traffic gets its own VLAN. That keeps Ceph replication away from the overlay network and makes tcpdump sessions short.

VLAN Bridge Network Used for
4010 mgtm0 172.18.18.0/24 management
4011 ovrl0 172.18.10.0/24 OVN overlay and Incus cluster API
4012 none public subnet optional public network pblc0
4013 strg0 172.18.80.0/24 Ceph public network
4014 strg1 172.18.81.0/24 Ceph cluster network

The last octet of each address is the node ID: 32 for node01, 33 for node02 and 34 for node03. The node script reads it from /etc/hosts, so the same script runs on all three nodes.

The stack is IPv4-only, one stack is easier to keep track of. IPv6 can be added later.

MTU 1400, and 1342 inside OVN

Hetzner asks to limit the MTU on vSwitch interfaces to 1400, so every VLAN runs with 1400. OVN wraps instance traffic in Geneve, which costs another 58 bytes. The OVN network therefore gets bridge.mtu: 1342.

Get it wrong and small packets work while large ones vanish. ping is happy, apt update hangs.

Install Ubuntu on the nodes

Boot every node into the Hetzner rescue system and install Ubuntu 24.04 with installimage. Replace [HOSTNAME] with node01, node02 or node03, and check with lsblk that the system disk really is /dev/sda. Servers with NVMe disks call it /dev/nvme0n1.

# On each node, in the rescue system
cat > ~/install.conf <<EOF
DRIVE1 /dev/sda
SWRAID 0
SWRAIDLEVEL 6
HOSTNAME me
Ubuntu-2404-noble-amd64-base
USE_KERNEL_MODE_SETTING yes
PART /boot  ext3     1024M
PART /boot/efi esp 256M
PART lvm    system   all
LV system   root   /        xfs          16G
LV system   swap   swap     swap          4G
LV system   home   /home    xfs          64G
LV system   varlog /var/log xfs          16G
LV system   tmp    /tmp     xfs          16G
IMAGE /root/.oldroot/nfs/install/../images/Ubuntu-2404-noble-amd64-base.tar.gz
EOF

# install
installimage -c ~/install.conf -n [HOSTNAME] -d /dev/sda -s en -t yes

The system disk belongs to the OS, Ceph gets the extra disks.

Configure the nodes

Run this on every node after the first boot. It sets up /etc/hosts, Open vSwitch, the VLANs with their bridges, a firewall and the SSH client, then reboots.

The firewall drops everything from outside except SSH. Private networks are allowed, so the nodes talk to each other freely over the VLANs.

# On each node as root
##############################################################################
# /etc/hosts
# remove short name aliases, relevant for ceph deployment
sed -i -E "s/(node01|node02|node03)$//" /etc/hosts

# add entries for the storage0 (ceph public network)
cat >> /etc/hosts << EOF
172.18.80.32 node01  node01.strg0
172.18.80.33 node02  node02.strg0
172.18.80.34 node03  node03.strg0
EOF
##############################################################################
# vars
MY_ID=$(
  cat /etc/hosts | grep strg0    | grep "$(hostname -s)" \
    | awk '{print $1}' | awk -F. '{print $4}'
)
MY_NIC=$(ip route | grep default | awk '{print $5}')

##############################################################################
# network
hostnamectl set-hostname $(hostname -s)

# install some packages
apt update
apt install -y openvswitch-common openvswitch-switch python3-openvswitch

cp /etc/netplan/01-netcfg.yaml /etc/netplan/01-netcfg.yaml.orig

# enforcing an ipv4-only stack for predictability
sed '/- 2a01:4f8:/d; /- to: default/,/via: fe80::1/d; /- 2a01:4ff:/d' -i  /etc/netplan/01-netcfg.yaml
sed '/ethernets:/!b;n;a \      link-local: [ ]'                       -i  /etc/netplan/01-netcfg.yaml

cat > /etc/netplan/10-vswitch.yaml  <<EOF
---
#
# Bridges and vlan are working independet from main enterface enp7s0 setup (linux ip/ ovs)
#
network:
  bridges:
    mgtm0:
      interfaces: [vlan4010]
      addresses:
        - 172.18.18.$((MY_ID))/24
      dhcp4: false
      dhcp6: false
      link-local: []
      openvswitch: {}
    ovrl0:
      interfaces: [vlan4011]
      addresses:
        - 172.18.10.$((MY_ID))/24
      dhcp4: false
      dhcp6: false
      link-local: []
      openvswitch: {}
    strg0:
      interfaces: [vlan4013]
      addresses:
        - 172.18.80.$((MY_ID))/24
      dhcp4: false
      dhcp6: false
      link-local: []
      openvswitch: {}
    strg1:
      interfaces: [vlan4014]
      addresses:
        - 172.18.81.$((MY_ID))/24
      dhcp4: false
      dhcp6: false
      link-local: []
      openvswitch: {}
  vlans:
    vlan4010:
      id: 4010
      link: $MY_NIC
      mtu: 1400
      dhcp4: false
      dhcp6: false
      link-local: []
    vlan4011:
      id: 4011
      link: $MY_NIC
      mtu: 1400
      dhcp4: false
      dhcp6: false
      link-local: []
    vlan4012:
      id: 4012
      link: $MY_NIC
      mtu: 1400
      dhcp4: false
      dhcp6: false
      link-local: []
    vlan4013:
      id: 4013
      link: $MY_NIC
      mtu: 1400
      dhcp4: false
      dhcp6: false
      link-local: []
    vlan4014:
      id: 4014
      link: $MY_NIC
      mtu: 1400
      dhcp4: false
      dhcp6: false
      link-local: []
EOF
chmod 600 /etc/netplan/10-*
netplan get
netplan apply

##############################################################################
# firewall

cat > /etc/nftables.conf <<EOF
#!/usr/sbin/nft -f

flush ruleset

table inet filter {
        chain input {
                type filter hook input priority filter;
        }
        chain forward {
                type filter hook forward priority filter;
        }
        chain output {
                type filter hook output priority filter;
        }
        chain input {
                type filter hook input priority filter; policy drop;
                iif "lo" accept
                ct state established,related accept
                tcp dport 22 accept
                ip saddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 } accept
                drop
        }

    chain output {
        type filter hook output priority filter; policy accept;
    }
}
EOF
systemctl enable nftables
systemctl daemon-reload

##############################################################################
# misc

cat > ~/.ssh/config <<EOF
Host node01 node02 node03 *.localdomain
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
    LogLevel ERROR
EOF

# reboot
reboot

After the reboot, check that the nodes see each other on the storage VLAN:

ping -c3 <HOST_NAME>

Prepare Ansible

Ansible runs on one of the nodes, or on a fourth machine with access to the VLANs. It gets its own Python through pyenv, so nothing collides with the Python packages of Ubuntu.

# On the Ansible node as root
apt install -y make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

# Install pyenv
curl https://pyenv.run | bash
echo 'export PYENV_ROOT="$HOME/.pyenv"'                                >> ~/.bashrc
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"'                                          >> ~/.bashrc
source ~/.bashrc

# Install virtualenv with pyenv
pyenv install 3.13.1
pyenv virtualenv 3.13.1 incus_deploy
pyenv activate incus_deploy
pip install -U pip pipenv
source ~/.bashrc

# Clone the repository
git clone https://github.com/lxc/incus-deploy.git
cd incus-deploy/ansible/
# the tested state of the roles
git checkout a01755a4a917884fd67f996ca76d0ad64dd98e59
# setting to put dependencies within virtual env
sed "/\[defaults\]/a home=\$VIRTUAL_ENV/.ansible" -i ansible.cfg

# Install dependencies
pipenv install
ansible-galaxy install -r ansible_requirements.yml

The git checkout pins incus-deploy to the commit this guide was tested with, from March 2026. Newer commits may bring fixes, but also changes that no longer fit the inventory below.

Write the inventory

The inventory describes the three nodes and everything that runs on them. All three nodes carry every role: Ceph mon, mgr, mds and osd, OVN central and host, Incus cluster and ui. With three nodes there is no room for dedicated controllers, and every node can fail without taking a service with it.

Ceph talks on the storage VLANs, OVN and the Incus API on the overlay VLAN. Nothing of it listens on the public interface.

Three things in it are yours to fill in:

The instances get three networks:

# On the Ansible node, in incus-deploy/ansible/
cat > hosts.yaml <<EOF
all:
  children:
    incus_servers:
      children:
        baremetal:
          hosts:
            node01:
              # TODO: put here disk ids available to create OSDs on
              #       you can run the following command on each node,
              #       to file the suitable candidates:
              #
              #       ls  /dev/disk/by-id/ | grep -i -E "nvme|ssd" | awk '{print "- data: "\$1}'
              #
              ceph_disks:
                - data: [SOME_ID]
                - data: [SOME_ID]
                - data: [SOME_ID]
            node02:
              ceph_disks:
                - data: [SOME_ID]
                - data: [SOME_ID]
                - data: [SOME_ID]
            node03:
              ceph_disks:
                - data: [SOME_ID]
                - data: [SOME_ID]
                - data: [SOME_ID]
  vars:
    #
    # ansible
    #
    ansible_python_interpreter: auto_silent
    #
    # ceph
    #
    ceph_fsid:             "a075071e-7fb2-4d49-a741-9ab391f2fbad"
    ceph_network_public:   '172.18.80.0/24'
    ceph_network_private:  '172.18.81.0/24'
    ceph_rbd_cache:        "2048Mi"
    ceph_rbd_cache_max:    "1792Mi"
    ceph_rbd_cache_target: "1536Mi"
    ceph_release:          "distro"
    ceph_ip_address:       ""
    # NOTE: all nodes will have the same roles in the ceph cluster
    ceph_roles:
      - client
      - mon
      - mds
      - mgr
      - osd
    #
    # OVN
    #
    ovn_name:       bolt
    ovn_release:    distro
    ovn_ip_address: ""
    ovn_roles:
      - central
      - host
    #
    # incus
    #
    incus_init:
      network:
        #
        # UPLINK network to connect with world outside of the platform
        #
        UPLINK:
          type: bridge
          config:
            bridge.driver:   openvswitch
            bridge.mtu:      1500
            ipv4.address:    172.18.20.1/24
            ipv4.ovn.ranges: 172.18.20.64-172.18.20.127
            ipv4.nat:        true
            ipv4.dhcp:       false
            ipv6.address:    none
          description: "Uplink via infra hosts"
        #
        # Just a network to create first workloads in, connected with UPLINK
        #
        ovn-test:
          type: ovn
          config:
            network:          UPLINK
            ipv4.address:     10.10.10.1/24
            ipv4.nat:         true
            ipv4.dhcp:        true
            ipv4.dhcp.ranges: 10.10.10.64-10.10.10.127
            ipv6.address:     none
            bridge.mtu:       1342 # NOTE: network backed by VLAN with MTU=1400
          description:        "OVN test network"
          default: true
        #
        # Additional public(internet) network can be used to get ips from
        #
        pblc0:
          type: bridge
          config:
            ipv4.dhcp:                  false
            ipv4.address:               116.202.68.154/29
            ipv4.dhcp.ranges:           116.202.68.155-116.202.68.158
            ipv4.nat:                   false
            ipv6.address:               none
            ipv6.nat:                   false
            dns.mode:                   none
            bridge.mtu:                 1400
            bridge.driver:              openvswitch
            bridge.external_interfaces: vlan4012
          description:                  "Hetzner additional network"
      storage:
        octopet:
          driver:              "ceph"
          description:         "Default"
          local_config:
            source:            "bolt"
            ceph.cluster_name: "ceph"
          default: true
    incus_ip_address: ":18443"
    incus_name:       bolt
    incus_release:    stable
    incus_roles:
      - cluster
      - ui
EOF

Write the playbook

The playbook replaces the deploy.yaml that ships with incus-deploy. It runs four roles in order: system settings, Ceph, OVN and Incus.

# On the Ansible node, in incus-deploy/ansible/
cat > deploy.yaml <<EOF
- hosts: all
  gather_facts: true
  gather_subset:
    - "default_ipv4"
    - "!default_ipv6"
    - "distribution_release"
  pre_tasks:
    - name: Inject inventory ceph_ip_address into the facts dictionary
      set_fact:
        ansible_ceph_ip_address: ""
  roles:
    - system_settings
    - ceph
    - ovn
    - incus
EOF

Deploy

# On the Ansible node, in incus-deploy/ansible/
ANSIBLE_ROLES_PATH=../roles ansible-playbook deploy.yaml

The run takes about an hour.

Check the result

# On any node
# list nodes in cluster
incus cluster ls

# list networks
incus network ls

# list storage
incus storage ls

# create a new instance in the default (ovn-test) network
incus launch images:ubuntu/24.04 test01 --vm  -c limits.cpu=1   -c limits.memory=4GiB

# test connectivity
incus exec test01 -- ping -c3 8.8.8.8

# install tools
incus exec test01 -- apt update
incus exec test01 -- apt install -y curl tcpdump arping net-tools

# test
incus exec test01 -- curl ifconfig.io

Three cluster members online, a Ceph pool as default storage, and a VM that answers from the outside world. curl ifconfig.io shows the public address of the node the VM leaves through.

When this setup fits

The setup fits when you need control over your data, don’t want to run a data center, and accept Hetzner as part of your infrastructure. It does not fit when you need guaranteed SLAs (contractual availability), compliance beyond data residency, or a team without the time to learn a new platform.