A UniFi firmware update wipes everything except /data

The Pi-hole container on my UDM had been dead for a while, and the reason was the firmware flash.

It took /usr and /var with it, so systemd-container was gone and nothing on the box could start a container any more.

/data survives, and it’s the only thing that does, so everything else has to hang off that.

/etc survives too, because / is an overlayfs and edits land in the writable upper layer, but that’s UniFi’s choice about what its updater clears and not a guarantee, so in my case I keep the source of truth in /data and copy it into /etc on every boot.

You cannot download anything at boot

The obvious fix is a boot hook that reinstalls the package, and the obvious boot hook starts with apt-get update.

That doesn’t work, and I measured why across two boots:

The hook runs about four minutes before the system is ready.

Its only ordering is After=network-online.target, which on UniFi OS is satisfied by systemd-networkd while the real WAN comes up later under ubios-udapi-server.

The upstream setup script is unsafe here for exactly that reason.

It starts with set -e and its second command is apt-get update, so with no internet it aborts before it ever reaches its own offline fallback.

So we cache the packages on the device and the network refresh becomes best effort:

apt download systemd-container libnss-mymachines debootstrap arch-test debian-archive-keyring
mv ./*.deb /data/custom/dpkg/

The version has to match exactly

systemd-container declares a strict dependency on the systemd it was built against, and the host runs 247.3-7+deb11u8.

Debian’s main pool only carries u4 and u5, so the u8 comes from bullseye-security, and bullseye-backports on the device already points at archive.debian.org.

The stale files I found cached on the device were u6 and would have failed the dependency anyway.

Check the version, not whether it’s installed

A firmware update reverts debian-archive-keyring to 2021.1.1+deb11u1, which is still installed, so a test that asks whether the keyring is there passes and debootstrap then refuses to build a trixie container.

I forced the downgrade to watch it happen.

The trixie keys went from six files to zero, and the next run logged installed 2021.1.1+deb11u1 is older than cached 2025.1 and put them back.

So we need to compare versions for all three build-time packages, and not just ask whether they are installed.

/usr/local/bin belongs to UniFi

Static binaries go in /data/custom/bin, because /usr/local/bin isn’t merely wiped, it’s repopulated by the firmware with addrinfo, sip-dig, stunc and friends.

So the boot script symlinks each name into /usr/local/bin, but only when a system binary isn’t already using it, and now a tool works in scripts, in systemd units and over non-login ssh.

Two boot runners is worse than none

If you want to drop anything into /data/on_boot.d, first take a look at what else walks it:

systemctl list-unit-files --state=enabled | grep -iE 'boot|setup'

Mine had udm-boot.service from unifi-common enabled alongside a leftover setup-system.service, both iterating that directory, so every script in there ran twice.

With set -e and an unconditional ip link add, the second pass errors out partway through and leaves the shim half configured.

The test that matters

Deleting the packages proves the reinstall works, and if you want to prove the rest of it, delete the cache as well and reboot.

It came back in 14 seconds. It waited 10 for the WAN, installed all four packages, repopulated the cache, and udm-boot.service exited 0.