Ilija Matoski

    ← Articles
    Containers

    Renaming a k3s node is not a cosmetic change

    I renamed four nodes because the old domain annoyed me. The cluster stayed green for weeks while sitting one reboot away from losing every worker.

    3 min read
    k3s kubernetes longhorn flannel tls turing pi

    I moved four k3s nodes from tpiN.ig3.c.example.com to tpiN.host.example.com because the old domain annoyed me, and I didn’t think about it any harder than that.

    k3s derives the node name from the full /etc/hostname, not the short form, and the running processes were still holding their pre rename names.

    So kubectl get nodes looked untouched, while the next restart was going to register all four under their new FQDNs. Longhorn keys replicas by node name, so that would have cut every volume loose from the node holding its data.

    The cluster stayed green for weeks while sitting one reboot away from losing every worker.

    Pin the hosts first

    I had deleted the old DNS records too, so the old names were NXDOMAIN and the agents were only still running because their load balancer held a cached address.

    cp -n /etc/hosts /etc/hosts.bak.pre-rename
    cat >> /etc/hosts <<'HOSTS'
    192.168.178.41 tpi1.host.example.com tpi1
    192.168.178.42 tpi2.host.example.com tpi2
    192.168.178.43 tpi3.host.example.com tpi3
    192.168.178.44 tpi4.host.example.com tpi4
    HOSTS

    We leave them there afterwards, so the cluster does not depend on DNS being up at all.

    Pin the node name

    This is the actual fix, and you want it in before the hostname changes rather than after.

    mkdir -p /etc/rancher/k3s
    printf 'node-name: tpi1\n' > /etc/rancher/k3s/config.yaml
    systemctl restart k3s

    On the workers the agent needs repointing as well, because the server address lives in the unit environment file rather than the config.

    sed -i "s|tpi1\.ig3\.c\.example\.com|tpi1.host.example.com|g" \
      /etc/systemd/system/k3s-agent.service.env
    systemctl daemon-reload && systemctl restart k3s-agent

    Clean up the duplicates

    Restarting the agents springs the trap, so four node objects become eight, the real ones going NotReady at 2y143d while impostors on the same hardware come up Ready beside them. Pinning the names brings them back with their ages intact.

    kubectl delete node tpi1.host.example.com
    kubectl patch node tpi1.host.example.com --type=merge \
      -p '{"metadata":{"finalizers":null}}'
    kubectl -n longhorn-system patch nodes.longhorn.io tpi1.host.example.com \
      --type=merge -p '{"spec":{"allowScheduling":false}}'

    The finalizer and the Longhorn scheduling flag are the two reasons a plain delete hangs on you.

    The certificate

    Three things that look like they should remove the old name do not: dropping --tls-san does nothing, deleting the certificate files does nothing because they come back from the etcd bootstrap bundle, and k3s certificate rotate gives you a new certificate carrying the same dead name.

    The annotations are only dynamiclistener’s request list. The certificate lives in the secret’s tls.crt, and dynamiclistener rewrites that to add a name, never to remove one.

    kubectl -n kube-system annotate secret k3s-serving \
      "listener.cattle.io/cn-tpi1.ig3.c.example.com-"
    kubectl -n kube-system delete secret k3s-serving
    rm -f /var/lib/rancher/k3s/server/tls/dynamic-cert.json
    systemctl restart k3s

    All three of those, in that order, and no CA material gets touched so nothing else needs reissuing.

    $ kubectl -n kube-system get secret k3s-serving -o jsonpath='{.data.tls\.crt}' \
      | base64 -d | openssl x509 -noout -text | grep -A2 "Subject Alternative Name"

    Ready is a weak signal

    After the reboots every node reported Ready and every pod was scheduled, while Longhorn volumes sat wedged in attaching because cross node pod traffic was gone.

    Routes were right, neighbour entries were right, and the forwarding database on flannel.1 was empty on all four nodes.

    $ bridge fdb show dev flannel.1
    2a:1b:3c:4d:5e:02 dst 192.168.178.42 self permanent
    2a:1b:3c:4d:5e:03 dst 192.168.178.43 self permanent
    2a:1b:3c:4d:5e:04 dst 192.168.178.44 self permanent

    That is one entry per peer, and empty means cross node pod traffic is dead however healthy the node list looks, with a k3s-agent restart repopulating it.

    The real gap this exposed is that node level config is the one part of this cluster with no version control and no backup, and that part is unchanged.