Proxmox High Availability: Fencing, HA Rules, and Limits

Before you enable HA on a single VM, four things have to be true:

• Three nodes minimum. HA rides on corosync quorum, and a two-node cluster loses quorum the moment either node drops, so the survivor fences itself too. A QDevice supplies a third vote as a tiebreaker.
• Shared storage, or replication you accept losing data to. A guest can only restart on a node that can reach its disk.
• A working watchdog. Fencing in Proxmox VE means a node resets itself. Nothing verifies that reset from outside: the CRM waits out a timeout and assumes it happened, which is exactly why the watchdog has to be real.
• Capacity for the failure. If one node’s guests cannot fit on the survivors, failover just moves the outage.

HA restarts a guest somewhere else. It is an unattended reboot, not a seamless handover, and it is not a backup.

Proxmox VE’s HA stack is included at no extra cost, and it is routinely switched on before its prerequisites exist. The failure mode is not subtle: a cluster less reliable with HA than without it, because a flaky link now makes nodes power-cycle themselves.

What the HA stack actually does

Two daemons do the work, and they talk through the cluster filesystem rather than to each other directly.

pve-ha-crm (Cluster Resource Manager) runs on every node, but only one holds the master lock at a time. The master decides which node runs which resource, when a node counts as dead, and when recovery starts. pve-ha-lrm (Local Resource Manager) runs everywhere and executes those decisions locally, reporting its own state back so the CRM knows the node is healthy.

The coordination medium is pmxcfs, the replicated cluster filesystem at /etc/pve, and it explains everything else about this stack: pmxcfs goes read-only on any node that loses quorum. A node without quorum cannot update its LRM status or hold a resource lock, so it cannot legitimately claim to be running anything. Quorum is not advisory here. It is enforced by the filesystem the whole stack writes to.

Resources are added individually, not cluster-wide. A VM becomes an HA resource as vm:101, a container as ct:200:

ha-manager add vm:101 --state started --max_restart 2 --max_relocate 1
ha-manager status

States worth knowing: started, stopped (still managed, deliberately off), disabled, and ignored (the stack leaves the guest alone without you removing it). A resource that repeatedly fails to start lands in an error state and stays there on purpose, because a guest that will not boot will not boot better on the fifth attempt. Clear it by disabling and re-enabling once you have fixed the cause.

To watch the state machine before trusting it, install pve-ha-simulator: it runs the real CRM and LRM logic against a fake cluster you can break freely.

Shared storage is the prerequisite people skip

A guest cannot restart on node B if node B cannot open its disk. That rules out local-lvm and unreplicated local ZFS immediately. Options, roughly in order of fit:

• Ceph RBD. The hyperconverged answer, and the one Proxmox pushes. Storage and compute redundancy fail independently, so losing a node costs a replica rather than the datastore. The price is real: three or more nodes, a separate fast network, and enough understanding that size=3 / min_size=2 means something to you. See the Ceph storage guide.
• NFS or CIFS. The simplest path to working HA, and the reason a lot of homelab HA exists at all. It also concentrates all your risk in one box.
• iSCSI or Fibre Channel with LVM on top. Boring, well-understood shared block storage. Thick LVM long meant no snapshots at that layer; Proxmox VE 9 added snapshot support for thick LVM as a technology preview, so confirm what your installed release actually supports before you plan around it.
• Local ZFS plus pvesr replication. Proxmox will let you run HA on replicated local ZFS, but be explicit about what you are buying: recovery restores the guest from the last replicated snapshot, so everything written since is gone. With a one minute interval you are choosing “restart with up to a minute of data loss” over “no restart at all.” Fine for stateless workloads, bad for a database. See the ZFS storage guide.

Whichever you pick, the storage ID must be identical on every node. HA recovery dies on trivial asymmetry: a storage defined on two of three nodes, a bridge named vmbr1 on one host and vmbr2 on another, an ISO from local that exists only where the guest started.

Fencing: the watchdog is the whole mechanism

Before the CRM starts a guest elsewhere, it must be certain the original node is not still running it, because two nodes writing the same virtual disk destroys the filesystem inside it. Fencing supplies that certainty, and Proxmox does it with self-fencing via a watchdog timer, not by cutting power from outside:

1. A node loses quorum (network failure, corosync death, hard hang).
2. /etc/pve becomes read-only on that node, so the LRM can no longer refresh its lock.
3. watchdog-mux stops resetting the watchdog timer.
4. The timer expires and the node hard-resets itself.
5. The CRM waits out the fence interval, then recovers the guests elsewhere.

The watchdog timeout alone is 60 seconds by default, and the CRM deliberately waits longer than that before it touches the guests. Budget minutes, not seconds, for the whole sequence. HA is not a substitute for anything that needs sub-second failover.

By default Proxmox uses softdog, the kernel’s software watchdog. It works, but it is code inside the kernel that may be the thing that has hung. A hardware watchdog (IPMI/BMC, or your board’s own) is independent of OS state and more trustworthy; enabling one means pointing /etc/default/pve-ha-manager at the right kernel module.

The other half of fencing reliability is the corosync network. Corosync is latency-sensitive and interprets delay as death. Sharing its link with backup jobs, migration traffic, or Ceph replication is a classic cause of “my cluster randomly rebooted at 2am.” Give it a dedicated physical link plus a second ring as fallback. Bridges and VLANs covers the interface layout; the rings are configured during cluster creation.

HA rules: where a resource is allowed to run

By default an HA resource may run on any node in the cluster. Constraining that is the job of HA rules, stored in /etc/pve/ha/rules.cfg, and they come in two kinds.

Node affinity rules list the nodes a resource should run on, with an optional priority per node, highest wins. The strict flag decides how hard that list binds:

• --strict 1 means the resource may run only on the listed nodes. If none are available it stays stopped rather than landing somewhere unsuitable. This is what you want for a guest that depends on node-specific hardware.
• Non-strict is the default and treats the list as a preference: if no listed node is available, the resource is allowed to start elsewhere rather than stay down.

ha-manager rules add node-affinity gpu-hosts --resources vm:101 --nodes pve1:2,pve2:1 --strict 1

Resource affinity rules describe relationships between guests instead of nodes. positive keeps a set of resources together on one node; negative forces them apart, which is how you stop two halves of a redundant pair from landing on the same host.

Failback is a per-resource option rather than a rule flag. It defaults to on, meaning a resource moves back as soon as a higher-priority node returns. Set --failback 0 on anything that should not be dragged along by a node that flaps.

Anything doing GPU passthrough belongs in a strict node affinity rule at minimum, and is often better left out of HA entirely: the passed-through card makes it unmigratable and unlikely to boot on a host that lacks one.

On older releases the same job was done by HA groups in /etc/pve/ha/groups.cfg, with restricted where strict now sits and a per-group nofailback in place of the per-resource failback. Groups are deprecated. Proxmox VE 9 migrates existing ones to node affinity rules on its own and drops the group option from the guest HA dialog, so if you are following an older write-up and cannot find groups in the UI, that is why.

Shutdown policy and planned maintenance

Reboots for updates are far more common than node failures, and the default surprises people. Datacenter, Options, HA Settings:

• conditional (default): a reboot freezes HA resources and waits for the node to come back; a shutdown triggers failover.
• freeze: never fail over on a local shutdown or reboot.
• failover: recover resources elsewhere rather than waiting.
• migrate: move HA resources off the node before it goes down.

For a planned kernel update, migrate is what you want: guests move while still healthy instead of being killed and restarted. There is also an explicit maintenance mode that evacuates a node and keeps resources off it until you clear the flag, which survives a reboot rather than resetting on the way back up:

ha-manager crm-command node-maintenance enable pve2
ha-manager crm-command node-maintenance disable pve2

Containers cannot be live-migrated the way VMs can, so an LXC guest under HA is always restarted on the target. If a workload must survive planned maintenance without a restart, that is an argument for a VM (LXC vs VM has the rest).

What HA does not protect against

It is not a backup. HA faithfully restarts whatever state sits on shared storage. A dropped database table, a ransomware pass, or a botched in-guest upgrade is preserved perfectly and rebooted onto a healthy node. Backups are a separate, non-negotiable system: Proxmox Backup Server setup and the wider backup and restore strategy.

It is not application availability. The guest experiences a power cut. Filesystems replay journals, databases run crash recovery, anything mid-transaction is gone. Workloads needing real continuity need clustering inside the guests, with HA as a floor rather than a ceiling.

It does not make single-box shared storage redundant. If every node mounts the same NFS server and that server dies, HA has nowhere to recover to. That is a single point of failure with extra complexity on top. Accept it consciously, or move to Ceph.

It can turn a partial outage into a total one. If one switch failure drops every node below quorum, every node fences itself, and a cluster that might have limped along goes fully dark instead. Redundant corosync links on physically separate switches are the mitigation.

It does not create capacity. Size for N+1. If three nodes each sit at 80% memory, losing one leaves the survivors unable to start what they inherit, and HA keeps trying anyway. Match the hardware plan to the failure you intend to survive, not to steady state.

It does not survive configuration drift. Mismatched bridge names, a storage defined on two of three nodes, a host PCI device present on one machine: each produces a guest that fails to start after failover, when you are least equipped to debug it. Test by pulling power from a node in a maintenance window.

It is not a DR plan. Corosync’s latency tolerance makes a cluster stretched across sites fragile. Cross-site protection is replication and off-site backups, not a wider HA cluster.

A minimum that actually works

• Three nodes, or two plus an external QDevice (a compromise, not a design).
• A dedicated corosync link plus a second ring on separate switching.
• Ceph, or NFS with clear eyes about the single point of failure.
• A hardware watchdog wherever the BMC allows one.
• HA on the handful of guests that justify it, not all of them. Every HA resource can restart without warning at 3am.
• One rehearsed failure test before you rely on any of it.

Next steps

• Build the foundation with the Proxmox cluster setup guide, then the Ceph storage guide.
• When it misbehaves, common Proxmox problems and fixes covers the usual quorum and node-state symptoms.

See also

• Docker homelab tutorials ↗
• Home network firewall guides ↗