Proxmox Cluster Setup: Corosync, Quorum, and QDevice

The short version of Proxmox VE clustering:

• Three nodes, or two nodes plus a QDevice. A bare two-node cluster is a trap, not a cluster.
• Corosync wants its own network. It needs low, boring latency far more than it needs bandwidth.
• Quorum is the whole game. Lose it and /etc/pve drops to read-only across the survivors.
• A QDevice is a vote, not a node. Any small always-on Linux box can supply it.

Clustering is a management and availability layer. It is not shared storage, and it is not a backup.

When a Proxmox cluster misbehaves, the fault is usually not in the hypervisor. It is in Corosync, because someone built two nodes and expected one to survive the other, or because Corosync was left sharing a link with Ceph replication and backup traffic.

What clustering actually buys you

Running pvecm create gets you one management plane (a single web UI and API across every node, with one set of users and roles), a replicated config filesystem at /etc/pve, migration between nodes, and optional high availability for guests you explicitly enroll.

It does not get you shared storage. If two nodes have no common storage backend, a cluster does not invent one; that is a separate decision, covered in the ZFS storage guide and the Ceph storage guide. It also does not get you backups. Cluster membership protects against a node dying, not against someone deleting the wrong VM. That is what Proxmox Backup Server is for.

Corosync is the cluster

The piece that makes a Proxmox cluster a cluster is Corosync, a cluster engine that keeps every node agreeing on who is currently a member. On top of it sits pmxcfs, the Proxmox Cluster File System: a small FUSE filesystem mounted at /etc/pve, backed by a local SQLite database, whose writes are replicated to every other node through Corosync.

That design has a consequence people meet the hard way. /etc/pve is not an ordinary directory. It is a distributed database with a filesystem interface, and it enforces one rule: if the node is not part of a quorate cluster, /etc/pve becomes read-only. Every symptom of a broken cluster flows from that rule.

Modern Proxmox releases run Corosync 3 on the knet transport, and two practical things follow. Communication is unicast by default, so the multicast and IGMP-snooping troubleshooting that dominated older guides is largely obsolete. And Corosync supports multiple redundant links with automatic failover, a better answer to link redundancy than a bond.

The cluster-wide config lives at /etc/pve/corosync.conf, with a local copy at /etc/corosync/corosync.conf kept in sync by the cluster filesystem.

Quorum, and what losing it actually does

Each node gets one vote. A cluster is quorate when more than half the total votes are present. Three nodes need two, four need three, five need three.

When a node loses quorum:

• /etc/pve goes read-only. You cannot create, delete, or edit a VM, or change storage and firewall config.
• You cannot start, stop, or migrate guests. All of those write config or claim a lock in /etc/pve.
• Already-running guests keep running. The QEMU and LXC processes are not children of pmxcfs, so they carry on serving traffic and their disks keep working. You just cannot manage them.
• The web UI fills with gray question marks, because nodes cannot see each other’s status.

That third point matters. Loss of quorum by itself is a management outage, not a workload outage. The exception is HA, which turns it into a real one on purpose.

Check state with pvecm status for votes and the quorate flag, and pvecm nodes for membership. There is a manual override, pvecm expected 1, which lowers the expected vote count on the node you run it on so it becomes quorate alone. Treat it as a recovery tool for a cluster you know is genuinely down, never as configuration. Run it on the wrong side of a network partition and you have manufactured a split brain.

Why two nodes is a trap

Two nodes means two votes, and a majority of two is two. Both nodes must be present for the cluster to be quorate. So reboot one node for updates and the other goes read-only until it returns. Lose one to hardware failure and the survivor cannot start the VMs that were on the dead node. The exact scenario people build a cluster for is the one it cannot handle.

Corosync’s votequorum layer does have a two_node option, which sets quorum to one so a lone surviving node stays quorate. It is the wrong answer, and Proxmox does not configure it for you. Its companion wait_for_all, enabled automatically alongside it, does not fix that: it only withholds quorum until every node has been seen at least once, after which either node can again go quorate on its own. Once they can, a failure of the link between them (rather than of a node) gives you two independently quorate clusters. If they share storage, both can start the same VM against the same disk image. That is not a degraded cluster, it is a corrupted one, and it is rarely obvious until you next boot the guest.

If you enable HA on a two-node cluster, the trap has teeth. Proxmox HA uses watchdog-based self-fencing: a node running HA-managed guests that loses quorum stops updating its watchdog and hard-resets itself after roughly a minute, so the cluster can safely restart those guests elsewhere. On a two-node cluster the survivor also loses quorum, so it fences itself too. One node failure, both nodes down. Read Proxmox high availability before you tick the HA box.

QDevice: the correct fix for an even node count

A QDevice is an external vote. A machine outside the cluster runs corosync-qnetd and hands its vote to whichever partition should win; the cluster nodes run corosync-qdevice and talk to it.

On a two-node cluster this makes expected votes three. One node plus the QDevice is a majority, so the survivor stays quorate and can start the dead node’s guests. A link failure between the two nodes cannot produce two winners, because the QDevice only votes for one side.

Setup, once the two nodes are already a cluster:

• Install corosync-qnetd on the external host. It does not have to be a Proxmox node: a small Debian VM, a Raspberry Pi, or anything that runs a Debian package works, as long as it is always on and reachable from both nodes.
• Install corosync-qdevice on every cluster node.
• Run pvecm qdevice setup <qdevice-ip> from one node. Proxmox generates and distributes certificates over SSH, so you may need to temporarily permit root SSH login on the QDevice host, and you should put that setting back afterwards.
• Confirm with pvecm status, which should now list the QDevice among the voting members.

Three things to be clear about:

• QDevices are for even-numbered clusters. Proxmox’s own documentation discourages using one on a cluster with an odd number of nodes. The reason is that the QDevice behaves differently there: on an even cluster it supplies one extra tie-breaking vote, while on an odd cluster it supplies as many votes as all but one of the nodes. That shifts the failure model onto the QDevice, so losing the QDevice itself leaves no margin for a node to fail as well.
• It is a witness, not a node. It stores no VM data, runs no guests, and does not need to be fast. It needs to be independent: not a VM on either cluster node, and not on the same switch or power circuit if you can avoid it.
• It does not solve storage. A quorate survivor still needs access to the guest’s disks.

The network a cluster actually needs

Corosync’s traffic volume is trivial. Its latency requirements are not. It is a real-time membership protocol: when packets are delayed it concludes nodes are gone and reforms the cluster without them. So the single most valuable thing you can do for stability is give Corosync its own physical network.

• Do not share the Corosync link with storage or migration traffic. A Ceph rebalance, a live migration, or a backup window can saturate a link and add enough delay to trigger a spurious membership change. That makes /etc/pve read-only for a moment, and with HA enabled it can fence a healthy node.
• Prefer multiple Corosync links to a bonded link. Corosync 3 handles link failure itself, and Proxmox’s documentation warns that a single link backed by a bond can be problematic in some failure scenarios. Give it link0 on the dedicated network and link1 on a different NIC as a fallback. A bond hides the failure from Corosync and can hand it a latency spike instead of a clean switch to the other link.
• A separate VLAN is the compromise. Dedicated switch and NICs is ideal. With only one physical network, at least isolate Corosync on its own VLAN and subnet and apply QoS so bulk traffic cannot starve it. See Proxmox network bridges and VLANs.
• Wired only, no NAT. Nodes must reach each other directly by IP. Wi-Fi and address translation between nodes are both non-starters.
• Open UDP 5405 to 5412 between nodes, which is the range Proxmox documents for Corosync, one port per link. If you run a QDevice, also open TCP 5403 to the corosync-qnetd host. Account for both in your firewall rules.
• Keep clocks and names consistent. Run chrony or systemd-timesyncd everywhere, and make sure every node name resolves correctly from every other node, via /etc/hosts if DNS is unreliable.

Stretching one cluster across a WAN link is the classic failure: variable latency makes Corosync reform the cluster every time the link hiccups. Two independent clusters with replication between them is almost always the better design.

Building the cluster

Get the prerequisites right first, because one of them cannot be fixed afterwards without a reinstall.

• All nodes on the same Proxmox VE major version, installed the same way, per how to install Proxmox VE.
• Unique hostnames and unique VMIDs across the estate.
• Joining nodes must have no VMs or containers. Joining replaces the node’s /etc/pve with the cluster’s copy, and any guest configs on it are gone. Build the cluster before the guests.

Then, on the first node:

pvecm create mycluster --link0 10.10.10.1

On each node you are adding, pointing at the first node’s Corosync address:

pvecm add 10.10.10.1 --link0 10.10.10.2

Verify with pvecm status after each addition rather than adding every node and checking at the end. If the Corosync network is wrong, find out on node two.

Mistakes that cost people their cluster

• Rejoining a removed node. After pvecm delnode, that machine must be reinstalled before it goes near the cluster again. Its stale state will fight with the live one. It is a tempting rule to skip, because the hardware is fine and a reinstall is inconvenient, and skipping it produces cluster faults that are hard to trace back to the cause.
• Editing corosync.conf without bumping config_version. Nodes use that integer to decide whose config is newer. Forget to increment it and your change is ignored or applied inconsistently. Edit the copy in /etc/pve, increment the version, and only while the cluster is quorate.
• Turning on HA before the Corosync network is clean. HA turns a transient latency spike into a hard node reset. Run without HA first and watch journalctl -u corosync for unexplained membership changes.

If a cluster is already misbehaving, common Proxmox problems and fixes covers the read-only /etc/pve recovery path.

Next steps

• Size the nodes before you buy with Proxmox hardware requirements.
• Once the cluster is stable and storage is shared or replicated, add high availability to the guests that genuinely need it, and only those.