Proxmox Ceph vs ZFS Replication: When Ceph Is Worth It

Short version, before you commit a weekend to this:

• Ceph is justified when you need genuinely shared storage: HA failover with no rollback, live migration that moves only RAM, capacity that grows by adding nodes.
• ZFS replication is enough if you tolerate losing the minutes since the last sync and want local NVMe latency.
• Node floor is three. Three is where Ceph starts working. Four or five is where it heals itself after a node dies.
• Network floor is a dedicated 10 GbE link, kept away from Corosync. 25 GbE if buying new.
• Disks: enterprise SSD or NVMe with power-loss protection, one OSD per device, no RAID controller in the path.

Ceph is the only shared storage Proxmox VE can build out of the cluster’s own disks. Proxmox supports plenty of other shared storage (NFS, CIFS, iSCSI, Fibre Channel, ZFS over iSCSI), but every one of those means an external box you buy, power and maintain separately. Ceph makes the cluster itself, rather than a node, the owner of your VM disks. That is a real capability, and it is also the largest single jump in operational complexity you can make to a Proxmox install. Most homelab clusters that adopt it did not need it.

What Ceph actually buys you in Proxmox VE

Proxmox ships Ceph as a first-class component. You install it from the node’s Ceph tab or with pveceph install, create monitors and managers, turn disks into OSDs, and create an RBD pool that appears as cluster-wide storage. No separate appliance, no external cluster to babysit. Three things follow:

Live migration stops copying disks. With local ZFS storage, migrating a running VM means shipping its disk image across the wire. With an RBD pool the disk is already visible from every node, so migration moves only RAM state. Migration becomes routine rather than an event.

HA failover loses no committed writes. This is the honest reason to run Ceph. When a node dies, Proxmox HA restarts the guest elsewhere against the same block device, holding every write that reached it before the node went down. Be precise about what that is and is not: the guest still cold-boots, exactly as if you had pulled its power, so RAM state and anything in flight are gone. What you avoid is rolling back to a snapshot. If your recovery point objective is genuinely zero, shared storage is the only way there, but recovery time is still a boot. Fencing, groups and priorities are covered in the Proxmox high availability guide.

Capacity becomes a cluster property. Add a node with disks, add its OSDs, and the pool grows, with Ceph rebalancing onto the new hardware on its own. There is no equivalent in a set of independent ZFS pools.

Ceph also gives you CephFS, which Proxmox uses for ISOs, container templates and snippets, so those stop living on one node as a quiet single point of failure. CephFS needs at least one metadata server (MDS) alongside the monitors and managers.

Why ZFS replication is usually the right answer

Proxmox’s built-in storage replication (pvesr, the Replication tab on a guest) takes periodic ZFS snapshots and sends the delta to other nodes on a schedule. Writes land on local disk at local latency. No network in the write path, no quorum to satisfy, no second distributed system to operate.

The cost is real, so say it out loud: failover rolls the guest back to the last replicated snapshot. Replicate every five minutes and an unplanned node loss costs up to five minutes of writes. For a media server or a lab Kubernetes node, that is a shrug. For a database other systems reconcile against, it is not.

Stay on ZFS replication if most of these are true:

• Two or three nodes, with no plans to grow past that.
• Workloads that tolerate minutes of rollback on an unplanned failure.
• A 1 GbE or 2.5 GbE network, with re-cabling off the table.
• Consumer SSDs.
• Nobody who wants to learn a second distributed system.

The two-node case deserves calling out. Ceph does no useful work on two nodes. Proxmox clustering does, provided you add a QDevice so quorum survives a node loss. Two nodes plus a QDevice plus ZFS replication is a boring, effective design, and it is what most homelabs should run. Start from the ZFS storage guide and the cluster setup guide.

Three nodes is a floor, not a target

Ceph’s default CRUSH failure domain is the host, and Proxmox creates pools with three replicas and a minimum of two (size=3, min_size=2). Those defaults are correct. Do not lower them because you are short of disks.

What they mean on exactly three nodes:

• All three healthy: three copies, everything fine.
• One node down: two copies remain and the pool keeps serving IO, but Ceph cannot restore the third copy. There is no fourth host to put it on. You run degraded until that node returns.
• While degraded, a single disk failure on either survivor can drop affected placement groups below min_size, at which point those groups stop serving IO and the guests touching them freeze.

That is not a theoretical corner. Maintenance is the ordinary case, and every reboot puts a three-node cluster into the state above. At four or five nodes, Ceph re-replicates onto surviving hosts and genuinely self-heals. If you are building for resilience, build five.

Capacity follows the same logic. Three replicas means usable space is roughly a third of raw, before headroom. Ceph warns when OSDs cross a nearfull threshold (85% by default) and blocks writes near the full ratio (95% by default), and a Ceph cluster that stops accepting writes freezes every guest on it. For the cluster to absorb a lost node’s data during rebalance, that data has to fit in remaining free space. Plan to sit well under half of usable capacity.

The network is the part people underestimate

Ceph acknowledges a write only once the replicas have it, so every write pays a network round trip and latency matters more than raw bandwidth here. Proxmox’s guidance is a dedicated network at 10 Gbit/s or better, and that is a floor, not a recommendation. 1 GbE Ceph clusters exist, technically function, and are uniformly miserable under real IO.

Two specifics bite Proxmox clusters in particular.

Corosync must not share a link with Ceph. Corosync is latency-sensitive and drives cluster membership. A backfill after a failed disk saturates whatever link it is on, Corosync starts missing tokens, and in an HA-enabled cluster nodes fence themselves and reboot in the middle of an incident you were already having. Give Corosync its own physical link, ideally with a second ring elsewhere. Bridge and VLAN layout for that separation is in Proxmox network bridges and VLANs.

Ceph itself has two networks. The public network carries client traffic between guests and OSDs. The cluster network carries replication, recovery and backfill between OSDs. Splitting them keeps a recovery storm from starving production IO. On a small cluster you can share one fast dedicated link, but know you made that trade.

Three nodes and no budget for a 25 GbE switch? A full mesh is the standard workaround: direct point-to-point links between each pair of nodes. The Proxmox wiki’s Full Mesh Network for Ceph Server page covers the variants, from a simple routed setup through OpenFabric routing with fallback to an RSTP loop or plain broadcast. Cheaper than the switch, does not scale far, good fit for three-node hyperconverged builds.

Disks, OSDs and RAM

• Enterprise SSD or NVMe with power-loss protection. Not snobbery. Ceph’s OSD backend issues durable writes constantly, and consumer drives without PLP cannot acknowledge them from cache, so they fall through to the flash and collapse to a fraction of advertised performance. A cheap consumer SSD can be slower under Ceph than a hard disk. This is the most common cause of “my Ceph cluster is unusably slow”.
• One OSD per physical device, whole disks, and never on the device holding the Proxmox install.
• No RAID controller. Ceph wants raw devices, so use an HBA in IT mode. A RAID card hiding disks behind virtual volumes takes away exactly what Ceph needs.
• Budget around 8 GB of RAM per OSD. BlueStore’s osd_memory_target defaults to 4 GiB, and it is tempting to budget from that number, but it is a steady-state target rather than a ceiling. Proxmox’s own guidance is to allow roughly 8 GiB per OSD for good performance, precisely because recovery, rebalance and backfill push an OSD past its target when you can least afford it. Then add RAM for monitors, managers, any MDS, and the guests themselves. Ceph is a real tenant on your nodes. Size hosts with the Proxmox hardware requirements guide.
• Spinning disks fit VM images poorly. If you use them, put the OSD’s database and write-ahead log on flash and keep expectations low for random IO.

The operational cost nobody quotes you

This is the section that should decide it.

• You now operate two distributed systems. The Proxmox cluster has its own quorum via Corosync, Ceph has its own via monitors. They fail independently and are debugged differently, and you must know which one is broken before you can fix anything.
• Ceph upgrades are a separate, ordered procedure from Proxmox VE upgrades, on their own release cadence, in a documented sequence: monitors, managers, OSDs, then metadata servers. You do not get to skip the upgrade notes.
• Routine maintenance grows a checklist. Setting noout before rebooting a node, so Ceph does not rebalance during a five-minute reboot, is not optional. Forgetting turns a patch window into hours of pointless recovery traffic.
• Scrubbing and recovery are permanent background load. Ceph continuously verifies data and rebalances after topology changes, and that traffic competes with your guests.
• You have to learn to read the cluster. ceph -s, ceph osd df tree, and what a placement group in degraded, undersized or incomplete state means. The GUI shows the health status. It will not interpret it for you.
• Single-guest latency will not beat local NVMe. Ceph scales aggregate throughput across many clients. One guest doing single-threaded synchronous writes pays network round trips it would not pay locally. If the goal is “make this one database faster”, Ceph is the wrong tool.
• Ceph is not a backup. Replication protects against hardware failure, not deletion, ransomware or a bad rm. You still need Proxmox Backup Server pointed somewhere else entirely.

None of this argues against Ceph. It argues against adopting it without budgeting for the attention it needs afterwards.

A sane starting configuration

If Ceph is justified, this is a defensible build:

• Nodes: five if resilience is the goal, four if budget is tight, three only if you accept degraded operation every maintenance window.
• Monitors: three, on three different nodes, plus two managers.
• Networks: a dedicated Corosync link plus a separate 10 GbE or faster Ceph link, or a full mesh on three nodes.
• OSDs: identical enterprise NVMe or SSD with PLP, whole disks, three or four per node so recovery has somewhere to go.
• Pool: keep the Proxmox defaults of size=3 and min_size=2. The PG autoscaler ships in warn mode, which only tells you the placement group count is wrong; set it to on if you want it corrected for you.
• CephFS: one filesystem plus a couple of MDS daemons for ISOs, templates and snippets.

Ceph does support erasure coding for a better raw-to-usable ratio, but it fits the random-write pattern of VM disks poorly. Use replication for RBD.

Next steps

• Build and validate the cluster first: Proxmox cluster setup guide.
• Price the simpler path: Proxmox ZFS storage guide.
• Know what failover does: Proxmox high availability.
• When it breaks: common Proxmox problems and fixes.