Proxmox ZFS Storage Setup: Zvols, volblocksize, and ARC

For running Proxmox VE on ZFS, the decisions that actually matter are:

• Root-on-ZFS earns its place through a bootable mirror and checksummed rollbacks, not through speed.
• Keep the root pool small and separate from the pool holding VM disks.
• volblocksize is fixed when a zvol is created. Set it on the storage before you create disks.
• RAIDZ plus a small volblocksize wastes capacity. Mirrors are the sane default for VM workloads.
• Cap zfs_arc_max explicitly. ARC does not shrink fast enough to keep up with a guest that allocates its memory in one go.

ZFS on a hypervisor is a different problem from ZFS on a file server. The unit of storage is a block device handed to a guest, not a directory of files, and the RAM ZFS wants for cache is the same RAM your virtual machines want for themselves. Almost every Proxmox ZFS mistake traces back to one of those two facts.

Root-on-ZFS: what you get and what it costs

The Proxmox installer will put the host itself on ZFS, offering pool topologies (mirror, striped mirrors, RAIDZ) on the disk selection screen. The payoff is a boot pool that survives a dead disk without mdraid, root snapshots you can take before anything risky, and checksums on the hypervisor’s own filesystem, the layer people forget to protect while carefully protecting guest data.

Proxmox handles the bootloader: on a ZFS root install it keeps boot files on each pool member and syncs them with proxmox-boot-tool, so losing one mirror disk does not stop the host booting. The detail that bites people: after replacing a failed member, zpool replace restores redundancy but does not make the new disk bootable. Initialising its boot partition is a separate proxmox-boot-tool format and proxmox-boot-tool init step, after the partition table has been copied over. Practise it before you need it at 2am.

You do not get managed boot environments: rolling the root dataset back to a snapshot is a manual console procedure, not a menu entry.

Two more things up front. Root-on-ZFS is not free in RAM: the host filesystem now competes for the same ARC as your guest disks. And avoid swap on a zvol. Proxmox’s documentation warns that swap space created on a zvol may cause trouble, such as blocking the server or generating a high IO load, and points you at a separate swap device instead. Use a non-ZFS partition, or no swap at all.

Either way, use a small mirrored pair for rpool and a separate pool for VM disks. You can then rebuild the hypervisor without touching guest data, give each pool the topology it wants, and stop runaway snapshots from also preventing boot. On a small host, ext4 on a modest SSD plus a ZFS pool for guests is defensible. Decide at install time, because changing it later means reinstalling: see hardware requirements and how to install Proxmox VE.

How Proxmox actually stores a VM disk on ZFS

ZFS storage (the zfspool type). Each VM disk becomes its own zvol, a raw block device handed straight to QEMU. There is no qcow2 layer, and snapshots and clones are native ZFS operations, which is why snapshotting a VM here is close to instant regardless of disk size. This is what you want for VM disks.

Directory storage on a ZFS dataset. Proxmox writes qcow2 or raw files into a directory that lives on ZFS. That is the right home for ISOs, templates and backups. For live VM disks it stacks qcow2’s copy-on-write on ZFS’s, which buys nothing and costs write amplification.

Containers are a separate case: on ZFS storage an LXC container gets a dataset with a refquota for its size, not a zvol, so the block-size discussion below does not apply to them at all. That is one more input into the LXC vs VM decision.

The storage’s Thin provision setting controls whether zvols are sparse. Useful, but it lets you oversubscribe a pool, and ZFS behaves badly as a pool approaches full. Alert on real usage.

volblocksize: the one setting you cannot change later

Every zvol has a volblocksize, set at creation and immutable for the life of that zvol. In Proxmox it comes from the ZFS storage’s Block Size field (blocksize in /etc/pve/storage.cfg). Change it and only disks created afterwards pick up the new value; existing disks keep what they were born with.

To change an existing disk you create a new one and copy: Disk Action > Move Storage on the VM’s Hardware tab, targeting a storage whose Block Size is the value you want, or restore a backup into it. There is no in-place conversion.

The tradeoff runs both ways:

• Too small and every guest write carries proportionally more metadata and checksum overhead, the pool sees more IOPS than the guest issued, and compression has less to work with because it operates per block.
• Too large and partial writes become read-modify-write. A 4K write into a 64K block means reading the whole block, modifying it, and writing it back. On a database guest that is a serious penalty.

Practical guidance:

• Set ashift correctly at pool creation (12, meaning 4K sectors, for anything modern). Like volblocksize it cannot be changed later, and it is the floor everything else sits on.
• General-purpose Linux and Windows guests are well served by 16K.
• Database guests want the block size lined up with the database’s page size, and the guest filesystem laid out to match. Getting this wrong is where most “ZFS is slow for VMs” complaints come from.
• OpenZFS raised its default zvol block size from 8K to 16K in 2.2. Proxmox only passes an explicit block size to ZFS when the storage defines one, so a storage with no Block Size set inherits whatever the installed OpenZFS defaults to. A storage definition carried over from an older install may still pin 8K, so read /etc/pve/storage.cfg rather than assuming.

RAIDZ turns volblocksize into a capacity problem

On mirrors a block is written whole to each side and the arithmetic is obvious. On RAIDZ, each block is carved into sectors spread across the data disks with parity added, then padded to keep allocations aligned. With 4K sectors and a small volblocksize, that parity-plus-padding overhead is far larger than the naive “one disk’s worth of parity” people budget for. Because zvols are usually thin provisioned, you often do not discover it until the pool is far fuller than predicted.

Mirrors sidestep it, and bring two other things a hypervisor cares about: better random IOPS, and much faster resilvering. Resilver speed is not an abstract virtue when your VMs are still serving traffic during the rebuild.

Our position: striped mirrors for anything backing VM disks. Keep RAIDZ for bulk datasets where you store files, such as ISO libraries and backup targets. If capacity pressure forces RAIDZ under zvols, use a larger volblocksize and verify real consumption before committing the pool. Weighing ZFS against a clustered option? Proxmox Ceph storage is the alternative, and Proxmox vs TrueNAS SCALE covers a dedicated storage box.

Sizing ARC when guests compete for the same RAM

ARC is ZFS’s read cache and it lives in host RAM. On a file server, letting it grow to consume most of the machine is correct. On a hypervisor it is not, because that RAM has another claimant.

Left alone, ARC on Linux grows toward half of system memory, or 62.5% on OpenZFS 2.3 and newer. Since Proxmox VE 8.1 the installer sets something far more conservative on new installs: 10% of host memory, clamped to a maximum of 16 GiB. Hosts installed before that and upgraded in place keep the old ZFS behaviour, so check rather than assume. Either way, do not leave it implicit on a hypervisor.

This is where “ARC is reclaimable, so it is fine” breaks down. ARC memory can be given back, but reclaim takes time, and a guest booting or allocating its heap wants pages immediately. If ARC holds most of the machine you get stalls, and in the worst case the OOM killer picks a QEMU process, which inside the guest looks like an unexplained power cut.

A budget that works: total the maximum memory of every guest you will run at once, add a few gigabytes for PVE services and the kernel, and cap ARC below what remains, with headroom rather than the exact remainder. Guest memory is the part you cannot claw back, since ballooning only helps cooperatively and only for KVM guests with a working driver. ARC is the knob you control.

How to set it. Put options zfs zfs_arc_max=<bytes> (a plain byte count) in /etc/modprobe.d/zfs.conf, regenerate the initramfs with update-initramfs -u -k all, and reboot. The initramfs step matters because root-on-ZFS mounts the pool before the normal filesystem exists. Writing to /sys/module/zfs/parameters/zfs_arc_max changes it at runtime but does not persist. Setting zfs_arc_min stops the cache collapsing to nothing under transient pressure.

arc_summary and arcstat tell you whether the cap is sensible. A high hit ratio while guests are starved means you can take RAM away from ARC; a poor hit ratio on a busy pool means ARC is too small for the working set.

Finally, do not stack caches. Leave the VM disk cache mode at the default, No cache, which bypasses the host page cache. Writeback is not unsafe in itself, since the guest is told there is a writeback cache and its flushes are honoured, but it keeps a second copy of the same data in host RAM on top of ARC, which is exactly the memory you just finished budgeting. The mode that does break the guarantees your guest filesystem relies on is Write back (unsafe), which ignores guest flushes outright: keep it for throwaway installs. Be sceptical of L2ARC too, since its index consumes ARC.

Settings worth changing, and ones to leave alone

• Compression: leave it on. LZ4 is effectively free and reduces bytes actually written. Consider zstd for backup or archive datasets, not live VM disks.
• Discard: enable it on the guest disk, with the SSD flag where appropriate, so deletions inside the guest release space in a thin zvol. Without it, a thin disk only ever grows.
• sync: leave it at standard. sync=disabled produces impressive numbers and silently discards writes the guest believes are committed. That is a data-loss setting, not a tuning setting.
• atime=off on VM-backing datasets removes pointless metadata writes.
• Special vdevs are sharp. Putting metadata and small blocks on fast devices helps, but losing that vdev loses the entire pool. Mirror it or skip it.

What ZFS on Proxmox does not do

ZFS is a local filesystem. Two nodes cannot mount the same pool, so a ZFS pool is never shared cluster storage. Proxmox’s built-in storage replication sends ZFS snapshots between nodes on a schedule, which is useful, but it is asynchronous: worst-case data loss equals the replication interval. That is disaster recovery, not shared storage, and the distinction matters as soon as you plan high availability or a cluster.

Snapshots are also not backups. They live in the same pool as the data they protect, so they do nothing about a pool loss or a mistake that takes the host with it. Pair ZFS with a real backup target, which is what Proxmox Backup Server exists for.

Next steps

• Decide root layout and pool topology before you install, not after.
• Set the storage Block Size before you create your first VM disk, then cap ARC against your real guest memory budget.