ProxmoxGuide
Isometric illustration of a Proxmox VE host running virtual machines above a SAS HBA card passed directly through to a storage guest
virtualization

Proxmox VE vs TrueNAS SCALE: Which Should Be the Host?

Proxmox VE vs TrueNAS SCALE: which one belongs on bare metal, when to run TrueNAS as a Proxmox VM, and why HBA passthrough is non-negotiable.

By ProxmoxGuide Editorial · · 7 min read

For a single box that has to do both virtualization and bulk storage:

  • Proxmox VE on bare metal, TrueNAS SCALE as a guest is the right default once you run more than two or three workloads.
  • The storage guest must own real disks through PCIe passthrough of a host bus adapter, not virtual disks carved out of host storage.
  • Passthrough costs you live migration for that guest, for as long as the card is attached to it. Plan the cluster around it before you build.
  • Never back Proxmox’s own guest storage with the NAS that Proxmox hosts. That is a boot deadlock waiting to happen.
  • Two separate boxes remains the least clever and most reliable answer if you have the power and rack space.

This gets framed as a product bake-off, which is the wrong frame. Proxmox VE is a hypervisor with good storage support. TrueNAS SCALE is a storage appliance that can also run VMs. Both are true, and every real design decision follows from which of those two jobs is your primary one. This guide covers the Proxmox side of that call: when Proxmox should be the base layer, what changes about a node once you virtualize a NAS on it, and the passthrough mechanics that separate a working build from an unrecoverable pool.

Hypervisor-first vs storage-first

Proxmox VE is Debian plus KVM/QEMU for full VMs, LXC for containers, and a clustering layer (corosync with the pmxcfs configuration filesystem) that ties nodes together. Its storage model is deliberately plural: ZFS, LVM-thin, plain directories, NFS, CIFS, iSCSI, and integrated Ceph all sit behind one storage abstraction, and any of them can back a guest disk. Storage there is a means to an end, and the end is running guests.

That plurality is the tell. A storage-first appliance commits to a single storage model and builds its UI, alerting, and upgrade path around that commitment. Proxmox does not commit, because it does not need to. What it commits to is the guest lifecycle: templates, linked clones, snapshots, backup jobs, migration, high availability, and per-guest resource limits.

So the question is not which product is better. It is which layer you want owning the boot process, the update cadence, and the failure domain of the machine.

When Proxmox should be the base layer

Put Proxmox on the metal when any of these hold:

  • You run more than a handful of guests, or you want a mix of containers and full VMs. LXC is a first-class citizen in Proxmox rather than a bolt-on, and the split matters for density. LXC vs VM on Proxmox covers where each one belongs.
  • You expect a second and third node. Cluster membership, shared-storage migration, and HA fencing are the entire reason the product exists. Retrofitting a cluster around an appliance you chose for storage reasons is painful.
  • You want the storage appliance to be replaceable. As a guest, TrueNAS is one VM among many: rebuild it, snapshot its boot disk before an upgrade, or swap it out without touching the machine’s boot path.
  • You are migrating off ESXi and want feature parity on guest management rather than on storage. Proxmox VE vs ESXi covers that transition specifically.

If your honest primary problem statement is “I have 60 TB and its integrity is the whole point, and I run three VMs,” the answer flips, and Proxmox is not pretending otherwise.

Why virtual disks are the wrong way to virtualize a NAS

The tempting shortcut is to give the storage guest a few large virtual disks on the host’s ZFS or LVM-thin pool. Do not do this for a pool you care about. Four things go wrong at once:

  • Stacked copy-on-write. The guest filesystem does its own allocation and checksumming on top of the host’s allocator, so every guest write is amplified by the layer underneath. You pay twice and gain nothing.
  • Double caching. The host’s cache (ARC, if the host pool is ZFS) and the guest’s ARC hold the same blocks in the same physical RAM. You are buying memory twice to hold one dataset.
  • No device visibility. The guest sees virtual devices, not drives. SMART attributes, real serial numbers, and firmware-level error reporting do not survive the abstraction, which means the guest cannot tell you which physical disk is failing.
  • Recovery is trapped. If the Proxmox host dies, the pool lives inside image files on host storage. A healthy pool on real disks can be pulled and imported on any machine with a controller. A pool living inside image files means rebuilding the host’s storage layer somewhere else first, before you can even reach the virtual disks the pool is made of.

Attaching individual block devices to the VM (qm set with a /dev/disk/by-id/... path) is better than image files but still wrong here. The guest talks to the hypervisor’s virtual SCSI layer rather than a controller, so ATA/SMART pass-through is unreliable and the host still mediates every IO. Use it for scratch data, not a redundant pool.

HBA passthrough: the mechanics

The correct approach is to hand the guest an entire storage controller so the guest’s driver talks to real hardware. On Proxmox that is standard VFIO passthrough:

  1. Enable IOMMU. Turn on VT-d or AMD-Vi in firmware first. Whether you also need a kernel command-line flag depends on your platform and kernel version, so verify rather than assume: check dmesg | grep -e DMAR -e IOMMU and confirm groups exist with find /sys/kernel/iommu_groups/ -type l | sort.
  2. Check the IOMMU group. Passthrough is group-granular, not device-granular. If your HBA shares a group with something the host needs, move the card to a different physical slot first. The ACS override kernel parameter is a last resort, and it is worth being clear about what it does: it adds no isolation, it tells the kernel to split the group anyway and assume an isolation guarantee the hardware never made. Acceptable on a single-user home box, not on anything multi-tenant.
  3. Keep the host off the card. Run lspci -nnk to get the PCI address, the vendor:device ID, and the driver currently bound (for Broadcom/LSI SAS cards this is typically mpt3sas). Blacklist that driver, or bind the device ID to vfio-pci, using a file in /etc/modprobe.d/. Then rebuild the initramfs with update-initramfs -u -k all, and on installs that use it, run proxmox-boot-tool refresh so the new initramfs actually reaches the boot partitions. Reboot, then confirm lspci -nnk reports vfio-pci as the driver in use.
  4. Attach the device to the VM. Add it as a hostpci entry (GUI: Hardware, Add, PCI Device) with All Functions selected so you get the whole card. Use the q35 machine type and OVMF/UEFI firmware; this is the well-trodden configuration for passthrough on Proxmox.
  5. Turn off ballooning. VFIO requires the guest’s memory to be pinned for DMA, so the full allocation is locked at boot regardless. Set a fixed memory size and leave the balloon device disabled. On a host that is otherwise overcommitted, that whole allocation has to be available up front or the guest will not start.
  6. Verify from inside the guest. Drives should appear with real model strings and serial numbers, and SMART queries should return data. If serials and model strings are missing, the guest is most likely still looking at virtualised disks rather than at the card.

One hardware rule that decides your build: do not pass through the controller your Proxmox boot disk sits on. On many boards the onboard SATA ports are a single controller, so if the host boots from SATA you cannot hand that controller to a guest. Boot Proxmox from NVMe or a separate controller, and put the pool drives on a dedicated add-in HBA. Plan this at purchase time; see Proxmox hardware requirements.

The same VFIO machinery is what drives Proxmox GPU passthrough, so if you get one working the other is mostly the same checklist with extra quirks.

What you give up on that node

Passthrough is not free, and the costs are structural rather than performance-related:

  • Live migration is gone for that guest. A VM holding a physical PCI device cannot be migrated to another node while running. Even in a cluster, that guest is pinned to the chassis holding the card.
  • HA cannot rescue it. High availability can restart a guest elsewhere, but the disks are physically in one box. There is nothing for the restarted guest to import.
  • Your Proxmox backups do not contain the NAS data. Backup jobs and Proxmox Backup Server capture the guest’s virtual disks, which here means the boot disk only. The pool lives on passed-through hardware Proxmox cannot see, so its protection has to be configured inside the guest and shipped to a separate target.
  • Host maintenance is now a storage outage. Every Proxmox reboot takes the NAS down with it, and everything that mounts those shares needs to tolerate it.

Start order, shutdown order, and the deadlock

Once storage is a guest, boot sequencing stops being cosmetic. Proxmox has per-guest start/shutdown ordering: set the storage guest to the lowest order number with a startup delay long enough for its pool to import and its shares to come up, then give dependent guests higher numbers. Shutdown runs in reverse, so the storage guest goes down last.

The failure mode to design out entirely: do not register an NFS or iSCSI export from the guest NAS as the Proxmox storage that holds your guests. On boot, Proxmox reaches for storage served by a VM that has not started; on shutdown it can hang flushing to storage it already stopped. Keep guest disks on local ZFS or LVM-thin. If you need shared storage across a cluster, that is Ceph’s job, not the NAS guest’s.

Mounting NAS shares inside guests for bulk data is fine, and so is using an export as a backup target, as long as you accept the ordering caveat and never make it load-bearing for the boot path.

When TrueNAS should be the base instead

Flip the layering when storage is the primary job and virtualization is occasional: a handful of long-lived VMs, no clustering ambitions, and a preference for one appliance with one update path. SCALE’s built-in VM support has changed its underlying implementation across releases, so treat hypervisor features there as a secondary capability rather than the product’s focus.

Running Proxmox as a guest of TrueNAS is possible but requires nested virtualization, complicates every troubleshooting session, and gives you the weaknesses of both designs. It is rarely the right build.

The unglamorous option

Two boxes: a small Proxmox node for compute, a dedicated NAS for storage. No passthrough, no IOMMU groups, no pinned guests, and either machine reboots without taking the other down. It costs watts and shelf space, and it is still the right answer more often than the single-box build people talk themselves into.

Next steps

See also

#proxmox#truenas#virtualization#pci-passthrough#homelab

Related

Comments