PCI Passthrough for NVMe Drives: Direct Storage Access for Storage VMs

NVMe passthrough is one of those ideas that sounds correct almost immediately.

If a virtual machine is supposed to behave like a storage system, why not let it touch the real drive?
Why keep stacking abstractions on top of abstractions?
Why hand a storage VM a virtual disk and then ask it to pretend that it still owns the storage story?

That instinct is not wrong.
It is just incomplete.

What finally made this topic click for me was realizing that NVMe passthrough feels clean because it changes who owns the truth. The guest stops seeing a curated storage object created by Proxmox and starts seeing a real PCIe storage device through VFIO and IOMMU isolation. For some storage workloads, that is exactly the right move. For others, it is an expensive way to feel more direct while quietly giving up host flexibility that was actually useful.

So this is not really a question about whether passthrough is more serious.
It is a question about where the storage logic should live.

For the technical baseline below, I use Proxmox VE documentation and separate documented platform behavior from my own placement and operating recommendations.

NVMe passthrough is not a faster virtual disk, it is a different ownership model

This distinction matters more than most first explanations make it sound.

In ordinary virtualization, the host remains the storage owner. Proxmox creates or exposes a virtual disk to the guest, and the VM works inside that abstraction. The guest may use that disk very effectively, but it is still living on top of host-managed storage decisions.

With PCI passthrough, the arrangement changes.

The Proxmox administration guide describes PCI passthrough as giving a virtual machine control over a PCI device from the host. The Linux VFIO documentation explains the deeper reason this works at all: the platform uses IOMMU-based isolation so a real device can be assigned safely to a guest instead of remaining a normal host-owned device.

That means an NVMe drive passed through as a PCI device is no longer merely “more direct storage.”
It is storage the guest actually owns at the hardware boundary.

This usually changes several things at once:

• the guest can use the device as a native NVMe drive rather than as a virtual block device;
• the host stops treating that drive as ordinary host storage;
• the VM becomes more tightly attached to that specific machine and slot topology;
• storage visibility, health checks, and filesystem decisions move toward the guest.

That last point is why passthrough feels so attractive for storage VMs in the first place.

Storage VMs are one of the few places where direct NVMe ownership often makes immediate sense

A storage VM is not just another application server.

Its whole job is to reason about storage.
It may want to build its own filesystem layout, manage redundancy, inspect device health more directly, or treat disks as first-class building blocks rather than as generic virtual volumes handed down from above.

That is where NVMe passthrough begins to look coherent instead of merely impressive.

If the guest is supposed to make the important storage decisions, then giving it a raw virtual disk can sometimes feel like giving a carpenter a photograph of wood instead of lumber. The image is useful, but it is not the material itself.

This is the strongest argument for passthrough.

The guest sees the real device.
Its storage stack can own the arrangement.
The mental model becomes simpler:

• Proxmox owns the host and the hypervisor layer;
• the storage VM owns the passed-through NVMe;
• the filesystem logic lives where the storage logic is expected to live.

That division can be much cleaner than making the guest believe it has low-level control while the host still quietly holds the real cards.

Virtual disks are still the better answer more often than enthusiasts like to admit

This is the part that saves people from overbuilding.

A storage VM does not automatically justify passthrough.
Sometimes what it really needs is dependable capacity, snapshot-friendly operation, easier migration, simpler backup behavior, and fewer node-specific hardware assumptions.

In that world, a virtual disk can be the wiser answer.

If the VM is mainly serving files, hosting an application stack, or exposing storage services without needing deep device-level ownership, then the host abstraction may be a feature rather than a compromise.

The design stays easier to move.
The host remains in charge of placement.
Proxmox storage tooling remains more naturally involved.

That matters because passthrough narrows your options very quickly.

The moment the guest owns the NVMe:

• the host cannot treat that drive as general Proxmox storage;
• the VM becomes less portable;
• node replacement gets more hardware-sensitive;
• your recovery plan starts depending on how that guest expects to see the device.

For many homelab workloads, that is a bigger cost than the performance argument is worth.

The first hard limit is not the VM. It is the platform

Before Proxmox settings even matter, the hardware platform decides whether the plan is realistic.

The Proxmox documentation is straightforward here: PCIe passthrough requires IOMMU support on the platform, which means Intel VT-d or AMD-Vi on x86. The Linux VFIO documentation adds the crucial nuance that devices are not always isolated in perfect one-device chunks. In practice, the unit that matters is often the IOMMU group.

This is the part that ruins many elegant whiteboard diagrams.

You may want to pass through one NVMe device.
The platform may decide that the meaningful isolation boundary includes more than that device.

That is why passthrough planning is partly motherboard planning, slot planning, and topology planning.
The NVMe drive itself is not the whole story.
The way the platform exposes it is the other half.

This matters even more with compact systems and mini PCs, where layout choices are often tighter and cleaner-looking than they are flexible.

Passing through the wrong NVMe can make the host less clever than it needs to be

This is the practical question I always come back to:

which NVMe are we talking about?

If it is a secondary drive dedicated to the storage VM, the design can be perfectly reasonable.
If it is the host boot NVMe, or a device the host is already relying on for its own storage stack, then passthrough stops being elegant and starts becoming self-defeating.

One of the easiest ways to make a homelab design brittle is to confuse “possible” with “architecturally kind.”

Just because a device can be assigned does not mean the host can afford to stop owning it.

That is why I like a very plain rule of thumb here:

• passthrough a device that has a clear guest-only purpose;
• do not passthrough a device whose removal makes the host awkward, blind, or needlessly fragile.

This is also where broader storage choices still matter. If you have not yet thought through the tradeoffs in Proxmox storage types or in which layer should own your storage backend, passthrough can arrive before the storage plan is actually mature.

The biggest cost after success is usually flexibility

The funny thing about passthrough is that the trouble does not always begin when it fails.

Sometimes the real cost appears only after everything works.

The Proxmox documentation notes that passed-through PCI devices block live migration. That one sentence explains a lot. Once a VM directly owns an NVMe device, that VM stops behaving like an ordinary movable workload and starts behaving like a machine with a local physical dependency.

That changes operations immediately:

• maintenance becomes node-specific;
• cluster mobility becomes smaller;
• HA expectations need to be more realistic;
• spare-node assumptions become weaker;
• disaster recovery planning becomes more deliberate.

None of that means passthrough is wrong.
It just means the design should be honest about what it becomes after the initial thrill wears off.

This is one reason I rarely think of NVMe passthrough as a “performance feature.”
I think of it as a local-ownership feature.
And local ownership always trades against portability.

Firmware and VM layout still matter because passthrough affects the whole guest model

Even though this article is about storage rather than GPUs, the broader Proxmox passthrough logic still applies.

The Proxmox QEMU documentation notes that OVMF and the Q35 machine type are often relevant for PCIe passthrough scenarios. That matters because once you start handing real PCIe devices into a VM, the guest is no longer just an abstract compute sandbox. Its firmware and bus model become part of how cleanly that device is represented.

That does not mean every NVMe passthrough setup becomes exotic.
It means passthrough reaches deeper than one checkbox.

The VM definition itself starts carrying more responsibility.

The cleanest use case is usually a dedicated storage VM with a dedicated NVMe

This is the scenario that feels most defensible to me.

A specific VM exists for storage duties.
A specific NVMe exists for that VM.
The host does not need the device for its own daily storage role.
The platform supports clean isolation.
The operator understands that migration flexibility is reduced.

That is a good passthrough story.

It is narrow.
It is deliberate.
It does not pretend that direct access is free.

What I distrust more are vague versions of the same idea:

• “Maybe I will pass through the host’s fastest drive and sort the rest out later.”
• “Maybe this VM will still move cleanly if I need it to.”
• “Maybe I do not need to think too hard about hardware replacement.”

That kind of optimism is exactly how a neat lab turns into an inherited puzzle.

How I think about NVMe passthrough now

I no longer see it as the advanced option by default.

I see it as the honest option in a narrow set of designs.

If the storage VM needs real device ownership for its own filesystem, health visibility, and storage logic, then NVMe passthrough can be beautifully coherent. The host stops pretending to be the storage brain for a workload that wants to think for itself.

If the VM mainly needs reliable capacity and the host benefits from staying in charge, then virtual disks remain the calmer answer.

That distinction made the whole topic much easier for me.
The question is not whether passthrough is more powerful in the abstract.
The question is which layer deserves to own the storage truth.

Conclusion

PCI passthrough for NVMe drives is a strong design when a storage VM genuinely needs direct ownership of the physical device and when the platform can isolate that hardware cleanly through IOMMU and VFIO. In that scenario, the guest gets a truer storage relationship, and the architecture becomes more honest about where storage decisions are really being made.

But passthrough is not a free upgrade. It takes the device away from the host, blocks the usual migration flexibility that makes virtualization so forgiving, and ties the VM more tightly to one machine’s hardware reality. That is why I think NVMe passthrough should be chosen not as a speed reflex, but as an ownership decision. When the guest deserves the device, it can be exactly right. When it only needs space, the virtual path is often the better one.