Knowledge Base
NVMe over Fabrics (NVMe-oF)
3 min read

Protocol extension that allows NVMe commands to be transported over a network fabric. NVMe/TCP is the standard choice for most deployments — it runs over standard Ethernet without special hardware, offers lower latency than iSCSI due to the leaner NVMe command set, and is well-supported since Linux 5.0.

Other transports (RDMA/RoCE, Fibre Channel) exist but require specialized NICs/switches and are not covered here.

NVMe/TCP

Exposes a local block device as an NVMe namespace accessible to remote hosts over TCP/IP.

NQN Naming

NVMe Qualified Names follow the format:

nqn.YYYY-MM.reverse-domain:identifier

Example: nqn.2024-01.com.example:storage-pool-1

Every host also has its own NQN (used for access control):

sh
cat /etc/nvme/hostnqn # view
nvme gen-hostnqn # generate if missing

Basic Setup (Linux)

Target (server):

sh
modprobe nvmet
modprobe nvmet-tcp

NQN="nqn.2024-01.com.example:storage-pool-1"

# Subsystem + namespace
mkdir /sys/kernel/config/nvmet/subsystems/$NQN
mkdir /sys/kernel/config/nvmet/subsystems/$NQN/namespaces/1
echo /dev/nvme0n1 > /sys/kernel/config/nvmet/subsystems/$NQN/namespaces/1/device_path
echo 1 > /sys/kernel/config/nvmet/subsystems/$NQN/namespaces/1/enable

# Port
mkdir /sys/kernel/config/nvmet/ports/1
echo 0.0.0.0 > /sys/kernel/config/nvmet/ports/1/addr_traddr
echo ipv4 > /sys/kernel/config/nvmet/ports/1/addr_adrfam
echo tcp > /sys/kernel/config/nvmet/ports/1/addr_trtype
echo 4420 > /sys/kernel/config/nvmet/ports/1/addr_trsvcid

ln -s /sys/kernel/config/nvmet/subsystems/$NQN \
/sys/kernel/config/nvmet/ports/1/subsystems/$NQN

Initiator (client):

sh
modprobe nvme-tcp

nvme discover -t tcp -a <server-ip> -s 4420
nvme connect -t tcp -a <server-ip> -s 4420 -n nqn.2024-01.com.example:storage-pool-1

nvme list # shows the connected device, e.g. /dev/nvme1n1

Disconnect:

sh
nvme disconnect -n nqn.2024-01.com.example:storage-pool-1
nvme disconnect-all # disconnects everything

Production

Host Access Control

Never use attr_allow_any_host 1 in production. Instead, explicitly allow each initiator by its host NQN:

sh
INITIATOR_NQN="nqn.2024-01.com.example:host-node1"
mkdir /sys/kernel/config/nvmet/subsystems/$NQN/allowed_hosts/$INITIATOR_NQN

Persistence

The configfs tree is lost on reboot. Use nvmetcli (Python tool) to save and restore it:

sh
nvmetcli save /etc/nvmet.json # snapshot current config
nvmetcli restore /etc/nvmet.json # restore it
nvmetcli clear # wipe all config

Systemd service to restore on boot (/etc/systemd/system/nvmet.service):

ini
[Unit]
Description=NVMe-oF Target
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=nvmetcli restore /etc/nvmet.json
ExecStop=nvmetcli clear

[Install]
WantedBy=multi-user.target

Multipath

NVMe native multipath merges multiple paths to the same subsystem into a single block device. Connect the same NQN over multiple IPs:

sh
nvme connect -t tcp -a <ip1> -s 4420 -n <nqn>
nvme connect -t tcp -a <ip2> -s 4420 -n <nqn>
# kernel presents a single /dev/nvmeXnY

Enable multipath in the kernel module config:

sh
# /etc/modprobe.d/nvme.conf
options nvme_core multipath=1

Check path status:

sh
nvme list-subsys

Backing Store

The namespace device_path accepts:

TypeExample
Physical NVMe / block device/dev/nvme0n1, /dev/sdb
LVM logical volume/dev/vg0/lv_storage
Regular file (kernel 5.x+)/mnt/pool/disk.img

Network Recommendations

• Dedicated storage VLAN — isolates storage traffic and simplifies access control.
• Jumbo frames (MTU 9000) — reduces CPU overhead for large sequential I/O; must be set on both ends and all switches in the path.
• Bind target to a specific IP — use the actual storage interface IP in addr_traddr instead of 0.0.0.0.
• IRQ affinity — spread NVMe queue interrupts across cores for high-throughput workloads (irqbalance or manual smp_affinity).

Security

By default NVMe/TCP has no encryption. Options:

• Network isolation — storage VLAN + firewall rules; simplest for trusted environments.
• TLS (kernel 6.3+) — native in-kernel TLS for NVMe/TCP 1.1; requires nvme connect --tls and pre-shared keys or certificates.