Knowledge Base

NVMe-oF

3 min read

NVMe over Fabrics (NVMe-oF)

Protocol extension that allows NVMe commands to be transported over a network fabric. NVMe/TCP is the standard choice for most deployments — it runs over standard Ethernet without special hardware, offers lower latency than iSCSI due to the leaner NVMe command set, and is well-supported since Linux 5.0.

Other transports (RDMA/RoCE, Fibre Channel) exist but require specialized NICs/switches and are not covered here.

NVMe/TCP

Exposes a local block device as an NVMe namespace accessible to remote hosts over TCP/IP.

NQN Naming

NVMe Qualified Names follow the format:

nqn.YYYY-MM.reverse-domain:identifier

Example: nqn.2024-01.com.example:storage-pool-1

Every host also has its own NQN (used for access control):

cat /etc/nvme/hostnqn   # view
nvme gen-hostnqn        # generate if missing

Basic Setup (Linux)

Target (server):

modprobe nvmet
modprobe nvmet-tcp
 
NQN="nqn.2024-01.com.example:storage-pool-1"
 
# Subsystem + namespace
mkdir /sys/kernel/config/nvmet/subsystems/$NQN
mkdir /sys/kernel/config/nvmet/subsystems/$NQN/namespaces/1
echo /dev/nvme0n1 > /sys/kernel/config/nvmet/subsystems/$NQN/namespaces/1/device_path
echo 1            > /sys/kernel/config/nvmet/subsystems/$NQN/namespaces/1/enable
 
# Port
mkdir /sys/kernel/config/nvmet/ports/1
echo 0.0.0.0 > /sys/kernel/config/nvmet/ports/1/addr_traddr
echo ipv4    > /sys/kernel/config/nvmet/ports/1/addr_adrfam
echo tcp     > /sys/kernel/config/nvmet/ports/1/addr_trtype
echo 4420    > /sys/kernel/config/nvmet/ports/1/addr_trsvcid
 
ln -s /sys/kernel/config/nvmet/subsystems/$NQN \
      /sys/kernel/config/nvmet/ports/1/subsystems/$NQN

Initiator (client):

modprobe nvme-tcp
 
nvme discover -t tcp -a <server-ip> -s 4420
nvme connect  -t tcp -a <server-ip> -s 4420 -n nqn.2024-01.com.example:storage-pool-1
 
nvme list   # shows the connected device, e.g. /dev/nvme1n1

Disconnect:

nvme disconnect -n nqn.2024-01.com.example:storage-pool-1
nvme disconnect-all   # disconnects everything

Production

Host Access Control

Never use attr_allow_any_host 1 in production. Instead, explicitly allow each initiator by its host NQN:

INITIATOR_NQN="nqn.2024-01.com.example:host-node1"
mkdir /sys/kernel/config/nvmet/subsystems/$NQN/allowed_hosts/$INITIATOR_NQN

Persistence

The configfs tree is lost on reboot. Use nvmetcli (Python tool) to save and restore it:

nvmetcli save /etc/nvmet.json     # snapshot current config
nvmetcli restore /etc/nvmet.json  # restore it
nvmetcli clear                    # wipe all config

Systemd service to restore on boot (/etc/systemd/system/nvmet.service):

[Unit]
Description=NVMe-oF Target
After=network.target
 
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=nvmetcli restore /etc/nvmet.json
ExecStop=nvmetcli clear
 
[Install]
WantedBy=multi-user.target

Multipath

NVMe native multipath merges multiple paths to the same subsystem into a single block device. Connect the same NQN over multiple IPs:

nvme connect -t tcp -a <ip1> -s 4420 -n <nqn>
nvme connect -t tcp -a <ip2> -s 4420 -n <nqn>
# kernel presents a single /dev/nvmeXnY

Enable multipath in the kernel module config:

# /etc/modprobe.d/nvme.conf
options nvme_core multipath=1

Check path status:

nvme list-subsys

Backing Store

The namespace device_path accepts:

TypeExample
Physical NVMe / block device/dev/nvme0n1, /dev/sdb
LVM logical volume/dev/vg0/lv_storage
Regular file (kernel 5.x+)/mnt/pool/disk.img

Network Recommendations

  • Dedicated storage VLAN — isolates storage traffic and simplifies access control.
  • Jumbo frames (MTU 9000) — reduces CPU overhead for large sequential I/O; must be set on both ends and all switches in the path.
  • Bind target to a specific IP — use the actual storage interface IP in addr_traddr instead of 0.0.0.0.
  • IRQ affinity — spread NVMe queue interrupts across cores for high-throughput workloads (irqbalance or manual smp_affinity).

Security

By default NVMe/TCP has no encryption. Options:

  • Network isolation — storage VLAN + firewall rules; simplest for trusted environments.
  • TLS (kernel 6.3+) — native in-kernel TLS for NVMe/TCP 1.1; requires nvme connect --tls and pre-shared keys or certificates.

Graph View