ZeroFS

23 May 20268 min read

Transforms S3 object storage into a functional filesystem. Exposes file-level access via NFS and 9P, and block-level access via NBD. Always-encrypted with XChaCha20-Poly1305 + LZ4/Zstd compression. No FUSE drivers or kernel modules required.

Built on top of SlateDB — an LSM-tree key-value store — which stores all file metadata and 32KB data chunks as key-value pairs directly in S3 objects.

Architecture

Clients (NFS / 9P / NBD / Web UI)

│

Protocol Servers

│

Virtual Filesystem (VFS)

│

Encryption + Compression

│

Multi-layer Cache (memory → disk → S3)

│

SlateDB (LSM tree)

│

Object Store (S3 / Azure / GCS / local)

Files are broken into 32KB chunks stored as key-value entries in the LSM tree. SlateDB compacts asynchronously, reclaiming space and optimizing reads.

Access Modes

Protocol	Use case
9P	Recommended — best POSIX semantics, accurate fsync, Linux/container-native
NFS	Broad OS compatibility (Linux, macOS)
NBD	Raw block device — run ZFS, ext4, btrfs, etc. directly on S3
Web UI	Browser-based file manager + terminal + live dashboard

Installation

bash

# Install script

curl -sSfL https://sh.zerofs.net | sh

# Via Cargo

cargo install zerofs

# Docker

docker pull ghcr.io/barre/zerofs:latest

Configuration

TOML format. Supports ${ENV_VAR} substitution throughout.

toml

[cache]

dir = "${HOME}/.cache/zerofs" # local cache directory

disk_size_gb = 10.0 # persistent disk cache size

memory_size_gb = 1.0 # optional in-memory block cache

[storage]

url = "s3://bucket/path" # backend URL

encryption_password = "${PASSWORD}" # required

[filesystem]

compression = "zstd-3" # zstd-{1..22} or lz4

max_size_gb = 100.0 # optional quota; default ~16 EiB

[servers.nfs]

addresses = ["127.0.0.1:2049"]

[servers.ninep]

addresses = ["127.0.0.1:5564"]

unix_socket = "/tmp/zerofs.9p.sock" # lower latency than TCP

[servers.nbd]

addresses = ["127.0.0.1:10809"]

unix_socket = "/tmp/zerofs.nbd.sock"

[servers.webui]

addresses = ["127.0.0.1:8080"]

uid = 1000

gid = 1000

[servers.rpc]

addresses = ["127.0.0.1:7000"]

unix_socket = "/tmp/zerofs.rpc.sock" # required for checkpoints & monitoring

[lsm]

wal_enabled = false # enable for fsync-heavy workloads

max_concurrent_compactions = 4

Storage Backends

AWS S3 / S3-compatible:

toml

[storage]

url = "s3://my-bucket/path"

[aws]

access_key_id = "${AWS_ACCESS_KEY_ID}"

secret_access_key = "${AWS_SECRET_ACCESS_KEY}"

endpoint = "https://s3.us-east-1.amazonaws.com"

default_region = "us-east-1"

allow_http = "true" # MinIO only

conditional_put = "redis://localhost:6379" # S3-compatible stores lacking conditional PUT

Azure Blob:

toml

[storage]

url = "azure://container/path"

[azure]

storage_account_name = "${ACCOUNT_NAME}"

storage_account_key = "${ACCOUNT_KEY}"

Google Cloud Storage:

toml

[storage]

url = "gs://my-bucket/path"

[gcp]

service_account = "/path/to/service-account-key.json"

# Or: GOOGLE_APPLICATION_CREDENTIALS env var

Local Filesystem:

toml

[storage]

url = "file:///path/to/storage"

Separate WAL Storage

Route the write-ahead log to a lower-latency store (e.g. local NVMe):

toml

[wal]

url = "file:///mnt/nvme/zerofs-wal"

[wal.aws]

access_key_id = "${WAL_KEY}"

secret_access_key = "${WAL_SECRET}"

CLI

bash

zerofs init # generate default zerofs.toml

zerofs run -c zerofs.toml # start filesystem

zerofs run -c zerofs.toml --read-only # read-only replica

zerofs run -c zerofs.toml --no-compactor # disable background compaction

zerofs run -c zerofs.toml --checkpoint NAME # start from a checkpoint

zerofs checkpoint create -c zerofs.toml NAME # create snapshot

zerofs checkpoint list -c zerofs.toml

zerofs checkpoint info -c zerofs.toml NAME

zerofs checkpoint delete -c zerofs.toml NAME

zerofs monitor -c zerofs.toml # real-time TUI dashboard

zerofs monitor -c zerofs.toml --interval 500 # refresh rate in ms

zerofs compactor -c zerofs.toml # standalone compaction process

zerofs change-password -c zerofs.toml # update encryption key (reads from stdin)

Mounting

9P (recommended)

bash

# TCP

mount -t 9p -o trans=tcp,port=5564,version=9p2000.L,cache=mmap,access=user 127.0.0.1 /mnt/9p

# Unix socket (lower latency)

mount -t 9p -o trans=unix,version=9p2000.L,cache=mmap,access=user /tmp/zerofs.9p.sock /mnt/9p

9P sends a proper fsync signal to ZeroFS, guaranteeing durability — unlike NFS which buffers writes in memory and reports them as stable.

NFS

bash

# Linux

mount -t nfs -o async,nolock,rsize=1048576,wsize=1048576,tcp,port=2049,mountport=2049,hard 127.0.0.1:/ /mnt

# macOS

mount -t nfs -o async,nolocks,rsize=1048576,wsize=1048576,tcp,port=2049,mountport=2049,hard 127.0.0.1:/ /mnt

NFS writes are reported as stable to clients even when still in memory/cache — trades durability for performance.

NBD

bash

# TCP

nbd-client 127.0.0.1 10809 /dev/nbd0 -N device1 -persist -timeout 600 -connections 4

# Unix socket

nbd-client -unix /tmp/zerofs.nbd.sock /dev/nbd0 -N device1 -persist -timeout 600 -connections 4

NBD devices are stored as files under .nbd/ in any mounted ZeroFS volume. Create a device before connecting:

bash

mount -t nfs 127.0.0.1:/ /mnt/zerofs

truncate -s 100G /mnt/zerofs/.nbd/storage # create 100G block device

umount /mnt/zerofs

Encryption & Security

Always on — cannot be disabled.

Layer	Detail
Algorithm	XChaCha20-Poly1305 (authenticated encryption)
Key derivation	Argon2id from password → 256-bit DEK
Scope	All file contents, metadata, permissions, timestamps
Granularity	Per SST block (32KB chunks)

What is NOT encrypted (visible in plaintext):

• Per-SST first/last key markers — leaks samples of directory entry names

• SlateDB manifest — SST IDs, segment prefixes, sizes

• S3 object listing — SST blob IDs and counts

The local cache directory contains decrypted, decompressed blocks. Treat it as sensitive data.

Password change:

bash

echo "new-password" | zerofs change-password -c zerofs.toml

Compression

Applied before encryption, reducing S3 bandwidth and storage costs.

Option	Tradeoff
`zstd-{1..22}`	Default `zstd-3`. Higher = better ratio, slower writes
`lz4`	Faster writes, lower compression ratio

Format is auto-detected on read — can be changed anytime without migrating data.

Caching

Three-layer pipeline from hot to cold:

1. Memory cache — in-process block cache (memory_size_gb)

2. Disk cache — persistent, stores decrypted+decompressed blocks (disk_size_gb)

3. S3 — compressed+encrypted SST objects

SQLite microbenchmarks on warm cache:

• Sequential fill: ~19.4 µs/op

• Sequential read: ~0.94 µs/op

• Random read: ~1.6 µs/op

Write-Ahead Log (WAL)

Disabled by default. Enable for workloads with frequent fsyncs (databases, etc.):

toml

[lsm]

wal_enabled = true

Without WAL, each fsync flushes the in-memory memtable to a new SST file, generating many small SSTs. WAL absorbs fsyncs without flushing, reducing compaction churn.

Checkpoints

Named point-in-time snapshots stored in the same S3 bucket. Requires the RPC server to be configured.

bash

zerofs checkpoint create -c zerofs.toml before-migration

zerofs checkpoint list -c zerofs.toml

zerofs run -c zerofs.toml --checkpoint before-migration # restore

Read-Only Replicas

One writer + unlimited read-only readers on the same backend:

bash

zerofs run -c zerofs.toml # read-write

zerofs run -c zerofs.toml --read-only # replica, auto-follows writer, returns EROFS on writes

Standalone Compaction

Decouple compaction from the main process to reduce latency and costs:

bash

zerofs run -c zerofs.toml --no-compactor # main instance

zerofs compactor -c zerofs.toml # separate compactor (run near S3)

Useful for running the compactor on a spot instance co-located with S3 to minimize egress fees.

NBD Advanced Usage

TRIM / Discard

Frees S3 space by deleting LSM chunks for discarded blocks:

bash

fstrim /mnt/block # manual TRIM

mount -o discard /dev/nbd0 /mnt/block # automatic on-write discard

zpool set autotrim=on mypool && zpool trim mypool # ZFS

ZFS on S3

bash

zpool create mypool /dev/nbd0

zpool set autotrim=on mypool

Geo-Distributed ZFS Mirror

Run ZeroFS in different S3 regions, mirror across them with ZFS:

bash

# Connect to 3 regional ZeroFS instances

nbd-client 10.0.1.5 10809 /dev/nbd0 -N storage -persist -timeout 600 -connections 8 # US East

nbd-client 10.0.2.5 10809 /dev/nbd1 -N storage -persist -timeout 600 -connections 8 # EU West

nbd-client 10.0.3.5 10809 /dev/nbd2 -N storage -persist -timeout 600 -connections 8 # AP

zpool create global-pool mirror /dev/nbd0 /dev/nbd1 /dev/nbd2

ZFS L2ARC Tiering

Add local NVMe as L2ARC in front of ZeroFS-backed vdevs:

bash

zpool create mypool /dev/nbd0 /dev/nbd1

zpool add mypool cache /dev/nvme0n1

ZFS manages placement: NVMe for hot data, ZeroFS disk cache for warm, S3 for cold.

Web UI

Configure:

toml

[servers.webui]

addresses = ["127.0.0.1:8080"]

uid = 1000

gid = 1000

Features:

• File manager — drag-and-drop uploads (files and folders), talks to ZeroFS via 9P over WebSocket

• Live dashboard — throughput, IOPS, storage usage, operation counters, GC stats; streams via gRPC-web

• In-browser terminal — runs a Linux VM (v86 emulator) with root filesystem mounted over 9P WebSocket

GitHub Actions

yaml

- uses: Barre/zerofs@main

with:

object-store-url: "s3://bucket/path"

encryption-password: ${{ secrets.ZEROFS_PASSWORD }}

aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}

aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Enables persistent storage and shared artifacts across CI workflow runs.

Performance Benchmarks

All tests: Azure Standard D48lds v6 (48 vCPUs, 96 GiB RAM), Cloudflare R2 backend.

vs JuiceFS (10,000 ops each)

Operation	ZeroFS	JuiceFS	Ratio
Sequential writes	984 ops/s	5.6 ops/s	175×
Data modifications	1,099 ops/s	6 ops/s	183×
File append	1,204 ops/s	5.3 ops/s	227×

Real-world task	ZeroFS	JuiceFS
`git clone`	2.6 s	34.4 s
TAR extraction	8.2 s	10m 26s
`cargo build`	3m 4s	DNF (>69 min)

JuiceFS also recorded 9,206–9,743 failures per 10,000 operations on modification/append tests (ZeroFS: 0). JuiceFS consumed 239 GB vs ZeroFS 7.6 GB for the same workload (31.6× more storage) and made 898,000 S3 API calls vs ZeroFS's 8,000.

JuiceFS requires a separate metadata database (SQLite/Redis/TiKV). ZeroFS requires only S3.

vs AWS Mountpoint-s3 (100 ops each)

Operation	ZeroFS	Mountpoint-s3	Ratio
Sequential writes	663.87 ops/s	0.70 ops/s	948×
Write latency	1.42 ms	1,435 ms	1,011×
Empty file creation	888.66 ops/s	0.09 ops/s	9,874×
Directory creation	985.98 ops/s	2.08 ops/s	474×
Random reads	1,000.84 ops/s	3.20 ops/s	313×
Read latency	0.90 ms	312 ms	347×

Real-world task	ZeroFS	Mountpoint-s3
`git clone`	3.1 s	failed
TAR extraction	13.5 s	~2 hours (est.)
S3 API calls	639	14,640 (23×)

Mountpoint-s3 maps S3 objects 1:1 to files — it does not support full POSIX semantics or arbitrary writes.

PostgreSQL (pgbench, 50 concurrent clients)

Workload	TPS	Latency
Mixed read/write	53,000	0.943 ms
Read-only	413,000	0.121 ms

Filesystem Limits

Limit	Value
Max file size	16 EiB
Max files (lifetime)	2⁶⁴ (~18 quintillion)
Max hard links per file	~4 billion (2³²)
Max filesystem size	16 EiB (or `max_size_gb` quota)

Testing & Maturity

• Passes all 8,662 pjdfstest_nfs POSIX compliance tests

• ZFS pool creation, Linux kernel extraction, and zpool scrub verified via CI

• Parallel make -j$(nproc) Linux kernel compilation stress test

• Ubuntu boot environment hosted entirely on S3