git-annex
12 min read
git-annex lets you manage large files with Git without storing their contents in Git. Git tracks only lightweight pointers (symlinks to checksum-named objects), while the actual file data lives in .git/annex/ and can be copied to, dropped from, and synced between many repositories and storage backends. This keeps the Git history small and fast even when managing terabytes of data.
It was written by Joey Hess and is the spiritual ancestor of approaches like Git LFS, but is far more flexible: it's fully distributed, tracks where every file's content lives across all your drives and remotes, and never requires a central server.
Concept
When you git annex add a file, git-annex:
1. Moves the content into .git/annex/objects/, named after a hash (a key) of its content.
2. Replaces the working-tree file with a symlink pointing to that object.
3. The symlink (a tiny text pointer) is what gets committed to Git — not the data.
Because Git only ever sees the symlink, the repository stays small. Multiple clones can share the same Git history but each hold a different subset of the actual file contents. git-annex tracks which repositories hold which content in a special git-annex branch, so it always knows where to fetch a file from and refuses to drop the last copy of anything.
working tree: photo.jpg ──symlink──▶ .git/annex/objects/.../SHA256E-...-photo.jpg
git commit: stores the symlink (a ~100 byte pointer), never the JPEG bytes
git-annex branch: logs "this content exists in: laptop, usb-drive, s3-backup"
Two file modes: In the default direct/unlocked workflows files can be present as normal files (via pointer files), but the classic model uses locked files — read-only symlinks that protect content from accidental modification. Use git annex unlock to edit, git annex lock to re-protect.
Setup
sh
# Inside an existing git repo (or after git init)
git annex init "my-laptop" # description identifies THIS repo among all clones
# Add files — content goes to the annex, a symlink is committed
git annex add bigfile.iso
git commit -m "add iso"
Every clone gets its own description (my-laptop, usb-backup, server) so whereis output is human-readable.
Core Commands
| Command | Purpose |
|---|---|
git annex init "desc" | Initialize annex in the current repo |
git annex add <file> | Store content in the annex, commit a pointer |
git annex get <file> | Download content from another repo/remote that has it |
git annex drop <file> | Remove local content (only if copies exist elsewhere) |
git annex copy <file> --to <remote> | Copy content to a remote, keeping the local copy |
git annex move <file> --to <remote> | Transfer content (get + drop) to a remote |
git annex whereis <file> | List which repos/remotes hold this content |
git annex sync [--content] | Sync git metadata (and optionally content) with remotes |
git annex unlock / lock | Make a file editable / protect it again |
git annex find | List files whose content is present locally |
git annex status | Show working-tree status (annex-aware) |
git annex info | Show repo info: size, counts, remotes |
git annex fsck | Verify content integrity against checksums |
Remote Management
git-annex works with two kinds of remotes:
• Normal git remotes — other clones of the repo (over SSH, a USB drive, a server). These hold both the git history and (some of) the annexed content.
• Special remotes — storage backends that hold only file content, never git history. You can't git clone from them; they're pure content stores (S3, a plain directory, WebDAV, …). See Special Remotes below.
Both are used the same way once set up: git annex copy/move/get --to/--from <remote>.
sh
# Add a normal git remote (a clone on a server you can push history to)
git remote add server ssh://user@host/srv/annex
git annex sync server # exchanges git history + branches
# List all remotes git-annex knows about, with their descriptions and sizes
git annex info
# Give a remote a human-readable description (shown by `whereis`)
git annex describe server "office server"
git annex describe here "my-laptop" # `here` = the current repo
Trust levels
git-annex tracks how much it trusts each remote not to lose data. This governs when it's willing to drop the last copy of a file. Levels: trusted > semitrusted (default) > untrusted > dead.
sh
git annex trust offsite-backup # count it as a reliable copy
git annex untrust usb-stick # don't rely on it for numcopies
git annex dead lost-drive # a drive that's gone for good — stop tracking it
git annex semitrust here # back to the default
A trusted remote is assumed to still have whatever git-annex last recorded, without re-checking — only mark truly reliable storage as trusted. Untrusted remotes never count toward numcopies, so git-annex won't let them hold the only copy.
enableremote
Special remotes live in the git-annex branch, so a fresh clone knows about them but must opt in to use them (re-supplying credentials where needed):
sh
git clone ssh://server/repo && cd repo
git annex enableremote s3-backup # activate an existing special remote here
Syncing
git annex sync is the workhorse command. It is not just git push/pull — in one shot it:
1. Commits any staged changes (to a local synced/<branch> and the git-annex branch).
2. Pulls from each remote and merges their git-annex branch (the location-tracking data) and your working branch.
3. Pushes the merged result back to each remote.
By default it moves only git metadata — branches, location logs, pointers — not the file contents. That's what makes it cheap to run often.
sh
git annex sync # sync metadata with all remotes
git annex sync server backup # sync with only these remotes
Syncing content
To also transfer the actual file data, add --content. git-annex then moves bytes around to satisfy your preferred-content rules:
sh
git annex sync --content # sync metadata AND content everywhere
git annex sync --content backup # push/pull content with one remote
git annex assist is the modern convenience wrapper: it's sync --content plus auto-importing new files (roughly what the old "git-annex assistant" daemon did, but on demand).
Mental model: git annex sync keeps every repo agreeing on where files are; git annex sync --content additionally makes the files actually be there. The location-tracking data in the git-annex branch is what lets any clone answer "where is this file?" — see whereis.
Special Remotes
Special remotes store file content only (never git history) in arbitrary backends. This is git-annex's superpower: back up the annex to S3, an external drive, or a cloud bucket — usually with client-side encryption.
The general pattern is always the same:
sh
git annex initremote <name> type=<backend> <key=value...> encryption=<mode>
git annex copy --to <name> # upload content
git annex get --from <name> <file> # download content
git annex enableremote <name> # re-activate in another clone
Two options apply to almost every backend:
• encryption= — none, shared, hybrid, pubkey, or sharedpubkey (see Encryption). Must be set explicitly, even if none.
• chunk= — split content into fixed-size chunks (e.g. chunk=50MiB). Helps with size limits, resumable transfers, and hides exact file sizes when encrypted.
Below are the most relevant backends.
directory
Stores content in a local filesystem path — ideal for an external/USB drive or a NAS mount ("sneakernet" backups). The directory must already exist.
sh
git annex initremote usbdrive type=directory directory=/mnt/usb encryption=none
git annex describe usbdrive "usb drive"
git annex copy --to usbdrive
rsync
Stores content on any host reachable by rsync, typically over SSH. Good lightweight choice for your own server.
sh
# Over SSH
git annex initremote myserver type=rsync \
rsyncurl=user@host.example.com:/srv/annex encryption=shared
# Via an rsync daemon URL
git annex initremote myserver type=rsync \
rsyncurl=rsync://host.example.com/annex encryption=shared
S3
Amazon S3 and any S3-compatible service (MinIO, Ceph, Wasabi, Backblaze via S3 API). Credentials come from AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars at initremote time (then cached, encrypted, in the repo).
sh
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
# Amazon S3, encrypted, infrequent-access tier, chunked
git annex initremote s3 type=S3 bucket=my-annex-bucket \
datacenter=EU storageclass=STANDARD_IA \
chunk=50MiB encryption=hybrid keyid=me@example.com
# Non-AWS S3-compatible endpoint (e.g. MinIO)
git annex initremote minio type=S3 host=minio.example.com port=9000 \
protocol=https requeststyle=path bucket=data encryption=shared
Useful params: bucket, datacenter/region, storageclass (STANDARD_IA, GLACIER, DEEP_ARCHIVE), host/port/protocol/requeststyle=path (for non-AWS), signature=v4, fileprefix= (share one bucket between remotes), partsize= (multipart upload size), publicurl.
webdav
Stores content on a WebDAV server — works with Nextcloud/ownCloud, Box, and similar. Use HTTPS (basic auth). Credentials via WEBDAV_USERNAME / WEBDAV_PASSWORD.
sh
WEBDAV_USERNAME=user WEBDAV_PASSWORD=secret \
git annex initremote nextcloud type=webdav \
url=https://cloud.example.com/remote.php/webdav/annex \
chunk=50MiB encryption=hybrid keyid=me@example.com embedcreds=yes
embedcreds=yes stores the credentials (encrypted, if using GPG) in the repo so other clones can enableremote without re-entering them.
web
A built-in, always-available, read-only remote named web. Instead of git-annex storing the content, you register an existing public URL as the source for a file. git-annex downloads from that URL on get, but can never upload or delete there.
sh
# Track a remote file by URL — content is fetched on demand from the web
git annex addurl --file dataset.zip https://example.com/dataset.zip
# Later, on any clone:
git annex get dataset.zip # downloads from the registered URL
git annex whereis dataset.zip # shows the web URL as a location
Great for datasets, releases, or anything already hosted online that you want versioned alongside your repo without re-hosting it.
rclone
Bridges git-annex to rclone's 70+ cloud providers (Google Drive, Dropbox, OneDrive, B2, …). Modern git-annex + rclone have built-in support via type=rclone (older setups used the external git-annex-remote-rclone script with type=external). Configure the underlying provider in rclone first (rclone config).
sh
git annex initremote gdrive type=rclone \
rcloneremotename=gdrive rcloneprefix=git-annex rclonelayout=lower \
chunk=50MiB encryption=shared
Here rcloneremotename is the name of a remote you defined in rclone; rcloneprefix is the subpath it writes under.
Other backends also exist: Glacier, Google Cloud Storage, Azure, Backblaze B2 (native), git-lfs, Borg/bup/ddar, IPFS, BitTorrent, Tahoe-LAFS, and a generic external protocol for writing your own.
Encryption
For untrusted backends (cloud, someone else's server), git-annex can transparently encrypt content with GnuPG and HMAC-hash the filenames before upload, so the provider sees only opaque blobs. Set encryption= at initremote; it can't be changed afterward.
| Mode | How it works | Who can decrypt |
|---|---|---|
none | No encryption | anyone with backend access |
shared | One symmetric key, stored (unencrypted) in the git repo | anyone who can clone the repo |
hybrid (recommended) | Symmetric key, itself encrypted to one or more GPG keys; keys can be added later | holders of any enrolled GPG private key |
pubkey | Content encrypted directly to GPG public keys | holders of those private keys (can't add keys retroactively) |
sharedpubkey | Anyone with the public key can upload; only private-key holders download | upload: public key; download: private key |
sh
# Shared: simplest; security = whoever can read the git repo
git annex initremote backup type=directory directory=/mnt/usb encryption=shared
# Hybrid: encrypt to your GPG key; collaborators added later without re-encrypting
git annex initremote s3 type=S3 bucket=b encryption=hybrid keyid=me@example.com
# Add another person's GPG key to an existing hybrid/pubkey remote
git annex enableremote s3 keyid+=788A3F4C
git annex enableremote s3 keyid-=OLDKEYID # revoke access
keyid accepts a GPG key id, fingerprint, or email. Check a remote's settings with git annex info <remote>.
shared vs hybrid: shared is convenient — every clone can decrypt automatically — but anyone who obtains the repo gets the data. hybrid ties decryption to specific GPG keys, so a leaked repo clone is useless without a private key. Prefer hybrid for anything sensitive going to third-party storage.
Preferred Content & numcopies
git-annex can decide automatically which files each repo should keep, instead of you running get/drop by hand.
sh
# Never let fewer than 2 copies of any content exist (refuses to drop below this)
git annex numcopies 2
# Make this repo a "backup" that wants ALL content
git annex wanted . anything
# Make a small laptop only want markdown + small files
git annex wanted . "include=*.md or smallerthan=100mb"
Then git annex sync --content (or git annex assist) moves data to satisfy these rules — dropping where over-replicated, fetching where wanted, always respecting numcopies so it never destroys the last copy. Groups (git annex group, git annex groupwanted) let you express this across many repos at once (e.g. a backup group that wants everything, a client group that wants nothing by default).
End-to-End Example
A complete "laptop + server + encrypted S3 backup" setup, the way you'd actually use it:
sh
# --- On the laptop: create the annex ---
git init photos && cd photos
git annex init "laptop"
git annex add 2024/ # ingest a year of photos
git commit -m "add 2024 photos"
# --- Add a server as a normal git remote (holds history + content) ---
git remote add server ssh://me@host/srv/photos
git annex sync --content server # push history + photo content
# --- Add an encrypted S3 special remote as offsite backup ---
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...
git annex initremote s3 type=S3 bucket=my-photos \
encryption=hybrid keyid=me@example.com chunk=50MiB storageclass=STANDARD_IA
git annex trust s3 # count it as a reliable copy
git annex copy --to s3 # upload (encrypted) content
# --- Require 2 copies, then reclaim laptop space safely ---
git annex numcopies 2
git annex drop 2024/ # only drops files with >=2 copies elsewhere
git annex whereis 2024/01/img_001.jpg # -> server, s3 (content gone from laptop)
# --- On a new machine: clone and selectively fetch ---
git clone ssh://me@host/srv/photos && cd photos
git annex enableremote s3 # reactivate the S3 remote here
git annex get 2024/01/ # pull just January, from whichever remote is closest
The laptop now holds pointers to everything but content for nothing, while the server and (encrypted) S3 each hold full copies — and whereis can always tell you so.
Git LFS vs git-annex
Both keep large blobs out of Git history, but differ fundamentally:
| git-annex | Git LFS | |
|---|---|---|
| Architecture | Fully distributed, P2P | Centralized server |
| Location tracking | Tracks content across all remotes/drives | Single LFS server |
| Partial checkout | First-class (get/drop per file) | Files fetched on checkout |
| Storage backends | Many special remotes + encryption | LFS-compatible servers |
| Complexity | Higher, more powerful | Simpler, GitHub-integrated |
Choose Git LFS for simple "big files in a GitHub repo" needs; choose git-annex for distributed archives, multi-drive backups, offline workflows, and encrypted cloud storage.
See Also