K8up
7 min read
Kubernetes backup operator built on restic. Backs up PersistentVolumeClaims and application data to any S3-compatible (or other) storage backend on a configurable schedule via CRDs.
CRDs Overview
| CRD | Purpose |
|---|---|
Schedule | Orchestrates recurring Backup / Check / Prune / Archive jobs via cron |
Backup | One-shot backup of PVCs (and PreBackupPods) in a namespace |
Restore | Restore data from a snapshot to a PVC or S3 bucket |
Archive | Export existing restic snapshots to a separate storage backend |
Check | Validate restic repository integrity |
Prune | Apply retention policies and remove old snapshots |
PreBackupPod | Ephemeral pod spawned before backup to dump app data to stdout |
PodConfig | Reusable pod template (node selector, tolerations, security context, etc.) |
Snapshot | Read-only metadata about a restic snapshot (id, date, paths) |
Install
sh
helm repo add k8up https://k8up-io.github.io/k8up
helm install k8up k8up/k8up --namespace k8up-system --create-namespace
Supported Backends
| Backend | Notes |
|---|---|
| S3 (AWS / compatible) | Most common; endpoint, bucket, access key, secret key |
| GCS | Google Cloud Storage |
| Azure Blob Storage | |
| OpenStack Swift | Extra vars via envFrom |
| Backblaze B2 | |
| REST server | Generic restic REST backend |
| Local filesystem | Mount path inside the backup pod |
All backends share:
• repoPasswordSecretRef — restic repository encryption password
• tlsOptions — caCert, clientCert, clientKey for mTLS / self-signed CAs
• envFrom — inject additional env vars from a ConfigMap or Secret (useful for Swift auth, custom options, etc.)
Credentials Secret
yaml
apiVersion: v1
kind: Secret
metadata:
name: backup-credentials
stringData:
accessKey: my-access-key
secretKey: my-secret-key
repoPassword: my-restic-repo-password
Schedule
Runs Backup, Prune, Check, and/or Archive on cron expressions. One Schedule per namespace is the typical pattern.
yaml
apiVersion: k8up.io/v1
kind: Schedule
metadata:
name: backup-schedule
spec:
backend:
repoPasswordSecretRef:
name: backup-credentials
key: repoPassword
s3:
endpoint: https://s3.example.com
bucket: my-backups
accessKeyIDSecretRef:
name: backup-credentials
key: accessKey
secretAccessKeySecretRef:
name: backup-credentials
key: secretKey
# Resource defaults for all spawned job pods (overridable per-job below)
resourceRequirementsTemplate:
requests:
cpu: 50m
memory: 128Mi
limits:
memory: 512Mi
backup:
schedule: "@daily-random" # randomized daily time (avoids thundering herd)
failedJobsHistoryLimit: 3
successfulJobsHistoryLimit: 3
promURL: http://prometheus-pushgateway:9091
check:
schedule: "@weekly-random"
prune:
schedule: "0 3 * * 0" # Sunday 3am
retention:
keepLast: 5
keepDaily: 7
keepWeekly: 4
keepMonthly: 6
keepYearly: 2
archive:
schedule: "@monthly-random"
restoreMethod:
s3:
endpoint: https://s3.example.com
bucket: my-archives
accessKeyIDSecretRef:
name: backup-credentials
key: accessKey
secretAccessKeySecretRef:
name: backup-credentials
key: secretKey
Schedule Syntax
Standard cron (* * * * *) plus K8up-specific shortcuts:
| Expression | Meaning |
|---|---|
@hourly | Every hour on the hour |
@daily / @midnight | Daily at midnight |
@weekly | Sundays at midnight |
@monthly | 1st of month at midnight |
@yearly / @annually | Jan 1st at midnight |
@every <duration> | Fixed interval, e.g. @every 6h30m |
@hourly-random | Random minute each hour |
@daily-random | Random time each day |
@weekly-random | Random weekday + time |
@monthly-random | Random day (≤27) + time |
@yearly-random | Random date + time once per year |
Random schedules prevent multiple namespaces from hitting the backend simultaneously.
Backup (one-shot)
yaml
apiVersion: k8up.io/v1
kind: Backup
metadata:
name: manual-backup
spec:
backend:
repoPasswordSecretRef:
name: backup-credentials
key: repoPassword
s3:
endpoint: https://s3.example.com
bucket: my-backups
accessKeyIDSecretRef:
name: backup-credentials
key: accessKey
secretAccessKeySecretRef:
name: backup-credentials
key: secretKey
failedJobsHistoryLimit: 1
successfulJobsHistoryLimit: 1
tags:
- manual
# labelSelectors: # target only specific PVCs / PreBackupPods
# - matchLabels:
# app: myapp
PVC Inclusion / Exclusion
By default all PVCs in the namespace are backed up.
| Annotation on PVC | Effect |
|---|---|
k8up.io/backup: "false" | Exclude this PVC |
k8up.io/backup: "true" | Explicitly include (required when BACKUP_SKIP_WITHOUT_ANNOTATION=true) |
Application-Aware Backups
Annotate a Pod (or Deployment/StatefulSet pod template) to run a dump command and stream it to stdout instead of — or in addition to — backing up the PVC:
yaml
metadata:
annotations:
k8up.io/backupcommand: "pg_dump -U postgres mydb"
k8up.io/file-extension: ".sql"
# k8up.io/backupcommand-container: "postgres" # required if multiple containers
| Annotation | Description |
|---|---|
k8up.io/backupcommand | Shell command to run; must write backup data to stdout |
k8up.io/file-extension | Extension for the resulting file (e.g. .sql, .archive, .tar.gz) |
k8up.io/backupcommand-container | Target container name when pod has multiple containers |
Works with: PostgreSQL (pg_dump), MySQL/MariaDB (mysqldump), MongoDB (mongodump), and any tool that writes to stdout.
K8up picks one pod per owner reference (Deployment / StatefulSet) to execute the command.
PreBackupPod
Use when the dump tool is not in an existing pod (e.g. managed cloud databases) or when you need a completely separate container.
yaml
apiVersion: k8up.io/v1
kind: PreBackupPod
metadata:
name: mariadb-dump
spec:
backupCommand: '/bin/bash -c ''mysqldump -uroot -p"${MARIADB_ROOT_PASSWORD}" --all-databases'''
pod:
spec:
containers:
- name: mariadb-dump
image: mariadb:11
env:
- name: MARIADB_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mariadb-secret
key: rootPassword
The operator starts this pod before the backup, streams its stdout, then cleans it up automatically.
Restore
Restore to a PVC
yaml
apiVersion: k8up.io/v1
kind: Restore
metadata:
name: restore-to-pvc
spec:
restoreMethod:
folder:
claimName: restored-data-pvc # must already exist
snapshot: latest # or a specific restic snapshot ID
# restoreFilter: "/data/myapp" # optional path filter
# restoreTimeFilter: "2024-03-01 00:00:00" # restore as of this point in time
backend:
repoPasswordSecretRef:
name: backup-credentials
key: repoPassword
s3:
endpoint: https://s3.example.com
bucket: my-backups
accessKeyIDSecretRef:
name: backup-credentials
key: accessKey
secretAccessKeySecretRef:
name: backup-credentials
key: secretKey
Restore to S3
yaml
spec:
restoreMethod:
s3:
endpoint: https://s3.example.com
bucket: my-restore-target
accessKeyIDSecretRef:
name: backup-credentials
key: accessKey
secretAccessKeySecretRef:
name: backup-credentials
key: secretKey
Non-root permissions
yaml
spec:
podSecurityContext:
fsGroup: 65532
fsGroupChangePolicy: OnRootMismatch
Manual restore via restic CLI
sh
export RESTIC_REPOSITORY=s3:https://s3.example.com/my-backups
export RESTIC_PASSWORD=my-restic-repo-password
export AWS_ACCESS_KEY_ID=my-access-key
export AWS_SECRET_ACCESS_KEY=my-secret-key
restic snapshots
restic restore latest --target /tmp/restore
restic mount /mnt/restic # FUSE mount for browsing
Archive
Exports existing restic snapshots to a separate long-term storage location (accepts slower restore speed for cheaper storage).
yaml
apiVersion: k8up.io/v1
kind: Archive
metadata:
name: quarterly-archive
spec:
restoreMethod:
s3:
endpoint: https://s3.example.com
bucket: my-archives
accessKeyIDSecretRef:
name: backup-credentials
key: accessKey
secretAccessKeySecretRef:
name: backup-credentials
key: secretKey
backend:
repoPasswordSecretRef:
name: backup-credentials
key: repoPassword
s3:
endpoint: https://s3.example.com
bucket: my-backups
accessKeyIDSecretRef:
name: backup-credentials
key: accessKey
secretAccessKeySecretRef:
name: backup-credentials
key: secretKey
If no snapshots exist yet the archive job completes silently without error.
Check
Validates the restic repository integrity (runs restic check).
yaml
apiVersion: k8up.io/v1
kind: Check
metadata:
name: repo-check
spec:
promURL: http://prometheus-pushgateway:9091
backend:
repoPasswordSecretRef:
name: backup-credentials
key: repoPassword
s3:
endpoint: https://s3.example.com
bucket: my-backups
accessKeyIDSecretRef:
name: backup-credentials
key: accessKey
secretAccessKeySecretRef:
name: backup-credentials
key: secretKey
Prune
Applies retention policy and removes unreferenced data from the restic repository.
yaml
apiVersion: k8up.io/v1
kind: Prune
metadata:
name: prune-old-snapshots
spec:
retention:
keepLast: 5
keepHourly: 0
keepDaily: 7
keepWeekly: 4
keepMonthly: 6
keepYearly: 2
# keepTags:
# - production
backend:
repoPasswordSecretRef:
name: backup-credentials
key: repoPassword
s3:
endpoint: https://s3.example.com
bucket: my-backups
accessKeyIDSecretRef:
name: backup-credentials
key: accessKey
secretAccessKeySecretRef:
name: backup-credentials
key: secretKey
PodConfig
Reusable pod configuration applied to job pods (node affinity, tolerations, security context, etc.). Cannot override image, command, container name, or args.
yaml
apiVersion: k8up.io/v1
kind: PodConfig
metadata:
name: backup-pod-config
spec:
template:
spec:
nodeSelector:
kubernetes.io/os: linux
tolerations:
- key: dedicated
operator: Equal
value: backup
effect: NoSchedule
securityContext:
runAsNonRoot: true
runAsUser: 65532
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
Reference it from a Backup or Schedule:
yaml
spec:
backup:
podConfigRef:
name: backup-pod-config
Pod Resources — Hierarchy
Resources are resolved in order (highest priority last):
1. Global defaults — operator env vars (BACKUP_GLOBAL_MEMORY_LIMIT, etc.)
2. Schedule resourceRequirementsTemplate — applies to all jobs in that Schedule
3. Per-job resources field — in Backup, Prune, Check, Archive, Restore
Global defaults only apply to jobs created by a Schedule. Standalone Backup / Restore objects ignore global defaults.
Monitoring Status
sh
# Wait for backup to start
kubectl wait --for condition=progressing backup/my-backup
# Wait for backup to finish
kubectl wait --for condition=completed backup/my-backup
# Quick status overview
kubectl get backup,restore,check,prune
# Detailed conditions
kubectl describe backup/my-backup
kubectl logs job/<job-name>
Prometheus Metrics
Operator metrics (:8080/metrics)
| Metric | Type | Labels |
|---|---|---|
k8up_jobs_total | Counter | namespace, jobType |
k8up_jobs_successful_counter | Counter | namespace, jobType |
k8up_jobs_failed_counter | Counter | namespace, jobType |
k8up_schedules_gauge | Gauge | namespace |
jobType values: backup, check, archive, restore, prune
Backup push metrics (Pushgateway via promURL)
| Metric | Labels |
|---|---|
k8up_backup_restic_new_files_during_backup | pvc, namespace |
k8up_backup_restic_changed_files_during_backup | pvc, namespace |
k8up_backup_restic_unmodified_files_during_backup | pvc, namespace |
k8up_backup_restic_new_directories_during_backup | pvc, namespace |
k8up_backup_restic_changed_directories_during_backup | pvc, namespace |
k8up_backup_restic_unmodified_directories_during_backup | pvc, namespace |
k8up_backup_restic_data_transferred_during_backup | pvc, namespace |
k8up_backup_restic_available_snapshots | — |
k8up_backup_restic_last_errors | pvc, namespace |
Operator Configuration (env vars)
Key environment variables for the K8up operator deployment:
| Variable | Default | Description |
|---|---|---|
BACKUP_IMAGE | ghcr.io/k8up-io/k8up:latest | Restic/k8up image used for job pods |
BACKUP_DATAPATH | /data | Mount path for PVCs inside backup pods |
BACKUP_SKIP_WITHOUT_ANNOTATION | false | Only back up PVCs with k8up.io/backup=true |
BACKUP_GLOBAL_BACKOFF_LIMIT | 6 | Job retry backoff limit |
BACKUP_GLOBAL_FAILED_JOBS_HISTORY_LIMIT | 3 | Failed jobs retained |
BACKUP_GLOBAL_SUCCESSFUL_JOBS_HISTORY_LIMIT | 3 | Successful jobs retained |
BACKUP_GLOBAL_CPU_REQUEST | — | Global CPU request for job pods |
BACKUP_GLOBAL_CPU_LIMIT | — | Global CPU limit |
BACKUP_GLOBAL_MEMORY_REQUEST | — | Global memory request |
BACKUP_GLOBAL_MEMORY_LIMIT | — | Global memory limit |
BACKUP_GLOBAL_CONCURRENT_BACKUP_JOBS_LIMIT | unlimited | Max concurrent backup jobs |
BACKUP_GLOBAL_CONCURRENT_PRUNE_JOBS_LIMIT | unlimited | Max concurrent prune jobs |
BACKUP_GLOBAL_CONCURRENT_CHECK_JOBS_LIMIT | unlimited | Max concurrent check jobs |
BACKUP_GLOBAL_CONCURRENT_RESTORE_JOBS_LIMIT | unlimited | Max concurrent restore jobs |
BACKUP_GLOBAL_CONCURRENT_ARCHIVE_JOBS_LIMIT | unlimited | Max concurrent archive jobs |
BACKUP_GLOBALREPOPASSWORD | — | Global restic repo password |
BACKUP_GLOBALACCESSKEYID | — | Global S3 access key ID |
BACKUP_GLOBALSECRETACCESSKEY | — | Global S3 secret access key |
BACKUP_GLOBALS3BUCKET | — | Global S3 bucket |
BACKUP_GLOBALS3ENDPOINT | — | Global S3 endpoint |
BACKUP_PROMURL | http://127.0.0.1/ | Default Prometheus push gateway |
BACKUP_METRICS_BINDADDRESS | :8080 | Operator metrics bind address |
BACKUP_CHECKSCHEDULE | 0 0 * * 0 | Default check cron if not specified |
BACKUP_RESTIC_OPTIONS | — | Extra restic options as key=value,key2=value2 |
CLUSTER_NAME | default | Cluster name label in metrics |