Spegel

5 June 20265 min read

Stateless cluster-local OCI registry mirror for Kubernetes. ("Spegel" = "mirror" in Swedish.) Runs as a DaemonSet — each node shares already-pulled container images with peers via P2P, eliminating redundant upstream pulls.

Only works with containerd. Does not support Docker or CRI-O.

Why

• Avoid Docker Hub / registry pull rate limits

• Faster pod startup — pull from a local peer instead of the internet

• Survive external registry outages (images already in cluster stay available)

• Reduce egress costs

• Edge deployments with limited bandwidth

Architecture

Each node runs a Spegel pod acting as both a local OCI registry mirror and a P2P peer. Containerd is configured to route image pulls through the local Spegel instance first.

P2P layer uses Kademlia DHT — each node advertises the digests of layers it holds locally. Resolution happens at the layer level (each chunk has its own digest), not the image level.

Pull flow:

1. Containerd requests a layer from local Spegel (via certs.d mirror config)

2. Spegel queries the DHT for peers holding that digest (20ms timeout, 3 retries)

3. If a peer is found → content is streamed from that peer back to containerd

4. If no peer responds → Spegel returns 404, containerd falls back to the upstream registry

Bootstrapping: A leader is elected (lease 10s, renewal 5s, retry 2s). New nodes connect to the leader to get the current peer list, then establish direct P2P connections.

Advertisement TTL: Nodes re-advertise their content every 9 minutes with a 10-minute TTL. Image add/update/delete events from containerd trigger immediate DHT updates.

Requirements

• containerd as the cluster runtime

• discard_unpacked_layers = false — layers must be kept on disk for peers to serve them

• Registry mirror config path set to /etc/containerd/certs.d

Minimum containerd config (/etc/containerd/config.toml):

toml

[plugins."io.containerd.cri.v1.images".registry]

config_path = "/etc/containerd/certs.d"

[plugins."io.containerd.cri.v1.images"]

discard_unpacked_layers = false

Installation

bash

# Helm

helm upgrade --create-namespace --namespace spegel --install spegel \

oci://ghcr.io/spegel-org/helm-charts/spegel

yaml

# Flux CD

apiVersion: source.toolkit.fluxcd.io/v1beta2

kind: HelmRepository

metadata:

name: spegel

spec:

type: oci

url: oci://ghcr.io/spegel-org/helm-charts

---

apiVersion: helm.toolkit.fluxcd.io/v2beta1

kind: HelmRelease

metadata:

name: spegel

namespace: spegel

spec:

chart:

spec:

chart: spegel

Distribution-Specific Setup

k3s / RKE2

Spegel is embedded in k3s and RKE2 — consult their own docs for enabling it.

k0s

Write containerd config before k0s starts (cannot be patched at runtime):

/etc/k0s/containerd.d/spegel.toml

Override paths in Helm values:

yaml

spegel:

containerdSock: "/run/k0s/containerd.sock"

containerdContentPath: "/var/lib/k0s/containerd/io.containerd.content.v1.content"

Talos

Machine config:

yaml

machine:

kubelet:

extraConfig:

imageGCHighThresholdPercent: 85

containerd:

extraArgs:

- --discard-unpacked-layers=false

Helm values:

yaml

spegel:

containerdRegistryConfigPath: /etc/cri/conf.d/hosts

Label the namespace:

bash

kubectl label namespace spegel pod-security.kubernetes.io/enforce=privileged

EKS (AL2023)

Use nodeadm configuration to set discard_unpacked_layers = false.

EKS (Bottlerocket)

Requires Bottlerocket v1.56+. Use bootstrap containers to write the mirror config and set containerdMirrorAdd: false in Helm values.

Kind

yaml

containerdConfigPatches:

- |-

[plugins."io.containerd.cri.v1.images".registry]

config_path = "/etc/containerd/certs.d"

[plugins."io.containerd.cri.v1.images"]

discard_unpacked_layers = false

VKE / CNI Port Conflicts

If the CNI conflicts with the default NodePort range:

yaml

service:

registry:

hostPort: 3020

Helm Values

Value	Default	Description
`spegel.containerdSock`	`/run/containerd/containerd.sock`	containerd socket path
`spegel.containerdContentPath`	`/var/lib/containerd/io.containerd.content.v1.content`	containerd content store
`spegel.containerdRegistryConfigPath`	`/etc/containerd/certs.d`	mirror config directory
`spegel.containerdMirrorAdd`	`true`	write mirror config via init container
`spegel.resolveTags`	`true`	resolve tags to digests locally
`spegel.additionalMirrorTargets`	`[]`	extra upstream mirrors to chain
`service.registry.nodePort`	`30020`	NodePort for the registry
`kvick.nodeReadiness.enabled`	`false`	hold node scheduling until Spegel is ready

Tag Resolution

By default Spegel resolves tags to digests locally. Limitation: once a tag is pulled, it resolves to the same digest for as long as the image is cached — rolling tags like latest won't update automatically.

Options:

Disable tag resolution entirely — tags always resolved upstream, only layer content served locally:

yaml

spegel:

resolveTags: false

k8s-digester — Google's mutating webhook that rewrites pod image references to digests at admission time. Recommended scope: pods only.

Disabling tag resolution does not protect against registry outages — only images already cached by digest remain available offline.

Chaining with Another Registry Cache

yaml

spegel:

additionalMirrorTargets:

- https://zot.example.com

Multiple Spegel Deployments

For multi-region or isolated node groups — deploy separate instances with unique selectors and NodePorts:

bash

helm upgrade --install spegel-us \

--set "nodeSelector.region=us" \

--set "service.registry.nodePort=30021" \

oci://ghcr.io/spegel-org/helm-charts/spegel

helm upgrade --install spegel-eu \

--set "nodeSelector.region=eu" \

--set "service.registry.nodePort=30022" \

oci://ghcr.io/spegel-org/helm-charts/spegel

Verifying

bash

kubectl port-forward -n spegel <pod> 9090

# http://localhost:9090/debug/web

# "Last Mirror Success" should show a recent timestamp

Metrics

Exposed at :9090/metrics.

Metric	Type	Description
`spegel_advertised_images`	Gauge	Images advertised by registry
`spegel_advertised_keys`	Gauge	DHT keys advertised
`spegel_advertised_image_tags`	Gauge	Tags advertised
`spegel_advertised_image_digests`	Gauge	Digests advertised
`spegel_resolve_duration_seconds`	Histogram	Tag-to-digest resolution latency
`spegel_mirror_requests_total`	Counter	Mirror requests — labels: `registry`, `cache` (hit/miss), `source` (internal/external)
`http_request_duration_seconds`	Histogram	HTTP handler latency
`http_response_size_bytes`	Histogram	Response sizes
`http_requests_inflight`	Gauge	Concurrent in-flight requests

Operational Notes

Private images in multi-tenant clusters: Spegel distributes cached images without re-checking pull secrets — a pod without credentials can pull a cached private image from a peer. Mitigate with imagePullPolicy: Always, though that re-introduces registry dependency.

Node failure: Failed nodes stay advertised for up to 10 minutes (TTL). Other peers may attempt the failed node before falling back. Usually minimal impact since layers are fetched in parallel, but can worsen with packet loss or a single large dominant layer.

Disk pressure: Spegel relies entirely on kubelet image GC for cleanup — it adds no cleanup of its own. Ensure kubelet GC thresholds are configured. Default Helm memory limit is 128Mi to prevent VFS cache growth.

OCI volumes: Supported via the CRI API (requires compatible Kubernetes and containerd versions).