cert-manager
11 min read

cert-manager is a Kubernetes operator that automates the issuance, storage and renewal of X.509 TLS certificates. It adds a set of CRDs so certificates become ordinary declarative cluster resources: you describe which certificate you want and who should sign it, and controllers keep the resulting Kubernetes Secret valid forever — including renewal well before expiry.

It is the de-facto standard way to get Let's Encrypt certificates into a cluster, but it is equally used to run a private internal PKI (self-signed root → intermediate → workload certs for mTLS), or to front a corporate CA such as HashiCorp Vault or Venafi.

Core idea

Two objects carry the whole model:

• An Issuer / ClusterIssuer is a certificate authority configuration — "how do I obtain signed certs" (ACME account, CA keypair, Vault endpoint, …).
• A Certificate is a request for a certificate — "I want a cert for these DNS names, signed by that issuer, stored in this Secret".

cert-manager reconciles the Certificate: it generates a private key, builds a CSR, drives whatever protocol the issuer needs, writes the result into a kubernetes.io/tls Secret, and re-runs the whole loop before expiry. Your workload only ever consumes the Secret and never knows a CA exists.

Certificate ──▶ CertificateRequest ──▶ (ACME only) Order ──▶ Challenge
│ │
└────────────── writes ──▶ Secret (tls.crt / tls.key) ◀────┘

That chain is also the debugging path — see Troubleshooting.

Components

Installing cert-manager deploys three Deployments into the cert-manager namespace:

ComponentRole
controllerThe actual reconcilers (Certificate, CertificateRequest, Order, Challenge) plus the ingress-shim
webhookValidating/mutating admission webhook — rejects malformed cert-manager resources at kubectl apply time
cainjectorInjects CA bundles into ValidatingWebhookConfiguration, APIService and CRD caBundle fields

Two short-lived helpers also exist: acmesolver (a tiny pod spawned to serve HTTP-01 challenge tokens) and startupapicheck (a Job that verifies the webhook is reachable before declaring the install healthy).

Installation

The CRDs must exist before any Certificate is applied. With Helm (recommended, works well with GitOps tools):

shell
helm repo add jetstack https://charts.jetstack.io --force-update
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager --create-namespace \
--version v1.21.0 \
--set crds.enabled=true

Or the static manifest (CRDs included):

shell
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.0/cert-manager.yaml

Helm does not upgrade CRDs on helm upgrade unless crds.enabled=true is kept set. When managing cert-manager declaratively, apply the CRD manifest separately so upgrades are explicit.

Useful chart values:

yaml
crds:
enabled: true
keep: true # don't delete CRDs (and all Certificates!) on uninstall
extraArgs:
# use public resolvers for the DNS-01 self-check instead of cluster DNS
- --dns01-recursive-nameservers=1.1.1.1:53,9.9.9.9:53
- --dns01-recursive-nameservers-only
prometheus:
enabled: true

The --dns01-recursive-nameservers-only flag is almost always needed in split-horizon / internal-DNS setups, where CoreDNS would otherwise answer the propagation self-check from the cluster's internal view.

CRD Overview

All in API group cert-manager.io/v1 unless noted.

CRDScopePurpose
IssuerNamespacedCA configuration usable only inside its own namespace
ClusterIssuerClusterSame, but usable from every namespace (secrets are read from the cert-manager namespace)
CertificateNamespacedDeclares a wanted certificate and the Secret it lands in — the object you normally write
CertificateRequestNamespacedA single CSR + issuance attempt. Created by the Certificate controller; rarely hand-written
OrderNamespacedOne ACME order (acme.cert-manager.io/v1). Created per CertificateRequest against an ACME issuer
ChallengeNamespacedOne ACME domain validation (acme.cert-manager.io/v1), HTTP-01 or DNS-01, one per identifier

Order and Challenge are internal machinery — you never create them, you only read them when something is stuck.

shell
kubectl get certificates -A
kubectl get certificaterequests,orders,challenges -A
kubectl get clusterissuers

Issuers

SelfSigned

Signs a certificate with its own key. Useless on its own for serving traffic, but it is the bootstrap step for an internal PKI: it is what signs your root CA certificate.

yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}

CA

Signs certificates with a CA keypair you supply as a Secret. This is the workhorse for internal/mTLS certificates — no ACME, no rate limits, instant issuance.

yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: internal-ca
spec:
ca:
secretName: internal-ca-key-pair # must live in the cert-manager namespace

The Secret must be of type kubernetes.io/tls and contain tls.crt + tls.key (optionally ca.crt for a chain).

ACME

Talks the ACME protocol to Let's Encrypt, ZeroSSL, Buypass, an internal step-ca, etc. Requires an account key (created automatically into privateKeySecretRef) and at least one solver that proves domain control.

yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: admin@example.com
privateKeySecretRef:
name: letsencrypt-account-key
# profile: tlsserver # ACME profiles, cert-manager v1.18+
solvers:
# DNS-01 for the wildcard zone
- selector:
dnsZones: ["example.com"]
dns01:
cloudflare:
apiTokenSecretRef:
name: cloudflare-token
key: token
# HTTP-01 for everything else
- http01:
ingress:
ingressClassName: traefik

Always develop against the staging directory first — production has hard rate limits (50 certs/registered domain/week):

https://acme-staging-v02.api.letsencrypt.org/directory

HTTP-01

cert-manager spins up a temporary acmesolver Pod, Service and Ingress serving /.well-known/acme-challenge/<token> on port 80, then asks the CA to fetch it.

• Requires the domain to resolve publicly to your ingress controller and port 80 to be reachable (HTTP → HTTPS redirects must not break the challenge path).
• Cannot issue wildcard certificates.
• With Traefik / ingress-nginx, set ingressClassName; use acme.cert-manager.io/http01-edit-in-place: "true" if your setup needs the challenge path added to the existing Ingress instead of a new one (e.g. when DNS points at a single host rule).

DNS-01

cert-manager writes a _acme-challenge.<domain> TXT record via a DNS provider API, waits for propagation (self-check), then asks the CA to resolve it.

• Works for hosts that are not publicly reachable — the only option for internal services and the only way to get wildcard certs (*.example.com).
• Built-in providers: Cloudflare, Route53, AzureDNS, CloudDNS, DigitalOcean, AcmeDNS, RFC2136 (any BIND/knot server with TSIG). Anything else is a webhook solver (a small external service implementing the cert-manager webhook API) — this is how providers like Hetzner, deSEC, Netcup, Gandi etc. are supported.

yaml
solvers:
- dns01:
rfc2136:
nameserver: 10.0.0.53:53
tsigKeyName: acme-key
tsigAlgorithm: HMACSHA256
tsigSecretSecretRef:
name: tsig-secret
key: tsig-secret-key

Solver selection

With several solvers, the most specific matching selector wins, evaluated per DNS name:

SelectorMatches
dnsNamesExact DNS names (most specific)
dnsZonesThe zone and everything below it (example.com → a.b.example.com)
matchLabelsLabels on the Certificate resource

An empty selector: {} (or none) is the catch-all.

External Account Binding

CAs like ZeroSSL or Google Trust Services require binding the ACME account to an existing account:

yaml
spec:
acme:
externalAccountBinding:
keyID: <kid>
keySecretRef:
name: eab-hmac
key: secret

Vault / Venafi / external issuers

spec.vault issues from a Vault PKI mount (auth via AppRole, Kubernetes SA token or a static token); spec.venafi targets Venafi TPP/Cloud. Beyond that, cert-manager has an external issuer interface: third-party controllers (AWS Private CA, step-issuer, Google CAS, Origin CA…) register their own CRDs and cert-manager hands the CertificateRequest over. Reference them with issuerRef.group/issuerRef.kind:

yaml
issuerRef:
name: aws-pca
kind: AWSPCAClusterIssuer
group: awspca.cert-manager.io

The Certificate resource

yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: example-com
namespace: web
spec:
# ── output ────────────────────────────────────────────────
secretName: example-com-tls # Secret created in this namespace
secretTemplate: # labels/annotations copied onto the Secret
labels:
app: example
additionalOutputFormats:
- type: CombinedPEM # tls-combined.pem (key + cert)
- type: DER # key.der
keystores:
pkcs12: # for JVM/.NET workloads
create: true
profile: Modern2023
passwordSecretRef:
name: keystore-pass
key: password

# ── identity ──────────────────────────────────────────────
dnsNames:
- example.com
- www.example.com
- "*.apps.example.com" # wildcard → requires a DNS-01 solver
# ipAddresses / uris / emailAddresses are also available
commonName: example.com # avoid for public certs; SANs are what matters
subject:
organizations: ["Example Ltd"]

# ── crypto ────────────────────────────────────────────────
privateKey:
algorithm: ECDSA # RSA | ECDSA | Ed25519
size: 256
encoding: PKCS1 # PKCS1 | PKCS8
rotationPolicy: Always # new key on every issuance (default since v1.18)
usages:
- server auth
- client auth
isCA: false

# ── lifecycle ─────────────────────────────────────────────
duration: 2160h # 90d — must be honoured by the issuer
renewBefore: 360h # renew 15d before expiry
revisionHistoryLimit: 3 # kept CertificateRequests

# ── who signs it ──────────────────────────────────────────
issuerRef:
name: letsencrypt
kind: ClusterIssuer # Issuer | ClusterIssuer | external kind
group: cert-manager.io

duration/renewBefore are requests: a public ACME CA ignores duration (Let's Encrypt always issues 90 days), while a CA or Vault issuer honours it. renewBefore may not exceed duration.

Resulting Secret

tls.crt leaf certificate + intermediate chain (PEM)
tls.key private key (PEM)
ca.crt the issuing CA certificate, when the issuer provides one

Type kubernetes.io/tls — directly consumable by every Ingress controller, and mountable as a volume for any workload doing its own TLS.

shell
kubectl get secret example-com-tls -o jsonpath='{.data.tls\.crt}' \
| base64 -d | openssl x509 -noout -text

(See OpenSSL for inspecting certificates; cmctl inspect secret does the same natively.)

Ingress integration (ingress-shim)

You usually don't write Certificate objects at all. The ingress-shim watches Ingress resources and generates a Certificate from the annotation plus spec.tls:

yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: web
annotations:
cert-manager.io/cluster-issuer: letsencrypt
spec:
ingressClassName: traefik
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app
port:
number: 80
tls:
- hosts: ["app.example.com"]
secretName: app-tls # ← cert-manager creates & fills this

The generated Certificate takes its dnsNames from tls[].hosts and its secretName from tls[].secretName. Delete the annotation and the Certificate is garbage-collected with the Ingress.

Common annotations (all cert-manager.io/… unless noted):

AnnotationEffect
issuer / cluster-issuerWhich issuer to use (namespaced vs. cluster-wide)
issuer-kind / issuer-groupTarget an external issuer type
common-name, duration, renew-beforePassed through to the Certificate spec
private-key-algorithm / -size / -encoding / -rotation-policyKey settings
usages, revision-history-limitPassed through
subject-organizations, subject-countries, …X.509 subject fields
acme.cert-manager.io/http01-edit-in-placeSolve HTTP-01 on the existing Ingress
acme.cert-manager.io/http01-ingress-classIngress class for the challenge Ingress
kubernetes.io/tls-acme: "true"Legacy kube-lego style; needs a configured default issuer

Gateway API

The same shim exists for Gateway API: annotate a Gateway with cert-manager.io/cluster-issuer and cert-manager fills the listener's certificateRefs. HTTP-01 can also solve via an HTTPRoute (solvers[].http01.gatewayHTTPRoute) instead of an Ingress. Gateway support requires the feature gate (--enable-gateway-api, chart value config.enableGatewayAPI).

Internal PKI

A self-contained CA hierarchy in three objects — this is the standard pattern for mTLS between services, signing webhook certs, or handing out certificates to non-public internal hosts.

yaml
# 1. bootstrap issuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}
---
# 2. the root CA certificate, signed by itself
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-ca
namespace: cert-manager
spec:
isCA: true
commonName: internal-ca
secretName: internal-ca-key-pair
duration: 87600h # 10 years
renewBefore: 8760h
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: selfsigned
kind: ClusterIssuer
---
# 3. an issuer backed by that CA
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: internal-ca
spec:
ca:
secretName: internal-ca-key-pair

Workloads then simply issuerRef: {name: internal-ca, kind: ClusterIssuer}. The remaining problem is distributing the root CA to clients — that is what the sibling project trust-manager does: a Bundle resource collects CA certificates (your root, plus optionally the public trust store) and syncs them into a ConfigMap/Secret in every selected namespace.

yaml
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
name: internal-trust
spec:
sources:
- useDefaultCAs: true
- secret:
name: internal-ca-key-pair
key: ca.crt
target:
configMap:
key: ca-bundle.crt
namespaceSelector:
matchLabels:
trust: enabled

The cainjector solves the same problem for the API server: annotate a webhook/APIService with cert-manager.io/inject-ca-from: <namespace>/<certificate> and it keeps the caBundle field in sync.

Renewal

Renewal is fully automatic: at notAfter - renewBefore (default ⅔ of the lifetime if unset) the controller creates a new CertificateRequest and rewrites the Secret in place. Consumers must pick the new cert up:

• Ingress controllers watch the Secret and reload — nothing to do.
• Pods mounting the Secret as a volume see the update within the kubelet sync period (~1 min), but only if the application re-reads the file. Use Reloader or a SIGHUP/watch in the app if it doesn't.
• Secrets consumed via env are not updated — never do this for certificates.

Force an early renewal:

shell
cmctl renew example-com -n web
cmctl renew --all -n web

Troubleshooting

Walk the chain downwards; kubectl describe events say more than the controller log.

shell
kubectl describe certificate example-com -n web
kubectl describe certificaterequest -n web # the CSR + issuer errors
kubectl describe order -n web # ACME order state
kubectl describe challenge -n web # the actual validation failure
kubectl describe clusterissuer letsencrypt # account registration problems
kubectl logs -n cert-manager deploy/cert-manager -f

cmctl (the cert-manager CLI) collapses most of this:

shell
cmctl status certificate example-com -n web # full chain: cert → request → order → challenge
cmctl inspect secret example-com-tls -n web # decode the issued certificate
cmctl renew example-com -n web
cmctl check api # is the webhook up and serving?

Frequent failure modes:

SymptomCause
Certificate stuck Ready: False, no CertificateRequestWebhook not reachable / Issuer in the wrong namespace
Challenge pending, "propagation check failed"Split-horizon DNS — set --dns01-recursive-nameservers-only
Challenge 404 on /.well-known/acme-challenge/…HTTP→HTTPS redirect, wrong ingressClassName, or port 80 blocked
too many certificates already issuedLet's Encrypt rate limit — you were not testing against staging
Wildcard cert never issuesHTTP-01 cannot do wildcards; a DNS-01 solver is required
ClusterIssuer can't read its SecretClusterIssuer secrets are read from the cert-manager namespace, not the app's
Everything vanished after helm uninstallCRD deletion cascades to all Certificates — keep crds.keep=true


NoteRelation
KubernetesThe platform; Ingress/Secret/CRD concepts used here
kubectlInspecting and debugging the resources
TraefikIngress controller consuming the TLS Secrets (has its own built-in ACME as an alternative)
DNSDNS-01 validation and the _acme-challenge TXT record
CoreDNSCluster DNS — relevant to DNS-01 self-check failures
OpenSSLInspecting the issued certificates
RSAKey algorithm behind privateKey.algorithm
HTTPProtocol secured by the certificates; HTTP-01 validation
sopsEncrypting issuer credentials (DNS API tokens) in Git
k3sCommon lightweight cluster to install cert-manager on