Kubernetes
9 min read
Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers into logical units called Pods, which run on Nodes in a cluster. A simple solution to get up and running is k3s.
You manage k8s clusters via kubectl (or a TUI like k9s). Most things are defined declaratively via YAML manifest files. You apply them to your cluster with kubectl apply -f FILE.
Architecture
A cluster is split into the control plane (the brain) and worker nodes (where workloads run).
| Component | Role |
|---|---|
kube-apiserver | Front door for the cluster; everything talks to it (control plane) |
etcd | Distributed key-value store holding all cluster state (control plane) |
kube-scheduler | Assigns Pods to Nodes based on resources/constraints (control plane) |
kube-controller-manager | Runs control loops reconciling desired vs. actual state (control plane) |
kubelet | Node agent; starts/stops containers and reports status (node) |
kube-proxy | Programs node networking rules for Services (node) |
| Container runtime | Runs containers (containerd, CRI-O) — inspect via crictl |
The control loop is the core idea: you declare desired state, controllers continuously reconcile the cluster toward it.
Nodes
A Node is a machine (physical or virtual) in the cluster running the kubelet. Pods are scheduled onto nodes.
bash
kubectl get nodes -o wide
kubectl describe node <node>
kubectl cordon <node> # mark unschedulable
kubectl drain <node> # evict pods (for maintenance)
kubectl uncordon <node> # mark schedulable again
Labels
Labels are key-value pairs attached to any object, used to select and group them. Nodes are commonly labeled by hardware, zone, or role:
bash
kubectl label node node-1 disktype=ssd
kubectl label node node-1 gpu=true
kubectl get nodes -l disktype=ssd
Built-in node labels include kubernetes.io/hostname, kubernetes.io/arch, topology.kubernetes.io/zone, and node-role.kubernetes.io/control-plane.
Scheduling
Control which nodes a Pod lands on.
nodeSelector
Simplest form — Pod only schedules onto nodes matching all listed labels:
yaml
spec:
nodeSelector:
disktype: ssd
Affinity / anti-affinity
More expressive rules, including soft (preferred) and hard (required) constraints, and co-locating/spreading Pods relative to each other:
yaml
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: disktype
operator: In
values: ["ssd"]
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: web
topologyKey: kubernetes.io/hostname
Taints & tolerations
Taints repel Pods from a node unless the Pod has a matching toleration — the inverse of affinity. Used to reserve nodes (e.g. GPU or control-plane nodes):
bash
kubectl taint node node-1 dedicated=gpu:NoSchedule
yaml
spec:
tolerations:
- key: dedicated
operator: Equal
value: gpu
effect: NoSchedule
Effects: NoSchedule (block new Pods), PreferNoSchedule (best effort), NoExecute (also evict running Pods).
Workload Resources
Pod
The smallest deployable unit. You rarely create bare Pods — they're managed by higher-level controllers — but the full anatomy is worth knowing:
yaml
apiVersion: v1
kind: Pod
metadata:
name: full-example-pod
namespace: example-namespace
labels:
app: web
tier: frontend
annotations:
description: "A full-featured pod example for demonstration purposes"
spec:
restartPolicy: Always
# Init container (runs before main containers)
initContainers:
- name: init-permissions
image: busybox
command: ["sh", "-c", "chmod 777 /mnt/data"]
volumeMounts:
- name: data-volume
mountPath: /mnt/data
containers:
- name: main-app
image: nginx:1.25
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
name: http
env:
# Environment
- name: ENVIRONMENT
value: production
# Env from ConfigMap
- name: CONFIG_TIMEOUT
valueFrom:
configMapKeyRef:
name: example-config
key: TIMEOUT
# Env from Secret
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: example-secret
key: password
volumeMounts:
- name: data-volume
mountPath: /usr/share/nginx/html
- name: config-volume
mountPath: /etc/config
readOnly: true
resources:
limits:
cpu: "500m"
memory: "256Mi"
requests:
cpu: "250m"
memory: "128Mi"
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
periodSeconds: 5
lifecycle:
preStop:
exec:
command: ["sh", "-c", "echo stopping..."]
- name: sidecar-logger
image: busybox
args: ["sh", "-c", "tail -f /var/log/app.log"]
volumeMounts:
- name: log-volume
mountPath: /var/log
# Volumes
volumes:
# ConfigMap - inject config files
- name: config-volume
configMap:
name: example-config
items:
- key: config.json
path: config.json
# Secret - inject sensitive data
- name: secret-volume
secret:
secretName: example-secret
items:
- key: password
path: password.txt
# EmptyDir - ephemeral shared storage between containers
- name: log-volume
emptyDir:
medium: ""
sizeLimit: 500Mi
# HostPath - access host node's filesystem (example: logs)
- name: host-logs
hostPath:
path: /var/log
type: Directory
Deployment
Ensures a specified number of identical Pods are running and up-to-date. Supports rolling updates and rollbacks — the workhorse for stateless apps.
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-deployment
namespace: example-namespace
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: example
template:
metadata:
labels:
app: example
spec:
containers:
- name: web
image: nginx:alpine
ports:
- containerPort: 80
bash
kubectl rollout status deployment/example-deployment
kubectl rollout undo deployment/example-deployment # roll back
kubectl scale deployment/example-deployment --replicas=5
StatefulSet
Like a Deployment, but for workloads requiring stable network IDs, persistent per-Pod storage, and ordered startup/shutdown (databases, queues). Each Pod gets a stable name (name-0, name-1, …) and its own PersistentVolumeClaim via volumeClaimTemplates.
yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: example-statefulset
namespace: example-namespace
spec:
serviceName: "example"
replicas: 2
selector:
matchLabels:
app: stateful-app
template:
metadata:
labels:
app: stateful-app
spec:
containers:
- name: web
image: nginx:alpine
volumeMounts:
- name: data
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
DaemonSet
Ensures a copy of a Pod runs on all (or some) Nodes. Ideal for log collectors, monitoring agents, or CNI/storage plugins.
yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: example-daemonset
namespace: example-namespace
spec:
selector:
matchLabels:
name: ds-app
template:
metadata:
labels:
name: ds-app
spec:
containers:
- name: node-monitor
image: busybox
args: ["sh", "-c", "while true; do echo hello; sleep 10; done"]
Job
Runs a Pod (or multiple) to completion. Used for batch processing or one-off tasks.
yaml
apiVersion: batch/v1
kind: Job
metadata:
name: example-job
namespace: example-namespace
spec:
template:
spec:
containers:
- name: pi
image: perl
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
restartPolicy: Never
backoffLimit: 4
CronJob
Schedules Jobs to run periodically, similar to UNIX cron.
yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: example-cronjob
namespace: example-namespace
spec:
schedule: "*/5 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: busybox
args: ["echo", "Hello from the CronJob"]
restartPolicy: OnFailure
Note: You can quickly run a CronJob immediately as a Job with: kubectl create job --from=cronjob.batch/my_cron_job new_job
Namespace
Logical separation of resources within a cluster. Most namespaced resources reference a metadata.namespace; cluster-scoped resources (Nodes, PersistentVolumes, ClusterRoles) do not.
yaml
apiVersion: v1
kind: Namespace
metadata:
name: example-namespace
Networking
Service
A stable network endpoint fronting a set of Pods (selected by label). The Pods behind it come and go; the Service IP/DNS name stays constant.
yaml
apiVersion: v1
kind: Service
metadata:
name: example-service
namespace: example-namespace
spec:
selector:
app: example
ports:
- protocol: TCP
port: 80
targetPort: 80
type: ClusterIP
| Type | Reachable from |
|---|---|
ClusterIP | Inside the cluster only (default) |
NodePort | A static port on every node's IP |
LoadBalancer | External LB; bare-metal clusters need MetalLB |
ExternalName | Maps to an external DNS name (CNAME) |
Pods reach a Service by DNS: example-service.example-namespace.svc.cluster.local.
Ingress
Routes external HTTP(S) traffic to Services based on host/path. Requires an ingress controller running in the cluster (e.g. Traefik, ingress-nginx). TLS certificates for the secretName below are usually issued automatically by cert-manager.
yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
namespace: example-namespace
annotations:
cert-manager.io/cluster-issuer: letsencrypt
spec:
ingressClassName: traefik
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: example-service
port:
number: 80
tls:
- hosts: ["app.example.com"]
secretName: app-tls
NetworkPolicy
Firewall rules for Pod traffic. By default all Pods can talk to each other; a NetworkPolicy selecting a Pod switches it to default-deny for the directions it specifies.
yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-external
namespace: example-namespace
spec:
podSelector:
matchLabels:
app: example
policyTypes: ["Ingress"]
ingress:
- from:
- podSelector:
matchLabels:
tier: frontend
ports:
- protocol: TCP
port: 80
Configuration
ConfigMap
Injects non-sensitive configuration (key-value pairs or whole files) into Pods, decoupling config from image.
yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: example-config
namespace: example-namespace
data:
APP_ENV: production
TIMEOUT: "30"
Usage in a Pod:
yaml
envFrom:
- configMapRef:
name: example-config
Secret
Like a ConfigMap, but for sensitive data (passwords, tokens, keys). Values are base64-encoded (not encrypted) at rest in etcd by default. For encrypted manifests in Git, see sops.
yaml
apiVersion: v1
kind: Secret
metadata:
name: example-secret
namespace: example-namespace
type: Opaque
data:
username: YWRtaW4= # base64 of 'admin'
password: cGFzc3dvcmQ= # base64 of 'password'
Usage in a Pod:
yaml
env:
- name: USERNAME
valueFrom:
secretKeyRef:
name: example-secret
key: username
Storage
Decouples a Pod's storage request from how it's physically provisioned.
• PersistentVolume (PV) — a piece of storage in the cluster (cluster-scoped).
• PersistentVolumeClaim (PVC) — a request for storage by a Pod; binds to a matching PV.
• StorageClass — defines a provisioner that creates PVs dynamically when a PVC asks for that class.
yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-pvc
namespace: example-namespace
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: longhorn
resources:
requests:
storage: 5Gi
yaml
# In a Pod spec
volumes:
- name: data
persistentVolumeClaim:
claimName: data-pvc
Access modes: ReadWriteOnce (one node), ReadOnlyMany, ReadWriteMany (shared), ReadWriteOncePod.
RBAC
Role-Based Access Control governs who can do what. A ServiceAccount is an in-cluster identity for Pods; Role/ClusterRole define permissions; RoleBinding/ClusterRoleBinding grant them to subjects.
yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-sa
namespace: example-namespace
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: example-namespace
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: example-namespace
subjects:
- kind: ServiceAccount
name: app-sa
namespace: example-namespace
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
A Role is namespaced; a ClusterRole applies cluster-wide (or to cluster-scoped resources like Nodes).
Autoscaling
A HorizontalPodAutoscaler scales a Deployment's replica count based on observed metrics (requires metrics-server).
yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: example-hpa
namespace: example-namespace
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: example-deployment
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
Extending Kubernetes
Custom Resources & the Operator pattern
A CustomResourceDefinition (CRD) teaches the API server a new resource kind. An operator is a custom controller that watches those resources and reconciles real-world state — encoding the operational knowledge a human would otherwise apply ("run loop forever: observe → diff → act").
yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: backups.example.com
spec:
group: example.com
names:
kind: Backup
plural: backups
singular: backup
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
schedule: { type: string }
target: { type: string }
Once installed, users create Backup objects like any built-in resource, and the operator acts on them. Many self-hosted stacks ship as operators — e.g. Rook (Ceph), KubeVirt (VMs), k8up (backups).
| Tool | Purpose |
|---|---|
| k3s | Lightweight single-binary Kubernetes distribution |
| kubectl | Primary CLI for interacting with clusters |
| k9s | Terminal UI for navigating and managing clusters |
| crictl | CLI for the underlying CRI container runtime |
| MetalLB | LoadBalancer implementation for bare-metal clusters |
| Rook / Longhorn | Cluster storage (Ceph / replicated block) |
| k8up | Backup operator for PVCs |
| KubeVirt | Run virtual machines as Kubernetes workloads |
| Traefik | Ingress controller / reverse proxy |
| cert-manager | Automated TLS certificate issuance and renewal |
| sops | Encrypt secrets in manifests for GitOps |
| spegel | Stateless cluster-local image mirror |
| kubenix | Define manifests with Nix |