# CKAD study guide
The 24 objectives of the Certified Kubernetes Application Developer curriculum, in its own words and order: what to know, the fastest commands, the traps, the documentation pages the exam lets you open, and the tasks in this simulator that practise each one.
_Generated by `./exam --track ckad notes all --md` from `tracks/ckad/curriculum.py` - edit that file, not this one._
```
./exam objectives # the path and your progress
./exam study 3.2 # one objective: these notes, then its tasks, no clock, solutions open
./exam drill 3.2 # the same tasks against the clock
```
## Application Design and Build (20%)
### 1.1 Define, build and modify container images
`./exam study 1.1` (or `images`)
**Know**
- Read and change a Dockerfile: FROM, COPY, RUN, ENV, ARG (build time only), WORKDIR, USER, LABEL, EXPOSE, ENTRYPOINT, CMD.
- ENTRYPOINT is the program, CMD its default arguments. In a Pod, `command` replaces ENTRYPOINT and `args` replaces CMD.
- Build, tag, list, inspect, run, save and remove images. The exam terminal has docker or podman - the verbs are the same.
- A tag is only a pointer: one image can carry several tags, and rebuilding under an old tag moves it.
**Fast**
```
podman build -t localhost/app:v1 . # -f path/Dockerfile --build-arg KEY=val
podman tag localhost/app:v1 localhost/app:latest
podman images | grep app
podman inspect localhost/app:v1 --format '{{.Config.User}} {{.Config.Entrypoint}} {{.Config.Cmd}} {{.Config.Env}}'
podman run --rm localhost/app:v1 [args replacing CMD] # -d --name x keeps it running; podman logs x
podman save --format oci-archive -o app.tar localhost/app:v1 # docker save -o app.tar ...
podman rmi localhost/app:v0
```
**Traps**
- An ARG is gone at run time unless you copy it into an ENV (`ARG V` then `ENV APP_VERSION=$V`).
- `podman save` writes a docker-archive unless you pass `--format oci-archive`.
- Build before you edit when a task wants the old version kept under its own tag.
- imagePullPolicy defaults to Always for `:latest` (or no tag) and IfNotPresent for any other tag.
**Docs**
-
-
**Tasks**
- Q13 Build, change and export a container image
- Q60 Entrypoint, default argument and a non-root user
- Q41 Build with a build argument, run it detached _(Simulator Set A (killer.sh-style))_
### 1.2 Choose and use the right workload resource (Deployment, DaemonSet, CronJob, etc.)
`./exam study 1.2` (or `workloads`)
**Know**
- Choose by behaviour: Deployment (interchangeable replicas, rolling updates), StatefulSet (stable name, DNS and storage per replica), DaemonSet (one Pod per node), Job (run to completion), CronJob (Jobs on a schedule), bare Pod (one-off).
- Job: completions, parallelism, backoffLimit, activeDeadlineSeconds, ttlSecondsAfterFinished; restartPolicy must be Never or OnFailure.
- CronJob: schedule, startingDeadlineSeconds, concurrencyPolicy (Allow, Forbid, Replace), suspend, successfulJobsHistoryLimit, failedJobsHistoryLimit.
- DaemonSet: no replicas; a toleration for node-role.kubernetes.io/control-plane:NoSchedule puts it on control-plane nodes too.
**Fast**
```
k create deploy web --image=nginx:1.27 --replicas=3 $do > deploy.yaml
k create job render --image=busybox:1.36 $do -- sh -c 'echo done' > job.yaml
k create cronjob digest --image=busybox:1.36 --schedule='*/5 * * * *' $do -- sh -c 'echo digest' > cj.yaml
k create job digest-manual --from=cronjob/digest # run a CronJob once, now
k run tmp --image=busybox:1.36 --restart=Never --rm -it -- sh
# DaemonSet has no generator: create a Deployment with $do, change kind, delete replicas/strategy/status
k explain cronjob.spec.jobTemplate.spec --recursive | less
```
**Traps**
- Three nested specs in a CronJob: spec (schedule, startingDeadlineSeconds) > jobTemplate.spec (completions, activeDeadlineSeconds) > template.spec (containers, restartPolicy).
- A Job's Pod template is immutable - delete the Job and create it again instead of editing it.
- A StatefulSet needs `serviceName` naming a headless Service (clusterIP: None) before its Pods get their own DNS names.
**Docs**
-
-
-
**Tasks**
- Q3 CronJob with deadlines
- Q14 Parallel batch Job
- Q15 DaemonSet on every node
- Q39 A suspended CronJob and a manual run _(Simulator Set A (killer.sh-style))_
- Q38 A Pod and a status one-liner _(Simulator Set A (killer.sh-style))_
- Q44 From a single Pod to a hardened Deployment _(Simulator Set A (killer.sh-style))_
- Q56 Stable DNS names for a StatefulSet _(Simulator Set A (killer.sh-style))_
### 1.3 Understand multi-container Pod design patterns (e.g. sidecar, init and others)
`./exam study 1.3` (or `multicontainer`)
**Know**
- Containers of a Pod share its network (they reach each other on localhost) and any volume both mount - that is all a sidecar, adapter or ambassador is.
- Init containers run one at a time, in order, each to completion, before any app container starts.
- A native sidecar is an init container with `restartPolicy: Always`: it starts before the app containers, keeps running beside them, and does not keep a Job from completing.
- Address one container with `-c`: `k logs POD -c NAME`, `k exec POD -c NAME -- ...`. Status `Init:0/1` means an init container is still running.
**Fast**
```
k run app --image=nginx:1.27 $do > pod.yaml # then add the second container by hand
spec:
initContainers:
- {name: wait, image: busybox:1.36, command: [sh, -c, "until wget -qO- -T2 http://backend; do sleep 2; done"]}
- {name: shipper, image: busybox:1.36, restartPolicy: Always, command: [sh, -c, "tail -F /logs/app.log"],
volumeMounts: [{name: logs, mountPath: /logs}]} # native sidecar
containers:
- {name: app, image: nginx:1.27, volumeMounts: [{name: logs, mountPath: /var/log/app}]}
volumes: [{name: logs, emptyDir: {}}]
k logs app -c shipper; k logs app --all-containers --prefix
```
**Traps**
- `kubectl run` creates one container only - generate the Pod and add the others in the YAML.
- A Service with no endpoints still resolves in DNS, so an init container waiting on `nslookup` is not waiting for the app. Wait on a real request.
- A classic sidecar that never exits keeps a Job running for ever; make it a native sidecar.
**Docs**
-
-
**Tasks**
- Q1 Adapter sidecar
- Q2 Init container waits for a dependency
- Q16 Native sidecar lets a Job finish
- Q40 An init container writes the page _(Simulator Set A (killer.sh-style))_
### 1.4 Utilize persistent and ephemeral volumes
`./exam study 1.4` (or `volumes`)
**Know**
- Ephemeral: emptyDir (lives as long as the Pod; `medium: Memory` is tmpfs, `sizeLimit` caps it), and the configMap, secret, downwardAPI and projected volumes.
- Persistent: a PersistentVolumeClaim asks for storage (accessModes, resources.requests.storage, storageClassName); a StorageClass provisions the PersistentVolume, or a pre-made one binds.
- Access modes: ReadWriteOnce, ReadOnlyMany, ReadWriteMany, ReadWriteOncePod.
- volumeMounts: mountPath, readOnly, and subPath to mount one file or directory without hiding the rest of the target directory.
**Fast**
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: data}
spec:
accessModes: [ReadWriteOnce]
storageClassName: standard
resources: {requests: {storage: 100Mi}}
--- # in the Pod spec:
volumes:
- {name: data, persistentVolumeClaim: {claimName: data}}
- {name: scratch, emptyDir: {medium: Memory, sizeLimit: 32Mi}}
containers:
- volumeMounts: [{name: data, mountPath: /data}, {name: scratch, mountPath: /scratch}]
k get pvc,pv; k describe pvc data # Pending? the Events say why
```
**Traps**
- There is no `kubectl create pvc` - copy the claim from the docs.
- With a WaitForFirstConsumer StorageClass (`standard` here) a claim stays Pending until a Pod uses it. That is not an error.
- emptyDir survives a container restart but not a new Pod. To prove data persists, delete the Pod and read the file again.
- A non-root container cannot write to a fresh volume unless the Pod sets `securityContext.fsGroup`.
**Docs**
-
-
-
**Tasks**
- Q17 Persistent and ephemeral volumes
- Q10 Projected volume: config, secret and token
- Q52 Two users, one volume: fsGroup _(Simulator Set A (killer.sh-style))_
- Q40 An init container writes the page _(Simulator Set A (killer.sh-style))_
- Q50 An immutable Secret, one file into a busy directory _(Simulator Set A (killer.sh-style))_
## Application Deployment (20%)
### 2.1 Use Kubernetes primitives to implement common deployment strategies (e.g. blue/green or canary)
`./exam study 2.1` (or `strategies`)
**Know**
- Blue/green: two Deployments, and the Service selector decides which one gets all the traffic. Cut over by changing the selector; roll back by changing it back.
- Canary: two Deployments whose Pods share the label the Service selects. Traffic splits by the number of ready Pods: 1 canary beside 4 stable is 20%.
- A Deployment's own strategy is RollingUpdate (default) or Recreate (every old Pod stops first - downtime).
**Fast**
```
k get svc web -o wide; k get pods --show-labels
k set selector svc web 'app=web,version=green' # blue/green cut-over
k scale deploy web-stable --replicas=4 # canary ratio: 4 stable + 1 canary = 20%
k get endpointslices -l kubernetes.io/service-name=web -o wide
k get pods -l app=web -L version,track
```
**Traps**
- Wait until the new Deployment is fully available before you switch the selector.
- `kubectl set selector` replaces the whole selector - name every label you want in it.
- A canary only shares traffic if the Service selector leaves out the label that tells the two versions apart.
**Docs**
-
-
**Tasks**
- Q5 Blue/green cutover
- Q18 Canary by replica ratio
### 2.2 Understand Deployments and how to perform rolling updates
`./exam study 2.2` (or `rollouts`)
**Know**
- Any change to `spec.template` starts a rollout: a new ReplicaSet scales up while the old one scales down, within maxSurge and maxUnavailable (25% each by default).
- minReadySeconds (how long a Pod must be Ready to count as available), revisionHistoryLimit (old ReplicaSets kept), progressDeadlineSeconds.
- `kubectl rollout`: status, history, undo, pause, resume, restart.
**Fast**
```
k set image deploy/web nginx=nginx:1.27 # CONTAINER=IMAGE
k rollout status deploy/web
k rollout history deploy/web # --revision=3 shows that revision's template
k rollout undo deploy/web # --to-revision=2
k rollout restart deploy/web # same spec, new Pods
k annotate deploy/web kubernetes.io/change-cause='pick 1.27'
k patch deploy web -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
```
**Traps**
- `--record` is deprecated: write the kubernetes.io/change-cause annotation yourself.
- An undo makes a new revision out of the old one, so the active revision number is not the one you rolled back to.
- A rollout stuck on a bad image does not fail by itself - `rollout status` just waits, and the old Pods keep serving.
- maxSurge and maxUnavailable cannot both be 0.
**Docs**
-
**Tasks**
- Q4 Roll forward, then back
- Q19 Rolling update without losing capacity
- Q42 Back to the last good revision _(Simulator Set A (killer.sh-style))_
- Q44 From a single Pod to a hardened Deployment _(Simulator Set A (killer.sh-style))_
### 2.3 Use the Helm package manager to deploy existing packages
`./exam study 2.3` (or `helm`)
**Know**
- A chart is the package, a release is one installation of it in a Namespace, and every install, upgrade or rollback adds a revision.
- Values: the chart's defaults (`helm show values`), overridden with `-f file.yaml` and `--set key=value`; `helm get values` shows what a release was given.
- Life cycle: repo add/update, search, install, upgrade, rollback, uninstall, list, history, status.
**Fast**
```
helm repo add NAME URL && helm repo update; helm search repo nginx --versions
helm show values CHART > values.yaml
helm install REL CHART -n NS --create-namespace -f values.yaml --set replicaCount=2 # --version X
helm upgrade REL CHART -n NS --reuse-values --set replicaCount=4
helm list -n NS # -A for every Namespace; --failed, --pending filter by status
helm history REL -n NS; helm rollback REL 1 -n NS; helm uninstall REL -n NS
helm get values REL -n NS # --all adds the defaults; helm get manifest REL -n NS
```
**Traps**
- Releases live in a Namespace: without `-n` Helm looks in `default` and finds nothing.
- An upgrade without `--reuse-values` (or the same `-f`/`--set` again) puts every value back to the chart's default.
- A rollback is a new revision, not a step back in the numbering.
- Helm 3's `helm list` leaves out pending releases unless you add `-a`; Helm 4 (on this machine) lists every status and has no `-a`. `--pending` and `--failed` work in both.
**Docs**
-
-
**Tasks**
- Q6 Helm release lifecycle
- Q21 Helm: values file and an upgrade that keeps values
- Q43 Helm housekeeping _(Simulator Set A (killer.sh-style))_
### 2.4 Kustomize
`./exam study 2.4` (or `kustomize`)
**Know**
- A kustomization.yaml lists `resources` (files or other kustomizations) and transforms them: namespace, namePrefix, nameSuffix, labels, images, replicas, patches.
- Generators build ConfigMaps and Secrets from files, env files or literals, add a content hash to the name and rewrite every reference to it.
- Base and overlays: an overlay lists `../../base` under resources and changes only what differs.
- `kubectl kustomize DIR` prints the result; `kubectl apply -k DIR` applies it.
**Fast**
```
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [../../base] # or files: [deployment.yaml, service.yaml]
namespace: prod
namePrefix: prod-
labels: [{pairs: {env: prod}, includeSelectors: false}]
images: [{name: nginx, newTag: "1.27"}]
replicas: [{name: web, count: 3}]
patches: [{path: patch.yaml}] # a partial Deployment with the same name
configMapGenerator: [{name: settings, envs: [settings.env]}]
secretGenerator: [{name: token, literals: [token=abc]}]
generatorOptions: {disableNameSuffixHash: true}
k kustomize DIR | less; k apply -k DIR; k diff -k DIR
```
**Traps**
- Always read the render before you apply it.
- `images`, `replicas` and patches name things as the base does, before any prefix or suffix.
- Refer to a generated ConfigMap by its plain name - Kustomize appends the hash everywhere it is used.
- `namespace:` does not create the Namespace.
**Docs**
-
**Tasks**
- Q20 Kustomize overlay
- Q61 A kustomization that does not build
- Q45 Kustomize: a patch and a generated ConfigMap _(Simulator Set A (killer.sh-style))_
## Application Observability and Maintenance (15%)
### 3.1 Understand API deprecations
`./exam study 3.1` (or `deprecations`)
**Know**
- Every object is a kind in an API group at a version (apps/v1 Deployment). Versions are deprecated, keep being served for some releases, then are removed.
- What this cluster serves: `kubectl api-resources` (the APIVERSION column), `kubectl api-versions`, `kubectl explain KIND`.
- A removed version fails with `no matches for kind "X" in version "g/v"`; a deprecated one still works and prints `Warning: ... is deprecated`.
- Migrating is the new apiVersion plus whatever changed shape: an Ingress backend becomes `service: {name, port: {number}}` with a required pathType, and an apps/v1 Deployment needs `spec.selector`.
**Fast**
```
k api-resources | grep -i -E 'cronjob|ingress' # NAME SHORTNAMES APIVERSION NAMESPACED KIND
k api-versions | grep batch
k explain hpa | head -4 # GROUP, KIND, VERSION
k apply -f old.yaml --dry-run=server # the error or warning, without changing anything
# removed, and what replaced them:
# extensions/v1beta1, networking.k8s.io/v1beta1 Ingress -> networking.k8s.io/v1 (1.22)
# extensions/v1beta1, apps/v1beta1, apps/v1beta2 Deployment -> apps/v1 (1.16)
# batch/v1beta1 CronJob -> batch/v1 (1.25)
# policy/v1beta1 PodDisruptionBudget -> policy/v1 (1.25)
# autoscaling/v2beta1, v2beta2 HorizontalPodAutoscaler -> autoscaling/v2 (1.25, 1.26)
```
**Traps**
- The `kubectl convert` plugin is usually not installed - know the Ingress and Deployment rewrites by hand.
- `kubectl get` shows an object in the API's preferred version whatever version it was written with.
- A warning is not an error: the object was created. Read stderr.
**Docs**
-
-
**Tasks**
- Q22 Manifests written for removed APIs
- Q58 Removed, and deprecated but still served
### 3.2 Implement probes and health checks
`./exam study 3.2` (or `probes`)
**Know**
- Liveness: failing restarts the container. Readiness: failing takes the Pod out of Service endpoints, no restart. Startup: holds the other two off until it has passed once.
- Handlers: httpGet (path, port), tcpSocket (port), exec (command), grpc (port).
- Timing: initialDelaySeconds (0), periodSeconds (10), timeoutSeconds (1), failureThreshold (3), successThreshold (1). A startup probe allows failureThreshold x periodSeconds to come up.
**Fast**
```
containers:
- name: app
startupProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5, failureThreshold: 12} # 60 s to start
readinessProbe: {httpGet: {path: /ready, port: 8080}, periodSeconds: 5, failureThreshold: 1}
livenessProbe: {tcpSocket: {port: 80}, initialDelaySeconds: 15, periodSeconds: 10}
# exec: readinessProbe: {exec: {command: [cat, /tmp/ready]}, periodSeconds: 5}
k describe pod POD | grep -E 'Liveness|Readiness|Startup'
k get events --field-selector reason=Unhealthy
k explain pod.spec.containers.livenessProbe
```
**Traps**
- No kubectl flag adds a probe: generate the YAML and edit it, or `kubectl edit`.
- Probes belong to a container, not to the Pod spec.
- Restarts climbing with `Unhealthy` and `Killing` events but nothing wrong in the logs: the liveness probe is wrong or too impatient, not the app.
- Running, 0/1 Ready and no restarts: the readiness probe is failing.
**Docs**
-
-
**Tasks**
- Q7 Three kinds of probe
- Q62 Health checks that check the wrong thing
- Q46 Probes that match the app's warm-up _(Simulator Set A (killer.sh-style))_
### 3.3 Use built-in CLI tools to monitor Kubernetes applications
`./exam study 3.3` (or `monitoring`)
**Know**
- `kubectl get` output: -o wide, yaml, json, name, jsonpath=, custom-columns=; narrowed with -l (labels), --field-selector, -n / -A; ordered with --sort-by.
- `kubectl top pod|node` shows live CPU and memory from metrics-server; `--containers` splits a Pod up.
- `kubectl describe` for the story of one object, `kubectl get events` for the Namespace.
**Fast**
```
k top pod -A --sort-by=memory; k top pod --containers; k top node
k get pods -o wide --sort-by=.spec.nodeName
k get pods -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,IMAGE:.spec.containers[0].image
k get pod POD -o jsonpath='{.status.phase}'
k get pods -l tier=web --field-selector=status.phase!=Running -o name
k get pods -A --sort-by='.status.containerStatuses[0].restartCount'
k get events --sort-by=.lastTimestamp; k events --types=Warning --for pod/POD
```
**Traps**
- `kubectl get --sort-by` sorts ascending (the largest is last); `kubectl top --sort-by` sorts descending (the largest is first).
- `kubectl top` shows usage, not requests or limits, and needs about a minute after a Pod starts.
- Quote jsonpath for the shell: '{.items[*].metadata.name}'.
**Docs**
-
-
**Tasks**
- Q23 Find the heaviest Pods with kubectl top
- Q63 An inventory with kubectl alone
- Q48 Restart and event forensics _(Simulator Set A (killer.sh-style))_
- Q38 A Pod and a status one-liner _(Simulator Set A (killer.sh-style))_
### 3.4 Utilize container logs
`./exam study 3.4` (or `logs`)
**Know**
- `kubectl logs` shows what a container wrote to stdout and stderr. The kubelet keeps it on the node, and it goes when the Pod is deleted.
- Pick the container with -c (init containers too), or take --all-containers; --previous is the run before the last restart.
- Narrow with --tail=N, --since=5m, --timestamps; follow with -f; by label with -l.
**Fast**
```
k logs POD -c CONTAINER # --all-containers --prefix
k logs POD --previous # the crashed run
k logs POD --tail=20 --since=10m --timestamps
k logs -l app=web --prefix --tail=5 # every Pod with the label
k logs deploy/web; k logs job/render # one Pod of it
k logs POD | grep ERROR > errors.log
```
**Traps**
- With -l, only the last 10 lines of each Pod are shown unless you set --tail (--tail=-1 for everything).
- `kubectl logs deploy/x` reads one Pod, not all replicas.
- --previous fails when the container has never restarted.
**Docs**
-
-
**Tasks**
- Q24 Logs of a container that keeps crashing
- Q64 One replica is logging errors
- Q49 RBAC, proven by the app's own log _(Simulator Set A (killer.sh-style))_
- Q54 Expose a Pod and prove it from inside _(Simulator Set A (killer.sh-style))_
- Q39 A suspended CronJob and a manual run _(Simulator Set A (killer.sh-style))_
### 3.5 Debugging in Kubernetes
`./exam study 3.5` (or `debugging`)
**Know**
- Always the same order: get (status), describe (Events at the bottom), logs (--previous), then exec or debug.
- Pending: nothing can schedule it (resources, taints, nodeSelector, an unbound claim). ImagePullBackOff / ErrImagePull: wrong image or tag. CreateContainerConfigError: a ConfigMap, Secret or key it references is missing.
- CrashLoopBackOff: the process exits - read the previous log and lastState.terminated (reason, exitCode). OOMKilled (137): the memory limit is too low.
- A Deployment with no Pods at all: describe its ReplicaSet - a quota, a LimitRange or Pod Security admission is rejecting them.
**Fast**
```
k get pods -o wide; k describe pod POD | tail -20
k get pod POD -o jsonpath='{.status.containerStatuses[0].state.waiting.reason}'
k get pod POD -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' # .exitCode
k logs POD --previous; k exec -it POD -- sh
k debug -it POD --image=busybox:1.36 --target=CONTAINER # ephemeral container, sees its processes
k run tmp --rm -it --restart=Never --image=busybox:1.36 -- wget -qO- -T3 http://SVC:PORT
k describe rs -l app=web --show-events | tail; k get events --sort-by=.lastTimestamp
k get pod POD -o yaml > pod.yaml && vi pod.yaml && k replace --force -f pod.yaml
```
**Traps**
- Most of a running Pod is immutable (the image is not): fix a bare Pod with `replace --force`, fix a Deployment's Pods in the Deployment.
- Fix what the task names and nothing else - graders check that the ConfigMap, Service or Namespace you were told to leave is unchanged.
- After fixing a Deployment wait for the rollout; old Pods linger in Terminating.
- `kubectl describe` with a selector that matches several objects leaves their Events out - add `--show-events`, or describe one by name.
**Docs**
-
-
-
**Tasks**
- Q8 Diagnose and fix a broken Pod
- Q25 Two faults behind a stuck Deployment
- Q26 OOMKilled
- Q36 A Service that routes nowhere
- Q47 Debugging a container with no shell _(Simulator Set A (killer.sh-style))_
- Q48 Restart and event forensics _(Simulator Set A (killer.sh-style))_
## Application Environment, Configuration and Security (25%)
### 4.1 Discover and use resources that extend Kubernetes (CRD, Operators)
`./exam study 4.1` (or `crds`)
**Know**
- A CustomResourceDefinition adds a kind to the API; its objects (custom resources) work with every kubectl verb. An operator is a CRD plus a controller acting on those objects.
- Discover: `k get crd`, `k api-resources --api-group=GROUP` (kind, short names, namespaced or not), `k explain KIND.spec`.
- The CRD's schema validates custom resources: required fields, enum values, minimum and maximum. The controller reports back in `.status`.
**Fast**
```
k get crd | grep -i backup # the CRD's name is PLURAL.GROUP
k api-resources --api-group=data.example.com
k explain backup.spec # --recursive for the whole tree
k get backups -A; k get bkp NAME -o yaml; k describe backup NAME
k get crd backups.data.example.com -o yaml | less # served versions, schema, enum, maximum
k apply -f cr.yaml --dry-run=server # every validation error, nothing created
```
**Traps**
- A custom resource's apiVersion is the CRD's GROUP/VERSION, never apiextensions.k8s.io/v1.
- Field names come from the CRD's schema - check with `kubectl explain` instead of guessing.
- Deleting a CRD deletes every object of that kind.
**Docs**
-
-
-
**Tasks**
- Q27 Discover and use a custom resource
- Q59 A custom resource the API server rejects
- Q58 Removed, and deprecated but still served
### 4.2 Understand authentication, authorization and admission control
`./exam study 4.2` (or `access`)
**Know**
- Every request passes authentication (who is it: a user certificate or token, a ServiceAccount token), authorization (may they: RBAC), then admission control (change or reject it: LimitRanger, ResourceQuota, PodSecurity).
- RBAC: a Role or ClusterRole lists verbs on resources (and subresources such as pods/log, deployments/scale); a RoleBinding or ClusterRoleBinding grants it to users, groups or ServiceAccounts.
- Pod Security admission: Namespace labels pod-security.kubernetes.io/enforce|warn|audit = privileged | baseline | restricted.
- `Forbidden: User ... cannot list` is authorization. `violates PodSecurity`, `exceeded quota` and `maximum memory usage` are admission.
**Fast**
```
k create role ci --verb=get,list,watch --resource=pods,pods/log -n NS
k create role ci --verb=get,update,patch --resource=deployments.apps,deployments.apps/scale -n NS
k create rolebinding ci --role=ci --serviceaccount=NS:SA -n NS # --user=jane --clusterrole=view
k auth can-i list pods -n NS --as=system:serviceaccount:NS:SA
k auth can-i --list -n NS --as=system:serviceaccount:NS:SA
k get ns NS --show-labels
# what `restricted` wants from every container:
securityContext: {allowPrivilegeEscalation: false, runAsNonRoot: true, capabilities: {drop: [ALL]},
seccompProfile: {type: RuntimeDefault}}
```
**Traps**
- The same ServiceAccount is `--serviceaccount=NS:NAME` in a binding and `system:serviceaccount:NS:NAME` after `--as`.
- `can-i create pods/exec` asks about a Pod named exec; use `can-i create pods --subresource=exec`.
- Admission rejects Pods, not Deployments: the Deployment exists and shows 0 Pods, and the reason is in its ReplicaSet's events.
**Docs**
-
-
-
**Tasks**
- Q28 RBAC for a CI ServiceAccount
- Q29 Pod Security admission blocks a Deployment
- Q49 RBAC, proven by the app's own log _(Simulator Set A (killer.sh-style))_
- Q53 Pods rejected by the Namespace's limits _(Simulator Set A (killer.sh-style))_
### 4.3 Understand requests, limits, quotas
`./exam study 4.3` (or `quotas`)
**Know**
- Requests are what the scheduler reserves on a node. Limits are the ceiling: CPU above it is throttled, memory above it gets the container OOMKilled.
- A ResourceQuota caps a Namespace's totals (requests.cpu, limits.memory, pods, count/...). Once it covers CPU or memory, every new Pod must state them.
- A LimitRange gives containers a default request and limit and enforces a min and max per container, at admission.
**Fast**
```
k create quota compute-quota -n NS --hard=requests.cpu=1,requests.memory=1Gi,limits.cpu=2,limits.memory=2Gi,pods=6
k describe quota -n NS # Used against Hard
k describe limitrange -n NS
apiVersion: v1
kind: LimitRange
metadata: {name: container-defaults}
spec:
limits:
- type: Container
defaultRequest: {cpu: 100m, memory: 128Mi} # the request a container gets
default: {cpu: 250m, memory: 256Mi} # the LIMIT a container gets
max: {cpu: 500m, memory: 512Mi}
```
**Traps**
- `default` is the default limit and `defaultRequest` the default request - they are easy to swap.
- Quotas and LimitRanges only act on Pods created afterwards.
- The rejection is not on the Deployment: `k describe rs` shows `exceeded quota` or `maximum memory usage per Container`.
**Docs**
-
-
**Tasks**
- Q30 Quota with LimitRange defaults
- Q26 OOMKilled
- Q53 Pods rejected by the Namespace's limits _(Simulator Set A (killer.sh-style))_
### 4.4 Understand ConfigMaps
`./exam study 4.4` (or `configmaps`)
**Know**
- Create from literals, files, directories or env files. Each key becomes an environment variable (configMapKeyRef, envFrom) or a file (a configMap volume; `items` picks keys).
- Mounted files follow a changed ConfigMap after a short delay. Environment variables and subPath mounts never do - restart the Pods.
- A Pod that references a missing ConfigMap or key waits in CreateContainerConfigError unless the reference is `optional: true`.
**Fast**
```
k create cm app --from-literal=TIER=backend --from-file=app.properties # --from-file=KEY=path --from-env-file=x.env
env:
- {name: APP_TIER, valueFrom: {configMapKeyRef: {name: app, key: TIER}}}
envFrom: [{configMapRef: {name: app}}]
volumes: [{name: cfg, configMap: {name: app, items: [{key: app.properties, path: app.properties}]}}]
volumeMounts: [{name: cfg, mountPath: /etc/app}] # one file into a full directory: add subPath: app.properties
k set env deploy/web --from=configmap/app; k exec POD -- env | grep TIER
k rollout restart deploy/web # pick up a change
```
**Traps**
- `--from-file` makes one key named after the file; `--from-env-file` makes one key per KEY=VALUE line.
- A volume mounted on a directory hides what the image had there.
- `immutable: true` cannot be edited or undone - delete the ConfigMap and create it again.
**Docs**
-
-
**Tasks**
- Q32 ConfigMap as an env var and as a file
- Q10 Projected volume: config, secret and token
- Q25 Two faults behind a stuck Deployment
- Q51 A ConfigMap-driven nginx that needs a restart _(Simulator Set A (killer.sh-style))_
- Q45 Kustomize: a patch and a generated ConfigMap _(Simulator Set A (killer.sh-style))_
### 4.5 Define resource requirements
`./exam study 4.5` (or `resources`)
**Know**
- Per container: resources.requests and resources.limits for cpu (cores or millicores: 250m) and memory (Mi, Gi).
- QoS class: Guaranteed (every container has requests equal to limits for both CPU and memory), Burstable (some are set), BestEffort (none). BestEffort is evicted first.
- A container with a limit but no request gets a request equal to the limit.
**Fast**
```
k set resources deploy/web -c app --requests=cpu=250m,memory=128Mi --limits=cpu=250m,memory=128Mi
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 250m, memory: 128Mi}
k get pod POD -o jsonpath='{.status.qosClass}'
k describe node NODE | grep -A8 'Allocated resources'
```
**Traps**
- `kubectl run` has no --requests or --limits any more: generate the YAML, or use `kubectl set resources` on a workload.
- Guaranteed needs every container of the Pod, sidecars included.
- Memory is Mi, not m: `128m` is a fraction of a byte.
- A request no node can fit leaves the Pod Pending with `Insufficient cpu` or `Insufficient memory`.
**Docs**
-
-
**Tasks**
- Q31 Resource requirements for Guaranteed QoS
- Q26 OOMKilled
- Q15 DaemonSet on every node
- Q53 Pods rejected by the Namespace's limits _(Simulator Set A (killer.sh-style))_
### 4.6 Create & consume Secrets
`./exam study 4.6` (or `secrets`)
**Know**
- A Secret's `data` is base64 - an encoding, not encryption. `stringData` takes plain text when you write one.
- Types: Opaque (generic), kubernetes.io/tls, kubernetes.io/dockerconfigjson, kubernetes.io/service-account-token.
- Consumed like a ConfigMap: secretKeyRef, envFrom secretRef, or a secret volume (`secretName`, `defaultMode`, `items`).
**Fast**
```
k create secret generic api --from-literal=username=svc --from-literal=password='S3cr3t!'
k create secret tls web-tls --cert=tls.crt --key=tls.key
k create secret docker-registry regcred --docker-server=REG --docker-username=U --docker-password=P
k get secret api -o jsonpath='{.data.password}' | base64 -d
env: [{name: API_USER, valueFrom: {secretKeyRef: {name: api, key: username}}}]
volumes: [{name: creds, secret: {secretName: api, defaultMode: 0400}}]
volumeMounts: [{name: creds, mountPath: /etc/creds, readOnly: true}]
```
**Traps**
- A secret volume names it with `secretName`; a configMap volume uses `name`.
- Encoding by hand: `echo -n value | base64`, or the newline becomes part of the secret.
- defaultMode is octal in YAML (0400) and shows as decimal in JSON (256).
**Docs**
-
-
**Tasks**
- Q33 Secrets: decode one, consume another
- Q10 Projected volume: config, secret and token
- Q37 Ingress with two paths and TLS
- Q50 An immutable Secret, one file into a busy directory _(Simulator Set A (killer.sh-style))_
### 4.7 Understand ServiceAccounts
`./exam study 4.7` (or `serviceaccounts`)
**Know**
- Every Pod runs as a ServiceAccount (`default` if you name none); its short-lived token is mounted at /var/run/secrets/kubernetes.io/serviceaccount.
- Choose it with spec.serviceAccountName. Stop the mount with `automountServiceAccountToken: false` on the ServiceAccount or on the Pod (the Pod wins).
- Tokens: `kubectl create token` for a short-lived one; a Secret of type kubernetes.io/service-account-token annotated with the account's name for a long-lived one.
- A ServiceAccount can do nothing until RBAC binds a Role to it.
**Fast**
```
k create sa ci -n NS
k set serviceaccount deploy/web ci -n NS # = spec.template.spec.serviceAccountName
k patch sa ci -n NS -p '{"automountServiceAccountToken": false}'
k create token ci -n NS --duration=1h
k auth can-i list pods -n NS --as=system:serviceaccount:NS:ci
apiVersion: v1
kind: Secret
metadata: {name: ci-token, annotations: {kubernetes.io/service-account.name: ci}}
type: kubernetes.io/service-account-token
```
**Traps**
- No token Secret is made for a new ServiceAccount (since 1.24) - create it yourself when a task wants one.
- serviceAccountName cannot be changed on a running Pod; on a Deployment it starts a rollout.
- A Pod naming a ServiceAccount that does not exist is never created - see the ReplicaSet's events.
**Docs**
-
-
**Tasks**
- Q34 ServiceAccount without an automounted token
- Q28 RBAC for a CI ServiceAccount
- Q10 Projected volume: config, secret and token
- Q49 RBAC, proven by the app's own log _(Simulator Set A (killer.sh-style))_
### 4.8 Understand Application Security (SecurityContexts, Capabilities, etc.)
`./exam study 4.8` (or `security`)
**Know**
- Pod level (spec.securityContext): runAsUser, runAsGroup, runAsNonRoot, fsGroup, supplementalGroups, seccompProfile - for every container.
- Container level adds allowPrivilegeEscalation, privileged, readOnlyRootFilesystem and capabilities, and overrides the Pod level.
- Capabilities are written without the CAP_ prefix: NET_BIND_SERVICE, NET_ADMIN, SYS_TIME. Drop ALL, then add back what the app needs.
**Fast**
```
spec:
securityContext: {runAsUser: 1000, runAsGroup: 1000, runAsNonRoot: true, fsGroup: 2000}
containers:
- name: app
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: [ALL], add: [NET_BIND_SERVICE]}
k exec POD -- id
k explain pod.spec.securityContext; k explain pod.spec.containers.securityContext
```
**Traps**
- capabilities exist only on a container, fsGroup only on the Pod.
- runAsNonRoot with an image that runs as root and no runAsUser: CreateContainerConfigError.
- readOnlyRootFilesystem breaks an app that writes to /tmp or /var/cache - mount an emptyDir there.
**Docs**
-
**Tasks**
- Q9 SecurityContext and capabilities
- Q29 Pod Security admission blocks a Deployment
- Q44 From a single Pod to a hardened Deployment _(Simulator Set A (killer.sh-style))_
- Q52 Two users, one volume: fsGroup _(Simulator Set A (killer.sh-style))_
## Services and Networking (20%)
### 5.1 Demonstrate basic understanding of NetworkPolicies
`./exam study 5.1` (or `netpol`)
**Know**
- Pods accept and send everything until a NetworkPolicy selects them. From then on, for the policyTypes it lists, only what some policy allows gets through.
- Policies only allow, and they add up: there is no deny rule and no ordering.
- Peers: podSelector (same Namespace), namespaceSelector, ipBlock. Two selectors in one list item mean AND; two list items mean OR.
- Default deny: `podSelector: {}` with a policyType and no rules. Locking down egress also needs DNS allowed (UDP and TCP 53).
**Fast**
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: db-allow, namespace: NS}
spec:
podSelector: {matchLabels: {app: db}}
policyTypes: [Ingress]
ingress:
- from:
- podSelector: {matchLabels: {app: api}} # OR
- namespaceSelector: {matchLabels: {purpose: monitoring}}
ports: [{protocol: TCP, port: 5432}]
# egress that keeps DNS: egress: [{ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]}, ...]
k describe netpol db-allow -n NS
k exec CLIENT -- wget -qO- -T2 http://db:5432 # test it with real traffic
```
**Traps**
- There is no `kubectl create networkpolicy` - copy the skeleton from the docs.
- `- podSelector: ...` followed by `namespaceSelector:` in the same item is AND; a second `- namespaceSelector:` is OR.
- A missing or empty `ingress` denies everything; `ingress: [{}]` allows everything.
- The policy lives in the Namespace of the Pods it selects, and a Namespace is matched by its labels (kubernetes.io/metadata.name=NAME always exists).
**Docs**
-
**Tasks**
- Q12 Restrict a namespace with NetworkPolicy
- Q35 Egress lockdown that keeps DNS working
- Q55 Two sources, one port _(Simulator Set A (killer.sh-style))_
### 5.2 Provide and troubleshoot access to applications via services
`./exam study 5.2` (or `services`)
**Know**
- A Service selects Pods by label and forwards `port` to `targetPort` (a number, or the name of a container port).
- Types: ClusterIP (inside the cluster), NodePort (30000-32767 on every node), LoadBalancer, ExternalName; headless (`clusterIP: None`) gives each Pod its own DNS record.
- DNS names: SVC, SVC.NS, SVC.NS.svc.cluster.local.
- Troubleshoot in this order: are there endpoints, does the selector match the Pod labels, are the Pods Ready, is targetPort the port the container listens on.
**Fast**
```
k expose deploy web --port=8080 --target-port=80 # --name=web-svc --type=NodePort
k create svc nodeport web-ext --tcp=80:80 --node-port=30080
k get svc,endpointslices -o wide; k describe svc web # Endpoints: = nothing selected
k get pods -l app=web --show-labels
k run tmp --rm -it --restart=Never --image=busybox:1.36 -- wget -qO- -T3 http://web:8080
k port-forward svc/web 8080:80
```
**Traps**
- `kubectl expose` copies the workload's selector; `kubectl create svc` selects app=, which may match nothing.
- `expose` cannot choose the nodePort - use `create svc nodeport --node-port` or edit the YAML.
- A Service without endpoints still resolves in DNS; the connection is what fails.
**Docs**
-
-
-
**Tasks**
- Q11 Expose a Deployment two ways
- Q36 A Service that routes nowhere
- Q54 Expose a Pod and prove it from inside _(Simulator Set A (killer.sh-style))_
- Q56 Stable DNS names for a StatefulSet _(Simulator Set A (killer.sh-style))_
### 5.3 Use Ingress rules to expose applications
`./exam study 5.3` (or `ingress`)
**Know**
- An Ingress routes HTTP(S) by host and path to Services of its own Namespace; an ingress controller, chosen with ingressClassName, does the work.
- Each path has a pathType (Prefix, Exact or ImplementationSpecific) and a backend: service.name plus service.port.number or .name.
- TLS: spec.tls lists hosts and the kubernetes.io/tls Secret that holds their certificate.
**Fast**
```
k create ingress shop --class=nginx \
--rule="shop.example.com/api*=api:8080,tls=shop-tls" --rule="shop.example.com/*=web:80,tls=shop-tls"
k create secret tls shop-tls --cert=tls.crt --key=tls.key
k get ingress,ingressclass; k describe ingress shop # a missing backend shows as
k explain ingress.spec.rules.http.paths
```
**Traps**
- In --rule a trailing * means pathType Prefix; without it the path is Exact.
- The backend port is the Service's port, not the container's.
- No ingress controller runs in this simulator, so Ingress tasks are graded on the object - on the exam, curl it.
**Docs**
-
**Tasks**
- Q37 Ingress with two paths and TLS
- Q11 Expose a Deployment two ways
- Q57 An Ingress pointing at the wrong backends _(Simulator Set A (killer.sh-style))_