44 KiB
CKAD study guide
The 24 objectives of the Certified Kubernetes Application Developer curriculum, in its own words and order: what to know, the fastest commands, the traps, the documentation pages the exam lets you open, and the tasks in this simulator that practise each one.
Generated by ./exam --track ckad notes all --md from tracks/ckad/curriculum.py - edit that file, not this one.
./exam objectives # the path and your progress
./exam study 3.2 # one objective: these notes, then its tasks, no clock, solutions open
./exam drill 3.2 # the same tasks against the clock
Application Design and Build (20%)
1.1 Define, build and modify container images
./exam study 1.1 (or images)
Know
- Read and change a Dockerfile: FROM, COPY, RUN, ENV, ARG (build time only), WORKDIR, USER, LABEL, EXPOSE, ENTRYPOINT, CMD.
- ENTRYPOINT is the program, CMD its default arguments. In a Pod,
commandreplaces ENTRYPOINT andargsreplaces CMD. - Build, tag, list, inspect, run, save and remove images. The exam terminal has docker or podman - the verbs are the same.
- A tag is only a pointer: one image can carry several tags, and rebuilding under an old tag moves it.
Fast
podman build -t localhost/app:v1 . # -f path/Dockerfile --build-arg KEY=val
podman tag localhost/app:v1 localhost/app:latest
podman images | grep app
podman inspect localhost/app:v1 --format '{{.Config.User}} {{.Config.Entrypoint}} {{.Config.Cmd}} {{.Config.Env}}'
podman run --rm localhost/app:v1 [args replacing CMD] # -d --name x keeps it running; podman logs x
podman save --format oci-archive -o app.tar localhost/app:v1 # docker save -o app.tar ...
podman rmi localhost/app:v0
Traps
- An ARG is gone at run time unless you copy it into an ENV (
ARG VthenENV APP_VERSION=$V). podman savewrites a docker-archive unless you pass--format oci-archive.- Build before you edit when a task wants the old version kept under its own tag.
- imagePullPolicy defaults to Always for
:latest(or no tag) and IfNotPresent for any other tag.
Docs
- https://kubernetes.io/docs/concepts/containers/images/
- https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/
Tasks
- Q13 Build, change and export a container image
- Q60 Entrypoint, default argument and a non-root user
- Q41 Build with a build argument, run it detached (Simulator Set A (killer.sh-style))
1.2 Choose and use the right workload resource (Deployment, DaemonSet, CronJob, etc.)
./exam study 1.2 (or workloads)
Know
- Choose by behaviour: Deployment (interchangeable replicas, rolling updates), StatefulSet (stable name, DNS and storage per replica), DaemonSet (one Pod per node), Job (run to completion), CronJob (Jobs on a schedule), bare Pod (one-off).
- Job: completions, parallelism, backoffLimit, activeDeadlineSeconds, ttlSecondsAfterFinished; restartPolicy must be Never or OnFailure.
- CronJob: schedule, startingDeadlineSeconds, concurrencyPolicy (Allow, Forbid, Replace), suspend, successfulJobsHistoryLimit, failedJobsHistoryLimit.
- DaemonSet: no replicas; a toleration for node-role.kubernetes.io/control-plane:NoSchedule puts it on control-plane nodes too.
Fast
k create deploy web --image=nginx:1.27 --replicas=3 $do > deploy.yaml
k create job render --image=busybox:1.36 $do -- sh -c 'echo done' > job.yaml
k create cronjob digest --image=busybox:1.36 --schedule='*/5 * * * *' $do -- sh -c 'echo digest' > cj.yaml
k create job digest-manual --from=cronjob/digest # run a CronJob once, now
k run tmp --image=busybox:1.36 --restart=Never --rm -it -- sh
# DaemonSet has no generator: create a Deployment with $do, change kind, delete replicas/strategy/status
k explain cronjob.spec.jobTemplate.spec --recursive | less
Traps
- Three nested specs in a CronJob: spec (schedule, startingDeadlineSeconds) > jobTemplate.spec (completions, activeDeadlineSeconds) > template.spec (containers, restartPolicy).
- A Job's Pod template is immutable - delete the Job and create it again instead of editing it.
- A StatefulSet needs
serviceNamenaming a headless Service (clusterIP: None) before its Pods get their own DNS names.
Docs
- https://kubernetes.io/docs/concepts/workloads/controllers/job/
- https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
- https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
Tasks
- Q3 CronJob with deadlines
- Q14 Parallel batch Job
- Q15 DaemonSet on every node
- Q39 A suspended CronJob and a manual run (Simulator Set A (killer.sh-style))
- Q38 A Pod and a status one-liner (Simulator Set A (killer.sh-style))
- Q44 From a single Pod to a hardened Deployment (Simulator Set A (killer.sh-style))
- Q56 Stable DNS names for a StatefulSet (Simulator Set A (killer.sh-style))
1.3 Understand multi-container Pod design patterns (e.g. sidecar, init and others)
./exam study 1.3 (or multicontainer)
Know
- Containers of a Pod share its network (they reach each other on localhost) and any volume both mount - that is all a sidecar, adapter or ambassador is.
- Init containers run one at a time, in order, each to completion, before any app container starts.
- A native sidecar is an init container with
restartPolicy: Always: it starts before the app containers, keeps running beside them, and does not keep a Job from completing. - Address one container with
-c:k logs POD -c NAME,k exec POD -c NAME -- .... StatusInit:0/1means an init container is still running.
Fast
k run app --image=nginx:1.27 $do > pod.yaml # then add the second container by hand
spec:
initContainers:
- {name: wait, image: busybox:1.36, command: [sh, -c, "until wget -qO- -T2 http://backend; do sleep 2; done"]}
- {name: shipper, image: busybox:1.36, restartPolicy: Always, command: [sh, -c, "tail -F /logs/app.log"],
volumeMounts: [{name: logs, mountPath: /logs}]} # native sidecar
containers:
- {name: app, image: nginx:1.27, volumeMounts: [{name: logs, mountPath: /var/log/app}]}
volumes: [{name: logs, emptyDir: {}}]
k logs app -c shipper; k logs app --all-containers --prefix
Traps
kubectl runcreates one container only - generate the Pod and add the others in the YAML.- A Service with no endpoints still resolves in DNS, so an init container waiting on
nslookupis not waiting for the app. Wait on a real request. - A classic sidecar that never exits keeps a Job running for ever; make it a native sidecar.
Docs
- https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
- https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/
Tasks
- Q1 Adapter sidecar
- Q2 Init container waits for a dependency
- Q16 Native sidecar lets a Job finish
- Q40 An init container writes the page (Simulator Set A (killer.sh-style))
1.4 Utilize persistent and ephemeral volumes
./exam study 1.4 (or volumes)
Know
- Ephemeral: emptyDir (lives as long as the Pod;
medium: Memoryis tmpfs,sizeLimitcaps it), and the configMap, secret, downwardAPI and projected volumes. - Persistent: a PersistentVolumeClaim asks for storage (accessModes, resources.requests.storage, storageClassName); a StorageClass provisions the PersistentVolume, or a pre-made one binds.
- Access modes: ReadWriteOnce, ReadOnlyMany, ReadWriteMany, ReadWriteOncePod.
- volumeMounts: mountPath, readOnly, and subPath to mount one file or directory without hiding the rest of the target directory.
Fast
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: data}
spec:
accessModes: [ReadWriteOnce]
storageClassName: standard
resources: {requests: {storage: 100Mi}}
--- # in the Pod spec:
volumes:
- {name: data, persistentVolumeClaim: {claimName: data}}
- {name: scratch, emptyDir: {medium: Memory, sizeLimit: 32Mi}}
containers:
- volumeMounts: [{name: data, mountPath: /data}, {name: scratch, mountPath: /scratch}]
k get pvc,pv; k describe pvc data # Pending? the Events say why
Traps
- There is no
kubectl create pvc- copy the claim from the docs. - With a WaitForFirstConsumer StorageClass (
standardhere) a claim stays Pending until a Pod uses it. That is not an error. - emptyDir survives a container restart but not a new Pod. To prove data persists, delete the Pod and read the file again.
- A non-root container cannot write to a fresh volume unless the Pod sets
securityContext.fsGroup.
Docs
- https://kubernetes.io/docs/concepts/storage/persistent-volumes/
- https://kubernetes.io/docs/concepts/storage/volumes/
- https://kubernetes.io/docs/tutorials/configuration/configure-persistent-volume-storage/
Tasks
- Q17 Persistent and ephemeral volumes
- Q10 Projected volume: config, secret and token
- Q52 Two users, one volume: fsGroup (Simulator Set A (killer.sh-style))
- Q40 An init container writes the page (Simulator Set A (killer.sh-style))
- Q50 An immutable Secret, one file into a busy directory (Simulator Set A (killer.sh-style))
Application Deployment (20%)
2.1 Use Kubernetes primitives to implement common deployment strategies (e.g. blue/green or canary)
./exam study 2.1 (or strategies)
Know
- Blue/green: two Deployments, and the Service selector decides which one gets all the traffic. Cut over by changing the selector; roll back by changing it back.
- Canary: two Deployments whose Pods share the label the Service selects. Traffic splits by the number of ready Pods: 1 canary beside 4 stable is 20%.
- A Deployment's own strategy is RollingUpdate (default) or Recreate (every old Pod stops first - downtime).
Fast
k get svc web -o wide; k get pods --show-labels
k set selector svc web 'app=web,version=green' # blue/green cut-over
k scale deploy web-stable --replicas=4 # canary ratio: 4 stable + 1 canary = 20%
k get endpointslices -l kubernetes.io/service-name=web -o wide
k get pods -l app=web -L version,track
Traps
- Wait until the new Deployment is fully available before you switch the selector.
kubectl set selectorreplaces the whole selector - name every label you want in it.- A canary only shares traffic if the Service selector leaves out the label that tells the two versions apart.
Docs
- https://kubernetes.io/docs/concepts/workloads/management/
- https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
Tasks
- Q5 Blue/green cutover
- Q18 Canary by replica ratio
2.2 Understand Deployments and how to perform rolling updates
./exam study 2.2 (or rollouts)
Know
- Any change to
spec.templatestarts a rollout: a new ReplicaSet scales up while the old one scales down, within maxSurge and maxUnavailable (25% each by default). - minReadySeconds (how long a Pod must be Ready to count as available), revisionHistoryLimit (old ReplicaSets kept), progressDeadlineSeconds.
kubectl rollout: status, history, undo, pause, resume, restart.
Fast
k set image deploy/web nginx=nginx:1.27 # CONTAINER=IMAGE
k rollout status deploy/web
k rollout history deploy/web # --revision=3 shows that revision's template
k rollout undo deploy/web # --to-revision=2
k rollout restart deploy/web # same spec, new Pods
k annotate deploy/web kubernetes.io/change-cause='pick 1.27'
k patch deploy web -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
Traps
--recordis deprecated: write the kubernetes.io/change-cause annotation yourself.- An undo makes a new revision out of the old one, so the active revision number is not the one you rolled back to.
- A rollout stuck on a bad image does not fail by itself -
rollout statusjust waits, and the old Pods keep serving. - maxSurge and maxUnavailable cannot both be 0.
Docs
Tasks
- Q4 Roll forward, then back
- Q19 Rolling update without losing capacity
- Q42 Back to the last good revision (Simulator Set A (killer.sh-style))
- Q44 From a single Pod to a hardened Deployment (Simulator Set A (killer.sh-style))
2.3 Use the Helm package manager to deploy existing packages
./exam study 2.3 (or helm)
Know
- A chart is the package, a release is one installation of it in a Namespace, and every install, upgrade or rollback adds a revision.
- Values: the chart's defaults (
helm show values), overridden with-f file.yamland--set key=value;helm get valuesshows what a release was given. - Life cycle: repo add/update, search, install, upgrade, rollback, uninstall, list, history, status.
Fast
helm repo add NAME URL && helm repo update; helm search repo nginx --versions
helm show values CHART > values.yaml
helm install REL CHART -n NS --create-namespace -f values.yaml --set replicaCount=2 # --version X
helm upgrade REL CHART -n NS --reuse-values --set replicaCount=4
helm list -n NS # -A for every Namespace; --failed, --pending filter by status
helm history REL -n NS; helm rollback REL 1 -n NS; helm uninstall REL -n NS
helm get values REL -n NS # --all adds the defaults; helm get manifest REL -n NS
Traps
- Releases live in a Namespace: without
-nHelm looks indefaultand finds nothing. - An upgrade without
--reuse-values(or the same-f/--setagain) puts every value back to the chart's default. - A rollback is a new revision, not a step back in the numbering.
- Helm 3's
helm listleaves out pending releases unless you add-a; Helm 4 (on this machine) lists every status and has no-a.--pendingand--failedwork in both.
Docs
Tasks
- Q6 Helm release lifecycle
- Q21 Helm: values file and an upgrade that keeps values
- Q43 Helm housekeeping (Simulator Set A (killer.sh-style))
2.4 Kustomize
./exam study 2.4 (or kustomize)
Know
- A kustomization.yaml lists
resources(files or other kustomizations) and transforms them: namespace, namePrefix, nameSuffix, labels, images, replicas, patches. - Generators build ConfigMaps and Secrets from files, env files or literals, add a content hash to the name and rewrite every reference to it.
- Base and overlays: an overlay lists
../../baseunder resources and changes only what differs. kubectl kustomize DIRprints the result;kubectl apply -k DIRapplies it.
Fast
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [../../base] # or files: [deployment.yaml, service.yaml]
namespace: prod
namePrefix: prod-
labels: [{pairs: {env: prod}, includeSelectors: false}]
images: [{name: nginx, newTag: "1.27"}]
replicas: [{name: web, count: 3}]
patches: [{path: patch.yaml}] # a partial Deployment with the same name
configMapGenerator: [{name: settings, envs: [settings.env]}]
secretGenerator: [{name: token, literals: [token=abc]}]
generatorOptions: {disableNameSuffixHash: true}
k kustomize DIR | less; k apply -k DIR; k diff -k DIR
Traps
- Always read the render before you apply it.
images,replicasand patches name things as the base does, before any prefix or suffix.- Refer to a generated ConfigMap by its plain name - Kustomize appends the hash everywhere it is used.
namespace:does not create the Namespace.
Docs
Tasks
- Q20 Kustomize overlay
- Q61 A kustomization that does not build
- Q45 Kustomize: a patch and a generated ConfigMap (Simulator Set A (killer.sh-style))
Application Observability and Maintenance (15%)
3.1 Understand API deprecations
./exam study 3.1 (or deprecations)
Know
- Every object is a kind in an API group at a version (apps/v1 Deployment). Versions are deprecated, keep being served for some releases, then are removed.
- What this cluster serves:
kubectl api-resources(the APIVERSION column),kubectl api-versions,kubectl explain KIND. - A removed version fails with
no matches for kind "X" in version "g/v"; a deprecated one still works and printsWarning: ... is deprecated. - Migrating is the new apiVersion plus whatever changed shape: an Ingress backend becomes
service: {name, port: {number}}with a required pathType, and an apps/v1 Deployment needsspec.selector.
Fast
k api-resources | grep -i -E 'cronjob|ingress' # NAME SHORTNAMES APIVERSION NAMESPACED KIND
k api-versions | grep batch
k explain hpa | head -4 # GROUP, KIND, VERSION
k apply -f old.yaml --dry-run=server # the error or warning, without changing anything
# removed, and what replaced them:
# extensions/v1beta1, networking.k8s.io/v1beta1 Ingress -> networking.k8s.io/v1 (1.22)
# extensions/v1beta1, apps/v1beta1, apps/v1beta2 Deployment -> apps/v1 (1.16)
# batch/v1beta1 CronJob -> batch/v1 (1.25)
# policy/v1beta1 PodDisruptionBudget -> policy/v1 (1.25)
# autoscaling/v2beta1, v2beta2 HorizontalPodAutoscaler -> autoscaling/v2 (1.25, 1.26)
Traps
- The
kubectl convertplugin is usually not installed - know the Ingress and Deployment rewrites by hand. kubectl getshows an object in the API's preferred version whatever version it was written with.- A warning is not an error: the object was created. Read stderr.
Docs
- https://kubernetes.io/docs/reference/using-api/deprecation-guide/
- https://kubernetes.io/docs/reference/deprecation-policy/
Tasks
- Q22 Manifests written for removed APIs
- Q58 Removed, and deprecated but still served
3.2 Implement probes and health checks
./exam study 3.2 (or probes)
Know
- Liveness: failing restarts the container. Readiness: failing takes the Pod out of Service endpoints, no restart. Startup: holds the other two off until it has passed once.
- Handlers: httpGet (path, port), tcpSocket (port), exec (command), grpc (port).
- Timing: initialDelaySeconds (0), periodSeconds (10), timeoutSeconds (1), failureThreshold (3), successThreshold (1). A startup probe allows failureThreshold x periodSeconds to come up.
Fast
containers:
- name: app
startupProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5, failureThreshold: 12} # 60 s to start
readinessProbe: {httpGet: {path: /ready, port: 8080}, periodSeconds: 5, failureThreshold: 1}
livenessProbe: {tcpSocket: {port: 80}, initialDelaySeconds: 15, periodSeconds: 10}
# exec: readinessProbe: {exec: {command: [cat, /tmp/ready]}, periodSeconds: 5}
k describe pod POD | grep -E 'Liveness|Readiness|Startup'
k get events --field-selector reason=Unhealthy
k explain pod.spec.containers.livenessProbe
Traps
- No kubectl flag adds a probe: generate the YAML and edit it, or
kubectl edit. - Probes belong to a container, not to the Pod spec.
- Restarts climbing with
UnhealthyandKillingevents but nothing wrong in the logs: the liveness probe is wrong or too impatient, not the app. - Running, 0/1 Ready and no restarts: the readiness probe is failing.
Docs
- https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
- https://kubernetes.io/docs/concepts/workloads/pods/probes/
Tasks
- Q7 Three kinds of probe
- Q62 Health checks that check the wrong thing
- Q46 Probes that match the app's warm-up (Simulator Set A (killer.sh-style))
3.3 Use built-in CLI tools to monitor Kubernetes applications
./exam study 3.3 (or monitoring)
Know
kubectl getoutput: -o wide, yaml, json, name, jsonpath=, custom-columns=; narrowed with -l (labels), --field-selector, -n / -A; ordered with --sort-by.kubectl top pod|nodeshows live CPU and memory from metrics-server;--containerssplits a Pod up.kubectl describefor the story of one object,kubectl get eventsfor the Namespace.
Fast
k top pod -A --sort-by=memory; k top pod --containers; k top node
k get pods -o wide --sort-by=.spec.nodeName
k get pods -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,IMAGE:.spec.containers[0].image
k get pod POD -o jsonpath='{.status.phase}'
k get pods -l tier=web --field-selector=status.phase!=Running -o name
k get pods -A --sort-by='.status.containerStatuses[0].restartCount'
k get events --sort-by=.lastTimestamp; k events --types=Warning --for pod/POD
Traps
kubectl get --sort-bysorts ascending (the largest is last);kubectl top --sort-bysorts descending (the largest is first).kubectl topshows usage, not requests or limits, and needs about a minute after a Pod starts.- Quote jsonpath for the shell: '{.items[*].metadata.name}'.
Docs
- https://kubernetes.io/docs/reference/kubectl/quick-reference/
- https://kubernetes.io/docs/reference/kubectl/jsonpath/
Tasks
- Q23 Find the heaviest Pods with kubectl top
- Q63 An inventory with kubectl alone
- Q48 Restart and event forensics (Simulator Set A (killer.sh-style))
- Q38 A Pod and a status one-liner (Simulator Set A (killer.sh-style))
3.4 Utilize container logs
./exam study 3.4 (or logs)
Know
kubectl logsshows what a container wrote to stdout and stderr. The kubelet keeps it on the node, and it goes when the Pod is deleted.- Pick the container with -c (init containers too), or take --all-containers; --previous is the run before the last restart.
- Narrow with --tail=N, --since=5m, --timestamps; follow with -f; by label with -l.
Fast
k logs POD -c CONTAINER # --all-containers --prefix
k logs POD --previous # the crashed run
k logs POD --tail=20 --since=10m --timestamps
k logs -l app=web --prefix --tail=5 # every Pod with the label
k logs deploy/web; k logs job/render # one Pod of it
k logs POD | grep ERROR > errors.log
Traps
- With -l, only the last 10 lines of each Pod are shown unless you set --tail (--tail=-1 for everything).
kubectl logs deploy/xreads one Pod, not all replicas.- --previous fails when the container has never restarted.
Docs
- https://kubernetes.io/docs/concepts/cluster-administration/logging/
- https://kubernetes.io/docs/reference/kubectl/quick-reference/
Tasks
- Q24 Logs of a container that keeps crashing
- Q64 One replica is logging errors
- Q49 RBAC, proven by the app's own log (Simulator Set A (killer.sh-style))
- Q54 Expose a Pod and prove it from inside (Simulator Set A (killer.sh-style))
- Q39 A suspended CronJob and a manual run (Simulator Set A (killer.sh-style))
3.5 Debugging in Kubernetes
./exam study 3.5 (or debugging)
Know
- Always the same order: get (status), describe (Events at the bottom), logs (--previous), then exec or debug.
- Pending: nothing can schedule it (resources, taints, nodeSelector, an unbound claim). ImagePullBackOff / ErrImagePull: wrong image or tag. CreateContainerConfigError: a ConfigMap, Secret or key it references is missing.
- CrashLoopBackOff: the process exits - read the previous log and lastState.terminated (reason, exitCode). OOMKilled (137): the memory limit is too low.
- A Deployment with no Pods at all: describe its ReplicaSet - a quota, a LimitRange or Pod Security admission is rejecting them.
Fast
k get pods -o wide; k describe pod POD | tail -20
k get pod POD -o jsonpath='{.status.containerStatuses[0].state.waiting.reason}'
k get pod POD -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' # .exitCode
k logs POD --previous; k exec -it POD -- sh
k debug -it POD --image=busybox:1.36 --target=CONTAINER # ephemeral container, sees its processes
k run tmp --rm -it --restart=Never --image=busybox:1.36 -- wget -qO- -T3 http://SVC:PORT
k describe rs -l app=web --show-events | tail; k get events --sort-by=.lastTimestamp
k get pod POD -o yaml > pod.yaml && vi pod.yaml && k replace --force -f pod.yaml
Traps
- Most of a running Pod is immutable (the image is not): fix a bare Pod with
replace --force, fix a Deployment's Pods in the Deployment. - Fix what the task names and nothing else - graders check that the ConfigMap, Service or Namespace you were told to leave is unchanged.
- After fixing a Deployment wait for the rollout; old Pods linger in Terminating.
kubectl describewith a selector that matches several objects leaves their Events out - add--show-events, or describe one by name.
Docs
- https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/
- https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/
- https://kubernetes.io/docs/tasks/debug/debug-application/determine-reason-pod-failure/
Tasks
- Q8 Diagnose and fix a broken Pod
- Q25 Two faults behind a stuck Deployment
- Q26 OOMKilled
- Q36 A Service that routes nowhere
- Q47 Debugging a container with no shell (Simulator Set A (killer.sh-style))
- Q48 Restart and event forensics (Simulator Set A (killer.sh-style))
Application Environment, Configuration and Security (25%)
4.1 Discover and use resources that extend Kubernetes (CRD, Operators)
./exam study 4.1 (or crds)
Know
- A CustomResourceDefinition adds a kind to the API; its objects (custom resources) work with every kubectl verb. An operator is a CRD plus a controller acting on those objects.
- Discover:
k get crd,k api-resources --api-group=GROUP(kind, short names, namespaced or not),k explain KIND.spec. - The CRD's schema validates custom resources: required fields, enum values, minimum and maximum. The controller reports back in
.status.
Fast
k get crd | grep -i backup # the CRD's name is PLURAL.GROUP
k api-resources --api-group=data.example.com
k explain backup.spec # --recursive for the whole tree
k get backups -A; k get bkp NAME -o yaml; k describe backup NAME
k get crd backups.data.example.com -o yaml | less # served versions, schema, enum, maximum
k apply -f cr.yaml --dry-run=server # every validation error, nothing created
Traps
- A custom resource's apiVersion is the CRD's GROUP/VERSION, never apiextensions.k8s.io/v1.
- Field names come from the CRD's schema - check with
kubectl explaininstead of guessing. - Deleting a CRD deletes every object of that kind.
Docs
- https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
- https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/
- https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
Tasks
- Q27 Discover and use a custom resource
- Q59 A custom resource the API server rejects
- Q58 Removed, and deprecated but still served
4.2 Understand authentication, authorization and admission control
./exam study 4.2 (or access)
Know
- Every request passes authentication (who is it: a user certificate or token, a ServiceAccount token), authorization (may they: RBAC), then admission control (change or reject it: LimitRanger, ResourceQuota, PodSecurity).
- RBAC: a Role or ClusterRole lists verbs on resources (and subresources such as pods/log, deployments/scale); a RoleBinding or ClusterRoleBinding grants it to users, groups or ServiceAccounts.
- Pod Security admission: Namespace labels pod-security.kubernetes.io/enforce|warn|audit = privileged | baseline | restricted.
Forbidden: User ... cannot listis authorization.violates PodSecurity,exceeded quotaandmaximum memory usageare admission.
Fast
k create role ci --verb=get,list,watch --resource=pods,pods/log -n NS
k create role ci --verb=get,update,patch --resource=deployments.apps,deployments.apps/scale -n NS
k create rolebinding ci --role=ci --serviceaccount=NS:SA -n NS # --user=jane --clusterrole=view
k auth can-i list pods -n NS --as=system:serviceaccount:NS:SA
k auth can-i --list -n NS --as=system:serviceaccount:NS:SA
k get ns NS --show-labels
# what `restricted` wants from every container:
securityContext: {allowPrivilegeEscalation: false, runAsNonRoot: true, capabilities: {drop: [ALL]},
seccompProfile: {type: RuntimeDefault}}
Traps
- The same ServiceAccount is
--serviceaccount=NS:NAMEin a binding andsystem:serviceaccount:NS:NAMEafter--as. can-i create pods/execasks about a Pod named exec; usecan-i create pods --subresource=exec.- Admission rejects Pods, not Deployments: the Deployment exists and shows 0 Pods, and the reason is in its ReplicaSet's events.
Docs
- https://kubernetes.io/docs/reference/access-authn-authz/rbac/
- https://kubernetes.io/docs/concepts/security/pod-security-admission/
- https://kubernetes.io/docs/concepts/security/pod-security-standards/
Tasks
- Q28 RBAC for a CI ServiceAccount
- Q29 Pod Security admission blocks a Deployment
- Q49 RBAC, proven by the app's own log (Simulator Set A (killer.sh-style))
- Q53 Pods rejected by the Namespace's limits (Simulator Set A (killer.sh-style))
4.3 Understand requests, limits, quotas
./exam study 4.3 (or quotas)
Know
- Requests are what the scheduler reserves on a node. Limits are the ceiling: CPU above it is throttled, memory above it gets the container OOMKilled.
- A ResourceQuota caps a Namespace's totals (requests.cpu, limits.memory, pods, count/...). Once it covers CPU or memory, every new Pod must state them.
- A LimitRange gives containers a default request and limit and enforces a min and max per container, at admission.
Fast
k create quota compute-quota -n NS --hard=requests.cpu=1,requests.memory=1Gi,limits.cpu=2,limits.memory=2Gi,pods=6
k describe quota -n NS # Used against Hard
k describe limitrange -n NS
apiVersion: v1
kind: LimitRange
metadata: {name: container-defaults}
spec:
limits:
- type: Container
defaultRequest: {cpu: 100m, memory: 128Mi} # the request a container gets
default: {cpu: 250m, memory: 256Mi} # the LIMIT a container gets
max: {cpu: 500m, memory: 512Mi}
Traps
defaultis the default limit anddefaultRequestthe default request - they are easy to swap.- Quotas and LimitRanges only act on Pods created afterwards.
- The rejection is not on the Deployment:
k describe rsshowsexceeded quotaormaximum memory usage per Container.
Docs
- https://kubernetes.io/docs/concepts/policy/resource-quotas/
- https://kubernetes.io/docs/concepts/policy/limit-range/
Tasks
- Q30 Quota with LimitRange defaults
- Q26 OOMKilled
- Q53 Pods rejected by the Namespace's limits (Simulator Set A (killer.sh-style))
4.4 Understand ConfigMaps
./exam study 4.4 (or configmaps)
Know
- Create from literals, files, directories or env files. Each key becomes an environment variable (configMapKeyRef, envFrom) or a file (a configMap volume;
itemspicks keys). - Mounted files follow a changed ConfigMap after a short delay. Environment variables and subPath mounts never do - restart the Pods.
- A Pod that references a missing ConfigMap or key waits in CreateContainerConfigError unless the reference is
optional: true.
Fast
k create cm app --from-literal=TIER=backend --from-file=app.properties # --from-file=KEY=path --from-env-file=x.env
env:
- {name: APP_TIER, valueFrom: {configMapKeyRef: {name: app, key: TIER}}}
envFrom: [{configMapRef: {name: app}}]
volumes: [{name: cfg, configMap: {name: app, items: [{key: app.properties, path: app.properties}]}}]
volumeMounts: [{name: cfg, mountPath: /etc/app}] # one file into a full directory: add subPath: app.properties
k set env deploy/web --from=configmap/app; k exec POD -- env | grep TIER
k rollout restart deploy/web # pick up a change
Traps
--from-filemakes one key named after the file;--from-env-filemakes one key per KEY=VALUE line.- A volume mounted on a directory hides what the image had there.
immutable: truecannot be edited or undone - delete the ConfigMap and create it again.
Docs
- https://kubernetes.io/docs/concepts/configuration/configmap/
- https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/
Tasks
- Q32 ConfigMap as an env var and as a file
- Q10 Projected volume: config, secret and token
- Q25 Two faults behind a stuck Deployment
- Q51 A ConfigMap-driven nginx that needs a restart (Simulator Set A (killer.sh-style))
- Q45 Kustomize: a patch and a generated ConfigMap (Simulator Set A (killer.sh-style))
4.5 Define resource requirements
./exam study 4.5 (or resources)
Know
- Per container: resources.requests and resources.limits for cpu (cores or millicores: 250m) and memory (Mi, Gi).
- QoS class: Guaranteed (every container has requests equal to limits for both CPU and memory), Burstable (some are set), BestEffort (none). BestEffort is evicted first.
- A container with a limit but no request gets a request equal to the limit.
Fast
k set resources deploy/web -c app --requests=cpu=250m,memory=128Mi --limits=cpu=250m,memory=128Mi
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 250m, memory: 128Mi}
k get pod POD -o jsonpath='{.status.qosClass}'
k describe node NODE | grep -A8 'Allocated resources'
Traps
kubectl runhas no --requests or --limits any more: generate the YAML, or usekubectl set resourceson a workload.- Guaranteed needs every container of the Pod, sidecars included.
- Memory is Mi, not m:
128mis a fraction of a byte. - A request no node can fit leaves the Pod Pending with
Insufficient cpuorInsufficient memory.
Docs
- https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
- https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/
Tasks
- Q31 Resource requirements for Guaranteed QoS
- Q26 OOMKilled
- Q15 DaemonSet on every node
- Q53 Pods rejected by the Namespace's limits (Simulator Set A (killer.sh-style))
4.6 Create & consume Secrets
./exam study 4.6 (or secrets)
Know
- A Secret's
datais base64 - an encoding, not encryption.stringDatatakes plain text when you write one. - Types: Opaque (generic), kubernetes.io/tls, kubernetes.io/dockerconfigjson, kubernetes.io/service-account-token.
- Consumed like a ConfigMap: secretKeyRef, envFrom secretRef, or a secret volume (
secretName,defaultMode,items).
Fast
k create secret generic api --from-literal=username=svc --from-literal=password='S3cr3t!'
k create secret tls web-tls --cert=tls.crt --key=tls.key
k create secret docker-registry regcred --docker-server=REG --docker-username=U --docker-password=P
k get secret api -o jsonpath='{.data.password}' | base64 -d
env: [{name: API_USER, valueFrom: {secretKeyRef: {name: api, key: username}}}]
volumes: [{name: creds, secret: {secretName: api, defaultMode: 0400}}]
volumeMounts: [{name: creds, mountPath: /etc/creds, readOnly: true}]
Traps
- A secret volume names it with
secretName; a configMap volume usesname. - Encoding by hand:
echo -n value | base64, or the newline becomes part of the secret. - defaultMode is octal in YAML (0400) and shows as decimal in JSON (256).
Docs
- https://kubernetes.io/docs/concepts/configuration/secret/
- https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/
Tasks
- Q33 Secrets: decode one, consume another
- Q10 Projected volume: config, secret and token
- Q37 Ingress with two paths and TLS
- Q50 An immutable Secret, one file into a busy directory (Simulator Set A (killer.sh-style))
4.7 Understand ServiceAccounts
./exam study 4.7 (or serviceaccounts)
Know
- Every Pod runs as a ServiceAccount (
defaultif you name none); its short-lived token is mounted at /var/run/secrets/kubernetes.io/serviceaccount. - Choose it with spec.serviceAccountName. Stop the mount with
automountServiceAccountToken: falseon the ServiceAccount or on the Pod (the Pod wins). - Tokens:
kubectl create tokenfor a short-lived one; a Secret of type kubernetes.io/service-account-token annotated with the account's name for a long-lived one. - A ServiceAccount can do nothing until RBAC binds a Role to it.
Fast
k create sa ci -n NS
k set serviceaccount deploy/web ci -n NS # = spec.template.spec.serviceAccountName
k patch sa ci -n NS -p '{"automountServiceAccountToken": false}'
k create token ci -n NS --duration=1h
k auth can-i list pods -n NS --as=system:serviceaccount:NS:ci
apiVersion: v1
kind: Secret
metadata: {name: ci-token, annotations: {kubernetes.io/service-account.name: ci}}
type: kubernetes.io/service-account-token
Traps
- No token Secret is made for a new ServiceAccount (since 1.24) - create it yourself when a task wants one.
- serviceAccountName cannot be changed on a running Pod; on a Deployment it starts a rollout.
- A Pod naming a ServiceAccount that does not exist is never created - see the ReplicaSet's events.
Docs
- https://kubernetes.io/docs/concepts/security/service-accounts/
- https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
Tasks
- Q34 ServiceAccount without an automounted token
- Q28 RBAC for a CI ServiceAccount
- Q10 Projected volume: config, secret and token
- Q49 RBAC, proven by the app's own log (Simulator Set A (killer.sh-style))
4.8 Understand Application Security (SecurityContexts, Capabilities, etc.)
./exam study 4.8 (or security)
Know
- Pod level (spec.securityContext): runAsUser, runAsGroup, runAsNonRoot, fsGroup, supplementalGroups, seccompProfile - for every container.
- Container level adds allowPrivilegeEscalation, privileged, readOnlyRootFilesystem and capabilities, and overrides the Pod level.
- Capabilities are written without the CAP_ prefix: NET_BIND_SERVICE, NET_ADMIN, SYS_TIME. Drop ALL, then add back what the app needs.
Fast
spec:
securityContext: {runAsUser: 1000, runAsGroup: 1000, runAsNonRoot: true, fsGroup: 2000}
containers:
- name: app
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: [ALL], add: [NET_BIND_SERVICE]}
k exec POD -- id
k explain pod.spec.securityContext; k explain pod.spec.containers.securityContext
Traps
- capabilities exist only on a container, fsGroup only on the Pod.
- runAsNonRoot with an image that runs as root and no runAsUser: CreateContainerConfigError.
- readOnlyRootFilesystem breaks an app that writes to /tmp or /var/cache - mount an emptyDir there.
Docs
Tasks
- Q9 SecurityContext and capabilities
- Q29 Pod Security admission blocks a Deployment
- Q44 From a single Pod to a hardened Deployment (Simulator Set A (killer.sh-style))
- Q52 Two users, one volume: fsGroup (Simulator Set A (killer.sh-style))
Services and Networking (20%)
5.1 Demonstrate basic understanding of NetworkPolicies
./exam study 5.1 (or netpol)
Know
- Pods accept and send everything until a NetworkPolicy selects them. From then on, for the policyTypes it lists, only what some policy allows gets through.
- Policies only allow, and they add up: there is no deny rule and no ordering.
- Peers: podSelector (same Namespace), namespaceSelector, ipBlock. Two selectors in one list item mean AND; two list items mean OR.
- Default deny:
podSelector: {}with a policyType and no rules. Locking down egress also needs DNS allowed (UDP and TCP 53).
Fast
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: db-allow, namespace: NS}
spec:
podSelector: {matchLabels: {app: db}}
policyTypes: [Ingress]
ingress:
- from:
- podSelector: {matchLabels: {app: api}} # OR
- namespaceSelector: {matchLabels: {purpose: monitoring}}
ports: [{protocol: TCP, port: 5432}]
# egress that keeps DNS: egress: [{ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]}, ...]
k describe netpol db-allow -n NS
k exec CLIENT -- wget -qO- -T2 http://db:5432 # test it with real traffic
Traps
- There is no
kubectl create networkpolicy- copy the skeleton from the docs. - podSelector: ...followed bynamespaceSelector:in the same item is AND; a second- namespaceSelector:is OR.- A missing or empty
ingressdenies everything;ingress: [{}]allows everything. - The policy lives in the Namespace of the Pods it selects, and a Namespace is matched by its labels (kubernetes.io/metadata.name=NAME always exists).
Docs
Tasks
- Q12 Restrict a namespace with NetworkPolicy
- Q35 Egress lockdown that keeps DNS working
- Q55 Two sources, one port (Simulator Set A (killer.sh-style))
5.2 Provide and troubleshoot access to applications via services
./exam study 5.2 (or services)
Know
- A Service selects Pods by label and forwards
porttotargetPort(a number, or the name of a container port). - Types: ClusterIP (inside the cluster), NodePort (30000-32767 on every node), LoadBalancer, ExternalName; headless (
clusterIP: None) gives each Pod its own DNS record. - DNS names: SVC, SVC.NS, SVC.NS.svc.cluster.local.
- Troubleshoot in this order: are there endpoints, does the selector match the Pod labels, are the Pods Ready, is targetPort the port the container listens on.
Fast
k expose deploy web --port=8080 --target-port=80 # --name=web-svc --type=NodePort
k create svc nodeport web-ext --tcp=80:80 --node-port=30080
k get svc,endpointslices -o wide; k describe svc web # Endpoints: <none> = nothing selected
k get pods -l app=web --show-labels
k run tmp --rm -it --restart=Never --image=busybox:1.36 -- wget -qO- -T3 http://web:8080
k port-forward svc/web 8080:80
Traps
kubectl exposecopies the workload's selector;kubectl create svcselects app=, which may match nothing.exposecannot choose the nodePort - usecreate svc nodeport --node-portor edit the YAML.- A Service without endpoints still resolves in DNS; the connection is what fails.
Docs
- https://kubernetes.io/docs/concepts/services-networking/service/
- https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/
- https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
Tasks
- Q11 Expose a Deployment two ways
- Q36 A Service that routes nowhere
- Q54 Expose a Pod and prove it from inside (Simulator Set A (killer.sh-style))
- Q56 Stable DNS names for a StatefulSet (Simulator Set A (killer.sh-style))
5.3 Use Ingress rules to expose applications
./exam study 5.3 (or ingress)
Know
- An Ingress routes HTTP(S) by host and path to Services of its own Namespace; an ingress controller, chosen with ingressClassName, does the work.
- Each path has a pathType (Prefix, Exact or ImplementationSpecific) and a backend: service.name plus service.port.number or .name.
- TLS: spec.tls lists hosts and the kubernetes.io/tls Secret that holds their certificate.
Fast
k create ingress shop --class=nginx \
--rule="shop.example.com/api*=api:8080,tls=shop-tls" --rule="shop.example.com/*=web:80,tls=shop-tls"
k create secret tls shop-tls --cert=tls.crt --key=tls.key
k get ingress,ingressclass; k describe ingress shop # a missing backend shows as <error: services "x" not found>
k explain ingress.spec.rules.http.paths
Traps
- In --rule a trailing * means pathType Prefix; without it the path is Exact.
- The backend port is the Service's port, not the container's.
- No ingress controller runs in this simulator, so Ingress tasks are graded on the object - on the exam, curl it.
Docs
Tasks
- Q37 Ingress with two paths and TLS
- Q11 Expose a Deployment two ways
- Q57 An Ingress pointing at the wrong backends (Simulator Set A (killer.sh-style))