"""CKA curriculum as a study path: the 27 objectives of the five domains, in the curriculum's own words and order (CKA_Curriculum_v1.35.pdf, cncf/curriculum), each with what to know, the fastest commands, the traps, the documentation pages the exam lets you open, and the tasks that practise it. ./exam objectives the path, with your progress per objective ./exam notes 4.4 the notes for one objective (id or key: 4.4, lifecycle) ./exam study 4.4 those notes, then its tasks with no clock and solutions visible ./exam drill 4.4 the same tasks against the clock ./exam next whatever is next on the path "tasks" lists every task that really exercises the objective - the plain pool tasks first, then the hard ones, then those of the fixed simulator sets (68-85 Set C, 86-102 Set D), which stay out of objective sessions unless you ask for them (--all). A task may sit under several objectives. Every task must be listed at least once and every objective needs two pool tasks - `./exam objectives` says so when that stops being true (tasks 103-113 exist because of that rule). An objective with many tasks takes several sittings: each one brings up to five tasks you have not solved yet, and never two that break the same part of the cluster. """ # The path walks the domains in this order, not the curriculum's: Troubleshooting is 30% of the # exam and every one of its tasks assumes you already know how the healthy thing is built. STUDY_ORDER = ["WORK", "NET", "STOR", "ARCH", "TSHOOT"] OBJECTIVES = [ # ------------------------------------------------------------------- 1. Storage (10%) ---- { "id": "1.1", "key": "storageclasses", "domain": "STOR", "text": "Implement storage classes and dynamic volume provisioning", "tasks": [16, 36, 59, 99], "know": [ "A StorageClass names a provisioner and its settings. A claim that asks for the class gets a PersistentVolume made for it - that is dynamic provisioning.", "Fields: provisioner, reclaimPolicy (Delete by default, or Retain), volumeBindingMode (Immediate or WaitForFirstConsumer), allowVolumeExpansion, parameters.", "The default class carries the annotation storageclass.kubernetes.io/is-default-class: \"true\"; a claim with no storageClassName gets it. `storageClassName: \"\"` asks for no class at all.", "A StatefulSet's volumeClaimTemplates make one claim per replica (data-web-0, data-web-1 ...).", ], "fast": """ k get sc # (default) marks the default class apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: {name: fast-local} provisioner: rancher.io/local-path reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer --- k patch sc standard -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}' k patch sc fast-local -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' k get pvc,pv; k describe pvc data # the Events say why a claim is Pending """, "traps": [ "With WaitForFirstConsumer a claim stays Pending until a Pod uses it. That is not an error.", "Leave exactly one class marked default: take the annotation off the old one as well as putting it on the new one.", "provisioner, reclaimPolicy and volumeBindingMode cannot be edited - delete the class and create it again.", ], "docs": ["https://kubernetes.io/docs/concepts/storage/storage-classes/", "https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/", "https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/"], }, { "id": "1.2", "key": "volumes", "domain": "STOR", "text": "Configure volume types, access modes and reclaim policies", "tasks": [15, 35, 58, 68, 72, 102], "know": [ "Volume types you meet: hostPath (a directory of whatever node the Pod lands on), local (a disk of one node, needs nodeAffinity), emptyDir, nfs, csi, and the configMap / secret / projected volumes.", "Access modes: ReadWriteOnce (one node), ReadOnlyMany, ReadWriteMany, ReadWriteOncePod (one Pod). A claim binds only to a volume that offers its mode.", "The reclaim policy is on the PersistentVolume: Retain keeps the volume and its data when the claim goes (it becomes Released), Delete removes both.", ], "fast": """ apiVersion: v1 kind: PersistentVolume metadata: {name: pv-data} spec: capacity: {storage: 1Gi} accessModes: [ReadWriteOnce] persistentVolumeReclaimPolicy: Retain storageClassName: manual hostPath: {path: /mnt/data} --- k patch pv pv-data -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' k patch pv pv-data --type=json -p '[{"op":"remove","path":"/spec/claimRef"}]' # Released -> Available k get pv -o custom-columns=NAME:.metadata.name,MODES:.spec.accessModes,RECLAIM:.spec.persistentVolumeReclaimPolicy,STATUS:.status.phase,CLAIM:.spec.claimRef.name """, "traps": [ "A Released volume never binds again by itself: remove its claimRef first.", "A `local` volume needs spec.nodeAffinity; a hostPath volume does not pin the Pod to the node that has the data.", "The local-path provisioner of this cluster only does ReadWriteOnce and ReadWriteOncePod - a ReadWriteMany claim stays Pending.", ], "docs": ["https://kubernetes.io/docs/concepts/storage/persistent-volumes/", "https://kubernetes.io/docs/concepts/storage/volumes/", "https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/"], }, { "id": "1.3", "key": "claims", "domain": "STOR", "text": "Manage persistent volumes and persistent volume claims", "tasks": [15, 36, 35, 58, 59, 68, 72, 81, 99, 102], "know": [ "A claim binds to one volume, and only if class, access mode and size fit (the volume may be larger); `volumeName` or a selector narrows it to one volume.", "Phases: a volume goes Available, Bound, Released; a claim Pending, Bound.", "A Pod mounts a claim of its own Namespace: volumes[].persistentVolumeClaim.claimName, then volumeMounts.", "A claim in use is protected: deleting it waits (Terminating) until no Pod references it, finished Pods included.", ], "fast": """ apiVersion: v1 kind: PersistentVolumeClaim metadata: {name: pvc-data, namespace: storage} spec: accessModes: [ReadWriteOnce] storageClassName: manual # omit for the default class resources: {requests: {storage: 500Mi}} --- # in the Pod: volumes: [{name: data, persistentVolumeClaim: {claimName: pvc-data}}] containers: [{..., volumeMounts: [{name: data, mountPath: /data}]}] k get pvc,pv -A; k describe pvc pvc-data # Used By: and Events: k describe pod POD | tail # unbound claim -> the Pod stays Pending """, "traps": [ "There is no `kubectl create pvc` or `create pv` - copy both from the docs.", "For a hand-made volume the claim's storageClassName has to equal the volume's exactly; leaving it out means the default class, not none.", "Apart from growing the request (and only if the class allows expansion) a claim cannot be edited.", "To move data between claims, run a Pod or Job that mounts both.", ], "docs": ["https://kubernetes.io/docs/concepts/storage/persistent-volumes/", "https://kubernetes.io/docs/tutorials/configuration/configure-persistent-volume-storage/"], }, # ------------------------------------------------------------ 2. Troubleshooting (30%) ---- { "id": "2.1", "key": "nodes", "domain": "TSHOOT", "text": "Troubleshoot clusters and nodes", "tasks": [17, 39, 60, 92, 90, 95], "know": [ "Start from the cluster: `k get nodes`, then `k describe node` - Conditions, Taints, Unschedulable, Events.", "A NotReady node is nearly always its kubelet: on the node, `systemctl status kubelet` and `journalctl -u kubelet` name the cause.", "What the kubelet needs: its unit and drop-in (which binary, which flags), /var/lib/kubelet/config.yaml, /etc/kubernetes/kubelet.conf (its kubeconfig), its certificates in /var/lib/kubelet/pki, and a running container runtime.", "Ready but SchedulingDisabled is a cordon, not a fault: `k uncordon`.", ], "fast": """ k get nodes -o wide; k describe node NODE | sed -n '/Conditions/,/Addresses/p' ssh NODE systemctl status kubelet; journalctl -u kubelet --no-pager | tail -30 systemctl cat kubelet # the unit and its drop-ins: ExecStart, EnvironmentFile cat /var/lib/kubelet/config.yaml; cat /var/lib/kubelet/kubeadm-flags.env systemctl daemon-reload; systemctl enable --now kubelet; systemctl restart kubelet systemctl status containerd; crictl ps -a k uncordon NODE; k describe node NODE | grep -i taint """, "traps": [ "A kubelet that was stopped is often disabled too: `enable --now`, or it is gone again after a reboot.", "After changing a unit file nothing happens until `systemctl daemon-reload`.", "There may be more than one fault. After every fix read the journal again instead of assuming you are done.", ], "docs": ["https://kubernetes.io/docs/tasks/debug/debug-cluster/", "https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/", "https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/"], }, { "id": "2.2", "key": "components", "domain": "TSHOOT", "text": "Troubleshoot cluster components", "tasks": [20, 38, 61, 66, 97, 86], "know": [ "kube-apiserver, kube-scheduler, kube-controller-manager and etcd run as static Pods: the kubelet starts whatever is in /etc/kubernetes/manifests and restarts it when the file changes.", "Read the symptom: Pods stay Pending with no events - scheduler. Deployments make no ReplicaSet or Pods - controller-manager. kubectl is refused - API server (or etcd behind it).", "With the API server up: `k -n kube-system get pods`, `logs`, `describe`. With it down: `crictl ps -a`, `crictl logs`, /var/log/pods and the kubelet's journal on the node.", ], "fast": """ k -n kube-system get pods; k -n kube-system logs kube-scheduler-NODE; k -n kube-system describe pod kube-scheduler-NODE ssh CONTROL-PLANE ls /etc/kubernetes/manifests; vi /etc/kubernetes/manifests/kube-scheduler.yaml crictl ps -a | grep -E 'kube-|etcd'; crictl logs CONTAINER-ID ls /var/log/pods/kube-system_kube-apiserver-*/kube-apiserver/ # its logs when kubectl cannot ask for them journalctl -u kubelet --no-pager | grep -i -E 'manifest|apiserver' | tail """, "traps": [ "The faults are small: a misspelt image, binary or flag, a wrong path in a volume or certificate flag, a wrong port in a probe.", "Do not leave a backup copy inside the manifests directory - the kubelet runs every file there.", "After saving a manifest give the kubelet up to a minute before deciding the fix did not work.", ], "docs": ["https://kubernetes.io/docs/tasks/debug/debug-cluster/", "https://kubernetes.io/docs/concepts/overview/components/", "https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/"], }, { "id": "2.3", "key": "usage", "domain": "TSHOOT", "text": "Monitor cluster and application resource usage", "tasks": [103, 23, 53, 85, 89], "know": [ "`kubectl top node|pod` shows live CPU and memory from metrics-server through the metrics API (metrics.k8s.io); `--containers` splits a Pod up, `--sort-by=cpu|memory` orders it.", "`k describe node` shows what is allocated - the sum of requests and limits - which is not what is used.", "`kubectl get` sorts and shapes everything else: --sort-by, -o custom-columns, -o jsonpath, -A.", ], "fast": """ k top node; k top pod -A --sort-by=memory | head; k top pod -n NS --containers k describe node NODE | grep -A8 'Allocated resources' k get pods -A --sort-by='.status.containerStatuses[0].restartCount' k -n kube-system get deploy metrics-server; k get apiservice v1beta1.metrics.k8s.io k -n kube-system logs deploy/metrics-server """, "traps": [ "`Metrics API not available` means metrics-server is missing or not Ready (on this cluster it needs --kubelet-insecure-tls).", "`kubectl top --sort-by` puts the largest first; `kubectl get --sort-by` puts it last.", "A new Pod has no metrics for about a minute.", ], "docs": ["https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/", "https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/", "https://kubernetes.io/docs/reference/kubectl/quick-reference/"], }, { "id": "2.4", "key": "logs", "domain": "TSHOOT", "text": "Manage and evaluate container output streams", "tasks": [19, 104, 8, 85], "know": [ "`kubectl logs` shows what a container wrote to stdout and stderr. The runtime keeps it on the node under /var/log/pods/NAMESPACE_POD_UID/CONTAINER/, and it goes when the Pod is deleted.", "Pick the container with -c (init containers too) or take --all-containers; --previous is the run before the last restart.", "Narrow with --tail=N, --since=10m, --timestamps; follow with -f; take every Pod of a label with -l and tell them apart with --prefix.", "On a node, without the API server: `crictl logs CONTAINER-ID`.", ], "fast": """ k logs POD -c CONTAINER # --all-containers --prefix k logs POD --previous # the crashed run k logs POD --tail=50 --since=10m --timestamps k logs -l app=web --prefix --tail=5 # every Pod with the label k logs deploy/web; k logs job/render # one Pod of it k logs POD | grep ERROR > errors.log crictl ps -a; crictl logs --tail=20 CONTAINER-ID; ls /var/log/pods/ """, "traps": [ "With -l, only the last 10 lines of each Pod are shown unless you set --tail (--tail=-1 for everything).", "`kubectl logs deploy/x` reads one Pod, not all replicas.", "--previous fails when the container has never restarted.", ], "docs": ["https://kubernetes.io/docs/concepts/cluster-administration/logging/", "https://kubernetes.io/docs/reference/kubectl/quick-reference/", "https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/"], }, { "id": "2.5", "key": "netdebug", "domain": "TSHOOT", "text": "Troubleshoot services and networking", "tasks": [18, 62, 37, 57, 73, 101, 70, 90], "know": [ "Work outwards from the Pod: is it Ready, does the Service have endpoints (selector against Pod labels), is targetPort the port the container listens on, does the name resolve, does a NetworkPolicy stand in the way.", "DNS is a Service like any other: the CoreDNS Pods, the kube-dns Service and its endpoints, the Corefile in ConfigMap coredns.", "Underneath: kube-proxy (a DaemonSet) turns Services into rules on every node, and the CNI plugin connects the Pods.", ], "fast": """ k get svc,endpointslices -o wide; k describe svc SVC # Endpoints: = nothing selected k get pods -l KEY=VALUE --show-labels -o wide k run tmp --rm -it --restart=Never --image=busybox:1.36 -- sh # wget -qO- -T3 http://SVC:PORT ; nslookup SVC.NS.svc.cluster.local k -n kube-system get pods -l k8s-app=kube-dns -o wide; k -n kube-system logs -l k8s-app=kube-dns k -n kube-system get cm coredns -o yaml; k -n kube-system get svc,endpointslices -l k8s-app=kube-dns k -n kube-system get ds kube-proxy; k -n kube-system logs ds/kube-proxy k get netpol -A """, "traps": [ "A Service with no endpoints still resolves - it is the connection that fails.", "Test names with the fully qualified form (SVC.NS.svc.cluster.local): busybox's nslookup misreports short ones.", "A NetworkPolicy in kube-system can cut DNS off for the whole cluster.", "Fix what is broken and leave the rest alone - the graders check what you did not have to touch.", ], "docs": ["https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/", "https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/"], }, # ------------------------------------------------- 3. Workloads & Scheduling (15%) ---- { "id": "3.1", "key": "rollouts", "domain": "WORK", "text": "Understand application deployments and how to perform rolling update and rollbacks", "tasks": [7, 51], "know": [ "Any change to `spec.template` starts a rollout: a new ReplicaSet scales up while the old one scales down, within maxSurge and maxUnavailable (25% each by default).", "Every rollout is a revision. `kubectl rollout`: status, history, undo, pause, resume, restart.", "minReadySeconds, revisionHistoryLimit and progressDeadlineSeconds tune how careful and how patient a rollout is.", ], "fast": """ k set image deploy/web nginx=nginx:1.27 # CONTAINER=IMAGE k scale deploy/web --replicas=4 k rollout status deploy/web k rollout history deploy/web; k rollout history deploy/web --revision=2 # that revision's template k rollout undo deploy/web --to-revision=2 k rollout pause deploy/web; k rollout resume deploy/web k patch deploy web -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' """, "traps": [ "To go back to a particular image, look at each revision's template first - \"the previous one\" is not always the one asked for.", "An undo makes a new revision out of the old one, so the numbers keep climbing.", "A paused Deployment records changes but rolls nothing out until it is resumed.", "maxSurge and maxUnavailable cannot both be 0.", ], "docs": ["https://kubernetes.io/docs/concepts/workloads/controllers/deployment/"], }, { "id": "3.2", "key": "config", "domain": "WORK", "text": "Use ConfigMaps and Secrets to configure applications", "tasks": [10, 29, 79, 93, 84], "know": [ "Both hold keys. A key becomes an environment variable (configMapKeyRef / secretKeyRef, or every key at once with envFrom) or a file (a configMap / secret volume; `items` picks and renames keys).", "Secret data is base64 - an encoding, not encryption. Types: Opaque, kubernetes.io/tls, kubernetes.io/dockerconfigjson.", "Mounted files follow a change after a short delay. Environment variables and subPath mounts never do: restart the Pods.", "`immutable: true` freezes one for good. To change the content, make a new object and point the workload at it.", ], "fast": """ k create cm app-config --from-literal=APP_MODE=production --from-file=app.properties k create secret generic db-creds --from-literal=username=admin --from-literal=password='S3cr3t!' k create secret tls web-tls --cert=tls.crt --key=tls.key k get secret db-creds -o jsonpath='{.data.password}' | base64 -d env: - {name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: APP_MODE}}} - {name: DB_PASS, valueFrom: {secretKeyRef: {name: db-creds, key: password}}} envFrom: [{configMapRef: {name: app-config}}, {secretRef: {name: db-creds}}] volumes: [{name: creds, secret: {secretName: db-creds, defaultMode: 0400}}] k set env deploy/web --from=configmap/app-config; k rollout restart deploy/web """, "traps": [ "A secret volume names it with `secretName`; a configMap volume uses `name`.", "A Pod that references a missing ConfigMap, Secret or key waits in CreateContainerConfigError.", "`--from-file` makes one key named after the file; `--from-env-file` makes one key per line.", ], "docs": ["https://kubernetes.io/docs/concepts/configuration/configmap/", "https://kubernetes.io/docs/concepts/configuration/secret/", "https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/"], }, { "id": "3.3", "key": "autoscaling", "domain": "WORK", "text": "Configure workload autoscaling", "tasks": [105, 23, 82], "know": [ "A HorizontalPodAutoscaler (autoscaling/v2) keeps a workload between minReplicas and maxReplicas by a metric - usually average CPU or memory utilisation, which is usage divided by the container's request.", "It reads the metrics API, so it needs metrics-server, and the containers need a request for the resource it scales on.", "`behavior` shapes the change: scaleUp and scaleDown each take a stabilizationWindowSeconds and policies (so many Pods or Percent per periodSeconds).", ], "fast": """ k set resources deploy web --requests=cpu=100m k autoscale deploy web --min=2 --max=6 --cpu=60% # kubectl before 1.34: --cpu-percent=60 k get hpa; k describe hpa web apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: {name: web} spec: scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: web} minReplicas: 2 maxReplicas: 6 metrics: - type: Resource resource: {name: cpu, target: {type: Utilization, averageUtilization: 60}} behavior: scaleDown: stabilizationWindowSeconds: 300 policies: [{type: Pods, value: 1, periodSeconds: 60}] """, "traps": [ "TARGETS ``: no metrics-server, or no request on the containers.", "`kubectl autoscale` cannot write `behavior` - edit the object or write the YAML.", "minReplicas is enforced at once, even without metrics.", ], "docs": ["https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/", "https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/", "https://kubernetes.io/docs/concepts/workloads/autoscaling/"], }, { "id": "3.4", "key": "selfhealing", "domain": "WORK", "text": "Understand the primitives used to create robust, self-healing, application deployments", "tasks": [106, 8, 28, 26, 27, 31, 52, 53, 76, 88, 70], "know": [ "Controllers put back what is missing: a Deployment its replicas, a DaemonSet one Pod per node, a StatefulSet named replicas with their own storage, a Job its completions (backoffLimit retries), a CronJob its Jobs.", "Probes tell the kubelet about health: liveness restarts the container, readiness takes the Pod out of Service endpoints, startup holds both off until the app is up.", "Init containers run to completion, in order, before the app starts; a native sidecar (an init container with restartPolicy: Always) keeps running beside it.", "A PodDisruptionBudget (minAvailable or maxUnavailable) limits how many Pods a drain may take away at once.", ], "fast": """ containers: - name: app startupProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5, failureThreshold: 12} readinessProbe: {httpGet: {path: /ready, port: 8080}, periodSeconds: 5} livenessProbe: {tcpSocket: {port: 80}, initialDelaySeconds: 15, periodSeconds: 10} # exec: {exec: {command: [cat, /tmp/ready]}} k create job render --image=busybox:1.36 -- sh -c 'echo done' k create cronjob digest --image=busybox:1.36 --schedule='*/5 * * * *' -- date; k create job run-1 --from=cronjob/digest k create pdb web --selector=app=web --min-available=2 k create deploy x --image=busybox:1.36 $do > ds.yaml # DaemonSet: change kind, drop replicas / strategy / status k describe pod POD | grep -E 'Liveness|Readiness|Startup'; k get events --field-selector reason=Unhealthy """, "traps": [ "Probes belong to a container. No kubectl flag adds one - edit the YAML.", "Restarts with Unhealthy and Killing events but a clean log: the liveness probe is wrong, not the app.", "A DaemonSet reaches control-plane nodes only with a toleration for node-role.kubernetes.io/control-plane.", "A Job's Pod template cannot be edited: delete the Job and create it again.", ], "docs": ["https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/", "https://kubernetes.io/docs/concepts/workloads/controllers/deployment/", "https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/", "https://kubernetes.io/docs/tasks/run-application/configure-pdb/"], }, { "id": "3.5", "key": "scheduling", "domain": "WORK", "text": "Configure Pod admission and scheduling (limits, node affinity, etc.)", "tasks": [9, 25, 24, 32, 50, 78, 96, 81, 97], "know": [ "Requests are what the scheduler reserves, limits what the node enforces. A LimitRange gives containers defaults and bounds; a ResourceQuota caps a Namespace's totals - both act at admission.", "Where a Pod may go: nodeSelector, node affinity (required or preferred), Pod affinity and anti-affinity with a topologyKey, topologySpreadConstraints, or nodeName, which skips the scheduler.", "Taints keep Pods off a node (NoSchedule, PreferNoSchedule, NoExecute); a toleration lets a Pod through.", "A PriorityClass orders Pods when room is short; the Pod names it in priorityClassName.", ], "fast": """ k label node NODE disktype=ssd k taint node NODE dedicated=gpu:NoSchedule; k taint node NODE dedicated- # the trailing - removes it tolerations: [{key: dedicated, operator: Equal, value: gpu, effect: NoSchedule}] affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: [{matchExpressions: [{key: disktype, operator: In, values: [ssd]}]}] podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - {labelSelector: {matchLabels: {app: web}}, topologyKey: kubernetes.io/hostname} k create priorityclass high --value=100000 --description='critical workloads' k create quota team -n NS --hard=pods=5,requests.cpu=1,limits.memory=2Gi k describe pod POD | tail # FailedScheduling: 0/4 nodes are available: ... """, "traps": [ "A toleration lets a Pod onto a tainted node but does not send it there - pair it with a nodeSelector or affinity.", "Once a quota covers CPU or memory, every new Pod must state them or get them from a LimitRange.", "In a LimitRange, `default` is the limit and `defaultRequest` the request.", "Scheduling fields of a running Pod cannot be changed: change the Deployment, or delete and recreate the Pod.", ], "docs": ["https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/", "https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/", "https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/", "https://kubernetes.io/docs/concepts/policy/resource-quotas/"], }, # ------------------ 4. Cluster Architecture, Installation & Configuration (25%) ---- { "id": "4.1", "key": "rbac", "domain": "ARCH", "text": "Manage role based access control (RBAC)", "tasks": [1, 33, 6, 44, 47, 75, 91], "know": [ "A Role (one Namespace) or ClusterRole (everywhere, and cluster-scoped resources) lists verbs on resources; a RoleBinding or ClusterRoleBinding grants it to users, groups or ServiceAccounts.", "A RoleBinding may point at a ClusterRole: the rules then apply in that one Namespace only.", "A user is a certificate: CN is the name, O the groups. Create a key and CSR, submit a CertificateSigningRequest, approve it, put the certificate in a kubeconfig.", "An aggregated ClusterRole has no rules of its own: the controller fills them in from every ClusterRole carrying the label it selects.", ], "fast": """ k create role deploy-manager -n NS --verb=get,list,watch,create,update,patch,delete --resource=deployments.apps k create rolebinding deploy-bot -n NS --role=deploy-manager --serviceaccount=NS:deploy-bot # --user=jane --group=devs k create clusterrole node-reader --verb=get,list,watch --resource=nodes k create clusterrolebinding node-reader --clusterrole=node-reader --serviceaccount=NS:SA k auth can-i create deployments -n NS --as=system:serviceaccount:NS:deploy-bot; k auth can-i --list --as=jane -n NS openssl genrsa -out jane.key 2048; openssl req -new -key jane.key -subj '/CN=jane/O=developers' -out jane.csr # CertificateSigningRequest: spec.request = base64 -w0 of the .csr, signerName kubernetes.io/kube-apiserver-client, usages [client auth] k certificate approve jane; k get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt k config set-credentials jane --client-certificate=jane.crt --client-key=jane.key --embed-certs """, "traps": [ "\"Exactly these permissions\" also means finding what is already granted: `k get rolebindings,clusterrolebindings -A -o wide | grep NAME`.", "Deployments are in the group apps: --resource=deployments.apps; a subresource is deployments.apps/scale, pods/log.", "The same ServiceAccount is `--serviceaccount=NS:NAME` in a binding and `system:serviceaccount:NS:NAME` after `--as`.", "Nodes, PersistentVolumes and Namespaces are cluster-scoped: they need a ClusterRole and a ClusterRoleBinding.", ], "docs": ["https://kubernetes.io/docs/reference/access-authn-authz/rbac/", "https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/"], }, { "id": "4.2", "key": "infra", "domain": "ARCH", "text": "Prepare underlying infrastructure for installing a Kubernetes cluster", "tasks": [107, 108, 41], "know": [ "Every machine needs, before kubeadm: a container runtime (containerd) using the same cgroup driver as the kubelet (systemd), the kernel modules overlay and br_netfilter, the sysctls net.bridge.bridge-nf-call-iptables and net.ipv4.ip_forward, and swap off or accounted for.", "kubeadm, kubelet and kubectl come from the package repository of one minor version (pkgs.k8s.io) and are held, so that a system update does not move them.", "Also: a unique hostname, MAC address and product_uuid per node, and the ports open - 6443 (API), 10250 (kubelet), 2379-2380 (etcd).", ], "fast": """ printf 'overlay\\nbr_netfilter\\n' > /etc/modules-load.d/k8s.conf; modprobe overlay; modprobe br_netfilter cat > /etc/sysctl.d/k8s.conf </dev/null | openssl dgst -sha256 -hex # the CA hash kubeadm certs check-expiration; openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate k -n kube-system get cm kubeadm-config -o yaml grep staticPodPath /var/lib/kubelet/config.yaml; k run web --image=nginx:1.27 $do > /etc/kubernetes/manifests/web.yaml """, "traps": [ "Tokens expire (24 hours unless you say otherwise) - make a new one rather than hunting for the old.", "Check staticPodPath before you write a static Pod: it is not always /etc/kubernetes/manifests.", "Deleting a static Pod with kubectl only deletes its mirror; the kubelet puts it straight back.", ], "docs": ["https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/", "https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-token/", "https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/", "https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/"], }, { "id": "4.4", "key": "lifecycle", "domain": "ARCH", "text": "Manage the lifecycle of Kubernetes clusters", "tasks": [109, 2, 65, 45, 80], "know": [ "Upgrade one minor version at a time, control plane first: the kubeadm package, `kubeadm upgrade plan` and `apply`, drain, the kubelet and kubectl packages, restart the kubelet, uncordon. Workers: `kubeadm upgrade node` in place of `apply`.", "Maintenance on a node: `kubectl drain` cordons it and evicts its Pods; `kubectl uncordon` gives it back.", "etcd: `etcdctl snapshot save` backs it up; `etcdutl snapshot restore --data-dir` unpacks a snapshot into a new directory that etcd's manifest must then point at.", "`kubeadm certs renew all` renews the control-plane certificates; the static Pods have to be restarted to load them.", ], "fast": """ k drain NODE --ignore-daemonsets --delete-emptydir-data; k uncordon NODE apt-get install -y kubeadm='1.36.*'; kubeadm upgrade plan; kubeadm upgrade apply v1.36.x # workers: kubeadm upgrade node apt-get install -y kubelet='1.36.*' kubectl='1.36.*'; systemctl daemon-reload; systemctl restart kubelet etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup.db etcdutl snapshot restore /opt/backup.db --data-dir /var/lib/etcd-restore # then etcd.yaml: the hostPath of volume etcd-data kubeadm certs renew all; kubeadm certs check-expiration """, "traps": [ "drain stops at DaemonSet Pods (--ignore-daemonsets), at emptyDir data (--delete-emptydir-data) and at bare Pods (--force, which deletes them for good).", "A drained node stays cordoned until you uncordon it.", "With etcd 3.6, as here, restore exists only in etcdutl - `etcdctl snapshot restore` is gone.", "Restoring etcd takes the whole cluster back to the snapshot: everything created since is lost.", ], "docs": ["https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/", "https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/", "https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/", "https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/"], }, { "id": "4.5", "key": "ha", "domain": "ARCH", "text": "Implement and configure a highly-available control plane", "tasks": [110, 111, 86, 87], "know": [ "Highly available means several control-plane nodes (three as a rule) behind one stable address - the controlPlaneEndpoint, a load balancer or virtual IP, fixed at `kubeadm init`.", "etcd is either stacked (a member on every control-plane node) or external (hosts of its own). It needs a majority to accept writes: 3 members survive the loss of 1, 5 of 2.", "More control-plane nodes join with `kubeadm join --control-plane --certificate-key KEY`; the key comes from `kubeadm init phase upload-certs --upload-certs` and is good for two hours.", "Every kube-apiserver serves at once. Of the schedulers and controller-managers only the leader acts - they elect it through a Lease in kube-system.", ], "fast": """ k get nodes -l node-role.kubernetes.io/control-plane k -n kube-system get cm kubeadm-config -o yaml | grep controlPlaneEndpoint kubeadm init --control-plane-endpoint=LB:6443 --upload-certs kubeadm init phase upload-certs --upload-certs # prints the certificate key kubeadm token create --print-join-command --certificate-key KEY # ... --control-plane --certificate-key KEY k -n kube-system get lease kube-scheduler kube-controller-manager # HOLDER = NODE_uuid etcdctl member list -w table; etcdctl endpoint status -w table; etcdctl endpoint health # with the four etcd flags k -n kube-system get pods -l component=etcd -o wide """, "traps": [ "A cluster initialised without a controlPlaneEndpoint cannot simply be given a second control-plane node.", "An even number of etcd members tolerates no more failures than the odd number below it.", "The certificate key expires after two hours - upload the certificates again rather than reusing an old key.", ], "docs": ["https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/", "https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/"], }, { "id": "4.6", "key": "helmkustomize", "domain": "ARCH", "text": "Use Helm and Kustomize to install cluster components", "tasks": [21, 3, 48, 69, 98], "know": [ "Helm: a chart is the package, a release one installation of it in a Namespace, and each install, upgrade or rollback a revision. Values come from the chart, then -f files, then --set.", "`helm template` renders a chart without installing it; `helm show values` prints its defaults; `helm get values` what a release was given.", "Kustomize: a kustomization.yaml lists resources and changes them - namespace, namePrefix, labels, images, replicas, patches - and generates ConfigMaps and Secrets. An overlay lists a base and changes only what differs.", ], "fast": """ helm repo add NAME URL && helm repo update; helm search repo NAME --versions helm install REL CHART -n NS --create-namespace --set replicaCount=2 -f values.yaml # --version X helm upgrade REL CHART -n NS --reuse-values --set image.tag=1.27 helm template REL CHART -n NS --set replicaCount=3 > manifest.yaml helm list -A; helm history REL -n NS; helm rollback REL 1 -n NS; helm uninstall REL -n NS k kustomize DIR | less; k apply -k DIR # kustomization.yaml: resources: [../../base] namespace: prod namePrefix: prod- images: [{name: nginx, newTag: "1.27"}] # replicas: [{name: web, count: 3}] patches: [{path: patch.yaml}] configMapGenerator: [{name: cfg, literals: [k=v]}] """, "traps": [ "Releases live in a Namespace: without -n Helm looks in `default`.", "An upgrade without --reuse-values (or the same -f and --set again) puts every value back to the chart's default.", "In a kustomization, `images`, `replicas` and patches use the names of the base, before any prefix.", "`namespace:` in a kustomization does not create the Namespace.", ], "docs": ["https://helm.sh/docs/intro/using_helm/", "https://helm.sh/docs/helm/helm_template/", "https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/"], }, { "id": "4.7", "key": "interfaces", "domain": "ARCH", "text": "Understand extension interfaces (CNI, CSI, CRI, etc.)", "tasks": [112, 108, 41, 63, 77, 87], "know": [ "CRI: the kubelet drives the container runtime (containerd, CRI-O) through a socket; `crictl` speaks the same protocol, so it works on a node when kubectl does not.", "CNI: for every Pod the runtime calls the plugins named in /etc/cni/net.d (binaries in /opt/cni/bin) to give it an address and wire it up. The plugin - kindnet, Calico, Cilium, Flannel - usually runs as a DaemonSet.", "CSI: a storage driver runs as Pods, registers a CSIDriver object, and a StorageClass names it as provisioner.", ], "fast": """ k get nodes -o wide # CONTAINER-RUNTIME column crictl version; crictl pods; crictl ps -a; crictl images; crictl inspect CONTAINER-ID grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml; cat /etc/crictl.yaml ls /etc/cni/net.d; cat /etc/cni/net.d/*; ls /opt/cni/bin k -n kube-system get ds # the CNI's DaemonSet beside kube-proxy k get csidrivers; k get csinodes; k get sc # PROVISIONER """, "traps": [ "Not every CNI plugin enforces NetworkPolicy (Flannel does not; kindnet in this cluster does).", "Without a CNI plugin the nodes stay NotReady: `cni plugin not initialized`.", "The runtime takes the first configuration file in /etc/cni/net.d, in alphabetical order.", ], "docs": ["https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/", "https://kubernetes.io/docs/concepts/containers/cri/", "https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/"], }, { "id": "4.8", "key": "crds", "domain": "ARCH", "text": "Understand CRDs, install and configure operators", "tasks": [4, 34, 49, 91], "know": [ "A CustomResourceDefinition adds a kind to the API: group, names (kind, plural, singular, shortNames), scope (Namespaced or Cluster) and versions, each with an OpenAPI v3 schema.", "The schema validates custom resources - types, required fields, enum, minimum, maximum, defaults - and feeds `kubectl explain`.", "An operator is a CRD plus a controller (a Deployment with RBAC) that acts on those objects. Installing one means applying its manifests or its Helm chart, then creating a custom resource.", ], "fast": """ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: {name: backups.data.example.com} # PLURAL.GROUP spec: group: data.example.com scope: Namespaced names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]} versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object required: [schedule] properties: schedule: {type: string} retention: {type: integer, minimum: 1, default: 7} k get crd; k api-resources --api-group=data.example.com; k explain backup.spec --recursive """, "traps": [ "The CRD's name must be PLURAL.GROUP, and exactly one version has `storage: true`.", "A custom resource's apiVersion is GROUP/VERSION of the CRD, not apiextensions.k8s.io/v1.", "`default` and validation only work inside a structural schema: every level needs its `type`.", "Deleting a CRD deletes every object of that kind.", ], "docs": ["https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/", "https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/", "https://kubernetes.io/docs/concepts/extend-kubernetes/operator/"], }, # -------------------------------------------------- 5. Services & Networking (20%) ---- { "id": "5.1", "key": "connectivity", "domain": "NET", "text": "Understand connectivity between Pods", "tasks": [113, 63, 71, 77, 87], "know": [ "Every Pod has its own IP address, and any Pod reaches any other Pod on any node at that address without NAT. The CNI plugin makes that true.", "Each node hands out Pod addresses from its own range (spec.podCIDR), a slice of the cluster's Pod CIDR; Services get theirs from the separate service CIDR.", "A hostNetwork Pod lives in the node's network: it has the node's address and shares its ports.", "Containers of one Pod share a network namespace and talk over localhost.", ], "fast": """ k get pods -o wide -A; k get nodes -o custom-columns=NODE:.metadata.name,PODCIDR:.spec.podCIDR,IP:.status.addresses[0].address k exec POD -- wget -qO- -T3 http://POD-IP:PORT k -n kube-system get cm kubeadm-config -o yaml | grep -E 'podSubnet|serviceSubnet' k -n kube-system get pods -o wide | grep -E 'kube-proxy|kindnet'; k -n kube-system get cm kube-proxy -o yaml | grep mode """, "traps": [ "A Pod's address changes whenever the Pod is replaced - that is what Services are for.", "Reaching a Pod by IP proves the network; reaching it by Service name also needs endpoints and DNS.", "If only the Pods of one node are unreachable, look at that node's CNI and kube-proxy Pods.", ], "docs": ["https://kubernetes.io/docs/concepts/services-networking/", "https://kubernetes.io/docs/concepts/cluster-administration/networking/"], }, { "id": "5.2", "key": "netpol", "domain": "NET", "text": "Define and enforce Network Policies", "tasks": [11, 42, 54, 83, 100, 77], "know": [ "Pods accept and send everything until a NetworkPolicy selects them. From then on, for the policyTypes it lists, only what some policy allows gets through.", "Policies only allow, and they add up: there is no deny rule and no ordering.", "Peers: podSelector (same Namespace), namespaceSelector, ipBlock. Two selectors in one list item mean AND; two list items mean OR.", "Default deny: `podSelector: {}` with the policyTypes and no rules. An egress lock-down also has to allow DNS (UDP and TCP 53).", ], "fast": """ apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: {name: backend-policy, namespace: NS} spec: podSelector: {matchLabels: {app: backend}} policyTypes: [Ingress, Egress] ingress: - from: [{podSelector: {matchLabels: {app: frontend}}}] ports: [{protocol: TCP, port: 8080}] egress: - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}] ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}] k describe netpol backend-policy -n NS; k exec CLIENT -- wget -qO- -T2 http://backend:8080 """, "traps": [ "There is no `kubectl create networkpolicy` - copy the skeleton from the docs.", "`namespaceSelector` and `podSelector` in the same item: Pods with that label in those Namespaces. In two items: either.", "A missing `ingress` denies everything; `ingress: [{}]` allows everything.", "A Namespace is matched by its labels; kubernetes.io/metadata.name=NAME is always there.", ], "docs": ["https://kubernetes.io/docs/concepts/services-networking/network-policies/"], }, { "id": "5.3", "key": "services", "domain": "NET", "text": "Use ClusterIP, NodePort, LoadBalancer service types and endpoints", "tasks": [12, 30, 43, 64, 71, 73, 94], "know": [ "A Service selects Pods by label and forwards `port` to `targetPort` - a number, or the name of a container port.", "ClusterIP is reachable inside the cluster; NodePort adds a port (30000-32767) on every node; LoadBalancer adds an external address from the cloud; ExternalName is a DNS alias; headless (clusterIP: None) gives every Pod its own DNS record.", "The Pods behind a Service are its endpoints, kept in EndpointSlices; only Ready Pods are in them.", ], "fast": """ k expose deploy shop --name=shop-svc --port=80 --target-port=80 --type=NodePort k create svc nodeport shop-svc --tcp=80:80 --node-port=30080 # then check the selector k create svc externalname legacy --external-name=db.example.com k get svc,endpointslices -o wide; k describe svc shop-svc # Endpoints: k get pods -l app=shop --show-labels k run tmp --rm -it --restart=Never --image=busybox:1.36 -- wget -qO- -T3 http://shop-svc """, "traps": [ "`kubectl expose` copies the workload's selector; `kubectl create svc` selects app=, which may match nothing.", "`expose` cannot choose the nodePort - use `create svc nodeport --node-port` or edit the YAML.", "A LoadBalancer stays where nothing provides one; it still works as a NodePort.", "A headless Service for a StatefulSet has to carry the name in the StatefulSet's serviceName.", ], "docs": ["https://kubernetes.io/docs/concepts/services-networking/service/", "https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/"], }, { "id": "5.4", "key": "gateway", "domain": "NET", "text": "Use the Gateway API to manage Ingress traffic", "tasks": [22, 56, 74], "know": [ "Three kinds, three owners: a GatewayClass names the controller, a Gateway opens listeners (port, protocol, hostname, which routes may attach), an HTTPRoute attaches to a Gateway (parentRefs) and sends requests to Services (backendRefs).", "An HTTPRoute rule matches on path, header, method or query, and may split traffic between backends by weight - that is a canary.", "The kinds are CRDs (gateway.networking.k8s.io): they have to be installed, and a controller has to implement them.", ], "fast": """ apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: {name: web-gw, namespace: NS} spec: gatewayClassName: example listeners: [{name: http, protocol: HTTP, port: 80, hostname: shop.example.com}] --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: {name: web, namespace: NS} spec: parentRefs: [{name: web-gw}] hostnames: [shop.example.com] rules: - matches: [{path: {type: PathPrefix, value: /}}] backendRefs: [{name: v1, port: 80, weight: 90}, {name: v2, port: 80, weight: 10}] k get gatewayclass,gateway,httproute -A; k describe httproute web -n NS """, "traps": [ "A route in another Namespace attaches only if the listener's allowedRoutes lets it.", "A header match belongs in `matches`, beside or instead of the path - and several rules are tried most specific first.", "No Gateway controller runs in this simulator, so these tasks are graded on the objects.", ], "docs": ["https://gateway-api.sigs.k8s.io/guides/user-guides/http-routing/", "https://kubernetes.io/docs/concepts/services-networking/gateway/"], }, { "id": "5.5", "key": "ingress", "domain": "NET", "text": "Know how to use Ingress controllers and Ingress resources", "tasks": [13, 55], "know": [ "An Ingress routes HTTP(S) by host and path to Services of its own Namespace. It does nothing without an ingress controller, chosen with ingressClassName (an IngressClass names the controller).", "Each path has a pathType (Prefix, Exact or ImplementationSpecific) and a backend: service.name plus service.port.number or .name.", "TLS: spec.tls lists hosts and the kubernetes.io/tls Secret that holds their certificate.", ], "fast": """ k create ingress app-ingress -n NS --class=nginx \\ --rule='shop.example.com/api*=api:8080,tls=shop-tls' --rule='shop.example.com/*=web:80,tls=shop-tls' openssl req -x509 -nodes -newkey rsa:2048 -days 365 -subj '/CN=shop.example.com' -keyout tls.key -out tls.crt k create secret tls shop-tls -n NS --cert=tls.crt --key=tls.key k get ingress,ingressclass -A; k describe ingress app-ingress -n NS """, "traps": [ "In --rule a trailing * means pathType Prefix; without it the path is Exact.", "The backend port is the Service's port, not the container's.", "No ingress controller runs in this simulator, so Ingress tasks are graded on the object - on the exam, curl it.", ], "docs": ["https://kubernetes.io/docs/concepts/services-networking/ingress/", "https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/"], }, { "id": "5.6", "key": "coredns", "domain": "NET", "text": "Understand and use CoreDNS", "tasks": [14, 37, 57, 101], "know": [ "CoreDNS is a Deployment in kube-system behind the Service kube-dns; its configuration, the Corefile, is in ConfigMap coredns: the `kubernetes` plugin answers for cluster.local, `forward` sends the rest upstream.", "Names: SVC.NS.svc.cluster.local for a Service, POD.SVC.NS.svc.cluster.local for the Pods of a headless Service.", "A Pod's /etc/resolv.conf points at the kube-dns address and lists the search domains that make short names work. dnsPolicy decides: ClusterFirst (default), Default (the node's resolver), None with dnsConfig.", ], "fast": """ k -n kube-system get deploy coredns; k -n kube-system get pods -l k8s-app=kube-dns -o wide k -n kube-system get svc kube-dns; k -n kube-system get endpointslices -l k8s-app=kube-dns k -n kube-system get cm coredns -o yaml; k -n kube-system edit cm coredns k -n kube-system logs -l k8s-app=kube-dns; k -n kube-system rollout restart deploy coredns k run tmp --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc.cluster.local k exec POD -- cat /etc/resolv.conf """, "traps": [ "A typo in the Corefile puts CoreDNS in CrashLoopBackOff - its log names the plugin and the line.", "DNS can be broken in several places at once: the Corefile, the Deployment (replicas, image), the kube-dns Service (selector, ports), a NetworkPolicy.", "A Pod with dnsPolicy: Default resolves the internet but not cluster names.", ], "docs": ["https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/", "https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/", "https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/"], }, ]