diff --git a/CHANGELOG.md b/CHANGELOG.md index 9fed8d2..1d411f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,78 @@ Notable changes to kubestronaut-sim, newest first. A version is the `VERSION` st (year.month.day.n); `./ksim help` and the top bar of the web page show the one you are running. Every entry names the commit it came in. +## Unreleased — 2026.10.19.1, on branch `feat/lessons` + +### Changed + +- **CKA and CKS run Kubernetes v1.37** (CKAD already did). CKA follows the real exam; CKS runs + ahead of it - the Linux Foundation still listed v1.35 for CKS on 2026-10-19. The CKA curriculum + is unchanged (`CKA_Curriculum_v1.37.pdf` is the v1.35 document). Tasks and notes that named + 1.35/1.36 now name 1.37 and 1.38, and the etcd ones etcd 3.7. + - CKS Q37 and Q39 hold their worker back on **v1.36** - the v1.37 repository has nothing older + than the cluster's 1.37.0 - so upgrading it is a minor upgrade: the repository line first. +- **Exam hosts and the desktop are Ubuntu 24.04**, not Debian. The host image is kind's node + rebuilt on `ubuntu:24.04`: kind's packages and systemd setup, and its containerd, runc, crictl, + CNI plugins and preloaded images copied from the pinned kind image. The desktop gets Firefox ESR + from Mozilla's APT repository (Ubuntu's own `firefox` is a snap). +- **Every host has Kubernetes as held apt packages** from pkgs.k8s.io, with the repository line of + its minor, as an exam node has - before, only CKS hosts did. The `kubeadm upgrade` wrapper moved + into the common image. Hosts also gained man pages, `hexdump` (bsdextrautils) and `xxd`. +- A task or lesson may name a Kubernetes minor (`"k8s"`) or host flavour (`"image"`) of its own; + `./ksim build` builds what any of them names. v1.36 is pinned for the upgrade task and lesson. +- `./ksim build --vm` builds VM disks only for the tracks that have VM tasks. + +### Added + +- **CKA lessons for all 27 objectives** (`cka/lessons_storage.py`, `_tshoot`, `_work`, `_arch`, + `_net`; 135 steps). Besides each competency, they cover what a full CKA course teaches under it + - checked against the outline of KubeDojo's CKA course, written from scratch: projected volumes, + volume expansion, snapshots and restores, eviction thresholds, debug containers, kube-proxy's + chains, encryption at rest, in-place resize, + PodDisruptionBudgets, QoS classes, ServiceAccount tokens, a real upgrade from v1.36 to v1.37 after + an etcd backup and restore, an HA join, EndpointSlices, an operator. `./ksim lesson audit`: 135 + of 135 steps proven. +- **CKA tasks 132-145** (`cka/bank_course.py`) on the same topics: a real cluster upgrade to + v1.37.1, a drain under a PodDisruptionBudget, a container without a shell, ServiceAccount tokens + and automounting, a projected volume, QoS classes, in-place resize, eviction thresholds, + kube-proxy's chains, a Service backed by a hand-written EndpointSlice, encryption at rest, + required anti-affinity (and the rollout it blocks), a StatefulSet canary by partition, an + operator whose RBAC lacks `backups/status`, and (146-147) growing a claim in use, snapshot, + restore and clone on `csi-hostpath-sc`. CKA now has 147 tasks, 112 of them in the pool. +- `ensure_csi_hostpath` for setups: the CSI hostpath driver (v1.18.0) with external-snapshotter's + CRDs and controller, its `csi-hostpath-snapclass`, and StorageClass `csi-hostpath-sc` with + `allowVolumeExpansion: true` - the class the real CKA's storage tasks use. + +- **Lessons**: guided labs, one per curriculum objective, on a host of their own and with no + clock. A lesson is an introduction, steps you do in the desktop's terminal (each with a + **Check step** button, a hint and **Show me the commands**), and a wrap-up that sends you to the + objective's timed drill. Steps count in order. A lesson writes nothing to the history or the + scores; one whose every step was checked is marked done on its objective. + - The pilot covers CKAD's Application Deployment domain: 2.1 blue/green and canary, 2.2 + rolling updates and rollbacks, 2.3 Helm, 2.4 Kustomize (`ckad/lessons.py`). + - Browser: a lesson screen in the exam screen's frame (Introduction, Step N of M, Wrap-up), a + **Lesson** button and a `lesson ✓` mark on objectives, **Lesson first** on the path card, a + *Take a lesson* session in the Start exam dialog, and a summary page after a lesson. + - Terminal: `./ksim lesson [OBJ]`, `./ksim lesson step | check | hint | show [N]`, `./ksim end`. + - `./ksim lesson audit [OBJ..]` proves lessons on real clusters: every step must fail its check, + be done by its own commands as `candidate`, then pass. CKAD 2.1–2.4: 24 of 24 steps proven. + - Engine: lessons are sessions of mode `lesson` (hosts `ckad9201`.., ids 9000 + objective), + `POST /api/lesson/check`, done stamps under `lessons` in `study-.json`, and `step()` + in `sim/bank.py` for writing them. +- Tasks and lessons render fenced code blocks (kept as written, a click copies them) and + paragraphs of `- ` lines as lists. + +### Fixed + +- CKA Q4 and Q49: their solutions wait for the new CRD to be `Established` before creating an + object of its kind (on v1.37 it lost that race: "no matches for kind"). +- CKS Q4 gave 3 of its 12 points untouched: its token check took any Pod of the Deployment, and + an untouched Deployment's Pods run as `default`, with a token mounted. It now takes a Pod that + runs as `-sa`. +- Building a host image for a kind image not pulled yet: podman's pull progress was read as the + etcd image's name. +- A solution or lesson step that prints binary data (an etcd value) no longer stops the audit. + ## 2026.10.15.3 — 2026-10-15 ### Added diff --git a/README.md b/README.md index e1cf689..6de7994 100644 --- a/README.md +++ b/README.md @@ -35,15 +35,19 @@ restore could only be drilled alone). All of that is gone: - tasks are provisioned, seeded and graded in parallel; - you work as `candidate` on the task's host and use `sudo -i` for node work, as on the day. -Each host is a kind node built from `images/node/` — a kubeadm cluster (real etcd, static-pod -control plane, systemd kubelet) with sshd, `candidate` with passwordless sudo, `k` and kubectl -completion preset, vim (2-space tabs), tmux, jq, yq, helm, podman (also as `docker`) for the -image tasks, and `etcdctl`/`etcdutl` at the version of the cluster's etcd. +Each host is a kind node built from `images/node/` on **Ubuntu 24.04**, as the exam's hosts are — +a kubeadm cluster (real etcd, static-pod control plane, systemd kubelet). kubelet, kubeadm and +kubectl are apt packages from pkgs.k8s.io, held, with the repository line of the cluster's minor +(the CNI plugins come with them, as `kubernetes-cni`): a node is upgraded with `apt-get install` +and `kubeadm upgrade`, for real, and a task can hold one back on an older release. containerd, +runc, crictl, kind's entrypoint and the preloaded control-plane images are copied from the pinned +kind node image, which kind builds on Debian. On top: sshd, `candidate` with passwordless sudo, +`k` and kubectl completion preset, vim (2-space tabs), tmux, jq, yq, helm, man pages, podman (also +as `docker`) for the image tasks, and `etcdctl`/`etcdutl` at the version of the cluster's etcd. **CKS hosts** add what that exam's hosts carry (`images/node-cks/`, built on the common image): -kube-bench, trivy, kubesec, kube-linter, bom, cosign, istioctl, gVisor (`runsc`, as a containerd -runtime handler) and Kubernetes as apt packages from pkgs.k8s.io — so a node can be held back on -an older patch release and upgraded with `apt-get install` and `kubeadm upgrade`, for real. +kube-bench, trivy, kubesec, kube-linter, bom, cosign, istioctl and gVisor (`runsc`, as a containerd +runtime handler). **VM hosts.** A container shares this machine's kernel, and under rootless podman it can neither load eBPF programs nor use a kernel feature this machine lacks: no Falco, no AppArmor. Tasks that @@ -77,8 +81,9 @@ refused before anything is created if a (more cautious) estimate does not fit (` overrides). Clusters are provisioned one per four cores at a time, at most 8 (`KSIM_PARALLEL=N`). A VM host is given 2 GB (it used about 1.6 GB right after starting) and is up in about 30 s. -Disk: the images take 1.3 GB per Kubernetes minor, 2.8 GB for the CKS host, 1.2 GB for the -desktop; the VM host adds a 3.5 GB base disk and the 0.6 GB Ubuntu cloud image under `.state/vm/`. +Disk: the images take 1.5 GB per Kubernetes minor (v1.37, and v1.36 for the tasks that upgrade +their cluster), 2.7 GB for the CKS host, 1.2 GB for the desktop; the VM host adds a 3.5 GB base +disk and the 0.6 GB Ubuntu cloud image under `.state/vm/`. ## Run @@ -130,6 +135,8 @@ host name such as `ckad0013` only resolves in the desktop's terminal: from your ./ksim sim c|d # a fixed simulator set (CKA: c, d · CKAD: a) ./ksim objectives | next | focus # the curriculum as a path, one objective at a time against the clock ./ksim notes 3.2 # an objective's notes: what to know, fastest commands, traps, docs +./ksim lesson [OBJ] # the track's lessons, or start one: a guided lab, no clock (CKA: all 27 · CKAD: 2.1-2.4) +./ksim lesson step | check | hint | show [N] # inside a lesson; ./ksim end finishes it ./ksim skip DESIGN | unskip DESIGN # "I am good with this domain (or objective 2.3)": off the path, and back ./ksim drill OBS:3 | next OBS:3 # three tasks of a domain, each from another objective - alone, or added ./ksim list | show N | grade N | solution N | timer | end @@ -141,8 +148,26 @@ host name such as `ckad0013` only resolves in the desktop's terminal: from your ./ksim coverage # tasks per domain and objective against the 17–20 target ./ksim audit [all|DOMAIN|N..] # bank check: seed each task on its own cluster, grade it untouched, # then solve it with its script and grade again (--settle N, --keep) +./ksim lesson audit [OBJ..] # lesson check: every step fails, is done by its own commands, then passes ``` +**Lessons.** A lesson teaches one objective as a guided lab on a host of its own: an introduction, +then steps you do in the desktop's terminal, each with a **Check step** button, a hint and **Show me +the commands**, and a wrap-up that sends you to the objective's timed drill. Lessons are the one +untimed part of the simulator — they teach, the drills test — and they write nothing to the +history, the statistics or the weak list; a lesson whose every step was checked is marked done on +its objective (`lesson ✓` in the objective list). Steps count in order. Start one from an +objective's **Lesson** button, **Lesson first** on the path card, the *Take a lesson* session in the +Start exam dialog, or `./ksim lesson 2.1`. **CKA has a lesson for each of its 27 objectives** +(`cka/lessons_*.py`, 135 steps), which also cover what a full CKA course teaches under them — +checked against the outline of KubeDojo's CKA course and written from scratch: projected volumes, +volume expansion, snapshots and restores, eviction thresholds, debug containers, kube-proxy's +rules, encryption at rest, in-place resize, PodDisruptionBudgets, QoS classes, ServiceAccount +tokens, a real upgrade from v1.36 to v1.37, EndpointSlices, an operator. Tasks 132–147 test those +topics against the clock. The CKAD pilot +covers the Application Deployment domain (2.1 blue/green and canary, 2.2 rolling updates, 2.3 +Helm, 2.4 Kustomize). + **Skipping what you know.** `./ksim skip DESIGN` (or the "I know this — skip it" link beside a domain in the browser's objective list) marks a domain or an objective as known: the path, the no-argument drill and `drill all` leave it out, and it no longer has to be drilled before a full @@ -163,29 +188,33 @@ Each track is a directory: `cka/`, `ckad/`, `cks/`. | `curriculum.py` | the exam's objectives in order, each with study notes and the tasks that practise it | | `solutions.py` | reference solutions (CKA keeps them apart; the others inline them) | | `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved | +| `lessons.py` | guided labs, one per objective (CKA: all, one `lessons_*.py` per domain · CKAD: 2.1–2.4): introduction, steps with their check, hint and commands, wrap-up | | Track | Kubernetes | Bank today | Target (17–20 pool tasks per domain) | |---|---|---|---| -| CKA | v1.35 | 131 tasks: 96 pool (17–25 per domain) + sets C, D (35) | reached in all five domains | +| CKA | v1.37 | 147 tasks: 112 pool (18–28 per domain) + sets C, D (35) | reached in all five domains | | CKAD | v1.37 | 105 tasks: 85 pool (17 per domain) + set A (20) | reached in all five domains | -| CKS | v1.35 | 110 tasks, all pool: 17 per domain on container hosts, plus 5 Falco and 3 AppArmor tasks on VM hosts | reached in all six domains | +| CKS | v1.37 | 110 tasks, all pool: 17 per domain on container hosts, plus 5 Falco and 3 AppArmor tasks on VM hosts | reached in all six domains | `./ksim coverage` prints this per domain and per objective. -**CKA, as audited on 2026-10-13:** all 131 tasks seed cleanly on a host of their own, score -nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted -solution is run on the host as `candidate` — for 66 of them that script is the reference solution -exactly as a drill shows it once the task is graded. +**CKA, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 147 tasks seed cleanly on a host of +their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when +their scripted solution is run on the host as `candidate` — for 80 of them that script is the +reference solution exactly as a drill shows it once the task is graded. Q132 really upgrades its +cluster from v1.36.4 to v1.37.1, worker included. -**CKAD, as audited on 2026-10-12:** all 105 tasks seed cleanly on a host of their own, score -nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted -solution is run on the host as `candidate`. +**CKAD, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 105 tasks seed cleanly on a host of +their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when +their scripted solution is run on the host as `candidate`. -**CKS, as audited on 2026-10-12:** all 110 tasks seed cleanly on a host of their own, score nothing -before anyone has worked on them, and reach full marks when their scripted solution is run on the -host as `candidate`. The 26 objectives are the curriculum's own (`cks/curriculum.py`), each with -notes, commands, traps and the documentation the exam allows — which, per the Linux Foundation's -list read that day, is the Kubernetes docs and blog plus Falco, bom, etcd, the NGINX Ingress +**CKS, as audited on 2026-10-19** (v1.37, Ubuntu hosts): 109 of 110 tasks seed cleanly on a host +of their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks +when their scripted solution is run on the host as `candidate`. Q103, a Falco task on a VM host, +failed to seed once — Falco raised no alert naming the Pod within a minute — and is still to be +audited again. The 26 objectives are the curriculum's own (`cks/curriculum.py`), each with notes, +commands, traps and the documentation the exam allows — which, per the Linux Foundation's list +read on 2026-10-12, is the Kubernetes docs and blog plus Falco, bom, etcd, the NGINX Ingress Controller, Cilium and Istio. Trivy's and AppArmor's own documentation are not on it. ### A task @@ -197,7 +226,8 @@ Controller, Cilium and Istio. Trivy's and AppArmor's own documentation are not o "setup": "bash that seeds the scenario on the task's own cluster", "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...], "solution": "...", - "workers": 2} # optional: worker nodes of its cluster + "workers": 2, # optional: worker nodes of its cluster + "k8s": "v1.36"} # optional: a minor of its own (an upgrade task) ``` Setup and check scripts run on this machine against the task's own cluster and get: @@ -213,7 +243,9 @@ Setup and check scripts run on this machine against the task's own cluster and g its checks compare against later (the UID of an object that must not be re-created, say); - `podman ...` — root's podman on the task's host, the store the candidate's `podman`/`docker` use; - `j '{jsonpath}'`, `eq a b`, `mkns` (from `sim.bank.NS`), `ensure_gateway_crds`, - `ensure_metrics_server`. + `ensure_metrics_server`, and `ensure_csi_hostpath` — the CSI hostpath driver with the snapshot + controller, and StorageClass `csi-hostpath-sc` (expandable) as the exam's clusters have it, for + what local-path cannot do: growing a claim, snapshots, clones. Task text may use `{WS}` (`/opt/course`), `{CP}`, `{W1}`.., and `{SHELL_CP}`, `{SHELL_W1}`.. (`ssh cka0114`, `ssh cka0114-worker`). @@ -242,15 +274,32 @@ In `cka/solve.py`, `AS_WRITTEN` lists the tasks whose reference solution is a sc for those the audit runs exactly the text the candidate is shown, so a solution that stops working with a new Kubernetes release fails the next audit instead of misleading you after a drill. +**Writing a lesson.** A lesson in `/lessons.py` is a dict with `id` (9000 plus the objective, +`9201` for 2.1 — its host is `ckad9201`, which no task's can be), `objective`, `title`, `workers`, +`intro`, `setup` (bash, as a task's), `steps` and `outro` - and, like a task, `k8s` or `image` if +it needs a host of its own kind (lesson 4.4 runs on v1.36 to upgrade it). Each step comes from +`step(title, text, check, hint, show)` in `sim/bank.py`: `check` runs like a task's check, and +`show` is what the learner would type. `./ksim lesson audit` seeds each lesson on its own host and, +step by step, requires the check to fail, runs `show` there as `candidate`, and requires the check +to pass — so a lesson can always be done exactly as it shows, and no step is done before you do it. +Work on another node is shown as `ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain +`ssh NODE` would read the rest of the commands as its own input. **As audited on 2026-10-19** +(v1.37, Ubuntu hosts): CKA's 27 lessons, 135 of 135 steps, and CKAD 2.1–2.4, 24 of 24 steps, +proven. + ## What is different from the real exam -- The desktop's base machine and the container hosts are Debian, not Ubuntu; `apt`, systemd and - kubeadm behave the same. VM hosts are Ubuntu 24.04. +- The desktop, the container hosts and the VM hosts are Ubuntu 24.04 LTS. The Linux Foundation does + not publish which Ubuntu release the exam runs; 24.04 is the current LTS. The container hosts' + containerd and runc are kind's builds, not Ubuntu's packages. +- CKA and CKAD run v1.37, as the real exams do. CKS runs v1.37 too, ahead of the real exam: on + 2026-10-19 the Linux Foundation still listed v1.35 for it. - Container hosts share this machine's kernel. That is invisible for CKA and CKAD, and for most of CKS — gVisor, seccomp, Istio's sidecars, kube-bench, real kubeadm upgrades all work in them. Falco and AppArmor do not, which is what the VM hosts are for. - No Cilium runs anywhere (kindnet is the CNI, and it does enforce NetworkPolicy): the one Cilium - task is graded on the object. No Ingress controller either: Ingress tasks are graded on the + task is graded on the object. The CSI hostpath driver is installed only on the hosts of the tasks + and lessons that use it. No Ingress controller either: Ingress tasks are graded on the objects, with their certificates checked by openssl. - On a container host `kubeadm upgrade` passes one preflight check it would fail (a rootless container has no cpuset cgroup controller); a wrapper adds the flag for that, nothing else. diff --git a/cka/bank_course.py b/cka/bank_course.py new file mode 100644 index 0000000..3ccba9c --- /dev/null +++ b/cka/bank_course.py @@ -0,0 +1,1033 @@ +"""CKA pool tasks 132-147: what a full CKA course teaches beyond the bank as it was - checked on +2026-10-19 against the topics of KubeDojo's CKA course, and written here from scratch. Each has a +lesson that teaches it (cka/lessons_*.py); these are the timed tasks that test it. + +132 upgrades its cluster from v1.36 to v1.37 for real ("k8s": "v1.36"). 133 drains a node under a +PodDisruptionBudget, 134 debugs a container without a shell, 135 issues and withholds +ServiceAccount tokens, 136 builds a projected volume, 137 reads QoS classes, 138 resizes a Pod in +place, 139 sets the kubelet's eviction thresholds, 140 follows a Service into kube-proxy's rules, +141 backs a Service with a hand-made EndpointSlice, 142 encrypts Secrets at rest, 143 spreads +replicas with anti-affinity, 144 rolls a StatefulSet out behind a partition, 145 makes an +operator with broken RBAC do its job, and 146-147 grow, snapshot, restore and clone volumes on +the CSI hostpath driver's class `csi-hostpath-sc`, as the real exam's clusters have it +(ensure_csi_hostpath). +""" +from cka.common import C, CAN, NET, NS, ONE, backup_operator, ns_pick, name_pick + +_ETCD = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt " + "--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key") + +# packages of a node, and whether they are held +APT = r""" +pkgs() { nx "$1" "dpkg-query -W -f='\${Version} ' kubelet kubeadm kubectl" /dev/null +$K -n %{ns} rollout status deploy web --timeout=180s >/dev/null +""", + "checks": [ + C("the API server runs v1.37.1", 3, r""" +eq "$($K version -o json 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["serverVersion"]["gitVersion"])')" v1.37.1"""), + C("{CP}: kubelet v1.37.1; kubelet, kubeadm, kubectl packages 1.37.1 and held", 2, APT + r""" +eq "$(j node "$CP" '{.status.nodeInfo.kubeletVersion}')" v1.37.1 && eq "$(pkgs "$CP")" "1.37.1-1.1 1.37.1-1.1 1.37.1-1.1 " && held "$CP" """), + C("{W1}: kubelet v1.37.1; kubelet, kubeadm, kubectl packages 1.37.1 and held", 3, APT + r""" +eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" v1.37.1 && eq "$(pkgs "$W1")" "1.37.1-1.1 1.37.1-1.1 1.37.1-1.1 " && held "$W1" """), + C("both nodes Ready and schedulable, Deployment web available", 2, r""" +for n in "$CP" "$W1"; do + eq "$(j node "$n" '{.status.conditions[?(@.type=="Ready")].status}')" True && [ "$(j node "$n" '{.spec.unschedulable}')" != true ] || exit 1 +done +eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" v1.37.1 && eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" 2"""), + ], + "solution": """ +# the control plane - this host +sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list # one repository per minor +sudo apt-get update +apt-cache madison kubeadm | head -3 +sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm +sudo kubeadm upgrade plan +sudo kubeadm upgrade apply v1.37.1 -y # a few minutes +k drain {CP} --ignore-daemonsets --delete-emptydir-data +sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl +sudo systemctl daemon-reload && sudo systemctl restart kubelet +k uncordon {CP} +# the worker +k drain {W1} --ignore-daemonsets --delete-emptydir-data +{SHELL_W1} +sudo -i +sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list +apt-get update +apt-mark unhold kubeadm kubelet kubectl +apt-get install -y kubeadm=1.37.1-1.1 +kubeadm upgrade node # a worker: node, not apply +apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 +apt-mark hold kubeadm kubelet kubectl +systemctl daemon-reload && systemctl restart kubelet +exit; exit +k uncordon {W1} +k get nodes +# docs: "Upgrading kubeadm clusters". pkgs.k8s.io has one repository per minor, so 1.37 packages +# appear only once the line names v1.37. kubeadm first - it drives the upgrade and rewrites the +# kubelet's configuration; kubelet and kubectl after `upgrade apply`, then the restart: `k get +# nodes` shows each kubelet's running version. Workers run `kubeadm upgrade node`. +""", +}, +# ------------------------------------------------------------------ 133 ---- +{ + "id": 133, "domain": "WORK", "workers": 2, "title": "Maintenance under a disruption budget", + "params": lambda r: {"ns": ns_pick(r, "orders"), "dep": name_pick(r, "checkout")}, + "task": """ +Node `{W1}` goes down for maintenance tonight. Deployment `%{dep}` in `%{ns}` (3 replicas) must +never have fewer than **2** Pods available while that happens - not for a drain, and not for any +other voluntary disruption. + +1. Create a PodDisruptionBudget `%{dep}` in `%{ns}` that guarantees it. +2. Drain `{W1}`. DaemonSet Pods stay where they are; Pod `scratch` in `%{ns}` is disposable. +3. Leave `{W1}` unschedulable. All three Pods of `%{dep}` must be running, on other nodes. +""", + "setup": NS + r""" +mkns %{ns} +$K uncordon "$W1" >/dev/null 2>&1 +$K -n %{ns} apply -f - >/dev/null </dev/null && $K -n %{ns} wait --for=condition=Ready pod/scratch --timeout=120s >/dev/null +""", + "checks": [ + C("PDB %{dep} keeps 2 of the 3 Pods available", 2, r""" +eq "$(j -n %{ns} pdb %{dep} '{.spec.selector.matchLabels.app}')" %{dep} || exit 1 +m=$(j -n %{ns} pdb %{dep} '{.spec.minAvailable}'); u=$(j -n %{ns} pdb %{dep} '{.spec.maxUnavailable}') +[ "$m" = 2 ] || [ "$u" = 1 ]"""), + C("{W1} is drained and stays unschedulable", 2, r""" +eq "$(j node "$W1" '{.spec.unschedulable}')" true || exit 1 +[ -z "$($K -n %{ns} get pods -o jsonpath="{range .items[?(@.spec.nodeName==\"$W1\")]}{.metadata.name}{end}" 2>/dev/null)" ]"""), + C("all 3 Pods of %{dep} are available on other nodes", 3, r""" +eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 || exit 1 +for n in $($K -n %{ns} get pods -l app=%{dep} -o jsonpath='{.items[*].spec.nodeName}'); do [ "$n" != "$W1" ] || exit 1; done +eq "$(j node "$W1" '{.spec.unschedulable}')" true"""), + ], + "solution": """ +k -n %{ns} create pdb %{dep} --selector=app=%{dep} --min-available=2 +k -n %{ns} get pods -o wide # which of them run on {W1} +k drain {W1} --ignore-daemonsets --delete-emptydir-data --force +k -n %{ns} get pods -o wide +# drain = cordon + evict. Evictions respect the PDB: with 3 available and minAvailable 2, one Pod +# may go at a time, and the drain retries the others until their replacements are ready. --force +# is needed for `scratch`, which no controller would re-create - it is deleted for good. Leave the +# node cordoned: `k uncordon {W1}` is for after the maintenance. +""", +}, +# ------------------------------------------------------------------ 134 ---- +{ + "id": 134, "domain": "TSHOOT", "workers": 1, "title": "A container you cannot exec into", + "params": lambda r: {"ns": ns_pick(r, "vault"), "pod": name_pick(r, "sealer"), "mode": r.choice(["sealed-3", "sealed-7", "sealed-9"])}, + "task": """ +Pod `%{pod}` in `%{ns}` runs an image that holds nothing but its own program: `kubectl exec` finds +no shell. Its container `app` takes its environment from ConfigMap `%{pod}-env` - which someone has +edited since the container started. + +Without restarting or replacing the Pod: + +1. Attach an ephemeral container named `debugger` (image `busybox:1.36`) that sees the processes + of container `app`. +2. Find out which value of `MODE` the running process of `app` actually has, and write it to + `{WS}/q134/mode.txt`. +""", + "setup": NS + r""" +mkns %{ns}; rm -rf "$WS/q134"; mkdir -p "$WS/q134" +$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true --grace-period=1 >/dev/null 2>&1 +$K -n %{ns} create configmap %{pod}-env --from-literal=MODE=%{mode} --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: Pod +metadata: {name: %{pod}} +spec: + containers: + - name: app + image: registry.k8s.io/pause:3.10.2 + envFrom: [{configMapRef: {name: %{pod}-env}}] +EOF +$K -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=120s >/dev/null || exit 1 +$K -n %{ns} create configmap %{pod}-env --from-literal=MODE=open --dry-run=client -o yaml | $K apply -f - >/dev/null +j -n %{ns} pod %{pod} '{.metadata.uid}' > "$PRIV/q134.uid" +""", + "checks": [ + C("ephemeral container debugger (busybox:1.36) targets app - in the same Pod, never restarted", 3, r""" +eq "$(j -n %{ns} pod %{pod} '{.spec.ephemeralContainers[?(@.name=="debugger")].targetContainerName} {.spec.ephemeralContainers[?(@.name=="debugger")].image}')" "app busybox:1.36" \ + && eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q134.uid") 0" """), + C("mode.txt holds the value the process runs with", 3, ONE + r"""eq "$(one "$WS/q134/mode.txt")" %{mode}"""), + ], + "solution": """ +k -n %{ns} exec %{pod} -- sh # fails: no shell in the image +k -n %{ns} get cm %{pod}-env -o yaml # MODE=open - but that is the ConfigMap NOW +k -n %{ns} debug %{pod} --image=busybox:1.36 --target=app -c debugger --profile=general -- sleep 3600 +for i in $(seq 1 30); do k -n %{ns} exec %{pod} -c debugger -- true 2>/dev/null && break; sleep 2; done +k -n %{ns} exec %{pod} -c debugger -- ps # PID 1 is /pause, the app's process +k -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\\0" "\\n" < /proc/1/environ' | grep '^MODE=' +k -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\\0" "\\n" < /proc/1/environ' | sed -n 's/^MODE=//p' > {WS}/q134/mode.txt +# Environment variables from a ConfigMap are read once, when the container starts: editing the +# ConfigMap changes nothing for a running process. --target puts the debug container into app's +# process namespace, so /proc/1 is app's process; the `general` profile gives it SYS_PTRACE, +# which reading another user's /proc/PID/environ needs. Interactively: `k debug -it ... -- sh`. +""", +}, +# ------------------------------------------------------------------ 135 ---- +{ + "id": 135, "domain": "ARCH", "title": "A token for the pipeline, none for the app", + "params": lambda r: {"ns": ns_pick(r, "ci"), "h": r.choice([2, 3, 4])}, + "task": """ +In Namespace `%{ns}`: + +1. Create ServiceAccount `pipeline`, and allow it - through a Role and a RoleBinding, both named + `pipeline` - to `get`, `list`, `create`, `update` and `patch` Deployments, and nothing else. +2. Issue a token for `pipeline` that is valid for **%{h} hours**, and save just the token in + `{WS}/q135/pipeline.token`. +3. Deployment `frontend` never talks to the API server. Change it so that its Pods get **no** + ServiceAccount token at all. It keeps running as the `default` ServiceAccount. +""", + "setup": NS + r""" +mkns %{ns}; rm -rf "$WS/q135"; mkdir -p "$WS/q135" +$K -n %{ns} create deploy frontend --image=nginx:1.27 --replicas=2 >/dev/null +$K -n %{ns} rollout status deploy frontend --timeout=180s >/dev/null +""", + "checks": [ + C("pipeline may get, list, create, update, patch Deployments - nothing more", 2, CAN + r""" +S=system:serviceaccount:%{ns}:pipeline +for v in get list create update patch; do can --as=$S -- $v deployments.apps -n %{ns} || exit 1; done +cannot --as=$S -- delete deployments.apps -n %{ns} && cannot --as=$S -- get secrets -n %{ns} && cannot --as=$S -- list pods -n %{ns} \ + && eq "$(j -n %{ns} rolebinding pipeline '{.roleRef.kind}/{.roleRef.name}')" Role/pipeline"""), + C("pipeline.token authenticates as pipeline and lasts %{h} hours", 3, r""" +t=$(tr -d ' \r\n' < "$WS/q135/pipeline.token" 2>/dev/null); [ -n "$t" ] || exit 1 +srv=$($K config view --minify -o jsonpath='{.clusters[0].cluster.server}') +eq "$($K --kubeconfig=/dev/null --server="$srv" --insecure-skip-tls-verify --token="$t" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)" system:serviceaccount:%{ns}:pipeline || exit 1 +echo "$t" | cut -d. -f2 | tr '_-' '/+' | python3 -c ' +import base64, json, sys +s = sys.stdin.read().strip(); c = json.loads(base64.b64decode(s + "=" * (-len(s) % 4))) +sys.exit(0 if c["exp"] - c["iat"] == %{h} * 3600 else 1)'"""), + C("frontend's Pods run with no ServiceAccount token, as default", 3, r""" +eq "$(j -n %{ns} deploy frontend '{.status.availableReplicas} {.status.updatedReplicas}')" "2 2" || exit 1 +sa=$(j -n %{ns} deploy frontend '{.spec.template.spec.serviceAccountName}'); [ -z "$sa" ] || [ "$sa" = default ] || exit 1 +for p in $($K -n %{ns} get pods -l app=frontend -o name); do + $K -n %{ns} get "$p" -o json | python3 -c ' +import json, sys; s = json.load(sys.stdin)["spec"] +tok = [v for v in s.get("volumes", []) if any("serviceAccountToken" in src for src in v.get("projected", {}).get("sources", []))] +sys.exit(1 if tok else 0)' || exit 1 +done"""), + ], + "solution": """ +k -n %{ns} create serviceaccount pipeline +k -n %{ns} create role pipeline --verb=get,list,create,update,patch --resource=deployments.apps +k -n %{ns} create rolebinding pipeline --role=pipeline --serviceaccount=%{ns}:pipeline +k auth can-i delete deployments -n %{ns} --as=system:serviceaccount:%{ns}:pipeline # no +k -n %{ns} create token pipeline --duration=%{h}h > {WS}/q135/pipeline.token +k -n %{ns} patch deploy frontend -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":false}}}}' +k -n %{ns} rollout status deploy frontend +# `create token` asks the TokenRequest API for a short-lived, bound token - nothing is stored in a +# Secret. automountServiceAccountToken: false on the Pod template (or on the ServiceAccount; the +# Pod's setting wins) leaves out the projected volume kube-api-access-*. To try a token alone, use +# no kubeconfig: a client certificate in yours would be checked first. +""", +}, +# ------------------------------------------------------------------ 136 ---- +{ + "id": 136, "domain": "STOR", "title": "One mount, four sources", + "params": lambda r: {"ns": ns_pick(r, "portal"), "pod": name_pick(r, "portal"), "mem": r.choice([96, 128, 160])}, + "task": """ +Create Pod `%{pod}` in `%{ns}`: image `busybox:1.36`, command `sleep 86400`, label `app=%{pod}`, +memory limit `%{mem}Mi`. Mount a **single** volume, read-only, at `/etc/portal`, that holds: + +- every key of ConfigMap `portal-config`, each as a file of its name; +- key `password` of Secret `portal-db`, as the file `db/password`; +- the Pod's labels, as the file `labels`; +- the container's memory limit in MiB, as the file `mem-limit` (just the number). +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true --grace-period=1 >/dev/null 2>&1 +$K -n %{ns} create configmap portal-config --from-literal=theme=dark --from-literal=locale=en_GB --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} create secret generic portal-db --from-literal=password=pw-%{mem}-x --from-literal=user=portal --dry-run=client -o yaml | $K apply -f - >/dev/null +""", + "checks": [ + C("one projected volume, read-only at /etc/portal", 2, r""" +eq "$(j -n %{ns} pod %{pod} '{.spec.containers[0].volumeMounts[?(@.mountPath=="/etc/portal")].readOnly}')" true || exit 1 +v=$(j -n %{ns} pod %{pod} '{.spec.containers[0].volumeMounts[?(@.mountPath=="/etc/portal")].name}') +[ -n "$(j -n %{ns} pod %{pod} "{.spec.volumes[?(@.name==\"$v\")].projected.sources}")" ]"""), + C("the ConfigMap's keys and db/password are files there", 2, r""" +x() { $K -n %{ns} exec %{pod} -- cat "/etc/portal/$1" 2>/dev/null; } +eq "$(x theme)" dark && eq "$(x locale)" en_GB && eq "$(x db/password)" pw-%{mem}-x && ! $K -n %{ns} exec %{pod} -- test -e /etc/portal/user"""), + C("labels and mem-limit come from the downward API", 3, r""" +x() { $K -n %{ns} exec %{pod} -- cat "/etc/portal/$1" 2>/dev/null; } +x labels | grep -qx 'app="%{pod}"' && eq "$(x mem-limit)" %{mem}"""), + ], + "solution": """ +k -n %{ns} apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: %{pod} + labels: {app: %{pod}} +spec: + containers: + - name: portal + image: busybox:1.36 + command: [sleep, '86400'] + resources: + limits: {memory: %{mem}Mi} + volumeMounts: + - {name: portal, mountPath: /etc/portal, readOnly: true} + volumes: + - name: portal + projected: + sources: + - configMap: + name: portal-config + - secret: + name: portal-db + items: + - {key: password, path: db/password} + - downwardAPI: + items: + - path: labels + fieldRef: {fieldPath: metadata.labels} + - path: mem-limit + resourceFieldRef: {containerName: portal, resource: limits.memory, divisor: 1Mi} +EOF +k -n %{ns} wait --for=condition=Ready pod/%{pod} +k -n %{ns} exec %{pod} -- sh -c 'ls -R /etc/portal; cat /etc/portal/labels /etc/portal/mem-limit' +# `items` picks keys and renames them: only `password` from the Secret, as db/password. A +# resourceFieldRef needs the container's name in a volume (in an env var it is implied), and the +# divisor turns bytes into MiB. +""", +}, +# ------------------------------------------------------------------ 137 ---- +{ + "id": 137, "domain": "WORK", "title": "Who is evicted first", + "params": lambda r: {"ns": ns_pick(r, "batch")}, + "task": """ +Namespace `%{ns}` runs Pods `alpha`, `beta`, `gamma` and `delta`, and Deployment `critical`. + +1. Write the QoS class of each of the four Pods to `{WS}/q137/qos.txt`, one per line, as + `POD CLASS`. +2. When the node runs short of memory, which of the four does the kubelet evict **first**? Write + its name to `{WS}/q137/first.txt`. +3. Change Deployment `critical` so that its Pods are of class `Guaranteed`, with `200m` CPU and + `128Mi` memory. +""", + "setup": NS + r""" +mkns %{ns}; rm -rf "$WS/q137"; mkdir -p "$WS/q137" +$K -n %{ns} delete pod alpha beta gamma delta --ignore-not-found --wait=true --grace-period=1 >/dev/null 2>&1 +$K -n %{ns} apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: Pod +metadata: {name: alpha} +spec: + containers: [{name: c, image: busybox:1.36, command: [sleep, '86400'], resources: {requests: {cpu: 50m, memory: 32Mi}, limits: {cpu: 50m, memory: 32Mi}}}] +--- +apiVersion: v1 +kind: Pod +metadata: {name: beta} +spec: + containers: [{name: c, image: busybox:1.36, command: [sleep, '86400'], resources: {requests: {memory: 32Mi}}}] +--- +apiVersion: v1 +kind: Pod +metadata: {name: gamma} +spec: + containers: [{name: c, image: busybox:1.36, command: [sleep, '86400']}] +--- +apiVersion: v1 +kind: Pod +metadata: {name: delta} +spec: + containers: + - {name: main, image: busybox:1.36, command: [sleep, '86400'], resources: {requests: {cpu: 50m, memory: 32Mi}, limits: {cpu: 50m, memory: 32Mi}}} + - {name: helper, image: busybox:1.36, command: [sleep, '86400']} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: critical} +spec: + replicas: 1 + selector: {matchLabels: {app: critical}} + template: + metadata: {labels: {app: critical}} + spec: + containers: [{name: app, image: nginx:1.27, resources: {requests: {cpu: 100m, memory: 64Mi}}}] +EOF +$K -n %{ns} wait --for=condition=Ready pod --all --timeout=180s >/dev/null +""", + "checks": [ + C("qos.txt names each Pod's class", 3, r""" +eq "$(tr -s ' \t' ' ' < "$WS/q137/qos.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$(printf 'alpha Guaranteed\nbeta Burstable\ndelta Burstable\ngamma BestEffort')" """), + C("first.txt names the Pod evicted first", 2, ONE + r"""eq "$(one "$WS/q137/first.txt")" gamma"""), + C("critical's Pods are Guaranteed with 200m and 128Mi", 3, r""" +eq "$(j -n %{ns} deploy critical '{.status.availableReplicas} {.status.updatedReplicas}')" "1 1" || exit 1 +eq "$(j -n %{ns} deploy critical '{.spec.template.spec.containers[0].resources.requests.cpu} {.spec.template.spec.containers[0].resources.limits.cpu} {.spec.template.spec.containers[0].resources.requests.memory} {.spec.template.spec.containers[0].resources.limits.memory}')" "200m 200m 128Mi 128Mi" || exit 1 +for q in $($K -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'); do [ "$q" = Guaranteed ] || exit 1; done"""), + ], + "solution": """ +k -n %{ns} get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass +k -n %{ns} get pods alpha beta delta gamma -o jsonpath='{range .items[*]}{.metadata.name} {.status.qosClass}{"\\n"}{end}' > {WS}/q137/qos.txt +echo gamma > {WS}/q137/first.txt +k -n %{ns} set resources deploy critical --requests=cpu=200m,memory=128Mi --limits=cpu=200m,memory=128Mi +k -n %{ns} rollout status deploy critical +k -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'; echo +# Guaranteed: every container has requests equal to limits for both CPU and memory. BestEffort: no +# container has any. Everything else is Burstable - delta too, because one container sets nothing. +# Under memory pressure the kubelet evicts BestEffort first, then Burstable Pods using more than +# they requested; Guaranteed Pods last. +""", +}, +# ------------------------------------------------------------------ 138 ---- +{ + "id": 138, "domain": "WORK", "title": "More CPU, same Pod", + "params": lambda r: {"ns": ns_pick(r, "render"), "pod": name_pick(r, "encoder"), "req": r.choice(["250m", "300m"]), "lim": r.choice(["500m", "600m"])}, + "task": """ +Pod `%{pod}` in `%{ns}` is throttled at its CPU limit. Give its container `encoder` a CPU request +of `%{req}` and a CPU limit of `%{lim}` - **without** replacing the Pod and without restarting the +container. Its memory request and limit stay as they are. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true --grace-period=1 >/dev/null 2>&1 +$K -n %{ns} apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: Pod +metadata: {name: %{pod}} +spec: + containers: + - name: encoder + image: busybox:1.36 + command: [sh, -c, 'while true; do :; done'] + resources: {requests: {cpu: 100m, memory: 48Mi}, limits: {cpu: 200m, memory: 48Mi}} +EOF +$K -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=120s >/dev/null || exit 1 +j -n %{ns} pod %{pod} '{.metadata.uid}' > "$PRIV/q138.uid" +""", + "checks": [ + C("its spec asks for %{req} / %{lim} CPU, memory unchanged", 2, r""" +eq "$(j -n %{ns} pod %{pod} '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.limits.cpu} {.spec.containers[0].resources.requests.memory} {.spec.containers[0].resources.limits.memory}')" "%{req} %{lim} 48Mi 48Mi" """), + C("the running container has the new CPU limit", 2, r""" +eq "$(j -n %{ns} pod %{pod} '{.status.containerStatuses[0].resources.requests.cpu} {.status.containerStatuses[0].resources.limits.cpu}')" "%{req} %{lim}" """), + C("in the same Pod, and its container never restarted", 2, r""" +eq "$(j -n %{ns} pod %{pod} '{.spec.containers[0].resources.limits.cpu}')" %{lim} \ + && eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q138.uid") 0" """), + ], + "solution": """ +k -n %{ns} get pod %{pod} -o jsonpath='{.spec.containers[0].resources}'; echo +k -n %{ns} patch pod %{pod} --subresource resize --patch '{"spec":{"containers":[{"name":"encoder","resources":{"requests":{"cpu":"%{req}"},"limits":{"cpu":"%{lim}"}}}]}}' +k -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].resources}{"\\n"}{.status.containerStatuses[0].restartCount}{"\\n"}' +# A running Pod's CPU and memory can be changed in place, through the `resize` subresource - a +# plain `k edit` or `k apply` of a Pod's resources is refused. The spec +# says what you asked for; status.containerStatuses[].resources what the kubelet has applied. +# A container's resizePolicy decides whether a resource's change needs a restart (CPU: no). +""", +}, +# ------------------------------------------------------------------ 139 ---- +{ + "id": 139, "domain": "TSHOOT", "workers": 1, "title": "Evict before the node runs dry", + "params": lambda r: {"mem": r.choice([200, 250, 300]), "fs": r.choice([10, 12, 15]), "soft": r.choice([400, 500])}, + "task": """ +Node `{W1}` ran out of memory last week and froze before its kubelet did anything about it. +Configure the kubelet on `{W1}` - in its configuration file, so that it lasts - to: + +- evict Pods at once when less than `%{mem}Mi` of memory, or less than `%{fs}%` of the node + filesystem, is available; +- evict Pods when less than `%{soft}Mi` of memory has been available for **90 seconds**. + +Leave its other settings as they are. The node must be `Ready` afterwards. Reach it with +`{SHELL_W1}`. +""", + "setup": r""" +nx "$W1" 'cp -n /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.ksim 2>/dev/null; true' /dev/null | python3 -c ' +import json, sys; c = json.load(sys.stdin)["kubeletconfig"]; e = c.get("evictionHard") or {} +sys.exit(0 if e.get("memory.available") == "%{mem}Mi" and e.get("nodefs.available") == "%{fs}%" else 1)'"""), + C("soft threshold memory.available %{soft}Mi with a 90 s grace period", 3, r""" +$K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c ' +import json, re, sys; c = json.load(sys.stdin)["kubeletconfig"] +s = (c.get("evictionSoft") or {}).get("memory.available"); g = (c.get("evictionSoftGracePeriod") or {}).get("memory.available", "") +m = re.fullmatch(r"(?:(\d+)m)?(?:(\d+)s)?", g); secs = int(m.group(1) or 0) * 60 + int(m.group(2) or 0) if m and g else -1 +sys.exit(0 if s == "%{soft}Mi" and secs == 90 else 1)'"""), + C("set in the file, the other settings kept, the node Ready", 2, r""" +nx "$W1" 'grep -q "%{soft}Mi" /var/lib/kubelet/config.yaml && grep -q "%{mem}Mi" /var/lib/kubelet/config.yaml && grep -q "^staticPodPath:" /var/lib/kubelet/config.yaml && systemctl is-active -q kubelet' /dev/null +$K -n %{ns} expose deploy %{svc} --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} rollout status deploy %{svc} --timeout=180s >/dev/null +""", + "checks": [ + C("mode.txt: kube-proxy's mode", 2, ONE + r""" +m=$($K -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' | tr -d '"'); m=${m:-iptables} +eq "$(one "$WS/q140/mode.txt" | tr -d '"')" "$m" """), + C("chain.txt: the Service's KUBE-SVC chain", 3, ONE + r""" +ip=$(j -n %{ns} svc %{svc} '{.spec.clusterIP}') +want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" {WS}/q140/mode.txt +ip=$(k -n %{ns} get svc %{svc} -o jsonpath='{.spec.clusterIP}') +sudo iptables-save -t nat | grep -- "-d $ip/32" # KUBE-SERVICES ... -j KUBE-SVC-XXXX +chain=$(sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1) +echo $chain > {WS}/q140/chain.txt +sudo iptables-save -t nat | grep -- "^-A $chain " # one -j KUBE-SEP-... per endpoint +sudo iptables-save -t nat | grep -- "^-A $chain " | grep -c -- '-j KUBE-SEP-' > {WS}/q140/endpoints.txt +# KUBE-SERVICES matches ClusterIP:port and jumps to the Service's KUBE-SVC chain; that picks an +# endpoint chain at random (statistic mode random), and each KUBE-SEP chain DNATs to one Pod. +# One KUBE-SEP per ready endpoint. In nftables mode: `sudo nft list ruleset | grep -A5 %{svc}`. +""", +}, +# ------------------------------------------------------------------ 141 ---- +{ + "id": 141, "domain": "NET", "workers": 1, "title": "A Service for an API outside the cluster", + "params": lambda r: {"ns": ns_pick(r, "apps"), "port": r.choice([8080, 9090])}, + "task": """ +The billing API is not part of the cluster: it answers at the address and port written in +`{WS}/q141/billing.txt`. Give the applications in `%{ns}` a stable name for it: + +1. Create Service `billing` in `%{ns}`, port `80`, **without** a selector. +2. Its endpoints are yours to maintain: create EndpointSlice `billing-1` that sends its traffic + to that address and port. + +`wget -qO- billing` from Pod `client` in `%{ns}` must return the API's answer. +""", + "setup": NS + r""" +mkns %{ns}; mkns billing-host; rm -rf "$WS/q141"; mkdir -p "$WS/q141" +$K -n %{ns} delete svc billing --ignore-not-found >/dev/null 2>&1 +$K -n %{ns} run client --image=busybox:1.36 -- sleep 86400 >/dev/null 2>&1 +$K -n billing-host run api --image=busybox:1.36 --overrides="{\"spec\":{\"hostNetwork\":true,\"nodeName\":\"$W1\"}}" --command -- \ + sh -c 'mkdir -p /www && echo billing-api-%{port} > /www/index.html && exec httpd -f -p %{port} -h /www' >/dev/null 2>&1 +$K -n %{ns} wait --for=condition=Ready pod/client --timeout=120s >/dev/null && $K -n billing-host wait --for=condition=Ready pod/api --timeout=120s >/dev/null || exit 1 +echo "$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}'):%{port}" > "$WS/q141/billing.txt" +""", + "checks": [ + C("Service billing, port 80, has no selector", 1, r"""eq "$(j -n %{ns} svc billing '{.spec.ports[0].port}')" 80 && [ -z "$(j -n %{ns} svc billing '{.spec.selector}')" ]"""), + C("EndpointSlice billing-1 belongs to it and holds the API's address and port", 3, r""" +a=$(cut -d: -f1 "$WS/q141/billing.txt") +eq "$(j -n %{ns} endpointslice billing-1 '{.metadata.labels.kubernetes\.io/service-name} {.addressType} {.endpoints[0].addresses[0]} {.ports[0].port}')" "billing IPv4 $a %{port}" """), + C("wget billing from client returns the API's answer", 3, r""" +for i in 1 2 3 4 5; do [ "$($K -n %{ns} exec client -- wget -qO- -T 3 billing 2>/dev/null)" = billing-api-%{port} ] && exit 0; sleep 3; done; exit 1"""), + ], + "solution": """ +cat {WS}/q141/billing.txt # ADDRESS:PORT +addr=$(cut -d: -f1 {WS}/q141/billing.txt); port=$(cut -d: -f2 {WS}/q141/billing.txt) +k -n %{ns} apply -f - </dev/null +""", + "checks": [ + C("enc.yaml encrypts secrets with aescbc key1 first, identity after", 1, r""" +nx "$CP" 'cat /etc/kubernetes/enc/enc.yaml' /dev/null | python3 -c ' +import sys, yaml +c = yaml.safe_load(sys.stdin); r = [x for x in c["resources"] if "secrets" in x["resources"]][0]["providers"] +sys.exit(0 if "aescbc" in r[0] and r[0]["aescbc"]["keys"][0]["name"] == "key1" and any("identity" in p for p in r[1:]) else 1)' 2>/dev/null \ + || nx "$CP" 'grep -q aescbc /etc/kubernetes/enc/enc.yaml && grep -q "name: key1" /etc/kubernetes/enc/enc.yaml && grep -q identity /etc/kubernetes/enc/enc.yaml' /dev/null | head -c 64' /dev/null 2>&1 +$K -n %{ns} create secret generic ksim-probe --from-literal=x=y >/dev/null 2>&1 || exit 1 +nx "$CP" '""" + _ETCD + r""" get /registry/secrets/%{ns}/ksim-probe --print-value-only 2>/dev/null | head -c 64' /dev/null 2>&1; exit $r"""), + ], + "solution": """ +sudo mkdir -p /etc/kubernetes/enc +sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null </dev/null 2>&1 && break; sleep 3; done +k -n %{ns} get secret %{sec} -o json | k replace -f - # stored again: encrypted now +sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ + --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \\ + get /registry/secrets/%{ns}/%{sec} | od -c | head -3 # k 8 s : e n c : a e s c b c : v 1 : k e y 1 +# By hand: in kube-apiserver.yaml add the flag to the command, a volumeMount for /etc/kubernetes/enc +# and a hostPath volume of the same path - the API server runs in a container and sees only what is +# mounted. The FIRST provider encrypts; `identity` after it reads what was stored before. Every +# Secret: `k get secrets -A -o json | k replace -f -`. +""", +}, +# ------------------------------------------------------------------ 143 ---- +{ + "id": 143, "domain": "WORK", "workers": 3, "title": "Never two on one node", + "params": lambda r: {"ns": ns_pick(r, "edge"), "dep": name_pick(r, "gateway")}, + "task": """ +Deployment `%{dep}` in `%{ns}` runs 3 replicas, and nothing stops two of them from landing on the +same node. Change it so that: + +- no two of its Pods ever run on the same node - a hard rule, not a preference; +- among the nodes allowed, those labelled `tier=edge` are **preferred**. + +Label `{W2}` and `{W3}` with `tier=edge`. All three replicas must be running afterwards. +""", + "setup": NS + r""" +mkns %{ns} +for n in "$W1" "$W2" "$W3"; do $K label node "$n" tier- >/dev/null 2>&1; done +$K -n %{ns} create deploy %{dep} --image=nginx:1.27 --replicas=3 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} rollout status deploy %{dep} --timeout=180s >/dev/null +""", + "checks": [ + C("required pod anti-affinity on kubernetes.io/hostname for its own Pods", 3, r""" +$K -n %{ns} get deploy %{dep} -o json 2>/dev/null | python3 -c ' +import json, sys; d = json.load(sys.stdin); t = d["spec"]["template"] +a = t["spec"].get("affinity", {}).get("podAntiAffinity", {}).get("requiredDuringSchedulingIgnoredDuringExecution", []) +lab = t["metadata"]["labels"] +ok = any(x.get("topologyKey") == "kubernetes.io/hostname" and all(lab.get(k) == v for k, v in x.get("labelSelector", {}).get("matchLabels", {}).items()) and x.get("labelSelector", {}).get("matchLabels") for x in a) +sys.exit(0 if ok else 1)'"""), + C("preferred node affinity for tier=edge; {W2} and {W3} labelled", 2, r""" +eq "$(j node "$W2" '{.metadata.labels.tier}') $(j node "$W3" '{.metadata.labels.tier}')" "edge edge" || exit 1 +$K -n %{ns} get deploy %{dep} -o json 2>/dev/null | python3 -c ' +import json, sys; s = json.load(sys.stdin)["spec"]["template"]["spec"] +p = s.get("affinity", {}).get("nodeAffinity", {}).get("preferredDuringSchedulingIgnoredDuringExecution", []) +ok = any(e.get("key") == "tier" and e.get("operator") == "In" and "edge" in e.get("values", []) for x in p for e in x["preference"].get("matchExpressions", [])) +sys.exit(0 if ok else 1)'"""), + C("3 Pods of the new template running on 3 different nodes", 2, r""" +[ -n "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution}')" ] || exit 1 +eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas} {.status.updatedReplicas} {.status.replicas}')" "3 3 3" || exit 1 +[ "$($K -n %{ns} get pods -l app=%{dep} --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 3 ]"""), + ], + "solution": """ +k label node {W2} {W3} tier=edge +k -n %{ns} patch deploy %{dep} --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}},"template":{"spec":{"affinity":{ + "podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app":"%{dep}"}},"topologyKey":"kubernetes.io/hostname"}]}, + "nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"tier","operator":"In","values":["edge"]}]}}]}}}}}}' +k -n %{ns} rollout status deploy %{dep} +k -n %{ns} get pods -o wide +# The anti-affinity's labelSelector picks the Pods to stay away from - its own, by its template's +# labels - and topologyKey says per what: one per hostname = one per node. With 3 workers a 4th +# replica would stay Pending - and so does the rolling update's surge Pod, since every node already +# has one of the old Pods (same labels). The default maxUnavailable of 25% is 0 Pods for 3 +# replicas, so the rollout would wait for ever: let it replace one Pod at a time instead +# (maxSurge 0, maxUnavailable 1), in the same change. +""", +}, +# ------------------------------------------------------------------ 144 ---- +{ + "id": 144, "domain": "WORK", "title": "A canary for a StatefulSet", + "params": lambda r: {"ns": ns_pick(r, "store"), "sts": name_pick(r, "cache")}, + "task": """ +StatefulSet `%{sts}` in `%{ns}` runs `nginx:1.26` on 3 replicas. Roll out `nginx:1.27` as a canary: +only the replica with the **highest ordinal** gets the new image now; the other two keep running +`nginx:1.26`, untouched, until someone lowers the partition. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: Service +metadata: {name: %{sts}} +spec: {clusterIP: None, selector: {app: %{sts}}, ports: [{port: 80}]} +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: {name: %{sts}} +spec: + serviceName: %{sts} + replicas: 3 + selector: {matchLabels: {app: %{sts}}} + template: + metadata: {labels: {app: %{sts}}} + spec: + containers: [{name: nginx, image: nginx:1.26}] +EOF +$K -n %{ns} rollout status sts %{sts} --timeout=240s >/dev/null || exit 1 +for i in 0 1; do j -n %{ns} pod %{sts}-$i '{.metadata.uid}'; echo; done > "$PRIV/q144.uids" +""", + "checks": [ + C("updateStrategy RollingUpdate with partition 2", 2, r"""eq "$(j -n %{ns} sts %{sts} '{.spec.updateStrategy.type} {.spec.updateStrategy.rollingUpdate.partition}')" "RollingUpdate 2" """), + C("%{sts}-2 runs nginx:1.27 and is ready", 3, r""" +eq "$(j -n %{ns} sts %{sts} '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1 +eq "$(j -n %{ns} pod %{sts}-2 '{.spec.containers[0].image} {.status.containerStatuses[0].ready}')" "nginx:1.27 true" """), + C("%{sts}-0 and %{sts}-1 still run nginx:1.26, never replaced", 2, r""" +eq "$(j -n %{ns} pod %{sts}-0 '{.spec.containers[0].image}') $(j -n %{ns} pod %{sts}-1 '{.spec.containers[0].image}')" "nginx:1.26 nginx:1.26" || exit 1 +eq "$(for i in 0 1; do j -n %{ns} pod %{sts}-$i '{.metadata.uid}'; echo; done)" "$(cat "$PRIV/q144.uids")" \ + && eq "$(j -n %{ns} sts %{sts} '{.spec.template.spec.containers[0].image}')" nginx:1.27"""), + ], + "solution": """ +k -n %{ns} patch sts %{sts} -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}' +k -n %{ns} set image sts/%{sts} nginx=nginx:1.27 +k -n %{ns} rollout status sts %{sts} +k -n %{ns} get pods -l app=%{sts} -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image +# The partition FIRST: a StatefulSet updates from the highest ordinal down, and with no partition +# the new image would reach -1 and -0 within moments. Pods with an ordinal >= partition get the new +# template; the others keep theirs. Next step of the canary: partition 1, then 0. +""", +}, +# ------------------------------------------------------------------ 145 ---- +{ + "id": 145, "domain": "ARCH", "workers": 1, "title": "An operator that cannot finish its job", + "params": lambda r: {"ns": ns_pick(r, "reports"), "name": name_pick(r, "daily")}, + "task": """ +The manifests of a backup operator are in `{WS}/q145/`: its CustomResourceDefinition, its RBAC and +its controller. + +1. Install the operator. +2. Create Namespace `%{ns}` and, in it, Backup `%{name}` with the schedule `30 1 * * *`. +3. The operator should create ConfigMap `backup-%{name}` for it and set its `status.phase` to + `Scheduled`. Make sure it does - its manifests are not quite right. Change only what is wrong. +""", + "setup": r""" +rm -rf "$WS/q145"; mkdir -p "$WS/q145" +cat > "$WS/q145/operator.yaml" <<'EOF' +""" + backup_operator(crd=True, status_rbac=False) + r"""EOF +$K delete crd backups.data.example.com --ignore-not-found >/dev/null 2>&1 +$K delete ns backup-operator --ignore-not-found >/dev/null 2>&1 +""", + "checks": [ + C("the CRD is established and the controller runs", 2, r""" +eq "$(j crd backups.data.example.com '{.status.conditions[?(@.type=="Established")].status}')" True && eq "$(j -n backup-operator deploy controller '{.status.readyReplicas}')" 1"""), + C("Backup %{name} in %{ns} has the schedule", 1, r"""eq "$(j -n %{ns} backup %{name} '{.spec.schedule}')" "30 1 * * *" """), + C("the operator made ConfigMap backup-%{name}", 1, r"""eq "$(j -n %{ns} cm backup-%{name} '{.data.schedule}')" "30 1 * * *" """), + C("Backup %{name} is Scheduled: the controller may update its status", 3, r""" +eq "$(j -n %{ns} backup %{name} '{.status.phase}')" Scheduled && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller"""), + ], + "solution": """ +grep -E '^kind:|^ name:' {WS}/q145/operator.yaml +k apply -f {WS}/q145/operator.yaml +k wait --for condition=established crd/backups.data.example.com +k -n backup-operator rollout status deploy controller +k create ns %{ns} +k -n %{ns} apply -f - <<'EOF' +apiVersion: data.example.com/v1 +kind: Backup +metadata: + name: %{name} +spec: + schedule: "30 1 * * *" +EOF +sleep 10 +k -n %{ns} get cm backup-%{name} # made - but: +k -n %{ns} get backup %{name} -o jsonpath='{.status}'; echo # no phase +k -n backup-operator logs deploy/controller | tail -3 # ... cannot patch resource "backups/status" ... +k auth can-i patch backups.data.example.com --subresource=status --as=system:serviceaccount:backup-operator:controller -A +k patch clusterrole backup-operator --type json -p '[{"op":"add","path":"/rules/0/resources/-","value":"backups/status"}]' +sleep 10 +k -n %{ns} get backup %{name} -o jsonpath='{.status.phase}'; echo +# The CRD has a status subresource, and a subresource is a resource of its own for RBAC: +# `backups/status`. The controller could read Backups and write ConfigMaps, but not set a phase. +# The log of an operator's controller is where such a gap shows - here as `forbidden`. +""", +}, +# ------------------------------------------------------------------ 146 ---- +{ + "id": 146, "domain": "STOR", "workers": 1, "title": "Grow a claim that is in use", + "params": lambda r: {"ns": ns_pick(r, "media"), "size": r.choice(["70Mi", "80Mi", "100Mi"])}, + "task": """ +In Namespace `%{ns}`: + +1. Create PersistentVolumeClaim `pv-volume`: class `csi-hostpath-sc`, access mode `ReadWriteOnce`, + capacity `10Mi`. +2. Create Pod `web-server` (image `nginx:1.27`) that mounts the claim at `/usr/share/nginx/html`. +3. The site needs more room: expand the claim to `%{size}` while `web-server` keeps running. +""", + "setup": NS + r""" +ensure_csi_hostpath || { echo "could not install the CSI hostpath driver (no network?) - the task needs it" >&2; exit 1; } +mkns %{ns} +$K -n %{ns} delete pod web-server --ignore-not-found --wait=true --grace-period=1 >/dev/null 2>&1 +$K -n %{ns} delete pvc pv-volume --ignore-not-found --wait=true >/dev/null 2>&1 +""", + "checks": [ + C("pv-volume uses csi-hostpath-sc, ReadWriteOnce, and is Bound", 1, r""" +eq "$(j -n %{ns} pvc pv-volume '{.spec.storageClassName} {.spec.accessModes[0]} {.status.phase}')" "csi-hostpath-sc ReadWriteOnce Bound" """), + C("web-server runs nginx with the claim at /usr/share/nginx/html", 2, r""" +eq "$(j -n %{ns} pod web-server '{.status.phase} {.spec.containers[0].image}')" "Running nginx:1.27" || exit 1 +v=$(j -n %{ns} pod web-server '{.spec.containers[0].volumeMounts[?(@.mountPath=="/usr/share/nginx/html")].name}'); [ -n "$v" ] || exit 1 +eq "$(j -n %{ns} pod web-server "{.spec.volumes[?(@.name==\"$v\")].persistentVolumeClaim.claimName}")" pv-volume"""), + C("the claim has grown to %{size}, and the Pod kept running", 3, r""" +eq "$(j -n %{ns} pvc pv-volume '{.spec.resources.requests.storage} {.status.capacity.storage}')" "%{size} %{size}" || exit 1 +eq "$(j -n %{ns} pod web-server '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """), + ], + "solution": """ +k get sc csi-hostpath-sc # ALLOWVOLUMEEXPANSION: true - a claim of this class may grow +k -n %{ns} apply -f - <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: pv-volume +spec: + storageClassName: csi-hostpath-sc + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 10Mi} +--- +apiVersion: v1 +kind: Pod +metadata: + name: web-server +spec: + containers: + - name: web-server + image: nginx:1.27 + volumeMounts: + - {name: html, mountPath: /usr/share/nginx/html} + volumes: + - name: html + persistentVolumeClaim: {claimName: pv-volume} +EOF +k -n %{ns} wait --for=condition=Ready pod/web-server +k -n %{ns} patch pvc pv-volume -p '{"spec":{"resources":{"requests":{"storage":"%{size}"}}}}' +k -n %{ns} get pvc pv-volume # CAPACITY follows once the driver has resized it +# `k edit pvc pv-volume` works just as well. A claim only grows, and only when its class has +# allowVolumeExpansion: true and its driver can expand; spec.resources.requests is what you ask +# for, status.capacity what you have. Filesystem volumes of this driver grow online - the Pod +# keeps running. +""", +}, +# ------------------------------------------------------------------ 147 ---- +{ + "id": 147, "domain": "STOR", "workers": 1, "title": "Snapshot, restore, clone", + "params": lambda r: {"ns": ns_pick(r, "ledger"), "n": r.randint(100, 999)}, + "task": """ +Claim `ledger-data` in `%{ns}` (class `csi-hostpath-sc`, `50Mi`) holds `ledger.txt`, written by Pod +`writer`. + +1. Take a VolumeSnapshot `ledger-snap` of the claim, with snapshot class `csi-hostpath-snapclass`, + and wait until it is ready to use. +2. Create claim `ledger-restore` (`50Mi`, same class) **from that snapshot**. +3. Create claim `ledger-copy` (`50Mi`, same class) as a **clone of `ledger-data`** itself. +4. Run Pod `auditor` (image `busybox:1.36`, command `sleep 86400`) that mounts `ledger-restore` at + `/restore` and `ledger-copy` at `/copy`. +""", + "setup": NS + r""" +ensure_csi_hostpath || { echo "could not install the CSI hostpath driver (no network?) - the task needs it" >&2; exit 1; } +mkns %{ns} +$K -n %{ns} apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: {name: ledger-data} +spec: {storageClassName: csi-hostpath-sc, accessModes: [ReadWriteOnce], resources: {requests: {storage: 50Mi}}} +--- +apiVersion: v1 +kind: Pod +metadata: {name: writer} +spec: + containers: + - {name: writer, image: busybox:1.36, command: [sh, -c, 'echo entries-%{n} > /data/ledger.txt; sync; sleep 86400'], volumeMounts: [{name: d, mountPath: /data}]} + volumes: [{name: d, persistentVolumeClaim: {claimName: ledger-data}}] +EOF +$K -n %{ns} wait --for=condition=Ready pod/writer --timeout=180s >/dev/null +""", + "checks": [ + C("VolumeSnapshot ledger-snap of ledger-data is ready to use", 2, r""" +eq "$(j -n %{ns} volumesnapshot ledger-snap '{.spec.source.persistentVolumeClaimName} {.spec.volumeSnapshotClassName} {.status.readyToUse}')" "ledger-data csi-hostpath-snapclass true" """), + C("ledger-restore is Bound, made from the snapshot", 2, r""" +eq "$(j -n %{ns} pvc ledger-restore '{.spec.dataSource.kind}/{.spec.dataSource.name} {.status.phase}')" "VolumeSnapshot/ledger-snap Bound" """), + C("ledger-copy is Bound, a clone of ledger-data", 2, r""" +eq "$(j -n %{ns} pvc ledger-copy '{.spec.dataSource.kind}/{.spec.dataSource.name} {.status.phase}')" "PersistentVolumeClaim/ledger-data Bound" """), + C("auditor reads the ledger from both", 2, r""" +x() { $K -n %{ns} exec auditor -- cat "$1/ledger.txt" 2>/dev/null; } +eq "$(x /restore)" entries-%{n} && eq "$(x /copy)" entries-%{n} \ + && eq "$(j -n %{ns} pod auditor '{.spec.volumes[*].persistentVolumeClaim.claimName}' | tr ' ' '\n' | sort | tr '\n' ' ')" "ledger-copy ledger-restore " """), + ], + "solution": """ +k get volumesnapshotclass # csi-hostpath-snapclass, driver hostpath.csi.k8s.io +k -n %{ns} apply -f - <<'EOF' +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshot +metadata: + name: ledger-snap +spec: + volumeSnapshotClassName: csi-hostpath-snapclass + source: + persistentVolumeClaimName: ledger-data +EOF +k -n %{ns} wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/ledger-snap --timeout=120s +k -n %{ns} apply -f - <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ledger-restore +spec: + storageClassName: csi-hostpath-sc + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 50Mi} + dataSource: + apiGroup: snapshot.storage.k8s.io + kind: VolumeSnapshot + name: ledger-snap +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ledger-copy +spec: + storageClassName: csi-hostpath-sc + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 50Mi} + dataSource: + kind: PersistentVolumeClaim + name: ledger-data +--- +apiVersion: v1 +kind: Pod +metadata: + name: auditor +spec: + containers: + - name: auditor + image: busybox:1.36 + command: [sleep, '86400'] + volumeMounts: + - {name: restore, mountPath: /restore} + - {name: copy, mountPath: /copy} + volumes: + - name: restore + persistentVolumeClaim: {claimName: ledger-restore} + - name: copy + persistentVolumeClaim: {claimName: ledger-copy} +EOF +k -n %{ns} wait --for=condition=Ready pod/auditor --timeout=180s +k -n %{ns} exec auditor -- cat /restore/ledger.txt /copy/ledger.txt +# A snapshot is taken by the CSI driver (VolumeSnapshotClass -> driver); `dataSource` makes a new +# claim from it - apiGroup snapshot.storage.k8s.io - or from another claim of the same class and +# Namespace, a clone (no apiGroup: claims are core). Both need a CSI driver that can do it, and +# the new claim must be at least as big as the source. +""", +}, +] diff --git a/cka/bank_sim_c.py b/cka/bank_sim_c.py index 52742c7..f1b5897 100644 --- a/cka/bank_sim_c.py +++ b/cka/bank_sim_c.py @@ -896,19 +896,21 @@ grep -Eq 'kubeadm[[:space:]]+upgrade[[:space:]]+node' "$WS/q80/worker.txt" 2>/de k version -o json | jq -r .serverVersion.gitVersion > {WS}/q80/current.txt k get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion --no-headers > {WS}/q80/kubelets.txt cat > {WS}/q80/steps.sh <<'EOF' -sed -i 's|/v1.35/|/v1.36/|' /etc/apt/sources.list.d/kubernetes.list -apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.36.5-1.1 && apt-mark hold kubeadm +sed -i 's|/v1.37/|/v1.38/|' /etc/apt/sources.list.d/kubernetes.list +apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.38.0-1.1 && apt-mark hold kubeadm kubeadm upgrade plan -kubeadm upgrade apply v1.36.5 -y +kubeadm upgrade apply v1.38.0 -y kubectl drain --ignore-daemonsets -apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.36.5-1.1 kubectl=1.36.5-1.1 && apt-mark hold kubelet kubectl +apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl systemctl daemon-reload systemctl restart kubelet kubectl uncordon EOF echo 'kubeadm upgrade node' > {WS}/q80/worker.txt -# The cluster runs v1.35, so the next minor is 1.36 - and pkgs.k8s.io serves one apt repo per -# minor, so the repo line has to move to v1.36 before 1.36 packages exist for apt at all. +# The cluster runs v1.37, so the next minor is 1.38 - and pkgs.k8s.io serves one apt repo per +# minor, so the repo line has to move to v1.38 before 1.38 packages exist for apt at all. (1.38 +# is due in December 2026; until then that repo is empty, which is why this is only a plan. Take +# the newest 1.38 patch on offer then: `apt-cache madison kubeadm`.) # kubeadm goes first because it drives the whole thing; the control plane is upgraded by # `upgrade apply`, and only then do the node binaries (kubelet/kubectl) follow, ending with # uncordon. Workers run `kubeadm upgrade node` - they host no control-plane static pods, so diff --git a/cka/common.py b/cka/common.py index 116d6ad..66cc8e9 100644 --- a/cka/common.py +++ b/cka/common.py @@ -44,3 +44,112 @@ look() { $K -n "$1" exec "$2" -- nslookup "$3" 2>/dev/null | awk '/^Name:/ {n=1} # configuration by itself, but not at once) resolves() { local i; for i in $(seq 1 10); do [ "$(look "$1" "$2" "$3")" = "$4" ] && return 0; sleep 3; done; return 1; } """ + + +# A small operator for Backup objects (group data.example.com): a controller that gives each +# Backup a ConfigMap and marks it Scheduled. It is kubectl in a loop - the node's own kubectl, +# mounted into a busybox container - which is enough to show the pattern: a CRD, RBAC for its +# controller, and a Deployment that acts on it. With status_rbac=False the ClusterRole lacks +# `backups/status`, and the controller can make the ConfigMaps but not set the phase. +_BACKUP_CRD = r"""apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: backups.data.example.com +spec: + group: data.example.com + scope: Namespaced + names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]} + versions: + - name: v1 + served: true + storage: true + subresources: + status: {} + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + required: [schedule] + properties: + schedule: {type: string} + retention: {type: integer, minimum: 1, default: 7} + status: + type: object + properties: + phase: {type: string} +--- +""" + +def backup_operator(crd=False, status_rbac=True): + """The operator's manifests, as one YAML document stream (the CRD first, when asked for).""" + status = ", backups/status" if status_rbac else "" + return (_BACKUP_CRD if crd else "") + r"""apiVersion: v1 +kind: Namespace +metadata: + name: backup-operator +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: controller + namespace: backup-operator +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: backup-operator +rules: +- apiGroups: [data.example.com] + resources: [backups""" + status + r"""] + verbs: [get, list, watch, update, patch] +- apiGroups: [""] + resources: [configmaps] + verbs: [get, list, create, update, patch] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: backup-operator +roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: backup-operator} +subjects: +- {kind: ServiceAccount, name: controller, namespace: backup-operator} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: controller + namespace: backup-operator +spec: + replicas: 1 + selector: + matchLabels: {app: backup-operator} + template: + metadata: + labels: {app: backup-operator} + spec: + serviceAccountName: controller + containers: + - name: controller + image: busybox:1.36 + command: + - sh + - -c + - | + set -f + while true; do + for b in $(kubectl get backups -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'); do + ns=${b%/*}; name=${b#*/} + sched=$(kubectl -n "$ns" get backup "$name" -o jsonpath='{.spec.schedule}') + kubectl -n "$ns" create configmap "backup-$name" --from-literal=schedule="$sched" --dry-run=client -o yaml | kubectl apply -f - >/dev/null + kubectl -n "$ns" patch backup "$name" --subresource=status --type=merge -p '{"status":{"phase":"Scheduled"}}' >/dev/null + done + sleep 5 + done + volumeMounts: + - {name: kubectl, mountPath: /bin/kubectl, readOnly: true} + volumes: + - name: kubectl + hostPath: {path: /usr/bin/kubectl, type: File} +""" diff --git a/cka/curriculum.py b/cka/curriculum.py index 9d08715..53df97c 100644 --- a/cka/curriculum.py +++ b/cka/curriculum.py @@ -1,5 +1,5 @@ """CKA curriculum as a path to the exam: the 27 objectives of the five domains, in the curriculum's own -words and order (CKA_Curriculum_v1.35.pdf, cncf/curriculum), each with what to know, the fastest +words and order (CKA_Curriculum_v1.37.pdf, cncf/curriculum), each with what to know, the fastest commands, the traps, the documentation pages the exam lets you open, and the tasks that practise it. ./ksim objectives the path, with your progress per objective @@ -26,7 +26,7 @@ OBJECTIVES = [ { "id": "1.1", "key": "storageclasses", "domain": "STOR", "text": "Implement storage classes and dynamic volume provisioning", - "tasks": [16, 36, 114, 123, 124, 59, 118, 122, 99], + "tasks": [16, 36, 114, 123, 124, 146, 59, 118, 122, 99], "know": [ "A StorageClass names a provisioner and its settings. A claim that asks for the class gets a PersistentVolume made for it - that is dynamic provisioning.", "Fields: provisioner, reclaimPolicy (Delete by default, or Retain), volumeBindingMode (Immediate or WaitForFirstConsumer), allowVolumeExpansion, parameters.", @@ -58,7 +58,7 @@ k get pvc,pv; k describe pvc data # the Events say why a cla { "id": "1.2", "key": "volumes", "domain": "STOR", "text": "Configure volume types, access modes and reclaim policies", - "tasks": [15, 35, 114, 115, 124, 58, 119, 120, 68, 72, 102], + "tasks": [15, 35, 114, 115, 124, 136, 58, 119, 120, 68, 72, 102], "know": [ "Volume types you meet: hostPath (a directory of whatever node the Pod lands on), local (a disk of one node, needs nodeAffinity), emptyDir, nfs, csi, and the configMap / secret / projected volumes.", "Access modes: ReadWriteOnce (one node), ReadOnlyMany, ReadWriteMany, ReadWriteOncePod (one Pod). A claim binds only to a volume that offers its mode.", @@ -91,7 +91,7 @@ k get pv -o custom-columns=NAME:.metadata.name,MODES:.spec.accessModes,RECLAIM:. { "id": "1.3", "key": "claims", "domain": "STOR", "text": "Manage persistent volumes and persistent volume claims", - "tasks": [15, 36, 35, 116, 121, 123, 58, 59, 117, 118, 119, 120, 68, 72, 81, 99, 102], + "tasks": [15, 36, 35, 116, 121, 123, 146, 147, 58, 59, 117, 118, 119, 120, 68, 72, 81, 99, 102], "know": [ "A claim binds to one volume, and only if class, access mode and size fit (the volume may be larger); `volumeName` or a selector narrows it to one volume.", "Phases: a volume goes Available, Bound, Released; a claim Pending, Bound.", @@ -125,7 +125,7 @@ k describe pod POD | tail # unbound claim -> the Pod stays { "id": "2.1", "key": "nodes", "domain": "TSHOOT", "text": "Troubleshoot clusters and nodes", - "tasks": [17, 39, 60, 129, 92, 90, 95], + "tasks": [17, 39, 60, 129, 139, 92, 90, 95], "know": [ "Start from the cluster: `k get nodes`, then `k describe node` - Conditions, Taints, Unschedulable, Events.", "A NotReady node is nearly always its kubelet: on the node, `systemctl status kubelet` and `journalctl -u kubelet` name the cause.", @@ -205,7 +205,7 @@ k -n kube-system logs deploy/metrics-server { "id": "2.4", "key": "logs", "domain": "TSHOOT", "text": "Manage and evaluate container output streams", - "tasks": [19, 104, 8, 131, 85], + "tasks": [19, 104, 8, 131, 134, 85], "know": [ "`kubectl logs` shows what a container wrote to stdout and stderr. The runtime keeps it on the node under /var/log/pods/NAMESPACE_POD_UID/CONTAINER/, and it goes when the Pod is deleted.", "Pick the container with -c (init containers too) or take --all-containers; --previous is the run before the last restart.", @@ -233,7 +233,7 @@ crictl ps -a; crictl logs --tail=20 CONTAINER-ID; ls /var/log/pods/ { "id": "2.5", "key": "netdebug", "domain": "TSHOOT", "text": "Troubleshoot services and networking", - "tasks": [18, 62, 37, 57, 73, 101, 70, 90], + "tasks": [18, 62, 37, 57, 140, 73, 101, 70, 90], "know": [ "Work outwards from the Pod: is it Ready, does the Service have endpoints (selector against Pod labels), is targetPort the port the container listens on, does the name resolve, does a NetworkPolicy stand in the way.", "DNS is a Service like any other: the CoreDNS Pods, the kube-dns Service and its endpoints, the Corefile in ConfigMap coredns.", @@ -261,7 +261,7 @@ k get netpol -A { "id": "3.1", "key": "rollouts", "domain": "WORK", "text": "Understand application deployments and how to perform rolling update and rollbacks", - "tasks": [7, 51], + "tasks": [7, 51, 144], "know": [ "Any change to `spec.template` starts a rollout: a new ReplicaSet scales up while the old one scales down, within maxSurge and maxUnavailable (25% each by default).", "Every rollout is a revision. `kubectl rollout`: status, history, undo, pause, resume, restart.", @@ -287,7 +287,7 @@ k patch deploy web -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUn { "id": "3.2", "key": "config", "domain": "WORK", "text": "Use ConfigMaps and Secrets to configure applications", - "tasks": [10, 29, 79, 93, 84], + "tasks": [10, 29, 142, 79, 93, 84], "know": [ "Both hold keys. A key becomes an environment variable (configMapKeyRef / secretKeyRef, or every key at once with envFrom) or a file (a configMap / secret volume; `items` picks and renames keys).", "Secret data is base64 - an encoding, not encryption. Types: Opaque, kubernetes.io/tls, kubernetes.io/dockerconfigjson.", @@ -318,7 +318,7 @@ k set env deploy/web --from=configmap/app-config; k rollout restart deploy/web { "id": "3.3", "key": "autoscaling", "domain": "WORK", "text": "Configure workload autoscaling", - "tasks": [105, 23, 82], + "tasks": [105, 23, 138, 82], "know": [ "A HorizontalPodAutoscaler (autoscaling/v2) keeps a workload between minReplicas and maxReplicas by a metric - usually average CPU or memory utilisation, which is usage divided by the container's request.", "It reads the metrics API, so it needs metrics-server, and the containers need a request for the resource it scales on.", @@ -355,7 +355,7 @@ spec: { "id": "3.4", "key": "selfhealing", "domain": "WORK", "text": "Understand the primitives used to create robust, self-healing, application deployments", - "tasks": [106, 8, 28, 26, 27, 31, 52, 53, 76, 88, 70], + "tasks": [106, 133, 8, 28, 26, 27, 31, 52, 53, 76, 88, 70], "know": [ "Controllers put back what is missing: a Deployment its replicas, a DaemonSet one Pod per node, a StatefulSet named replicas with their own storage, a Job its completions (backoffLimit retries), a CronJob its Jobs.", "Probes tell the kubelet about health: liveness restarts the container, readiness takes the Pod out of Service endpoints, startup holds both off until the app is up.", @@ -389,7 +389,7 @@ k describe pod POD | grep -E 'Liveness|Readiness|Startup'; k get events --field { "id": "3.5", "key": "scheduling", "domain": "WORK", "text": "Configure Pod admission and scheduling (limits, node affinity, etc.)", - "tasks": [9, 25, 24, 32, 50, 78, 96, 81, 97], + "tasks": [9, 25, 24, 32, 50, 137, 143, 78, 96, 81, 97], "know": [ "Requests are what the scheduler reserves, limits what the node enforces. A LimitRange gives containers defaults and bounds; a ResourceQuota caps a Namespace's totals - both act at admission.", "Where a Pod may go: nodeSelector, node affinity (required or preferred), Pod affinity and anti-affinity with a topologyKey, topologySpreadConstraints, or nodeName, which skips the scheduler.", @@ -426,7 +426,7 @@ k describe pod POD | tail # FailedScheduling: 0/4 nodes are a { "id": "4.1", "key": "rbac", "domain": "ARCH", "text": "Manage role based access control (RBAC)", - "tasks": [1, 33, 6, 44, 47, 75, 91], + "tasks": [1, 33, 6, 44, 47, 135, 75, 91], "know": [ "A Role (one Namespace) or ClusterRole (everywhere, and cluster-scoped resources) lists verbs on resources; a RoleBinding or ClusterRoleBinding grants it to users, groups or ServiceAccounts.", "A RoleBinding may point at a ClusterRole: the rules then apply in that one Namespace only.", @@ -514,7 +514,7 @@ grep staticPodPath /var/lib/kubelet/config.yaml; k run web --image=nginx:1.27 $ { "id": "4.4", "key": "lifecycle", "domain": "ARCH", "text": "Manage the lifecycle of Kubernetes clusters", - "tasks": [109, 2, 65, 45, 80], + "tasks": [109, 2, 65, 45, 132, 80], "know": [ "Upgrade one minor version at a time, control plane first: the kubeadm package, `kubeadm upgrade plan` and `apply`, drain, the kubelet and kubectl packages, restart the kubelet, uncordon. Workers: `kubeadm upgrade node` in place of `apply`.", "Maintenance on a node: `kubectl drain` cordons it and evicts its Pods; `kubectl uncordon` gives it back.", @@ -523,8 +523,8 @@ grep staticPodPath /var/lib/kubelet/config.yaml; k run web --image=nginx:1.27 $ ], "fast": """ k drain NODE --ignore-daemonsets --delete-emptydir-data; k uncordon NODE -apt-get install -y kubeadm='1.36.*'; kubeadm upgrade plan; kubeadm upgrade apply v1.36.x # workers: kubeadm upgrade node -apt-get install -y kubelet='1.36.*' kubectl='1.36.*'; systemctl daemon-reload; systemctl restart kubelet +apt-get install -y kubeadm='1.38.*'; kubeadm upgrade plan; kubeadm upgrade apply v1.38.x # workers: kubeadm upgrade node +apt-get install -y kubelet='1.38.*' kubectl='1.38.*'; systemctl daemon-reload; systemctl restart kubelet etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup.db etcdutl snapshot restore /opt/backup.db --data-dir /var/lib/etcd-restore # then etcd.yaml: the hostPath of volume etcd-data @@ -533,7 +533,7 @@ kubeadm certs renew all; kubeadm certs check-expiration "traps": [ "drain stops at DaemonSet Pods (--ignore-daemonsets), at emptyDir data (--delete-emptydir-data) and at bare Pods (--force, which deletes them for good).", "A drained node stays cordoned until you uncordon it.", - "With etcd 3.6, as here, restore exists only in etcdutl - `etcdctl snapshot restore` is gone.", + "Since etcd 3.6 (v1.37 runs 3.7), restore exists only in etcdutl - `etcdctl snapshot restore` is gone.", "Restoring etcd takes the whole cluster back to the snapshot: everything created since is lost.", ], "docs": ["https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/", @@ -626,7 +626,7 @@ k get csidrivers; k get csinodes; k get sc # PROVISIONER { "id": "4.8", "key": "crds", "domain": "ARCH", "text": "Understand CRDs, install and configure operators", - "tasks": [4, 34, 49, 91], + "tasks": [4, 34, 49, 145, 91], "know": [ "A CustomResourceDefinition adds a kind to the API: group, names (kind, plural, singular, shortNames), scope (Namespaced or Cluster) and versions, each with an OpenAPI v3 schema.", "The schema validates custom resources - types, required fields, enum, minimum, maximum, defaults - and feeds `kubectl explain`.", @@ -658,6 +658,7 @@ k get crd; k api-resources --api-group=data.example.com; k explain backup.spec """, "traps": [ "The CRD's name must be PLURAL.GROUP, and exactly one version has `storage: true`.", + "A custom resource applied right after its CRD can fail with \"no matches for kind\": the new type is served a moment later. `k wait --for condition=established crd/NAME` first.", "A custom resource's apiVersion is GROUP/VERSION of the CRD, not apiextensions.k8s.io/v1.", "`default` and validation only work inside a structural schema: every level needs its `type`.", "Deleting a CRD deletes every object of that kind.", @@ -727,7 +728,7 @@ k describe netpol backend-policy -n NS; k exec CLIENT -- wget -qO- -T2 http://b { "id": "5.3", "key": "services", "domain": "NET", "text": "Use ClusterIP, NodePort, LoadBalancer service types and endpoints", - "tasks": [12, 30, 43, 64, 71, 73, 94], + "tasks": [12, 30, 43, 64, 141, 71, 73, 94], "know": [ "A Service selects Pods by label and forwards `port` to `targetPort` - a number, or the name of a container port.", "ClusterIP is reachable inside the cluster; NodePort adds a port (30000-32767) on every node; LoadBalancer adds an external address from the cloud; ExternalName is a DNS alias; headless (clusterIP: None) gives every Pod its own DNS record.", diff --git a/cka/lessons.py b/cka/lessons.py new file mode 100644 index 0000000..1e6ec74 --- /dev/null +++ b/cka/lessons.py @@ -0,0 +1,27 @@ +"""CKA lessons: guided labs, one per curriculum objective, on a cluster of your own and without a +clock. A lesson explains, then walks you through steps that you do on its host; each step has a +check, a hint and the commands that do it. It ends by sending you to the objective's timed drill. +Lessons write nothing to the history or the scores - only a "lesson done" mark on the objective. + +./ksim lesson the lessons of this track, and which you have done +./ksim lesson 1.1 start the lesson of objective 1.1 (id or key) +./ksim lesson check check the step you are on; the next one follows +./ksim lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after + +A lesson's host is named after its id - lesson 9101 runs on `cka9101` - which never clashes with a +task's. Its files go under {WS}/lesson-/. A step's `show` commands are what the learner +types; the audit runs exactly them, as `candidate` on the host, so what a lesson shows always +works. Work on another node is shown as `ssh NODE sudo bash <<'EOF' ... EOF`: the same commands +you would type after `ssh NODE` and `sudo -i`, in a form that runs unattended. A single command +there is `ssh -n NODE 'COMMAND'`: without -n, ssh would read the rest of the commands as its input. + +One file per domain, as with the task banks. Besides the curriculum's competencies, the lessons +cover what a full CKA course teaches under each of them (2026-10-19: the topics of KubeDojo's CKA +course, written here from scratch): projected volumes, eviction thresholds, debug containers, +kube-proxy's rules, encryption at rest, in-place resize, PodDisruptionBudgets, QoS classes, +ServiceAccount tokens, a real kubeadm upgrade, EndpointSlices. +""" +from cka import lessons_arch, lessons_net, lessons_storage, lessons_tshoot, lessons_work + +LESSONS = (lessons_storage.LESSONS + lessons_tshoot.LESSONS + lessons_work.LESSONS + + lessons_arch.LESSONS + lessons_net.LESSONS) diff --git a/cka/lessons_arch.py b/cka/lessons_arch.py new file mode 100644 index 0000000..f6a0a60 --- /dev/null +++ b/cka/lessons_arch.py @@ -0,0 +1,943 @@ +"""CKA lessons of the Cluster Architecture, Installation and Configuration domain (4.1-4.8). See +cka/lessons.py for what a lesson is.""" +from cka.common import backup_operator +from sim.bank import CAN, NS, PY, step + +_ETCDCTL = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt " + "--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key") + +# ---- 4.1 RBAC and ServiceAccounts ------------------------------------------------------------- +L41 = { + "id": 9401, "objective": "4.1", "workers": 0, + "title": "RBAC: Roles, bindings, ServiceAccounts and their tokens", + "intro": r"""Every request to the API server is made by someone - a **user** (a client certificate, or a token from outside), or a **ServiceAccount** (a token the cluster issues) - and RBAC decides what they may do: + +``` +Role / ClusterRole verbs on resources (in apiGroups) what may be done +RoleBinding subjects -> a Role or a ClusterRole in ONE Namespace +ClusterRoleBinding subjects -> a ClusterRole everywhere +kubectl auth can-i VERB RESOURCE --as=USER | --as=system:serviceaccount:NS:NAME [-n NS] [--list] +``` + +A RoleBinding may point at a ClusterRole: the ClusterRole's rules then apply only in the binding's Namespace - the usual way to reuse one set of rules in many Namespaces. + +Pods authenticate as their ServiceAccount, with a short-lived token the kubelet mounts at `/var/run/secrets/kubernetes.io/serviceaccount/`. `kubectl create token` issues one for you. + +Namespace `team` is empty.""", + "setup": NS + r""" +mkns team +mkdir -p "$WS/lesson-4.1" +""", + "steps": [ + step("A ServiceAccount that deploys", + r"""In `team`, create ServiceAccount `deployer`, Role `deploy-edit` that allows `get`, `list`, `watch`, `create`, `update`, `patch` and `delete` on **deployments** (API group `apps`) and `get`, `list` on **pods**, and RoleBinding `deployer` that gives the Role to the ServiceAccount. + +Prove it with `kubectl auth can-i ... --as=system:serviceaccount:team:deployer`.""", + check=CAN + r""" +S=system:serviceaccount:team:deployer +can --as=$S -- create deployments.apps -n team && can --as=$S -- list pods -n team && cannot --as=$S -- delete pods -n team \ + && cannot --as=$S -- create deployments.apps -n default""", + hint=r"""`kubectl create role deploy-edit --verb=... --resource=deployments.apps` - a Role with two resources and different verbs needs two rules, so write YAML or create it and `kubectl edit`. `kubectl create rolebinding NAME --role=ROLE --serviceaccount=team:deployer`.""", + show=r"""kubectl -n team create serviceaccount deployer +kubectl -n team apply -f - <<'EOF' +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: deploy-edit +rules: +- apiGroups: [apps] + resources: [deployments] + verbs: [get, list, watch, create, update, patch, delete] +- apiGroups: [""] + resources: [pods] + verbs: [get, list] +EOF +kubectl -n team create rolebinding deployer --role=deploy-edit --serviceaccount=team:deployer +kubectl auth can-i create deployments -n team --as=system:serviceaccount:team:deployer +kubectl auth can-i delete pods -n team --as=system:serviceaccount:team:deployer"""), + step("One ClusterRole, one Namespace", + r"""Create ClusterRole `pod-reader` (`get`, `list`, `watch` on pods). Give it to the user **`jane`** in Namespace `team` only - with a **RoleBinding** `jane-reads-pods`, not a ClusterRoleBinding. + +`jane` can then list Pods in `team`, and nowhere else.""", + check=CAN + r""" +eq "$(j clusterrole pod-reader '{.rules[0].verbs}')" '["get","list","watch"]' || exit 1 +eq "$(j -n team rolebinding jane-reads-pods '{.roleRef.kind}/{.roleRef.name}')" ClusterRole/pod-reader || exit 1 +can --as=jane -- list pods -n team && cannot --as=jane -- list pods -n default && cannot --as=jane -- list pods -A""", + hint=r"""`kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods`, then `kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane`. Users exist only in certificates and tokens: nothing to create for `jane`.""", + show=r"""kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods +kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane +kubectl auth can-i list pods -n team --as=jane +kubectl auth can-i list pods -n default --as=jane"""), + step("A token for the ServiceAccount", + r"""Issue a token for `deployer` that is valid for **one hour**, and save it in `{WS}/lesson-4.1/deployer.token`. Then use it - as the **only** credential: your kubeconfig carries the admin's client certificate, and the API server checks a certificate before a token, so `kubectl --token=...` alone would still be you. With no kubeconfig, just the server, the CA and the token, `kubectl auth whoami` says `system:serviceaccount:team:deployer`, `get deployments` works in `team` and `get secrets` is forbidden.""", + check=r""" +t=$(tr -d ' \r\n' < "$WS/lesson-4.1/deployer.token" 2>/dev/null); [ -n "$t" ] || exit 1 +srv=$($K config view --minify -o jsonpath='{.clusters[0].cluster.server}') +eq "$($K --kubeconfig=/dev/null --server="$srv" --insecure-skip-tls-verify --token="$t" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)" system:serviceaccount:team:deployer || exit 1 +echo "$t" | cut -d. -f2 | tr '_-' '/+' | python3 -c ' +import base64, json, sys +s = sys.stdin.read().strip(); c = json.loads(base64.b64decode(s + "=" * (-len(s) % 4))) +sys.exit(0 if c["exp"] - c["iat"] == 3600 else 1)'""", + hint=r"""`kubectl -n team create token deployer --duration=1h`. Then `kubectl --kubeconfig=/dev/null --server=URL --certificate-authority=/etc/kubernetes/pki/ca.crt --token=TOKEN ...`; the URL is in your kubeconfig. The token is a JWT: its middle part, base64-decoded, holds `sub`, `iat` and `exp`.""", + show=r"""kubectl -n team create token deployer --duration=1h > {WS}/lesson-4.1/deployer.token +server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}') +as_deployer() { kubectl --kubeconfig=/dev/null --server=$server --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$(cat {WS}/lesson-4.1/deployer.token) "$@"; } +as_deployer auth whoami +as_deployer -n team get deployments +as_deployer -n team get secrets || true"""), + step("A Pod without a token", + r"""Most Pods never talk to the API server, and do not need a token in their filesystem. Create ServiceAccount `quiet` in `team` with `automountServiceAccountToken: false`, and run Pod `notoken` (`busybox:1.36`, `sleep 86400`) as that ServiceAccount. Inside it, `/var/run/secrets/kubernetes.io/serviceaccount` does not exist.""", + check=r""" +eq "$(j -n team sa quiet '{.automountServiceAccountToken}')" false && eq "$(j -n team pod notoken '{.spec.serviceAccountName} {.status.phase}')" "quiet Running" || exit 1 +! $K -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount >/dev/null 2>&1""", + hint=r"""The field sits at the top level of the ServiceAccount (and can be set on a Pod's `spec` too, which wins). The Pod names its ServiceAccount with `spec.serviceAccountName`.""", + show=r"""kubectl -n team apply -f - <<'EOF' +apiVersion: v1 +kind: ServiceAccount +metadata: + name: quiet +automountServiceAccountToken: false +--- +apiVersion: v1 +kind: Pod +metadata: + name: notoken +spec: + serviceAccountName: quiet + containers: + - name: app + image: busybox:1.36 + command: [sleep, '86400'] +EOF +kubectl -n team wait --for=condition=Ready pod/notoken --timeout=120s +kubectl -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount || echo 'no token mounted'"""), + step("Everything one subject may do", + r"""Write the complete list of what `deployer` may do in `team` to `{WS}/lesson-4.1/deployer-can.txt` - `kubectl auth can-i` prints it with `--list`. It includes what every authenticated subject gets from the built-in ClusterRoles, such as creating `selfsubjectreviews`.""", + check=r"""f="$WS/lesson-4.1/deployer-can.txt"; grep -q '^deployments.apps' "$f" 2>/dev/null && grep -q '^pods ' "$f" && grep -qi 'verbs' "$f" """, + hint=r"""`kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer`.""", + show=r"""kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer > {WS}/lesson-4.1/deployer-can.txt +cat {WS}/lesson-4.1/deployer-can.txt"""), + ], + "outro": r"""You gave a ServiceAccount a Role, reused a ClusterRole in one Namespace, issued and used a token, ran a Pod without one, and listed a subject's permissions. + +**For the exam** + +- `kubectl create role|clusterrole NAME --verb=... --resource=...` and `kubectl create rolebinding|clusterrolebinding NAME --role|--clusterrole=... --user|--group|--serviceaccount=NS:NAME`. +- Resources in a group are written `RESOURCE.GROUP` on the command line (`deployments.apps`) and as `apiGroups: [apps]` in YAML; the core group is `""`. Subresources: `pods/log`, `pods/exec`, `deployments/scale`. +- A binding's `roleRef` cannot be changed: delete and re-create. +- `--as=system:serviceaccount:NS:NAME` to test a ServiceAccount; `--as=USER --as-group=GROUP` for users. +- Users are certificates: a CSR with `CN=user, O=group`, signed via a `CertificateSigningRequest` (`kubectl certificate approve`), then a kubeconfig with `kubectl config set-credentials --embed-certs`. Drill 4.1 has that end to end. + +Next, prove it against the clock: **drill 4.1**.""", +} + +# ---- 4.2 underlying infrastructure ------------------------------------------------------------ +L42 = { + "id": 9402, "objective": "4.2", "workers": 1, + "title": "Preparing a machine for kubeadm: modules, sysctls, swap, runtime", + "intro": r"""Before `kubeadm init` or `join`, each machine needs: + +``` +kernel modules overlay, br_netfilter /etc/modules-load.d/*.conf (modprobe now) +sysctls net.ipv4.ip_forward = 1, net.bridge.bridge-nf-call-iptables = 1 /etc/sysctl.d/*.conf (sysctl --system) +swap off, or the kubelet told to tolerate it (failSwapOn: false) +container runtime containerd, CRI enabled, its cgroup driver matching the kubelet's (systemd) +packages kubeadm, kubelet, kubectl from pkgs.k8s.io for ONE minor, held +ports 6443 (API server), 2379-2380 (etcd), 10250 (kubelet), 30000-32767 (NodePorts) +``` + +A setting made with `sysctl -w` or `modprobe` alone is gone after the next reboot; the files are what make it stick. + +Worker `{W1}` stands in for the new machine. (The nodes of this simulator are containers: a module cannot really be loaded there, but the files and what you read from the node are real.)""", + "setup": r""" +mkdir -p "$WS/lesson-4.2" +nx "$W1" 'rm -f /etc/modules-load.d/k8s.conf /etc/sysctl.d/k8s.conf' /dev/null | tr -d " " | grep . | sort | tr "\n" " "' /etc/modules-load.d/k8s.conf +cat /etc/modules-load.d/k8s.conf +EOF"""), + step("Kernel parameters, for good", + r"""On `{W1}`, put the three parameters Kubernetes needs in `/etc/sysctl.d/k8s.conf` - `net.bridge.bridge-nf-call-iptables = 1`, `net.bridge.bridge-nf-call-ip6tables = 1`, `net.ipv4.ip_forward = 1` - and apply every sysctl file now with `sysctl --system`.""", + check=r""" +f=$(nx "$W1" 'cat /etc/sysctl.d/k8s.conf 2>/dev/null' /etc/sysctl.d/k8s.conf <<'CONF' +net.bridge.bridge-nf-call-iptables = 1 +net.bridge.bridge-nf-call-ip6tables = 1 +net.ipv4.ip_forward = 1 +CONF +sysctl --system 2>/dev/null | grep -A3 k8s.conf +sysctl net.ipv4.ip_forward +EOF"""), + step("Swap and the cgroup driver", + r"""Two more things kubeadm's preflight checks. Write to `{WS}/lesson-4.2/node.txt`, in this format: + +``` +swap= +failSwapOn= +kubelet-cgroup= +containerd-systemd-cgroup= +``` + +The last two must agree, or Pods fail to start.""", + check=r""" +swap=$(nx "$W1" '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off' /dev/null | grep .)" "$want" """, + hint=r"""`cat /proc/swaps` (or `swapon --show`): any line below the header is swap. `grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml` - a field that is not there has its default (`failSwapOn` is `true`). `grep SystemdCgroup /etc/containerd/config.toml`; no such line means `false`.""", + show=r"""ssh {W1} sudo bash <<'EOF' +cat /proc/swaps +grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml +grep -n SystemdCgroup /etc/containerd/config.toml || echo 'SystemdCgroup not set' +EOF +swap=$(ssh -n {W1} '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off') +fso=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.failSwapOn) +cg=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver) +sc=$(ssh -n {W1} 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml | head -1 | tr -d " " | cut -d= -f2') +printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s\n' "$swap" "$fso" "$cg" "${sc:-false}" > {WS}/lesson-4.2/node.txt +cat {WS}/lesson-4.2/node.txt"""), + step("The packages, as kubeadm wants them", + r"""On an exam node, kubeadm, kubelet and kubectl come from the repository of **one** minor at `pkgs.k8s.io`, and are **held** so that an `apt upgrade` never moves them by accident. + +Write to `{WS}/lesson-4.2/packages.txt`: the line of `{W1}`'s Kubernetes repository (from `/etc/apt/sources.list.d/`), then the held packages, one per line, as `apt-mark` prints them.""", + check=r""" +want=$(nx "$W1" 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' /dev/null | tr -d '\r')" "$want" """, + hint=r"""`cat /etc/apt/sources.list.d/kubernetes.list` and `apt-mark showhold`. `apt-cache madison kubeadm` lists the versions that repository offers; `dpkg -l | grep kube` what is installed.""", + show=r"""ssh -n {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt +cat {WS}/lesson-4.2/packages.txt +ssh -n {W1} 'dpkg -l | grep -E "kube(adm|let|ctl)"'"""), + ], + "outro": r"""You set up modules and sysctls that survive a reboot, checked swap and the cgroup drivers, and read how Kubernetes is installed on a node. + +**For the exam** + +- The docs page "Installing kubeadm" and "Container Runtimes" have every file to copy - module list, sysctl file, the containerd `SystemdCgroup = true` line, the apt repository and `apt-mark hold`. +- `sysctl --system` applies every file under `/etc/sysctl.d`; `sysctl -p FILE` just one. +- containerd: `containerd config default > /etc/containerd/config.toml`, set `SystemdCgroup = true`, `systemctl restart containerd`. `crictl info` shows what it runs with. +- A machine with a CNI already configured has a file in `/etc/cni/net.d/` - the CNI itself is installed after `kubeadm init`, as a manifest. + +Next, prove it against the clock: **drill 4.2**.""", +} + +# ---- 4.3 kubeadm clusters --------------------------------------------------------------------- +L43 = { + "id": 9403, "objective": "4.3", "workers": 1, + "title": "kubeadm: the cluster's configuration, joining nodes, certificates, static Pods", + "intro": r"""`kubeadm init` builds a control plane from a `ClusterConfiguration` and leaves you everything needed to manage it: + +``` +/etc/kubernetes/manifests/ static Pods: etcd, kube-apiserver, kube-controller-manager, kube-scheduler +/etc/kubernetes/pki/ the cluster CA, etcd CA, front-proxy CA, and the certificates they signed +/etc/kubernetes/*.conf kubeconfigs: admin.conf, super-admin.conf, controller-manager.conf, scheduler.conf, kubelet.conf +ConfigMap kubeadm-config the ClusterConfiguration, in kube-system +``` + +Nodes join with a **bootstrap token** and the hash of the CA's public key: `kubeadm token create --print-join-command` prints the whole command. The certificates kubeadm makes last a year - the CA ten years - and `kubeadm certs check-expiration` / `renew` look after them. + +This cluster was made by kubeadm; its worker is `{W1}`.""", + "setup": r""" +mkdir -p "$WS/lesson-4.3" +nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' "$PRIV/lesson-4.3.before" +nx "$W1" 'rm -f /etc/kubernetes/manifests/node-web.yaml' /dev/null | tr -d '\r\n')" "$want" """, + hint=r"""`kubectl -n kube-system get cm kubeadm-config -o yaml` - look under `networking:`. `kubeadm config print init-defaults` shows every field such a file can have.""", + show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep -A4 networking +kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}' > {WS}/lesson-4.3/networks.txt +cat {WS}/lesson-4.3/networks.txt"""), + step("A join command for a new worker", + r"""Create a new bootstrap token that expires in **2 hours**, and save the complete `kubeadm join` command for a worker in `{WS}/lesson-4.3/join.sh`, on one line. Run it on the control-plane node.""", + check=r""" +f="$WS/lesson-4.3/join.sh"; grep -q '^kubeadm join .*--token .*--discovery-token-ca-cert-hash sha256:' "$f" 2>/dev/null || exit 1 +id=$(grep -o -- '--token [a-z0-9]*\.[a-z0-9]*' "$f" | awk '{print $2}' | cut -d. -f1); [ -n "$id" ] || exit 1 +exp=$(j -n kube-system secret "bootstrap-token-$id" '{.data.expiration}' | base64 -d); [ -n "$exp" ] || exit 1 +left=$(( $(date -d "$exp" +%s) - $(date +%s) )); [ "$left" -gt 6600 ] && [ "$left" -le 7300 ]""", + hint=r"""`sudo kubeadm token create --ttl 2h --print-join-command`. `kubeadm token list` shows the tokens and when they expire; each is a Secret `bootstrap-token-ID` in `kube-system`.""", + show=r"""sudo kubeadm token create --ttl 2h --print-join-command > {WS}/lesson-4.3/join.sh +cat {WS}/lesson-4.3/join.sh +sudo kubeadm token list"""), + step("Renew the API server's certificate", + r"""Check the certificates' expiry dates, then **renew** the API server's serving certificate - only that one - and make the API server use it. A running process keeps the certificate it read at start, so the static Pod has to be restarted: move its manifest out of `/etc/kubernetes/manifests`, wait until the container is gone, and move it back. + +Afterwards `kubeadm certs check-expiration` shows the new date, and the API server presents the new certificate.""", + check=r""" +b=$(cat "$PRIV/lesson-4.3.before") +nb=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' /dev/null | openssl x509 -noout -serial' /dev/null 2>&1 && break; sleep 3; done +sudo kubeadm certs check-expiration | grep '^apiserver '"""), + step("A static Pod on the worker", + r"""Every kubelet runs static Pods, not only the control plane's. On `{W1}`, find the kubelet's `staticPodPath` and put a manifest there for Pod `node-web` (`nginx:1.27`). + +The API server shows it as a **mirror Pod** named after the node: `node-web-{W1}`. Deleting the mirror Pod with kubectl does nothing lasting - the kubelet puts it back; only removing the file does.""", + check=r"""eq "$(j -n default pod "node-web-$W1" '{.status.phase} {.metadata.annotations.kubernetes\.io/config\.source}')" "Running file" """, + hint=r"""`grep staticPodPath /var/lib/kubelet/config.yaml` on the node. `kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml` gives you the manifest; write it with `sudo tee`.""", + show=r"""kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml > {WS}/lesson-4.3/node-web.yaml +ssh -n {W1} 'grep staticPodPath /var/lib/kubelet/config.yaml' +ssh {W1} 'sudo tee /etc/kubernetes/manifests/node-web.yaml >/dev/null' < {WS}/lesson-4.3/node-web.yaml +sleep 15 +kubectl get pods -o wide"""), + ], + "outro": r"""You read the cluster's kubeadm configuration, made a join command, renewed a certificate and made the API server load it, and ran a static Pod on a worker. + +**For the exam** + +- `kubeadm init --pod-network-cidr=... [--control-plane-endpoint=...] [--upload-certs]`, then copy `admin.conf` to `~/.kube/config`, then install a CNI. `kubeadm join` on the others. `kubeadm reset` undoes either. +- `kubeadm token create --print-join-command`; for a control-plane node add `--certificate-key` from `kubeadm init phase upload-certs --upload-certs`. +- `kubeadm certs check-expiration | renew all|apiserver|...`; restart the static Pods afterwards. `kubeadm kubeconfig user --client-name=NAME` writes a kubeconfig for a user. +- Static Pods: the kubelet's `staticPodPath` (`/etc/kubernetes/manifests`); mirror Pods are named `POD-NODE`. + +Next, prove it against the clock: **drill 4.3**.""", +} + +# ---- 4.4 cluster lifecycle: etcd and a real upgrade ---------------------------------------------- +_APT = r""" +pkgs() { nx "$1" "dpkg-query -W -f='\${Version} ' kubelet kubeadm kubectl" v1.37, apt install kubeadm kubeadm upgrade plan / apply v1.37.x + drain, apt install kubelet kubectl, systemctl daemon-reload, restart kubelet, uncordon +each worker drain (from the control plane); on the node: repo line, kubeadm, kubeadm upgrade node, + kubelet kubectl, restart kubelet; uncordon +``` + +pkgs.k8s.io has one repository per minor, so the first step on every node is the line in `/etc/apt/sources.list.d/kubernetes.list`. The packages are held: `apt-mark unhold` before, `hold` after. + +Namespace `life` has ConfigMap `marker`; etcd's certificates are under `/etc/kubernetes/pki/etcd/`.""", + "setup": NS + r""" +mkns life +$K -n life create configmap marker --from-literal=state=before-backup >/dev/null +nx "$CP" 'mkdir -p /opt/backup; rm -f /opt/backup/*.db' /dev/null 2>&1' /dev/null \ + && grep -qi 'total key' "$WS/lesson-4.4/snapshot.txt" """, + hint=r"""`etcdctl --endpoints=https://127.0.0.1:2379 --cacert=.../ca.crt --cert=.../server.crt --key=.../server.key snapshot save FILE`, as root. Since etcd 3.6, `status` and `restore` are `etcdutl` only.""", + show=r"""sudo grep -E -- '--(listen-client-urls|cert-file|key-file|trusted-ca-file)' /etc/kubernetes/manifests/etcd.yaml +sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup/etcd.db +sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/snapshot.txt"""), + step("Lose something", + r"""Now the mistake a backup is for: delete ConfigMap `marker` in `life`.""", + check=r"""[ -z "$(j -n life cm marker '{.metadata.name}')" ] && nx "$CP" 'test -s /opt/backup/etcd.db' /dev/null 2>&1 && break; sleep 3; done +kubectl -n life get configmap marker -o yaml | grep state"""), + step("Plan the upgrade", + r"""On the control-plane node, point the Kubernetes repository at **v1.37**, install the **newest 1.37 kubeadm** the repository offers (unhold it first, hold it again after), and save the output of `kubeadm upgrade plan` in `{WS}/lesson-4.4/plan.txt`. + +The plan shows two choices: the newest 1.36 patch, and the 1.37 release your kubeadm can go to.""", + check=_APT + r""" +t=$(newest); [ -n "$t" ] || exit 1 +eq "$(ver "$CP" kubeadm)" "$t" && held "$CP" && grep -q "kubeadm upgrade apply v${t%%-*}" "$WS/lesson-4.4/plan.txt" 2>/dev/null""", + hint=r"""`sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list`, `sudo apt-get update`, `apt-cache madison kubeadm` (newest first), `sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.X-1.1 && sudo apt-mark hold kubeadm`.""", + show=r"""sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list +sudo apt-get update -qq +apt-cache madison kubeadm | head -3 +v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}') +sudo apt-mark unhold kubeadm && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$v && sudo apt-mark hold kubeadm +kubeadm version -o short +sudo kubeadm upgrade plan | tee {WS}/lesson-4.4/plan.txt | tail -12"""), + step("Upgrade the control plane", + r"""Apply the upgrade to that 1.37 release (`kubeadm upgrade apply`, a few minutes - it upgrades etcd, the static Pods, CoreDNS and kube-proxy). Then drain the node, upgrade `kubelet` and `kubectl` to the same version, reload systemd, restart the kubelet, and uncordon. + +`kubectl get nodes` shows the kubelet's version: the control-plane node says v1.37 only after its kubelet has been upgraded and restarted.""", + check=_APT + r""" +t=$(newest); v="v${t%%-*}" +eq "$($K version -o json 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["serverVersion"]["gitVersion"])')" "$v" || exit 1 +eq "$(j node "$CP" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$CP")" "$t $t $t " && held "$CP" \ + && [ "$(j node "$CP" '{.spec.unschedulable}')" != true ]""", + hint=r"""`sudo kubeadm upgrade apply v1.37.X -y`; `kubectl drain {CP} --ignore-daemonsets`; `sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=... kubectl=... && sudo apt-mark hold kubelet kubectl`; `sudo systemctl daemon-reload && sudo systemctl restart kubelet`; `kubectl uncordon {CP}`.""", + show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}') +sudo kubeadm upgrade apply v${v%%-*} -y | tail -3 +kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data +sudo apt-mark unhold kubelet kubectl && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$v kubectl=$v && sudo apt-mark hold kubelet kubectl +sudo systemctl daemon-reload && sudo systemctl restart kubelet +for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done +kubectl get nodes"""), + step("Upgrade the worker", + r"""Now `{W1}`: drain it from the control-plane node; on `{W1}`, the repository line, kubeadm, **`kubeadm upgrade node`** (a worker has no control-plane Pods to replace - it only refreshes the kubelet's configuration), kubelet and kubectl, daemon-reload, restart; then uncordon it. + +Both nodes then report the new version, and every package is held again.""", + check=_APT + r""" +t=$(newest); v="v${t%%-*}" +eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$W1")" "$t $t $t " && held "$W1" \ + && [ "$(j node "$W1" '{.spec.unschedulable}')" != true ] && eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""", + hint=r"""The same packages, from the same repository line, as on the control plane - only `kubeadm upgrade node` instead of `apply`. Drain and uncordon are kubectl commands: run them on the control-plane node.""", + show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}') +kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data +ssh {W1} sudo bash -s $v <<'EOF' +sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list +apt-get update -qq +apt-mark unhold kubeadm kubelet kubectl >/dev/null +DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$1 +kubeadm upgrade node | tail -2 +DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$1 kubectl=$1 +apt-mark hold kubeadm kubelet kubectl +systemctl daemon-reload && systemctl restart kubelet +EOF +kubectl uncordon {W1} +kubectl get nodes"""), + ], + "outro": r"""You backed etcd up, restored it after a loss, and took a whole cluster from v1.36 to v1.37: control plane first, then the worker, with the packages held at every step. + +**For the exam** + +- etcd: `etcdctl snapshot save` (with `--endpoints --cacert --cert --key`), `etcdutl snapshot status`, `etcdutl snapshot restore --data-dir NEWDIR`, then the `etcd-data` `hostPath` in `etcd.yaml` - or restore in place, as here. +- Upgrade one minor at a time; kubeadm before `upgrade apply`, kubelet after it. `kubeadm upgrade plan` names the versions; take the exact patch the task asks for, or the newest. +- Workers: `kubeadm upgrade node`, never `apply`. Drain and uncordon from where kubectl works. +- Version skew: kubelets may be up to three minors behind the API server, never ahead. +- `apt-mark unhold` → `apt-get install PKG=VERSION` → `apt-mark hold`; `systemctl daemon-reload` before restarting the kubelet. + +Next, prove it against the clock: **drill 4.4**.""", +} + +# ---- 4.5 highly available control plane -------------------------------------------------------- +L45 = { + "id": 9405, "objective": "4.5", "workers": 0, + "title": "A highly available control plane: endpoint, etcd members, leaders, joining", + "intro": r"""An HA control plane is three (or five) control-plane nodes behind one address: + +``` +controlPlaneEndpoint one stable address - a load balancer or a VIP - for every kubelet and kubectl +etcd stacked: a member on each control-plane node external: hosts of its own +kube-apiserver active on every node at once, behind the endpoint +kube-scheduler, kube-controller-manager one leader at a time, chosen through a Lease in kube-system +``` + +etcd needs a majority (quorum): three members survive one failure, five survive two. A second control-plane node joins with the certificates the first uploads (`kubeadm init phase upload-certs --upload-certs`, valid two hours) and `kubeadm join ... --control-plane --certificate-key KEY`. + +This cluster has one control-plane node, `{CP}`, made with a `controlPlaneEndpoint` so that more can join.""", + "setup": r""" +mkdir -p "$WS/lesson-4.5" +$K -n kube-system delete secret kubeadm-certs --ignore-not-found >/dev/null 2>&1 +""", + "steps": [ + step("The one stable address", + r"""Write the cluster's control-plane endpoint, as kubeadm has it, to `{WS}/lesson-4.5/endpoint.txt`. It is also the `server:` of every kubeconfig on the nodes.""", + check=r""" +want=$($K -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}') +[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-4.5/endpoint.txt" 2>/dev/null)" "$want" """, + hint=r"""`controlPlaneEndpoint:` in the `ClusterConfiguration` of ConfigMap `kubeadm-config`.""", + show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}' > {WS}/lesson-4.5/endpoint.txt +cat {WS}/lesson-4.5/endpoint.txt +sudo grep server: /etc/kubernetes/kubelet.conf"""), + step("etcd's members and its leader", + r"""Ask etcd itself. Write to `{WS}/lesson-4.5/etcd.txt`: + +``` +members= +leader= +```""", + check=r""" +out=$(nx "$CP" '""" + _ETCDCTL + r""" endpoint status -w json' /dev/null | grep .)" "$(printf 'members=%s\nleader=%s' "$n" "$lead")" """, + hint=r"""`etcdctl member list -w table` and `etcdctl endpoint status -w table` (the IS LEADER column), with the endpoint and certificates as for a snapshot. `export ETCDCTL_ENDPOINTS=... ETCDCTL_CACERT=...` saves typing.""", + show=r"""sudo bash <<'EOF' +export ETCDCTL_API=3 ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \ + ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key +etcdctl member list -w table +etcdctl endpoint status -w table +EOF +printf 'members=1\nleader={CP}\n' > {WS}/lesson-4.5/etcd.txt"""), + step("Who leads the scheduler and the controller manager", + r"""Only one kube-scheduler and one kube-controller-manager act at a time; the others wait for the **Lease** in `kube-system` to expire. Write the node of each holder to `{WS}/lesson-4.5/leaders.txt`: + +``` +scheduler= +controller-manager= +```""", + check=r""" +s=$(j -n kube-system lease kube-scheduler '{.spec.holderIdentity}'); c=$(j -n kube-system lease kube-controller-manager '{.spec.holderIdentity}') +eq "$(tr -d ' \r' < "$WS/lesson-4.5/leaders.txt" 2>/dev/null | grep .)" "$(printf 'scheduler=%s\ncontroller-manager=%s' "${s%%_*}" "${c%%_*}")" """, + hint=r"""`kubectl -n kube-system get lease kube-scheduler kube-controller-manager` - HOLDER is `NODE_UUID`.""", + show=r"""kubectl -n kube-system get lease kube-scheduler kube-controller-manager +s=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}') +c=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}') +printf 'scheduler=%s\ncontroller-manager=%s\n' "${s%%_*}" "${c%%_*}" > {WS}/lesson-4.5/leaders.txt"""), + step("What a second control-plane node needs", + r"""Upload the control-plane certificates for a joining node, and save the complete `kubeadm join` command a new **control-plane** node would run - new token, CA hash, `--control-plane` and the certificate key - to `{WS}/lesson-4.5/join-cp.sh`, on one line. Do not run it anywhere.""", + check=r""" +f="$WS/lesson-4.5/join-cp.sh"; grep -q '^kubeadm join .*--token .*--discovery-token-ca-cert-hash sha256:.*--control-plane' "$f" 2>/dev/null || exit 1 +key=$(grep -o -- '--certificate-key [0-9a-f]\{64\}' "$f" | awk '{print $2}'); [ -n "$key" ] && [ -n "$(j -n kube-system secret kubeadm-certs '{.metadata.name}')" ]""", + hint=r"""`sudo kubeadm init phase upload-certs --upload-certs` prints the key as its last line; `sudo kubeadm token create --print-join-command --certificate-key KEY` adds `--control-plane --certificate-key` itself.""", + show=r"""key=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1) +sudo kubeadm token create --print-join-command --certificate-key $key > {WS}/lesson-4.5/join-cp.sh +cat {WS}/lesson-4.5/join-cp.sh +kubectl -n kube-system get secret kubeadm-certs"""), + ], + "outro": r"""You read the endpoint that makes HA possible, asked etcd for its members and leader, found the leaders of the scheduler and controller manager, and prepared a control-plane join. + +**For the exam** + +- HA starts at `kubeadm init --control-plane-endpoint LB:6443 --upload-certs`; without an endpoint, adding control-plane nodes later means re-issuing certificates. +- Stacked etcd: an etcd static Pod on each control-plane node. External: `etcd.external.endpoints` in the ClusterConfiguration, and the API server's `--etcd-servers` names other hosts. +- Quorum is a majority of members: an even number adds no tolerance. +- The uploaded certificates expire after two hours; re-run `upload-certs` for a fresh key. + +Next, prove it against the clock: **drill 4.5**.""", +} + +# ---- 4.6 Helm and Kustomize ------------------------------------------------------------------- +_CHART = r""" +c="$WS/lesson-4.6/chart"; rm -rf "$WS/lesson-4.6"; mkdir -p "$c/templates" "$WS/lesson-4.6/kustomize/base" +cat > "$c/Chart.yaml" <<'EOF' +apiVersion: v2 +name: shop +description: A small web shop +type: application +version: 0.1.0 +appVersion: "1.27" +EOF +cat > "$c/values.yaml" <<'EOF' +replicaCount: 1 +image: + repository: nginx + tag: "1.27" +service: + type: ClusterIP + port: 80 +EOF +cat > "$c/templates/deployment.yaml" <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ .Release.Name }}-shop + labels: {app: {{ .Release.Name }}-shop} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: {app: {{ .Release.Name }}-shop} + template: + metadata: + labels: {app: {{ .Release.Name }}-shop} + spec: + containers: + - name: shop + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + ports: [{containerPort: 80}] +EOF +cat > "$c/templates/service.yaml" <<'EOF' +apiVersion: v1 +kind: Service +metadata: + name: {{ .Release.Name }}-shop +spec: + type: {{ .Values.service.type }} + selector: {app: {{ .Release.Name }}-shop} + ports: [{port: {{ .Values.service.port }}, targetPort: 80}] +EOF +b="$WS/lesson-4.6/kustomize/base" +cat > "$b/deployment.yaml" <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: api +spec: + replicas: 1 + selector: + matchLabels: {app: api} + template: + metadata: + labels: {app: api} + spec: + containers: + - name: api + image: nginx:1.26 +EOF +cat > "$b/kustomization.yaml" <<'EOF' +resources: +- deployment.yaml +EOF +helm -n apps uninstall web >/dev/null 2>&1 || true +""" + +L46 = { + "id": 9406, "objective": "4.6", "workers": 1, + "title": "Helm releases and Kustomize overlays", + "intro": r"""Two ways to install cluster components from more than a single manifest: + +``` +Helm a chart = templates + values.yaml helm install | upgrade | rollback | uninstall + a release = one installed copy, with numbered revisions (stored as Secrets in its Namespace) +Kustomize a base + overlays that change it, no templates kubectl apply -k DIR / kubectl kustomize DIR +``` + +With Helm you change **values** (`--set`, `-f values.yaml`), never the templates; `helm template` renders without installing. With Kustomize you never edit the base; an overlay sets the Namespace, prefixes, images, replicas, labels, and patches the rest. + +A chart is at `{WS}/lesson-4.6/chart` (values `replicaCount`, `image.tag`, `service.type`, `service.port`); a Kustomize base is at `{WS}/lesson-4.6/kustomize/base`.""", + "setup": _CHART, + "steps": [ + step("Render before you install", + r"""Render the chart as release `web` with `replicaCount=2`, without installing anything, and save the manifests in `{WS}/lesson-4.6/rendered.yaml`. That is how you check what a set of values does.""", + check=r"""f="$WS/lesson-4.6/rendered.yaml"; grep -q 'name: web-shop' "$f" 2>/dev/null && grep -q 'replicas: 2' "$f" && [ -z "$(j -n apps deploy web-shop '{.metadata.name}')" ]""", + hint=r"""`helm template RELEASE CHART --set KEY=VALUE`.""", + show=r"""helm template web {WS}/lesson-4.6/chart --set replicaCount=2 > {WS}/lesson-4.6/rendered.yaml +grep -E 'kind:|replicas:|image:' {WS}/lesson-4.6/rendered.yaml"""), + step("Install a release", + r"""Install the chart as release `web` in Namespace `apps` (create it), with `replicaCount=2`.""", + check=r"""eq "$(helm -n apps status web -o json 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["info"]["status"], d["version"])')" "deployed 1" && eq "$(j -n apps deploy web-shop '{.status.readyReplicas}')" 2""", + hint=r"""`helm install web CHART -n apps --create-namespace --set replicaCount=2`; `helm list -n apps`.""", + show=r"""helm install web {WS}/lesson-4.6/chart -n apps --create-namespace --set replicaCount=2 +helm list -n apps +kubectl -n apps rollout status deploy web-shop --timeout=120s"""), + step("Upgrade it, keeping what you set", + r"""Upgrade release `web` so that its Service becomes a `NodePort` - and keep `replicaCount=2`. A plain `helm upgrade --set service.type=NodePort` would quietly put the replica count back to the chart's default.""", + check=r"""eq "$(j -n apps svc web-shop '{.spec.type}')" NodePort && eq "$(j -n apps deploy web-shop '{.spec.replicas}')" 2 \ + && [ "$(helm -n apps history web -o json 2>/dev/null | python3 -c 'import json,sys; print(len(json.load(sys.stdin)))')" -ge 2 ]""", + hint=r"""`--reuse-values` keeps the values of the last release and adds yours; `helm get values web -n apps` shows what is set.""", + show=r"""helm upgrade web {WS}/lesson-4.6/chart -n apps --reuse-values --set service.type=NodePort +helm get values web -n apps +kubectl -n apps get svc web-shop"""), + step("Roll the release back", + r"""The NodePort was a mistake: roll release `web` back to **revision 1**. A rollback is itself a new revision.""", + check=r""" +eq "$(j -n apps svc web-shop '{.spec.type}')" ClusterIP || exit 1 +helm -n apps history web -o json 2>/dev/null | python3 -c ' +import json, sys; h = json.load(sys.stdin) +sys.exit(0 if len(h) >= 3 and h[-1]["status"] == "deployed" and "Rollback to 1" in h[-1]["description"] else 1)'""", + hint=r"""`helm rollback web 1 -n apps`, then `helm history web -n apps`.""", + show=r"""helm rollback web 1 -n apps +helm history web -n apps +kubectl -n apps get svc web-shop"""), + step("An overlay for production", + r"""Create a Kustomize overlay at `{WS}/lesson-4.6/kustomize/prod` on top of the base, which: puts everything in Namespace `prod`, prefixes names with `prod-`, runs **3** replicas, and uses image `nginx:1.27`. Apply it. Leave the base as it is.""", + check=r"""eq "$(j -n prod deploy prod-api '{.spec.replicas} {.spec.template.spec.containers[0].image} {.status.readyReplicas}')" "3 nginx:1.27 3" \ + && grep -q 'image: nginx:1.26' "$WS/lesson-4.6/kustomize/base/deployment.yaml" """, + hint=r"""A `kustomization.yaml` with `resources: [../base]`, `namespace: prod`, `namePrefix: prod-`, `replicas: [{name: api, count: 3}]`, `images: [{name: nginx, newTag: "1.27"}]`. The Namespace itself must exist (or be one of the resources). `kubectl kustomize DIR` shows the result first.""", + show=r"""mkdir -p {WS}/lesson-4.6/kustomize/prod +cat > {WS}/lesson-4.6/kustomize/prod/kustomization.yaml <<'EOF' +resources: +- ../base +namespace: prod +namePrefix: prod- +replicas: +- name: api + count: 3 +images: +- name: nginx + newTag: "1.27" +EOF +kubectl kustomize {WS}/lesson-4.6/kustomize/prod +kubectl create namespace prod +kubectl apply -k {WS}/lesson-4.6/kustomize/prod +kubectl -n prod rollout status deploy prod-api --timeout=120s"""), + ], + "outro": r"""You rendered a chart, installed, upgraded and rolled back a release, and built a production overlay with Kustomize without touching its base. + +**For the exam** + +- `helm repo add NAME URL && helm repo update`, `helm search repo KEYWORD`, `helm show values CHART`, `helm pull CHART --untar`. +- `helm install|upgrade RELEASE CHART -n NS [--create-namespace] [--set k=v] [-f values.yaml] [--version X]`; `helm upgrade --install` for either. `--reuse-values` on an upgrade, or you lose earlier `--set`s. +- `helm list -A`, `helm history`, `helm rollback RELEASE REVISION`, `helm get values|manifest`, `helm uninstall`. +- Kustomize: `resources`, `namespace`, `namePrefix`/`nameSuffix`, `labels`, `images`, `replicas`, `configMapGenerator`/`secretGenerator`, `patches` (strategic merge or JSON 6902). `kubectl apply -k`, `kubectl delete -k`. + +Next, prove it against the clock: **drill 4.6**.""", +} + +# ---- 4.7 extension interfaces ----------------------------------------------------------------- +L47 = { + "id": 9407, "objective": "4.7", "workers": 1, + "title": "CNI, CRI and CSI: what plugs into this cluster", + "intro": r"""Kubernetes leaves three jobs to plugins, each behind an interface: + +``` +CRI Container Runtime Interface the kubelet asks the runtime (containerd, CRI-O) over a socket crictl +CNI Container Network Interface the runtime calls the plugins in /etc/cni/net.d for each Pod sandbox +CSI Container Storage Interface a driver (Deployment + DaemonSet) makes and mounts volumes CSIDriver, CSINode +``` + +A **RuntimeClass** names a handler the runtime has configured - `runc`, or a sandbox such as `runsc` (gVisor) - and a Pod picks it with `runtimeClassName`. + +This cluster runs containerd, kindnet as its CNI, and the local-path provisioner for storage. Its worker is `{W1}`.""", + "setup": NS + r""" +mkns plug +mkdir -p "$WS/lesson-4.7" +$K -n plug run probe --image=busybox:1.36 -- sleep 86400 >/dev/null +$K -n plug wait --for=condition=Ready pod/probe --timeout=180s >/dev/null +""", + "steps": [ + step("The CNI chain on a node", + r"""On `{W1}`, find the CNI configuration (the runtime uses the first file in `/etc/cni/net.d`, by name) and write to `{WS}/lesson-4.7/cni.txt`: + +``` +config= +plugins= +```""", + check=r""" +f=$(nx "$W1" 'ls /etc/cni/net.d/*.conf* 2>/dev/null | sort | head -1' /dev/null | grep .)" "$(printf 'config=%s\nplugins=%s' "$f" "$p")" """, + hint=r"""`ls /etc/cni/net.d/` and `cat` the file: a `.conflist` has a `"plugins"` list, each with a `"type"` - the binary under `/opt/cni/bin` that is run.""", + show=r"""ssh -n {W1} 'ls /etc/cni/net.d/; cat /etc/cni/net.d/10-kindnet.conflist; ls /opt/cni/bin' +f=$(ssh -n {W1} 'ls /etc/cni/net.d/*.conf* | sort | head -1') +p=$(ssh -n {W1} "cat $f" | jq -r '[.plugins[].type] | join(",")') +printf 'config=%s\nplugins=%s\n' "$f" "$p" > {WS}/lesson-4.7/cni.txt +cat {WS}/lesson-4.7/cni.txt"""), + step("The runtime, through CRI", + r"""Write to `{WS}/lesson-4.7/cri.txt`: + +``` +endpoint= +runtime= +```""", + check=r""" +e=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["containerRuntimeEndpoint"])') +r=$(j node "$W1" '{.status.nodeInfo.containerRuntimeVersion}') +eq "$(tr -d ' \r' < "$WS/lesson-4.7/cri.txt" 2>/dev/null | grep .)" "$(printf 'endpoint=%s\nruntime=%s' "$e" "$r")" """, + hint=r"""`containerRuntimeEndpoint` in the kubelet's configuration (`/var/lib/kubelet/config.yaml`, or `configz`); `kubectl get node -o wide` shows CONTAINER-RUNTIME.""", + show=r"""e=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.containerRuntimeEndpoint) +r=$(kubectl get node {W1} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}') +printf 'endpoint=%s\nruntime=%s\n' "$e" "$r" > {WS}/lesson-4.7/cri.txt +cat {WS}/lesson-4.7/cri.txt"""), + step("A Pod's container, from the runtime's side", + r"""Pod `probe` in `plug` runs on `{W1}`. Ask the runtime about it there, with `crictl`: find its container, and write the container's **process ID on the node** to `{WS}/lesson-4.7/pid.txt`.""", + check=r""" +want=$(nx "$W1" 'crictl inspect $(crictl ps --name probe -q | head -1) 2>/dev/null' /dev/null)" "$want" """, + hint=r"""`sudo crictl pods --name probe`, `sudo crictl ps --name probe`, `sudo crictl inspect ID` - the PID is under `info.pid`. `crictl` talks to the same socket as the kubelet.""", + show=r"""ssh -n {W1} 'sudo crictl pods --name probe; sudo crictl ps --name probe' +ssh -n {W1} 'sudo crictl inspect $(sudo crictl ps --name probe -q | head -1)' | jq .info.pid > {WS}/lesson-4.7/pid.txt +cat {WS}/lesson-4.7/pid.txt"""), + step("A RuntimeClass", + r"""Create RuntimeClass `plain` for containerd's handler `runc`, and run Pod `classy` in `plug` (`busybox:1.36`, `sleep 86400`) with that runtime class. A Pod that names a handler the node does not have stays in `ContainerCreating`.""", + check=r"""eq "$(j runtimeclass plain '{.handler}')" runc && eq "$(j -n plug pod classy '{.spec.runtimeClassName} {.status.phase}')" "plain Running" """, + hint=r"""A RuntimeClass is `node.k8s.io/v1` with a top-level `handler:` and no `spec`. The Pod: `spec.runtimeClassName: plain`.""", + show=r"""kubectl apply -f - <<'EOF' +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: plain +handler: runc +EOF +kubectl -n plug run classy --image=busybox:1.36 --overrides='{"spec":{"runtimeClassName":"plain"}}' -- sleep 86400 +kubectl -n plug wait --for=condition=Ready pod/classy --timeout=120s +kubectl get runtimeclass"""), + step("Storage drivers", + r"""Write to `{WS}/lesson-4.7/csi.txt`: + +``` +csidrivers= +provisioner= +``` + +A CSI driver registers a `CSIDriver` object and appears in each node's `CSINode`. The local-path provisioner is not one - it is an external provisioner of its own.""", + check=r""" +n=$($K get csidrivers --no-headers 2>/dev/null | grep -c .) +p=$($K get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.provisioner}{end}') +eq "$(tr -d ' \r' < "$WS/lesson-4.7/csi.txt" 2>/dev/null | grep .)" "$(printf 'csidrivers=%s\nprovisioner=%s' "$n" "$p")" """, + hint=r"""`kubectl get csidrivers`, `kubectl get csinodes`, `kubectl get sc`.""", + show=r"""kubectl get csidrivers +kubectl get csinodes +kubectl get sc +printf 'csidrivers=%s\nprovisioner=%s\n' "$(kubectl get csidrivers --no-headers 2>/dev/null | grep -c .)" "$(kubectl get sc standard -o jsonpath='{.provisioner}')" > {WS}/lesson-4.7/csi.txt +cat {WS}/lesson-4.7/csi.txt"""), + ], + "outro": r"""You read the CNI chain on a node, the runtime the kubelet talks to, a container as the runtime sees it, ran a Pod with a RuntimeClass, and counted the storage drivers. + +**For the exam** + +- CNI: `/etc/cni/net.d/` (config), `/opt/cni/bin/` (plugins); the CNI itself (Calico, Cilium, Flannel, ...) is installed as a manifest or a Helm chart after `kubeadm init`, and its Pods run in `kube-system` or a Namespace of its own. +- CRI: `crictl ps|pods|images|inspect|logs|exec`; the endpoint is in `/etc/crictl.yaml` and in the kubelet's `containerRuntimeEndpoint`. +- RuntimeClass `handler` must match a runtime configured in containerd (`[plugins..."cri".containerd.runtimes.NAME]`). +- CSI: `kubectl get csidrivers,csinodes`; a StorageClass names the driver as its `provisioner`. + +Next, prove it against the clock: **drill 4.7**.""", +} + +# ---- 4.8 CRDs and operators -------------------------------------------------------------------- +_OPERATOR = r""" +o="$WS/lesson-4.8/operator"; rm -rf "$WS/lesson-4.8"; mkdir -p "$o" +cat > "$o/operator.yaml" <<'EOF' +""" + backup_operator() + r"""EOF +$K delete crd backups.data.example.com --ignore-not-found >/dev/null 2>&1 +""" + +L48 = { + "id": 9408, "objective": "4.8", "workers": 1, + "title": "CustomResourceDefinitions, and an operator that acts on them", + "intro": r"""A **CustomResourceDefinition** adds a kind to the API: once it is `Established`, you create, list and watch its objects like any other, and its OpenAPI schema validates them and fills in defaults. + +``` +CRD name PLURAL.GROUP backups.data.example.com +names kind, plural, singular, shortNames +scope Namespaced | Cluster +versions each served, exactly one storage: true, each with a schema (and subresources: status) +``` + +An **operator** is a CRD plus a controller - a Deployment with RBAC - that watches those objects and makes the world match them. Installing one is applying its manifests or its Helm chart; using it is creating custom resources. + +The manifests of a small backup operator are in `{WS}/lesson-4.8/operator/`. It expects a `Backup` kind that you will define.""", + "setup": NS + _OPERATOR + r""" +mkns data +""", + "steps": [ + step("Define the kind", + r"""Create CRD `backups.data.example.com`: group `data.example.com`, kind `Backup` (plural `backups`, singular `backup`, short name `bk`), **Namespaced**, one version `v1` (served, storage) with: + +- `spec.schedule`: string, **required** +- `spec.retention`: integer, at least `1`, **default `7`** +- a `status` subresource, with `status.phase` a string""", + check=PY + r""" +c=backups.data.example.com +eq "$(j crd $c '{.spec.scope} {.spec.names.kind} {.spec.names.shortNames[0]}')" "Namespaced Backup bk" || exit 1 +py crd $c 's = o["spec"]["versions"][0]; p = s["schema"]["openAPIV3Schema"]["properties"]["spec"] +s["served"] and s["storage"] and "status" in s.get("subresources", {}) and "schedule" in p.get("required", []) and p["properties"]["retention"].get("default") == 7 and p["properties"]["retention"].get("minimum") == 1' \ + && eq "$(j crd $c '{.status.conditions[?(@.type=="Established")].status}')" True""", + hint=r"""The docs page "Extend the Kubernetes API with CustomResourceDefinitions" has the skeleton. `required:` sits beside `properties:`, at the level of `spec`'s schema; `default:` and `minimum:` inside `retention`. `subresources: {status: {}}` goes under the version.""", + show=r"""kubectl apply -f - <<'EOF' +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: backups.data.example.com +spec: + group: data.example.com + scope: Namespaced + names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]} + versions: + - name: v1 + served: true + storage: true + subresources: + status: {} + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + required: [schedule] + properties: + schedule: {type: string} + retention: {type: integer, minimum: 1, default: 7} + status: + type: object + properties: + phase: {type: string} +EOF +kubectl wait --for condition=established crd/backups.data.example.com +kubectl api-resources --api-group=data.example.com"""), + step("Defaults and validation", + r"""In Namespace `data`, create Backup `nightly` with `schedule: "0 2 * * *"` and no retention: the API server fills in `7`. Then try a Backup `broken` with `retention: 0` - it is refused. Save the refusal (kubectl writes it to **stderr**) in `{WS}/lesson-4.8/refused.txt`.""", + check=r""" +eq "$(j -n data bk nightly '{.spec.schedule} {.spec.retention}')" "0 2 * * * 7" && [ -z "$(j -n data bk broken '{.metadata.name}')" ] \ + && grep -q 'retention' "$WS/lesson-4.8/refused.txt" 2>/dev/null && grep -qi 'greater than or equal to 1' "$WS/lesson-4.8/refused.txt" """, + hint=r"""`apiVersion: data.example.com/v1`, `kind: Backup`. Redirect with `2>` - with `>` the file stays empty.""", + show=r"""kubectl -n data apply -f - <<'EOF' +apiVersion: data.example.com/v1 +kind: Backup +metadata: + name: nightly +spec: + schedule: "0 2 * * *" +EOF +kubectl -n data get bk nightly -o jsonpath='{.spec}'; echo +printf 'apiVersion: data.example.com/v1\nkind: Backup\nmetadata: {name: broken}\nspec: {schedule: "@hourly", retention: 0}\n' | kubectl -n data apply -f - 2> {WS}/lesson-4.8/refused.txt +cat {WS}/lesson-4.8/refused.txt"""), + step("What the schema says", + r"""`kubectl explain` reads the CRD's schema like any built-in kind's. Save the output of `kubectl explain backup.spec` in `{WS}/lesson-4.8/explain.txt`.""", + check=r"""f="$WS/lesson-4.8/explain.txt"; grep -q 'retention' "$f" 2>/dev/null && grep -q 'schedule' "$f" && grep -qi 'required' "$f" """, + hint=r"""`kubectl explain backup.spec` - or `--recursive` for the whole tree.""", + show=r"""kubectl explain backup.spec | tee {WS}/lesson-4.8/explain.txt"""), + step("Install the operator", + r"""Install the operator from `{WS}/lesson-4.8/operator/` and wait until its controller runs. Read the manifests first: a Namespace, a ServiceAccount, a ClusterRole for `backups` and ConfigMaps, the binding, and the controller's Deployment.""", + check=r"""eq "$(j -n backup-operator deploy controller '{.status.readyReplicas}')" 1 && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller""", + hint=r"""`kubectl apply -f DIR` applies every file in it; `kubectl -n backup-operator rollout status deploy controller`.""", + show=r"""grep -E '^kind:|^ name:' {WS}/lesson-4.8/operator/operator.yaml +kubectl apply -f {WS}/lesson-4.8/operator/ +kubectl -n backup-operator rollout status deploy controller --timeout=120s"""), + step("Let it work", + r"""Create Backup `weekly` in `data` with `schedule: "0 3 * * 0"`. Within a few seconds the controller has acted on it - and on `nightly`: each has `status.phase: Scheduled`, and a ConfigMap `backup-NAME` holds its schedule. `kubectl -n backup-operator logs deploy/controller` is where you would look if it did not.""", + check=r""" +eq "$(j -n data bk weekly '{.status.phase}')" Scheduled && eq "$(j -n data cm backup-weekly '{.data.schedule}')" "0 3 * * 0" && eq "$(j -n data bk nightly '{.status.phase}')" Scheduled""", + hint=r"""A Backup like `nightly`. `kubectl -n data get bk` - add `additionalPrinterColumns` to the CRD and the phase would show there too.""", + show=r"""kubectl -n data apply -f - <<'EOF' +apiVersion: data.example.com/v1 +kind: Backup +metadata: + name: weekly +spec: + schedule: "0 3 * * 0" +EOF +sleep 10 +kubectl -n data get bk -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PHASE:.status.phase +kubectl -n data get cm"""), + ], + "outro": r"""You defined a kind with validation and defaults, saw the schema refuse bad objects, installed an operator with its RBAC, and watched it act on your custom resources. + +**For the exam** + +- `kubectl get crd`, `kubectl api-resources --api-group=GROUP`, `kubectl explain KIND.spec --recursive` - the fastest way to learn an unfamiliar custom resource. +- Wait for `Established` before creating objects of a new kind. +- Installing an operator: its manifests (`kubectl apply -f URL|DIR`) or chart (`helm install`); then check its CRDs, its controller's Pods and logs, and create the custom resource the task asks for. +- Deleting a CRD deletes every object of its kind. + +Next, prove it against the clock: **drill 4.8**.""", +} + +LESSONS = [L41, L42, L43, L44, L45, L46, L47, L48] diff --git a/cka/lessons_net.py b/cka/lessons_net.py new file mode 100644 index 0000000..f529fd3 --- /dev/null +++ b/cka/lessons_net.py @@ -0,0 +1,676 @@ +"""CKA lessons of the Services and Networking domain (5.1-5.6). See cka/lessons.py for what a lesson is.""" +from cka.common import DNS +from sim.bank import NET, NS, PODS, step + +# ---- 5.1 connectivity between Pods ------------------------------------------------------------ +L51 = { + "id": 9501, "objective": "5.1", "workers": 2, + "title": "The Pod network: addresses, routes between nodes, and the host network", + "intro": r"""Kubernetes asks three things of a network, and leaves the rest to the CNI plugin: + +``` +every Pod has its own IP from the node's slice of the Pod CIDR (node.spec.podCIDR) +every Pod reaches every other Pod across nodes, without NAT +agents on a node reach its Pods the kubelet, for probes +``` + +How the packets get from one node to another is the plugin's business: kindnet, here, adds a route on each node for every other node's Pod CIDR, via that node's address. Others use overlays (VXLAN) or BGP. + +A Pod with `hostNetwork: true` gets no IP of its own - it uses the node's network namespace and address. + +Namespace `net` has Pod `a` on `{W1}` and Pod `b` on `{W2}`; each serves its own name on port 80.""", + "setup": PODS + r""" +mkns net +mkdir -p "$WS/lesson-5.1" +for p in a:$W1 b:$W2; do + $K -n net run "${p%%:*}" --image=busybox:1.36 --overrides="{\"spec\":{\"nodeName\":\"${p#*:}\"}}" --command -- \ + sh -c 'mkdir -p /www && echo "$HOSTNAME" > /www/index.html && exec httpd -f -p 80 -h /www' >/dev/null +done +ready net || exit 1 +""", + "steps": [ + step("Each node's share of the Pod network", + r"""Write each node's Pod CIDR to `{WS}/lesson-5.1/cidrs.txt`, one line per node: `NODE CIDR`. Then compare with the Pod IPs in `kubectl get pods -A -o wide`: every Pod's address is from its node's range.""", + check=r""" +want=$($K get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' | sort) +eq "$(tr -s ' \t' ' ' < "$WS/lesson-5.1/cidrs.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """, + hint=r"""`.spec.podCIDR` of each node: `-o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}'`, or `custom-columns`. The cluster-wide range is the `podSubnet` in kubeadm's configuration.""", + show=r"""kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' > {WS}/lesson-5.1/cidrs.txt +cat {WS}/lesson-5.1/cidrs.txt +kubectl -n net get pods -o wide"""), + step("Across nodes, Pod to Pod", + r"""From Pod `a`, send a request straight to Pod `b`'s **IP address** on port 80 - no Service - and save the answer in `{WS}/lesson-5.1/answer.txt`. It crossed from `{W1}` to `{W2}`.""", + check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/answer.txt" 2>/dev/null)" b""", + hint=r"""`kubectl -n net get pod b -o jsonpath='{.status.podIP}'`, then `kubectl -n net exec a -- wget -qO- -T3 http://IP`.""", + show=r"""ip=$(kubectl -n net get pod b -o jsonpath='{.status.podIP}') +kubectl -n net exec a -- wget -qO- -T3 http://$ip | tee {WS}/lesson-5.1/answer.txt"""), + step("The route that carries it", + r"""On `{W1}`, look at the routing table: there is a route for `{W2}`'s Pod CIDR, via `{W2}`'s node address. Write that route's destination and gateway to `{WS}/lesson-5.1/route.txt`, as `CIDR via ADDRESS`.""", + check=r""" +c=$(j node "$W2" '{.spec.podCIDR}'); gw=$(j node "$W2" '{.status.addresses[?(@.type=="InternalIP")].address}') +nx "$W1" "ip route | grep -q '^$c via $gw '" /dev/null | tr -d '\r\n' | sed 's/ *$//')" "$c via $gw" """, + hint=r"""`ip route` on the node. The node's address is its `InternalIP` (`kubectl get nodes -o wide`).""", + show=r"""ssh -n {W1} ip route +c=$(kubectl get node {W2} -o jsonpath='{.spec.podCIDR}') +ssh -n {W1} ip route | grep "^$c " | awk '{print $1, $2, $3}' > {WS}/lesson-5.1/route.txt +cat {WS}/lesson-5.1/route.txt"""), + step("A Pod on the host's network", + r"""Run Pod `hostnet` in `net` (`busybox:1.36`, `sleep 86400`) with `hostNetwork: true`, on `{W1}`. Its IP is not from the Pod CIDR: it is `{W1}`'s own address, and it sees the node's interfaces. Write its IP to `{WS}/lesson-5.1/hostnet.txt`.""", + check=r""" +want=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}') +eq "$(j -n net pod hostnet '{.spec.hostNetwork} {.status.podIP}')" "true $want" && eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/hostnet.txt" 2>/dev/null)" "$want" """, + hint=r"""`spec.hostNetwork: true` and `spec.nodeName: {W1}` (or a nodeSelector). `kubectl run ... --overrides='{"spec":{...}}'` adds both.""", + show=r"""kubectl -n net run hostnet --image=busybox:1.36 --overrides='{"spec":{"hostNetwork":true,"nodeName":"{W1}"}}' -- sleep 86400 +kubectl -n net wait --for=condition=Ready pod/hostnet --timeout=120s +kubectl -n net get pod hostnet -o jsonpath='{.status.podIP}' > {WS}/lesson-5.1/hostnet.txt +kubectl -n net exec hostnet -- ip -4 addr show eth0"""), + ], + "outro": r"""You read how the Pod network is divided among nodes, sent a request from one node's Pod to another's, found the route that carried it, and ran a Pod on the host network. + +**For the exam** + +- Pod IPs: `kubectl get pods -o wide`; the node's range: `.spec.podCIDR`; the cluster's: kubeadm's `podSubnet` (or the CNI's own configuration). +- Pods reach each other by IP across nodes with no NAT; Services and DNS are on top of that. +- Which CNI runs: its Pods in `kube-system` (or its own Namespace) and its file in `/etc/cni/net.d/`. NetworkPolicy works only if the plugin enforces it. +- `hostNetwork: true` Pods share the node's ports: two of them cannot listen on the same port on one node. + +Next, prove it against the clock: **drill 5.1**.""", +} + +# ---- 5.2 NetworkPolicies ---------------------------------------------------------------------- +L52 = { + "id": 9502, "objective": "5.2", "workers": 1, + "title": "NetworkPolicies: deny by default, then allow exactly what is needed", + "intro": r"""Without NetworkPolicies, every Pod accepts traffic from everywhere. A policy **selects** Pods (`podSelector`) and, for those Pods only, allows the traffic its rules list - everything else in that direction is dropped. Policies add up: a packet passes if any policy allows it. + +``` +podSelector: {} every Pod in the Namespace +from: [{podSelector}] Pods of THIS Namespace +from: [{namespaceSelector}] every Pod of the matching Namespaces +from: [{namespaceSelector, podSelector}] one element: those Pods IN those Namespaces (AND) +from: [{namespaceSelector}, {podSelector}] two elements: either (OR) +``` + +An egress policy also blocks DNS unless it allows UDP and TCP 53 to `kube-dns`. + +Namespace `prod` (label `team=prod`) runs Pods `db`, `api` and `web`, each with a Service of its name on port 80. Namespace `dev` (label `team=dev`) runs `api`, `tester` and `echo`. Nothing is restricted yet.""", + "setup": PODS + r""" +mkns prod; mkns dev +$K label ns prod team=prod --overwrite >/dev/null; $K label ns dev team=dev --overwrite >/dev/null +for p in db api web; do srv prod $p; done +for p in api tester echo; do srv dev $p; done +ready prod && ready dev || exit 1 +""", + "steps": [ + step("Close the Namespace", + r"""Create NetworkPolicy `default-deny` in `prod`: it selects every Pod and allows **no** ingress. From now on, nothing reaches the Pods of `prod` - not even from inside `prod`.""", + check=NET + r""" +eq "$(j -n prod netpol default-deny '{.spec.policyTypes[0]}')" Ingress && [ -z "$(j -n prod netpol default-deny '{.spec.ingress}')" ] || exit 1 +blocked prod api http://db && blocked dev tester http://web.prod""", + hint=r"""`podSelector: {}` and `policyTypes: [Ingress]`, with no `ingress:` list. Test with `kubectl -n prod exec api -- wget -qO- -T3 db`.""", + show=r"""kubectl -n prod apply -f - <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny +spec: + podSelector: {} + policyTypes: [Ingress] +EOF +kubectl -n prod exec api -- wget -qO- -T3 db || echo blocked"""), + step("Let the API reach the database", + r"""Create NetworkPolicy `db-from-api` in `prod`: Pods `app=db` accept TCP 80 from Pods `app=api` **of `prod`**. `web` still cannot reach `db`, and neither can `dev`'s `api` - a `podSelector` alone means "in this Namespace".""", + check=NET + r"""okr prod api http://db && blocked prod web http://db && blocked dev api http://db.prod""", + hint=r"""`podSelector: {matchLabels: {app: db}}`, `ingress: [{from: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}]`.""", + show=r"""kubectl -n prod apply -f - <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: db-from-api +spec: + podSelector: + matchLabels: {app: db} + policyTypes: [Ingress] + ingress: + - from: + - podSelector: + matchLabels: {app: api} + ports: + - {protocol: TCP, port: 80} +EOF +kubectl -n prod exec api -- wget -qO- -T3 db +kubectl -n prod exec web -- wget -qO- -T3 db || echo blocked"""), + step("Those Pods in that Namespace", + r"""Create NetworkPolicy `web-from-dev-api` in `prod`: Pods `app=web` accept TCP 80 from Pods `app=api` **in** Namespaces labelled `team=dev` - both conditions in **one** element of `from`. + +`dev`'s `api` gets in; `dev`'s `tester` does not, and neither does `prod`'s own `api`. Written as two elements, the rule would let every Pod of `dev` in, and every `api` Pod of `prod` too.""", + check=NET + r"""okr dev api http://web.prod && blocked dev tester http://web.prod && blocked prod api http://web""", + hint=r"""`from: [{namespaceSelector: {matchLabels: {team: dev}}, podSelector: {matchLabels: {app: api}}}]` - mind the single `-`.""", + show=r"""kubectl -n prod apply -f - <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: web-from-dev-api +spec: + podSelector: + matchLabels: {app: web} + policyTypes: [Ingress] + ingress: + - from: + - namespaceSelector: + matchLabels: {team: dev} + podSelector: + matchLabels: {app: api} + ports: + - {protocol: TCP, port: 80} +EOF +kubectl -n dev exec api -- wget -qO- -T3 web.prod +kubectl -n dev exec tester -- wget -qO- -T3 web.prod || echo blocked"""), + step("Egress, and DNS", + r"""Now limit what `tester` in `dev` may **send**: NetworkPolicy `tester-egress` in `dev` selects `app=tester` and allows egress only to `dev`'s `app=api` Pods on TCP 80 - and DNS, to `kube-dns` in `kube-system` on UDP and TCP 53. + +`tester` reaches `api` by its name; `echo` is now out of its reach. Without the DNS rule, even `api` would fail - the name would not resolve.""", + check=NET + r""" +eq "$(j -n dev netpol tester-egress '{.spec.policyTypes[0]}')" Egress || exit 1 +okr dev tester http://api && blocked dev tester http://echo""", + hint=r"""Two egress rules: `{to: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}` and `{to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}], ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]}`. Every Namespace carries the label `kubernetes.io/metadata.name`.""", + show=r"""kubectl -n dev apply -f - <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: tester-egress +spec: + podSelector: + matchLabels: {app: tester} + policyTypes: [Egress] + egress: + - to: + - podSelector: + matchLabels: {app: api} + ports: + - {protocol: TCP, port: 80} + - to: + - namespaceSelector: + matchLabels: {kubernetes.io/metadata.name: kube-system} + podSelector: + matchLabels: {k8s-app: kube-dns} + ports: + - {protocol: UDP, port: 53} + - {protocol: TCP, port: 53} +EOF +kubectl -n dev exec tester -- wget -qO- -T3 api +kubectl -n dev exec tester -- wget -qO- -T3 echo || echo blocked"""), + ], + "outro": r"""You closed a Namespace, opened single paths into it, used the AND form of a rule, and limited a Pod's egress without breaking its DNS. + +**For the exam** + +- Test every policy from the Pods in question: `kubectl exec POD -- wget -qO- -T3 TARGET` (or `nc -zv -w3 HOST PORT`). A timeout means dropped. +- `policyTypes` decides which directions the policy restricts; a policy with only `Ingress` leaves egress open. +- Ports in a policy are the Pod's ports (`targetPort`), not the Service's. +- `ipBlock: {cidr: ..., except: [...]}` for addresses outside the cluster. +- `kubectl describe netpol` spells the rules out in words - good for checking the AND/OR. + +Next, prove it against the clock: **drill 5.2**.""", +} + +# ---- 5.3 Services and endpoints --------------------------------------------------------------- +L53 = { + "id": 9503, "objective": "5.3", "workers": 1, + "title": "Service types, headless Services, ExternalName, and endpoints by hand", + "intro": r"""A Service is a stable name and address in front of a changing set of Pods - its **endpoints**, kept in EndpointSlices by a controller that follows the selector: + +``` +ClusterIP a virtual IP inside the cluster the default +NodePort ClusterIP + a port (30000-32767) on every node +LoadBalancer NodePort + an external address from a cloud controller (none here: it stays ) +headless clusterIP: None - DNS returns the Pod IPs themselves +ExternalName a DNS alias (CNAME) for a name outside the cluster; no proxying +no selector you write the EndpointSlice yourself - for something outside the cluster +``` + +Namespace `svc` runs Deployment `web` (nginx, 2 replicas), Pod `client`, and Pod `legacy-box`, which serves on port 80 but carries no label any Service selects.""", + "setup": PODS + r""" +mkns svc +mkdir -p "$WS/lesson-5.3" +$K -n svc create deploy web --image=nginx:1.27 --replicas=2 --port=80 >/dev/null +cli svc client +$K -n svc run legacy-box --image=busybox:1.36 --labels=role=legacy --command -- sh -c 'mkdir -p /www && echo legacy > /www/index.html && exec httpd -f -p 80 -h /www' >/dev/null +$K -n svc rollout status deploy web --timeout=180s >/dev/null && ready svc +""", + "steps": [ + step("ClusterIP", + r"""Expose Deployment `web` as Service `web`, port 80. From `client`, `wget -qO- web` returns nginx's page.""", + check=NET + r"""eq "$(j -n svc svc web '{.spec.type} {.spec.ports[0].port}')" "ClusterIP 80" && okr svc client http://web""", + hint=r"""`kubectl expose deploy web --port=80` - it copies the Deployment's selector.""", + show=r"""kubectl -n svc expose deploy web --port=80 +kubectl -n svc get svc,endpointslices +kubectl -n svc exec client -- wget -qO- -T3 web | head -4"""), + step("NodePort", + r"""Create Service `web-np` of type **NodePort** for the same Pods, with node port **30080**. Then reach it from the control-plane node: `curl http://{W1}-ADDRESS:30080` - every node answers on that port, whether or not it runs one of the Pods.""", + check=r""" +eq "$(j -n svc svc web-np '{.spec.type} {.spec.ports[0].nodePort}')" "NodePort 30080" || exit 1 +ip=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}'); nx "$CP" "curl -s -m 3 http://$ip:30080 | grep -q nginx" {WS}/lesson-5.3/web-np.yaml +yq -i '.spec.ports[0].nodePort = 30080' {WS}/lesson-5.3/web-np.yaml +kubectl -n svc apply -f {WS}/lesson-5.3/web-np.yaml +ip=$(kubectl get node {W1} -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') +curl -s -m 3 http://$ip:30080 | head -4"""), + step("Headless", + r"""Create Service `web-headless` with `clusterIP: None` for the `web` Pods. Look it up from `client` with `nslookup web-headless`: you get the two Pod IPs, not a Service IP. Write them, one per line, to `{WS}/lesson-5.3/headless.txt`.""", + check=DNS + r""" +eq "$(j -n svc svc web-headless '{.spec.clusterIP}')" None || exit 1 +want=$($K -n svc get pods -l app=web -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' | sort) +eq "$(tr -d ' \r' < "$WS/lesson-5.3/headless.txt" 2>/dev/null | grep . | sort)" "$want" """, + hint=r"""`kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80`. In busybox's `nslookup` output the addresses follow the `Name:` line.""", + show=r"""kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80 +sleep 5 +kubectl -n svc exec client -- nslookup web-headless +kubectl -n svc exec client -- nslookup web-headless | awk '/^Name:/ {n=1} n && /^Address/ {print $NF}' > {WS}/lesson-5.3/headless.txt +cat {WS}/lesson-5.3/headless.txt"""), + step("ExternalName", + r"""Create Service `docs` of type **ExternalName** pointing at `kubernetes.io`. Inside the cluster, `docs.svc.svc.cluster.local` is now a CNAME for that name - DNS only: nothing is proxied, and the port does not matter.""", + check=r"""eq "$(j -n svc svc docs '{.spec.type} {.spec.externalName}')" "ExternalName kubernetes.io" """, + hint=r"""`kubectl -n svc create service externalname docs --external-name=kubernetes.io`.""", + show=r"""kubectl -n svc create service externalname docs --external-name=kubernetes.io +kubectl -n svc get svc docs +kubectl -n svc exec client -- nslookup docs | head -6"""), + step("A Service without a selector", + r"""Create Service `legacy` (port 80) with **no selector**, and an EndpointSlice for it by hand that holds `legacy-box`'s Pod IP on port 80. From `client`, `wget -qO- legacy` then answers `legacy`. + +An EndpointSlice belongs to a Service by the label `kubernetes.io/service-name`; its port's name must match the Service port's name (here: none).""", + check=NET + r""" +[ -z "$(j -n svc svc legacy '{.spec.selector}')" ] || exit 1 +for i in 1 2 3 4 5; do [ "$($K -n svc exec client -- wget -qO- -T3 legacy 2>/dev/null)" = legacy ] && exit 0; sleep 3; done; exit 1""", + hint=r"""Service: `spec.ports: [{port: 80, targetPort: 80}]` and no `selector`. EndpointSlice (`discovery.k8s.io/v1`): `addressType: IPv4`, `endpoints: [{addresses: [IP]}]`, `ports: [{port: 80, protocol: TCP}]`, and the label.""", + show=r"""ip=$(kubectl -n svc get pod legacy-box -o jsonpath='{.status.podIP}') +kubectl -n svc apply -f - < backendRefs (Services, with weights) +``` + +A route attaches to a Gateway through `parentRefs`; a listener's `allowedRoutes` says from which Namespaces. Weights split traffic between backends - a canary without a second Deployment of tricks. + +The Gateway API's CRDs are installed. No controller runs here, so nothing will actually serve traffic: you are graded, as in many exam tasks, on the objects. Namespace `gw` has Services `web` (port 80), `web-v2` (80) and `api` (8080).""", + "setup": NS + r""" +ensure_gateway_crds || { echo "could not install the Gateway API CRDs" >&2; exit 1; } +mkns gw +for s in web:80 web-v2:80 api:8080; do $K -n gw create service clusterip "${s%%:*}" --tcp="${s#*:}:80" >/dev/null; done +mkdir -p "$WS/lesson-5.4" +""", + "steps": [ + step("A class and a Gateway", + r"""Create GatewayClass `example` (controller name `example.com/gateway-controller`) and, in `gw`, Gateway `shop` of that class with one listener: `http`, protocol HTTP, port 80, hostname `shop.example.com`, accepting routes from its own Namespace.""", + check=r""" +eq "$(j gatewayclass example '{.spec.controllerName}')" example.com/gateway-controller || exit 1 +eq "$(j -n gw gateway shop '{.spec.gatewayClassName} {.spec.listeners[0].name} {.spec.listeners[0].protocol} {.spec.listeners[0].port} {.spec.listeners[0].hostname} {.spec.listeners[0].allowedRoutes.namespaces.from}')" "example http HTTP 80 shop.example.com Same" """, + hint=r"""Both are `gateway.networking.k8s.io/v1`. `kubectl explain gateway.spec.listeners` once the CRDs are in; the docs at gateway-api.sigs.k8s.io have every example.""", + show=r"""kubectl apply -f - <<'EOF' +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass +metadata: + name: example +spec: + controllerName: example.com/gateway-controller +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: shop + namespace: gw +spec: + gatewayClassName: example + listeners: + - name: http + protocol: HTTP + port: 80 + hostname: shop.example.com + allowedRoutes: + namespaces: + from: Same +EOF +kubectl -n gw get gateway shop"""), + step("Route by path", + r"""Create HTTPRoute `shop` in `gw`, attached to Gateway `shop`: requests for paths under `/api` go to Service `api` port 8080, everything else to `web` port 80.""", + check=r""" +py() { $K -n gw get httproute shop -o json 2>/dev/null | python3 -c "import json,sys; r=json.load(sys.stdin)['spec']; $1"; } +py ' +p = r["parentRefs"][0]["name"]; rules = r["rules"] +def to(path): + for x in rules: + for m in x.get("matches", []): + if m.get("path", {}).get("value") == path and m["path"].get("type", "PathPrefix") == "PathPrefix": + return [(b["name"], b.get("port")) for b in x["backendRefs"]] +sys.exit(0 if p == "shop" and to("/api") == [("api", 8080)] and to("/") == [("web", 80)] else 1)'""", + hint=r"""`parentRefs: [{name: shop}]`, then two `rules`, each with `matches: [{path: {type: PathPrefix, value: ...}}]` and `backendRefs: [{name: ..., port: ...}]`.""", + show=r"""kubectl -n gw apply -f - <<'EOF' +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: shop +spec: + parentRefs: + - name: shop + hostnames: [shop.example.com] + rules: + - matches: + - path: {type: PathPrefix, value: /api} + backendRefs: + - {name: api, port: 8080} + - matches: + - path: {type: PathPrefix, value: /} + backendRefs: + - {name: web, port: 80} +EOF +kubectl -n gw describe httproute shop | grep -A3 'Backend Refs' | head -8"""), + step("A canary by weight", + r"""Change the `/` rule of route `shop` so that **90%** of its requests go to `web` and **10%** to `web-v2` (both port 80). The `/api` rule stays as it is.""", + check=r""" +$K -n gw get httproute shop -o json 2>/dev/null | python3 -c ' +import json, sys; rules = json.load(sys.stdin)["spec"]["rules"] +def by(path): return next((x for x in rules for m in x.get("matches", []) if m.get("path", {}).get("value") == path), None) +root, api = by("/"), by("/api") +w = sorted((b["name"], b.get("weight", 1)) for b in root["backendRefs"]) if root else [] +sys.exit(0 if w == [("web", 90), ("web-v2", 10)] and api and [b["name"] for b in api["backendRefs"]] == ["api"] else 1)'""", + hint=r"""Two `backendRefs` in the same rule, each with a `weight`. The weights are relative: 9 and 1 would do the same.""", + show=r"""kubectl -n gw get httproute shop -o yaml > {WS}/lesson-5.4/shop.yaml +yq -i '(.spec.rules[] | select(.matches[0].path.value == "/")).backendRefs = [{"name": "web", "port": 80, "weight": 90}, {"name": "web-v2", "port": 80, "weight": 10}]' {WS}/lesson-5.4/shop.yaml +kubectl apply -f {WS}/lesson-5.4/shop.yaml +kubectl -n gw get httproute shop -o jsonpath='{.spec.rules[1].backendRefs}'; echo"""), + step("Beta testers by header", + r"""Add HTTPRoute `shop-beta` in `gw`, attached to `shop`: requests carrying the header `x-env: beta` go **all** to `web-v2`. A more specific match wins, so these requests no longer take part in the 90/10 split.""", + check=r""" +$K -n gw get httproute shop-beta -o json 2>/dev/null | python3 -c ' +import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0] +h = r["matches"][0].get("headers", [{}])[0] +sys.exit(0 if s["parentRefs"][0]["name"] == "shop" and h.get("name", "").lower() == "x-env" and h.get("value") == "beta" and [b["name"] for b in r["backendRefs"]] == ["web-v2"] else 1)'""", + hint=r"""`matches: [{headers: [{name: x-env, value: beta}]}]` (type `Exact` by default).""", + show=r"""kubectl -n gw apply -f - <<'EOF' +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: shop-beta +spec: + parentRefs: + - name: shop + hostnames: [shop.example.com] + rules: + - matches: + - headers: + - {name: x-env, value: beta} + backendRefs: + - {name: web-v2, port: 80} +EOF +kubectl -n gw get httproute"""), + step("HTTPS on the Gateway", + r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `gw`, and add a second listener to Gateway `shop`: `https`, protocol HTTPS, port 443, the same hostname, TLS mode `Terminate` with that Secret as its certificate.""", + check=r""" +eq "$(j -n gw secret shop-tls '{.type}')" kubernetes.io/tls || exit 1 +eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="https")].protocol} {.spec.listeners[?(@.name=="https")].port} {.spec.listeners[?(@.name=="https")].tls.mode} {.spec.listeners[?(@.name=="https")].tls.certificateRefs[0].name}')" "HTTPS 443 Terminate shop-tls" \ + && eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="http")].port}')" 80""", + hint=r"""`openssl req -x509 ...`, `kubectl create secret tls`, then a listener with `tls: {mode: Terminate, certificateRefs: [{name: shop-tls}]}`. Keep the `http` listener.""", + show=r"""cd {WS}/lesson-5.4 +openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null +kubectl -n gw create secret tls shop-tls --cert=tls.crt --key=tls.key +kubectl -n gw patch gateway shop --type json -p '[{"op":"add","path":"/spec/listeners/-","value":{"name":"https","protocol":"HTTPS","port":443,"hostname":"shop.example.com","tls":{"mode":"Terminate","certificateRefs":[{"name":"shop-tls"}]},"allowedRoutes":{"namespaces":{"from":"Same"}}}}]' +kubectl -n gw get gateway shop -o jsonpath='{range .spec.listeners[*]}{.name} {.port}{"\n"}{end}' +cd"""), + ], + "outro": r"""You built a Gateway with HTTP and HTTPS listeners, routed by path, split traffic by weight, and sent one header's requests elsewhere. + +**For the exam** + +- The CRDs come from the Gateway API release (`standard-install.yaml`); the controller is separate (Envoy Gateway, NGINX Gateway Fabric, Istio, Cilium, ...). A Gateway's `status` stays empty without one. +- `parentRefs` names the Gateway (and `sectionName` a listener); `allowedRoutes.namespaces.from` is `Same`, `All` or `Selector`. +- Matches: `path` (`PathPrefix`, `Exact`, `RegularExpression`), `headers`, `queryParams`, `method`. Filters: `RequestRedirect` (http → https), `URLRewrite`, `RequestHeaderModifier`. +- Moving from Ingress: one Ingress rule ≈ one HTTPRoute rule; TLS moves from the Ingress to the Gateway's listener. + +Next, prove it against the clock: **drill 5.4**.""", +} + +# ---- 5.5 Ingress ------------------------------------------------------------------------------ +L55 = { + "id": 9505, "objective": "5.5", "workers": 1, + "title": "Ingress: classes, host and path rules, TLS, and the default backend", + "intro": r"""An **Ingress** routes HTTP(S) by host and path to Services. It does nothing on its own: an **Ingress controller** (ingress-nginx, Traefik, HAProxy, ...) watches Ingresses of its **IngressClass** and configures a proxy from them. + +``` +ingressClassName which controller (or the IngressClass marked default) +rules[].host shop.example.com (none = every host) +paths[].pathType Prefix | Exact | ImplementationSpecific +paths[].backend service: {name, port: {number | name}} +tls[] hosts + a kubernetes.io/tls Secret +defaultBackend what matches no rule +``` + +No controller runs here, so you are graded on the objects. Namespace `ing` has Services `web` (port 80), `api` (8080) and `health` (8081).""", + "setup": NS + r""" +mkns ing +for s in web:80 api:8080 health:8081; do $K -n ing create service clusterip "${s%%:*}" --tcp="${s#*:}:80" >/dev/null; done +$K delete ingressclass nginx --ignore-not-found >/dev/null 2>&1 +mkdir -p "$WS/lesson-5.5" +""", + "steps": [ + step("An IngressClass, the default", + r"""Create IngressClass `nginx` for controller `k8s.io/ingress-nginx`, and make it the cluster's **default** class: an Ingress without `ingressClassName` then belongs to it.""", + check=r"""eq "$(j ingressclass nginx '{.spec.controller} {.metadata.annotations.ingressclass\.kubernetes\.io/is-default-class}')" "k8s.io/ingress-nginx true" """, + hint=r"""`networking.k8s.io/v1`, kind `IngressClass`, `spec.controller`, and the annotation `ingressclass.kubernetes.io/is-default-class: "true"`. Installing ingress-nginx makes this object for you.""", + show=r"""kubectl apply -f - <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: IngressClass +metadata: + name: nginx + annotations: + ingressclass.kubernetes.io/is-default-class: "true" +spec: + controller: k8s.io/ingress-nginx +EOF +kubectl get ingressclass"""), + step("Host and paths, from the command line", + r"""Create Ingress `shop` in `ing` with class `nginx` and host `shop.example.com`: paths under `/api` go to `api:8080`, everything else to `web:80`. `kubectl create ingress` can do it in one line.""", + check=r""" +$K -n ing get ingress shop -o json 2>/dev/null | python3 -c ' +import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0] +p = {x["path"]: (x["pathType"], x["backend"]["service"]["name"], x["backend"]["service"]["port"]["number"]) for x in r["http"]["paths"]} +sys.exit(0 if s.get("ingressClassName") == "nginx" and r["host"] == "shop.example.com" and p.get("/api") == ("Prefix", "api", 8080) and p.get("/") == ("Prefix", "web", 80) else 1)'""", + hint=r"""`kubectl create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'` - a trailing `*` means `Prefix`, without it `Exact`.""", + show=r"""kubectl -n ing create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80' +kubectl -n ing describe ingress shop | sed -n '/Rules/,/Annotations/p'"""), + step("An exact path", + r"""Add a rule to Ingress `shop`: exactly `/healthz` (pathType `Exact`) goes to `health:8081`. `/healthz/deep` would not match it - it would fall through to the `/` prefix.""", + check=r""" +$K -n ing get ingress shop -o json 2>/dev/null | python3 -c ' +import json, sys; ps = json.load(sys.stdin)["spec"]["rules"][0]["http"]["paths"] +h = [x for x in ps if x["path"] == "/healthz"] +sys.exit(0 if h and h[0]["pathType"] == "Exact" and h[0]["backend"]["service"]["name"] == "health" and h[0]["backend"]["service"]["port"]["number"] == 8081 and len(ps) == 3 else 1)'""", + hint=r"""`kubectl edit ingress shop` and add a third entry under `paths:`, or a JSON patch that appends to `/spec/rules/0/http/paths`.""", + show=r"""kubectl -n ing patch ingress shop --type json -p '[{"op":"add","path":"/spec/rules/0/http/paths/-","value":{"path":"/healthz","pathType":"Exact","backend":{"service":{"name":"health","port":{"number":8081}}}}}]' +kubectl -n ing get ingress shop -o jsonpath='{range .spec.rules[0].http.paths[*]}{.pathType} {.path} -> {.backend.service.name}{"\n"}{end}'"""), + step("TLS", + r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `ing`, and add a `tls` section to Ingress `shop` for that host with that Secret. The controller would then serve HTTPS for it and, by default, redirect HTTP there.""", + check=r""" +eq "$(j -n ing secret shop-tls '{.type}')" kubernetes.io/tls && eq "$(j -n ing ingress shop '{.spec.tls[0].hosts[0]} {.spec.tls[0].secretName}')" "shop.example.com shop-tls" \ + && j -n ing secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q shop.example.com""", + hint=r"""`spec.tls: [{hosts: [shop.example.com], secretName: shop-tls}]`. The Secret must be in the Ingress's Namespace.""", + show=r"""cd {WS}/lesson-5.5 +openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null +kubectl -n ing create secret tls shop-tls --cert=tls.crt --key=tls.key +kubectl -n ing patch ingress shop --type merge -p '{"spec":{"tls":[{"hosts":["shop.example.com"],"secretName":"shop-tls"}]}}' +kubectl -n ing get ingress shop +cd"""), + step("What matches nothing", + r"""Give Ingress `shop` a **default backend**: `web:80`, for requests whose host or path no rule matches.""", + check=r"""eq "$(j -n ing ingress shop '{.spec.defaultBackend.service.name} {.spec.defaultBackend.service.port.number}')" "web 80" """, + hint=r"""`spec.defaultBackend: {service: {name: web, port: {number: 80}}}`; `kubectl create ingress --default-backend=web:80` does it at creation.""", + show=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"defaultBackend":{"service":{"name":"web","port":{"number":80}}}}}' +kubectl -n ing describe ingress shop | head -12"""), + ], + "outro": r"""You made a default IngressClass and an Ingress with host, prefix and exact paths, TLS and a default backend. + +**For the exam** + +- `kubectl create ingress NAME --class=C --rule='HOST/PATH*=SVC:PORT[,tls=SECRET]' [--default-backend=SVC:PORT]`. +- `pathType` is required; `Prefix` matches by path segment (`/api` matches `/api/v1`, not `/apix`). +- A backend port is the **Service's** port, by number or name. +- No address in `kubectl get ingress`? The controller is not running, or the class names another controller. Controller-specific behaviour lives in annotations (`nginx.ingress.kubernetes.io/rewrite-target`, `ssl-redirect`). + +Next, prove it against the clock: **drill 5.5**.""", +} + +# ---- 5.6 CoreDNS ------------------------------------------------------------------------------ +L56 = { + "id": 9506, "objective": "5.6", "workers": 1, + "title": "Cluster DNS: names of Services and Pods, the Corefile, and a Pod's resolver", + "intro": r"""CoreDNS (Deployment `coredns`, Service `kube-dns` in `kube-system`) answers every Pod's DNS queries. The names it knows: + +``` +SERVICE.NAMESPACE.svc.cluster.local the ClusterIP (or, headless, the Pod IPs) +POD.SERVICE.NAMESPACE.svc.cluster.local one Pod of a headless Service - a StatefulSet's db-0.db... +A-B-C-D.NAMESPACE.pod.cluster.local a Pod by its IP, dots as dashes +``` + +Each Pod's `/etc/resolv.conf` points at `kube-dns` and searches its own Namespace first, so a bare `web` works inside the Namespace and `web.other` across. CoreDNS itself is configured by the **Corefile** in ConfigMap `coredns`, and reloads it on its own when it changes. + +Namespace `dns-lab` has Service `web`, a StatefulSet `db` (2 replicas) behind headless Service `db`, and Pod `client`.""", + "setup": PODS + r""" +mkns dns-lab +mkdir -p "$WS/lesson-5.6" +srv dns-lab web; cli dns-lab client +$K -n dns-lab apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: Service +metadata: {name: db} +spec: + clusterIP: None + selector: {app: db} + ports: [{port: 5432}] +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: {name: db} +spec: + serviceName: db + replicas: 2 + selector: {matchLabels: {app: db}} + template: + metadata: {labels: {app: db}} + spec: + containers: [{name: db, image: busybox:1.36, command: [sleep, '86400']}] +EOF +$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > "$PRIV/lesson-5.6.corefile" +ready dns-lab && $K -n dns-lab rollout status sts db --timeout=180s >/dev/null +""", + "steps": [ + step("A Service by its full name", + r"""From `client`, look up `web` by its **fully qualified** name, and write the address you get to `{WS}/lesson-5.6/web.txt`. It is the Service's ClusterIP.""", + check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/web.txt" 2>/dev/null)" "$(j -n dns-lab svc web '{.spec.clusterIP}')" """, + hint=r"""`kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local`.""", + show=r"""kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local +kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local | awk '/^Name:/ {n=1} n && /^Address/ {print $NF; exit}' > {WS}/lesson-5.6/web.txt"""), + step("One Pod of a StatefulSet", + r"""A StatefulSet's Pods get stable names through their headless Service. Write the fully qualified DNS name of Pod `db-0` to `{WS}/lesson-5.6/db0.txt`, and check from `client` that it resolves to `db-0`'s IP.""", + check=DNS + r""" +eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/db0.txt" 2>/dev/null | sed 's/\.$//')" db-0.db.dns-lab.svc.cluster.local && resolves dns-lab client db-0.db.dns-lab.svc.cluster.local "$(j -n dns-lab pod db-0 '{.status.podIP}')" """, + hint=r"""`POD.SERVICE.NAMESPACE.svc.cluster.local` - the Service is the StatefulSet's `serviceName`.""", + show=r"""echo db-0.db.dns-lab.svc.cluster.local > {WS}/lesson-5.6/db0.txt +kubectl -n dns-lab exec client -- nslookup db-0.db.dns-lab.svc.cluster.local +kubectl -n dns-lab get pod db-0 -o wide"""), + step("Names of your own", + r"""Add a `hosts` block to CoreDNS's Corefile so that every Pod resolves **`legacy.internal`** to `10.96.0.250`, and every other name still resolves as before (`fallthrough`). Put it inside the `.:53 { ... }` server block, before `forward`. + +CoreDNS reloads the changed ConfigMap by itself within a minute; `client` then resolves the new name.""", + check=DNS + r""" +$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep -q 'kubernetes cluster.local' || exit 1 +resolves dns-lab client legacy.internal 10.96.0.250 && resolves dns-lab client web.dns-lab.svc.cluster.local "$(j -n dns-lab svc web '{.spec.clusterIP}')" """, + hint=r"""`kubectl -n kube-system edit cm coredns`. The block: `hosts {` / `10.96.0.250 legacy.internal` / `fallthrough` / `}`. A syntax error leaves CoreDNS on its old configuration: `kubectl -n kube-system logs -l k8s-app=kube-dns` says so.""", + show=r"""kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > {WS}/lesson-5.6/Corefile +sed -i '0,/^ forward/s// hosts {\n 10.96.0.250 legacy.internal\n fallthrough\n }\n forward/' {WS}/lesson-5.6/Corefile +cat {WS}/lesson-5.6/Corefile +kubectl -n kube-system create configmap coredns --from-file=Corefile={WS}/lesson-5.6/Corefile --dry-run=client -o yaml | kubectl apply -f - +sleep 40 +kubectl -n dns-lab exec client -- nslookup legacy.internal"""), + step("A Pod's own resolver settings", + r"""Run Pod `tuned` in `dns-lab` (`busybox:1.36`, `sleep 86400`) whose resolver you set yourself: `dnsPolicy: None`, with the `kube-dns` Service's IP as its only nameserver, the search domains `dns-lab.svc.cluster.local` and `svc.cluster.local`, and the option `ndots: 2`. Its `/etc/resolv.conf` shows exactly that.""", + check=r""" +ip=$(j -n kube-system svc kube-dns '{.spec.clusterIP}') +r=$($K -n dns-lab exec tuned -- cat /etc/resolv.conf 2>/dev/null) +echo "$r" | grep -qx "nameserver $ip" && echo "$r" | grep -qx 'search dns-lab.svc.cluster.local svc.cluster.local' && echo "$r" | grep -q 'ndots:2' \ + && eq "$(j -n dns-lab pod tuned '{.spec.dnsPolicy}')" None""", + hint=r"""`spec.dnsPolicy: None` and `spec.dnsConfig: {nameservers: [IP], searches: [...], options: [{name: ndots, value: "2"}]}`. The other policies: `ClusterFirst` (the default), `Default` (the node's resolver), `ClusterFirstWithHostNet`.""", + show=r"""ip=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}') +kubectl -n dns-lab apply -f - </dev/null | tr -d '\r\n' | sed 's/ *$//')" "$want" """, + hint=r"""`kubectl get sc` shows all three columns. For exactly the values: `-o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}'`.""", + show=r"""kubectl get sc +kubectl get sc standard -o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}' > {WS}/lesson-1.1/default.txt +cat {WS}/lesson-1.1/default.txt"""), + step("A class that keeps its volumes", + r"""Create StorageClass `keep`: the same provisioner as `standard` (`rancher.io/local-path`), binding mode `WaitForFirstConsumer`, and reclaim policy **`Retain`**, so a volume outlives its claim. + +There is no `kubectl create storageclass`: write the manifest. `kubectl get sc standard -o yaml` is a good start - drop the default annotation when you copy it.""", + check=r""" +eq "$(j sc keep '{.provisioner} {.reclaimPolicy} {.volumeBindingMode}')" "rancher.io/local-path Retain WaitForFirstConsumer" \ + && [ "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""", + hint=r"""A StorageClass has no `spec`: `provisioner`, `reclaimPolicy` and `volumeBindingMode` sit at the top level, beside `metadata`. Its `apiVersion` is `storage.k8s.io/v1`.""", + show=r"""kubectl apply -f - <<'EOF' +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: keep +provisioner: rancher.io/local-path +reclaimPolicy: Retain +volumeBindingMode: WaitForFirstConsumer +EOF +kubectl get sc"""), + step("A claim that waits", + r"""In Namespace `store`, create PersistentVolumeClaim `ledger`: class `keep`, `ReadWriteOnce`, `100Mi`. + +It stays `Pending`, and that is correct. `kubectl -n store describe pvc ledger` says why in its events; write the event's **reason** (one word) to `{WS}/lesson-1.1/why.txt`.""", + check=r""" +eq "$(j -n store pvc ledger '{.spec.storageClassName}')" keep && eq "$(j -n store pvc ledger '{.spec.resources.requests.storage}')" 100Mi \ + && grep -qx 'WaitForFirstConsumer' <(tr -d ' \r' < "$WS/lesson-1.1/why.txt" 2>/dev/null)""", + hint=r"""The events at the bottom of `describe` have a Reason column. A claim is namespaced; the class is not.""", + show=r"""kubectl -n store apply -f - <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ledger +spec: + storageClassName: keep + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 100Mi +EOF +kubectl -n store describe pvc ledger | tail -4 +echo WaitForFirstConsumer > {WS}/lesson-1.1/why.txt"""), + step("Give it a consumer", + r"""Run Pod `bookkeeper` in `store` (image `busybox:1.36`) that mounts claim `ledger` at `/data` and runs `sh -c 'echo first > /data/entry.txt && sleep 86400'`. + +As soon as the Pod is scheduled the provisioner makes a volume on its node and the claim turns `Bound`. Look at the new PersistentVolume: `kubectl get pv`. Its reclaim policy came from the class.""", + check=r""" +eq "$(j -n store pvc ledger '{.status.phase}')" Bound || exit 1 +pv=$(j -n store pvc ledger '{.spec.volumeName}'); eq "$(j pv "$pv" '{.spec.persistentVolumeReclaimPolicy}')" Retain || exit 1 +eq "$($K -n store exec bookkeeper -- cat /data/entry.txt 2>/dev/null)" first""", + hint=r"""`kubectl run bookkeeper --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'` gives you the Pod; add `volumes:` (with `persistentVolumeClaim.claimName`) at Pod level and `volumeMounts:` in the container.""", + show=r"""kubectl -n store apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: bookkeeper +spec: + containers: + - name: bookkeeper + image: busybox:1.36 + command: [sh, -c, 'echo first > /data/entry.txt && sleep 86400'] + volumeMounts: + - name: data + mountPath: /data + volumes: + - name: data + persistentVolumeClaim: + claimName: ledger +EOF +kubectl -n store wait --for=condition=Ready pod/bookkeeper --timeout=120s +kubectl -n store get pvc ledger +kubectl get pv"""), + step("Delete the claim, keep the volume", + r"""Delete Pod `bookkeeper` and claim `ledger`. Because of `Retain`, the volume stays, with status `Released`: its data is still on disk, and no new claim can bind it until an admin says so. + +Write the PersistentVolume's name to `{WS}/lesson-1.1/released.txt`.""", + check=r""" +[ -z "$(j -n store pvc ledger '{.metadata.name}')" ] || exit 1 +pv=$(tr -d ' \r\n' < "$WS/lesson-1.1/released.txt" 2>/dev/null); [ -n "$pv" ] || exit 1 +eq "$(j pv "$pv" '{.status.phase} {.spec.claimRef.namespace}/{.spec.claimRef.name}')" "Released store/ledger" """, + hint=r"""Read the volume name off the claim before you delete it (`kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}'`), or find it afterwards in `kubectl get pv`: the CLAIM column still names `store/ledger`.""", + show=r"""kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}' > {WS}/lesson-1.1/released.txt +kubectl -n store delete pod bookkeeper --grace-period=1 +kubectl -n store delete pvc ledger +kubectl get pv"""), + step("Change the default class", + r"""Make `keep` the cluster's default class, and `standard` no longer the default. Two defaults would be ambiguous: set the annotation to `"true"` on one and `"false"` on the other.""", + check=r""" +eq "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" true \ + && [ "$(j sc standard '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""", + hint=r"""`kubectl annotate sc NAME storageclass.kubernetes.io/is-default-class=true --overwrite`, or `kubectl patch` with a `metadata.annotations` map.""", + show=r"""kubectl annotate sc standard storageclass.kubernetes.io/is-default-class=false --overwrite +kubectl annotate sc keep storageclass.kubernetes.io/is-default-class=true --overwrite +kubectl get sc"""), + ], + "outro": r"""You read a class, made one that keeps its volumes, watched a claim wait for its Pod, kept a volume past its claim, and moved the default. + +**For the exam** + +- A StorageClass has no `spec`; `provisioner`, `reclaimPolicy`, `volumeBindingMode` and `allowVolumeExpansion` are top-level fields. +- `WaitForFirstConsumer`: a claim is `Pending` until a Pod uses it - not a fault. Look at `describe pvc` before you "fix" anything. +- The reclaim policy is copied into the volume when it is made. To keep an existing volume, patch the **PV**: `kubectl patch pv NAME -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'`. +- Growing a claim: the class needs `allowVolumeExpansion: true` and a driver that can expand, then raise `spec.resources.requests.storage` on the claim - claims only grow. The local-path provisioner can do neither that nor snapshots; a CSI driver such as the hostpath driver behind `csi-hostpath-sc` can, and lesson 1.3 grows, snapshots and restores a claim on it. +- One default class. `kubectl get sc` shows `(default)`. + +Next, prove it against the clock: **drill 1.1**.""", +} + +# ---- 1.2 volume types, access modes and reclaim policies ------------------------------------- +L12 = { + "id": 9102, "objective": "1.2", "workers": 0, + "title": "Volume types, access modes and reclaim policies", + "intro": r"""A Pod's `volumes:` list says where data comes from; each container's `volumeMounts:` says where it appears. The types you meet on the exam: + +``` +emptyDir a scratch directory that lives and dies with the Pod (medium: Memory = tmpfs) +hostPath a directory of the node - tied to that node +configMap keys of a ConfigMap as files secret the same for a Secret +projected several of those, plus downwardAPI and a ServiceAccount token, in ONE directory +persistentVolumeClaim a claim, bound to a PersistentVolume +``` + +A PersistentVolume has **access modes** - `ReadWriteOnce` (one *node* at a time), `ReadOnlyMany`, `ReadWriteMany`, `ReadWriteOncePod` (one *Pod*) - and a **reclaim policy**, `Delete` or `Retain`. A claim binds a volume whose modes, size and class fit it. + +Namespace `vols` has ConfigMap `app-config` (key `mode`) and Secret `app-secret` (key `password`).""", + "setup": NS + r""" +mkns vols +$K -n vols create configmap app-config --from-literal=mode=prod >/dev/null +$K -n vols create secret generic app-secret --from-literal=password=s3cr3t >/dev/null +mkdir -p "$WS/lesson-1.2" +""", + "steps": [ + step("Two containers, one emptyDir", + r"""Create Pod `pair` in `vols` with an `emptyDir` volume `shared`, mounted at `/shared` in two `busybox:1.36` containers: + +- `writer`: `sh -c 'echo hello > /shared/msg && sleep 86400'` +- `reader`: `sleep 86400` + +The reader sees what the writer wrote: an emptyDir is shared by every container of its Pod.""", + check=r"""eq "$($K -n vols exec pair -c reader -- cat /shared/msg 2>/dev/null)" hello \ + && [ -n "$(j -n vols pod pair '{.spec.volumes[?(@.name=="shared")].emptyDir}')" ]""", + hint=r"""Both containers list the same `volumeMounts` entry (`name: shared`); the Pod lists the volume once, as `emptyDir: {}`.""", + show=r"""kubectl -n vols apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: pair +spec: + containers: + - name: writer + image: busybox:1.36 + command: [sh, -c, 'echo hello > /shared/msg && sleep 86400'] + volumeMounts: + - {name: shared, mountPath: /shared} + - name: reader + image: busybox:1.36 + command: [sleep, '86400'] + volumeMounts: + - {name: shared, mountPath: /shared} + volumes: + - name: shared + emptyDir: {} +EOF +kubectl -n vols wait --for=condition=Ready pod/pair --timeout=120s +kubectl -n vols exec pair -c reader -- cat /shared/msg"""), + step("Scratch space in memory", + r"""Create Pod `scratch` in `vols` (`busybox:1.36`, `sleep 86400`) with an emptyDir at `/scratch` that is held in **memory**, at most `64Mi`. + +Inside the Pod, `/proc/mounts` shows the difference: `/scratch` is a `tmpfs`. What is written there counts against the container's memory.""", + check=r""" +eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.medium}')" Memory && eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.sizeLimit}')" 64Mi \ + && $K -n vols exec scratch -- grep -q ' /scratch tmpfs ' /proc/mounts 2>/dev/null""", + hint=r"""`emptyDir: {medium: Memory, sizeLimit: 64Mi}`.""", + show=r"""kubectl -n vols apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: scratch +spec: + containers: + - name: scratch + image: busybox:1.36 + command: [sleep, '86400'] + volumeMounts: + - {name: tmp, mountPath: /scratch} + volumes: + - name: tmp + emptyDir: {medium: Memory, sizeLimit: 64Mi} +EOF +kubectl -n vols wait --for=condition=Ready pod/scratch --timeout=120s +kubectl -n vols exec scratch -- grep /scratch /proc/mounts"""), + step("One directory from three sources", + r"""Create Pod `bundle` in `vols` (`busybox:1.36`, `sleep 86400`) with **one** `projected` volume mounted at `/etc/bundle`, made of: + +- ConfigMap `app-config` (gives the file `mode`), +- Secret `app-secret` (gives `password`), +- the downward API: the Pod's own name as the file `podname`. + +Separate volumes would need three mount points; a projected volume puts them side by side.""", + check=r""" +x() { $K -n vols exec bundle -- cat "/etc/bundle/$1" 2>/dev/null; } +[ -n "$(j -n vols pod bundle '{.spec.volumes[0].projected}')" ] && eq "$(x mode)" prod && eq "$(x password)" s3cr3t && eq "$(x podname)" bundle""", + hint=r"""`projected: {sources: [...]}` - each source is `configMap: {name: ...}`, `secret: {name: ...}` or `downwardAPI: {items: [{path: podname, fieldRef: {fieldPath: metadata.name}}]}`. `kubectl explain pod.spec.volumes.projected.sources` lists them.""", + show=r"""kubectl -n vols apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: bundle +spec: + containers: + - name: app + image: busybox:1.36 + command: [sleep, '86400'] + volumeMounts: + - {name: bundle, mountPath: /etc/bundle, readOnly: true} + volumes: + - name: bundle + projected: + sources: + - configMap: {name: app-config} + - secret: {name: app-secret} + - downwardAPI: + items: + - path: podname + fieldRef: {fieldPath: metadata.name} +EOF +kubectl -n vols wait --for=condition=Ready pod/bundle --timeout=120s +kubectl -n vols exec bundle -- ls /etc/bundle"""), + step("A volume for exactly one Pod", + r"""Create PersistentVolume `archive`: `hostPath` `/srv/archive` (type `DirectoryOrCreate`), `200Mi`, access mode **`ReadWriteOncePod`**, reclaim policy `Retain`, and `storageClassName: ""`. + +Then claim it: PVC `archive-claim` in `vols`, `ReadWriteOncePod`, `200Mi`, `storageClassName: ""`. + +The empty class matters on both sides. A claim with no class at all gets the default class, and its provisioner would make a fresh volume instead of binding yours.""", + check=r""" +eq "$(j pv archive '{.spec.accessModes[0]} {.spec.persistentVolumeReclaimPolicy} {.spec.hostPath.path}')" "ReadWriteOncePod Retain /srv/archive" \ + && eq "$(j -n vols pvc archive-claim '{.status.phase} {.spec.volumeName}')" "Bound archive" """, + hint=r"""`storageClassName: ""` is spelled out, with the quotes, under `spec` of both objects. A PersistentVolume is cluster-wide: no namespace.""", + show=r"""kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: PersistentVolume +metadata: + name: archive +spec: + capacity: {storage: 200Mi} + accessModes: [ReadWriteOncePod] + persistentVolumeReclaimPolicy: Retain + storageClassName: "" + hostPath: {path: /srv/archive, type: DirectoryOrCreate} +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: archive-claim + namespace: vols +spec: + storageClassName: "" + accessModes: [ReadWriteOncePod] + resources: + requests: {storage: 200Mi} +EOF +kubectl get pv archive +kubectl -n vols get pvc archive-claim"""), + step("Hand a Released volume to a new claim", + r"""Delete claim `archive-claim`. The volume becomes `Released`: it still remembers its old claim in `spec.claimRef`, and nothing binds it while it does. + +Make it `Available` again by removing `spec.claimRef` from the PV, then create claim `archive-claim-2` in `vols` (same modes, size and empty class). It binds `archive`, and the data on it is still there.""", + check=r""" +[ -z "$(j -n vols pvc archive-claim '{.metadata.name}')" ] && eq "$(j -n vols pvc archive-claim-2 '{.status.phase} {.spec.volumeName}')" "Bound archive" """, + hint=r"""`kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'` - or `kubectl edit pv archive` and delete the `claimRef:` block.""", + show=r"""kubectl -n vols delete pvc archive-claim +kubectl get pv archive +kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: archive-claim-2 + namespace: vols +spec: + storageClassName: "" + accessModes: [ReadWriteOncePod] + resources: + requests: {storage: 200Mi} +EOF +kubectl get pv archive"""), + ], + "outro": r"""You shared an emptyDir between containers, put one in memory, combined three sources in one projected volume, and took a `Retain` volume through `Bound`, `Released` and back. + +**For the exam** + +- `ReadWriteOnce` is one **node**, not one Pod; `ReadWriteOncePod` is one Pod. A claim's modes must be a subset of the volume's. +- `Retain` keeps the data and leaves the volume `Released`; to reuse it, remove `spec.claimRef` (or delete and re-create the PV). `Delete` removes the volume with its claim. `Recycle` is deprecated. +- `storageClassName: ""` binds a pre-made volume; leaving the field out means "the default class". +- configMap and secret volumes follow changes to their object after a short delay - but not when mounted with `subPath`, and environment variables never do. +- `kubectl explain pod.spec.volumes` lists every volume type; the docs page "Volumes" has an example of each. + +Next, prove it against the clock: **drill 1.2**.""", +} + +# ---- 1.3 PersistentVolumes and PersistentVolumeClaims --------------------------------------- +_PVS = r""" +$K apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: PersistentVolume +metadata: + name: pv-small + labels: {tier: bronze} +spec: + capacity: {storage: 100Mi} + accessModes: [ReadWriteOnce] + persistentVolumeReclaimPolicy: Retain + storageClassName: manual + hostPath: {path: /srv/pv-small, type: DirectoryOrCreate} +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: pv-large + labels: {tier: gold} +spec: + capacity: {storage: 1Gi} + accessModes: [ReadWriteOnce] + persistentVolumeReclaimPolicy: Retain + storageClassName: manual + hostPath: {path: /srv/pv-large, type: DirectoryOrCreate} +EOF +""" + +L13 = { + "id": 9103, "objective": "1.3", "workers": 0, + "title": "Claims: binding the right volume, protection, and claims that come and go", + "intro": r"""A claim binds **one** volume, and keeps it until the claim is deleted. Among the volumes that fit - access modes, class, at least the requested size - Kubernetes picks the smallest. You can steer it: + +``` +selector: matchLabels on the volume's labels "one of the gold volumes" +volumeName: the volume's name "exactly this volume" +``` + +Claims are also protected: one that a Pod still uses is not removed until the Pod is gone. And a StatefulSet makes claims of its own, one per replica, from `volumeClaimTemplates` - its `persistentVolumeClaimRetentionPolicy` says whether they go when replicas go. + +Some things only a **CSI driver** can do for a claim: grow it, snapshot it, make a new claim from a snapshot or from another claim. This cluster has the CSI hostpath driver and its class `csi-hostpath-sc` (expandable), as the exam's clusters do, besides kind's `standard`. + +Two volumes are ready, both of class `manual`: `pv-small` (100Mi, label `tier=bronze`) and `pv-large` (1Gi, `tier=gold`). Namespace `claims` has claim `growing` (`csi-hostpath-sc`, 10Mi), used by Pod `grower`.""", + "setup": NS + _PVS + r""" +ensure_csi_hostpath || { echo "could not install the CSI hostpath driver" >&2; exit 1; } +mkns claims +mkdir -p "$WS/lesson-1.3" +$K -n claims apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: {name: growing} +spec: {storageClassName: csi-hostpath-sc, accessModes: [ReadWriteOnce], resources: {requests: {storage: 10Mi}}} +--- +apiVersion: v1 +kind: Pod +metadata: {name: grower} +spec: + containers: + - {name: grower, image: busybox:1.36, command: [sh, -c, 'echo kept > /data/note.txt; sync; sleep 86400'], volumeMounts: [{name: d, mountPath: /data}]} + volumes: [{name: d, persistentVolumeClaim: {claimName: growing}}] +EOF +$K -n claims wait --for=condition=Ready pod/grower --timeout=180s >/dev/null +""", + "steps": [ + step("Bind by label", + r"""Create claim `gold-claim` in `claims`: class `manual`, `ReadWriteOnce`, `100Mi` - and a **selector** for `tier=gold`. + +Without the selector it would bind `pv-small`, the smallest volume that fits. With it, it binds `pv-large`.""", + check=r"""eq "$(j -n claims pvc gold-claim '{.status.phase} {.spec.volumeName}')" "Bound pv-large" """, + hint=r"""`spec.selector.matchLabels: {tier: gold}` on the claim.""", + show=r"""kubectl -n claims apply -f - <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: gold-claim +spec: + storageClassName: manual + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 100Mi} + selector: + matchLabels: {tier: gold} +EOF +kubectl get pv"""), + step("Bind by name, and use it", + r"""Create claim `pinned` in `claims` that binds `pv-small` **by name** (class `manual`, `ReadWriteOnce`, `100Mi`), and Pod `user` (`busybox:1.36`, `sleep 86400`) that mounts it at `/data`.""", + check=r"""eq "$(j -n claims pvc pinned '{.status.phase} {.spec.volumeName}')" "Bound pv-small" && eq "$(j -n claims pod user '{.status.phase}')" Running \ + && eq "$(j -n claims pod user '{.spec.volumes[0].persistentVolumeClaim.claimName}')" pinned""", + hint=r"""`spec.volumeName: pv-small` on the claim. The volume's class and modes must still match the claim's.""", + show=r"""kubectl -n claims apply -f - <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: pinned +spec: + storageClassName: manual + volumeName: pv-small + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 100Mi} +--- +apiVersion: v1 +kind: Pod +metadata: + name: user +spec: + containers: + - name: user + image: busybox:1.36 + command: [sleep, '86400'] + volumeMounts: + - {name: data, mountPath: /data} + volumes: + - name: data + persistentVolumeClaim: {claimName: pinned} +EOF +kubectl -n claims wait --for=condition=Ready pod/user --timeout=120s +kubectl -n claims get pvc"""), + step("A claim in use does not go", + r"""Delete claim `pinned` without waiting (`--wait=false`). It does not disappear: `kubectl -n claims get pvc` shows it `Terminating`, held by a **finalizer**. + +Write that finalizer's name to `{WS}/lesson-1.3/finalizer.txt`.""", + check=r""" +[ -n "$(j -n claims pvc pinned '{.metadata.deletionTimestamp}')" ] \ + && eq "$(tr -d ' \r\n' < "$WS/lesson-1.3/finalizer.txt" 2>/dev/null)" kubernetes.io/pvc-protection""", + hint=r"""`kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers}'`. Without `--wait=false`, `kubectl delete` would sit there until the claim is really gone.""", + show=r"""kubectl -n claims delete pvc pinned --wait=false +kubectl -n claims get pvc pinned +kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers[0]}' > {WS}/lesson-1.3/finalizer.txt +cat {WS}/lesson-1.3/finalizer.txt"""), + step("Let it go", + r"""Delete Pod `user`. With no Pod using it any more, the protection lifts and claim `pinned` is removed. `pv-small` (reclaim policy `Retain`) is now `Released`.""", + check=r"""[ -z "$(j -n claims pod user '{.metadata.name}')" ] && [ -z "$(j -n claims pvc pinned '{.metadata.name}')" ] && eq "$(j pv pv-small '{.status.phase}')" Released""", + hint=r"""`kubectl -n claims delete pod user`, then `kubectl -n claims get pvc` and `kubectl get pv`.""", + show=r"""kubectl -n claims delete pod user --grace-period=1 +kubectl -n claims get pvc +kubectl get pv pv-small"""), + step("Claims made by a StatefulSet", + r"""Create a StatefulSet `db` in `claims` with 2 replicas (`busybox:1.36`, `sleep 86400`) and its headless Service `db`. Give it a `volumeClaimTemplates` entry `data` (`ReadWriteOnce`, `50Mi`, class `standard`) mounted at `/var/lib/db`, and a `persistentVolumeClaimRetentionPolicy` that **deletes** a replica's claim when the StatefulSet is scaled down and **retains** the claims when the StatefulSet is deleted. + +Each replica gets a claim of its own: `data-db-0`, `data-db-1`.""", + check=r""" +eq "$(j -n claims sts db '{.spec.persistentVolumeClaimRetentionPolicy.whenScaled} {.spec.persistentVolumeClaimRetentionPolicy.whenDeleted}')" "Delete Retain" || exit 1 +eq "$(j -n claims sts db '{.status.readyReplicas}')" 2 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound && eq "$(j -n claims pvc data-db-1 '{.status.phase}')" Bound""", + hint=r"""`kubectl create deploy db --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400` is a start: change the kind, add `serviceName: db`, drop `strategy`, add `volumeClaimTemplates` beside `template`. The Service: `clusterIP: None`. `kubectl explain sts.spec.persistentVolumeClaimRetentionPolicy`.""", + show=r"""kubectl -n claims apply -f - <<'EOF' +apiVersion: v1 +kind: Service +metadata: + name: db +spec: + clusterIP: None + selector: {app: db} + ports: + - port: 5432 +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: db +spec: + serviceName: db + replicas: 2 + selector: + matchLabels: {app: db} + persistentVolumeClaimRetentionPolicy: + whenScaled: Delete + whenDeleted: Retain + template: + metadata: + labels: {app: db} + spec: + containers: + - name: db + image: busybox:1.36 + command: [sleep, '86400'] + volumeMounts: + - {name: data, mountPath: /var/lib/db} + volumeClaimTemplates: + - metadata: + name: data + spec: + storageClassName: standard + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 50Mi} +EOF +kubectl -n claims rollout status sts db --timeout=180s +kubectl -n claims get pvc"""), + step("Scale down, and the claim goes with the replica", + r"""Scale `db` to 1 replica. Pod `db-1` goes, and because `whenScaled` is `Delete`, so does `data-db-1`. `data-db-0` stays. + +The default for both fields is `Retain`: without the policy, `data-db-1` would wait for the replica to come back.""", + check=r""" +eq "$(j -n claims sts db '{.spec.replicas}')" 1 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound || exit 1 +for i in $(seq 1 10); do [ -z "$(j -n claims pvc data-db-1 '{.metadata.name}')" ] && exit 0; sleep 3; done; exit 1""", + hint=r"""`kubectl -n claims scale sts db --replicas=1`, then `kubectl -n claims get pvc -w`.""", + show=r"""kubectl -n claims scale sts db --replicas=1 +kubectl -n claims rollout status sts db --timeout=120s +kubectl -n claims get pvc"""), + step("Grow a claim in use", + r"""Claim `growing` is full. Its class allows expansion (`kubectl get sc csi-hostpath-sc`: `ALLOWVOLUMEEXPANSION true`), so make it **50Mi** - while Pod `grower` keeps running. Raise the request; the driver does the rest, and `status.capacity` follows.""", + check=r"""eq "$(j -n claims pvc growing '{.spec.resources.requests.storage} {.status.capacity.storage}')" "50Mi 50Mi" && eq "$(j -n claims pod grower '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """, + hint=r"""`kubectl -n claims edit pvc growing`, or `kubectl patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}'`. A claim never shrinks; with `standard` (local-path) the same patch is refused.""", + show=r"""kubectl get sc csi-hostpath-sc +kubectl -n claims patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}' +sleep 10 +kubectl -n claims get pvc growing"""), + step("Snapshot it, and restore the snapshot", + r"""Take VolumeSnapshot `growing-snap` of claim `growing` (snapshot class `csi-hostpath-snapclass`), wait until it is ready to use, then create claim `restored` (`csi-hostpath-sc`, 50Mi) **from** the snapshot: its `dataSource` names the VolumeSnapshot. The new volume holds what `growing` held when the snapshot was taken.""", + check=r""" +eq "$(j -n claims volumesnapshot growing-snap '{.spec.source.persistentVolumeClaimName} {.status.readyToUse}')" "growing true" && eq "$(j -n claims pvc restored '{.spec.dataSource.kind}/{.spec.dataSource.name} {.status.phase}')" "VolumeSnapshot/growing-snap Bound" """, + hint=r"""`apiVersion: snapshot.storage.k8s.io/v1`, `kind: VolumeSnapshot`, `spec: {volumeSnapshotClassName: ..., source: {persistentVolumeClaimName: growing}}`. The claim: `dataSource: {apiGroup: snapshot.storage.k8s.io, kind: VolumeSnapshot, name: growing-snap}`. A clone is the same with `dataSource: {kind: PersistentVolumeClaim, name: growing}`.""", + show=r"""kubectl -n claims apply -f - <<'EOF' +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshot +metadata: + name: growing-snap +spec: + volumeSnapshotClassName: csi-hostpath-snapclass + source: + persistentVolumeClaimName: growing +EOF +kubectl -n claims wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/growing-snap --timeout=120s +kubectl -n claims apply -f - <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: restored +spec: + storageClassName: csi-hostpath-sc + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 50Mi} + dataSource: + apiGroup: snapshot.storage.k8s.io + kind: VolumeSnapshot + name: growing-snap +EOF +kubectl -n claims get volumesnapshot,pvc"""), + ], + "outro": r"""You steered binding with a selector and with a volume's name, watched claim protection hold a claim until its Pod went, let a StatefulSet's claims follow its replicas, and grew, snapshotted and restored a claim on a CSI class. + +**For the exam** + +- A claim binds the smallest volume that fits, unless `selector` or `volumeName` say otherwise. The class must match on both sides. +- A `Pending` claim: `describe pvc` - no volume fits (size, modes, class, selector), the class has no provisioner, or it waits for its first consumer. +- `Terminating` claim: some Pod still uses it (`kubernetes.io/pvc-protection`). Find the Pod, not the finalizer. +- StatefulSet claims are named `TEMPLATE-STATEFULSET-ORDINAL` and kept by default; `persistentVolumeClaimRetentionPolicy` (`whenScaled`, `whenDeleted`) can delete them. +- A Pod's `ephemeral.volumeClaimTemplate` makes a claim that lives and dies with the Pod - a generic ephemeral volume, named `POD-VOLUME`. +- Expansion, snapshots and clones need a CSI driver that does them: `kubectl get sc` (ALLOWVOLUMEEXPANSION), `kubectl get volumesnapshotclass`, `kubectl get csidrivers`. + +Next, prove it against the clock: **drill 1.3**.""", +} + +LESSONS = [L11, L12, L13] diff --git a/cka/lessons_tshoot.py b/cka/lessons_tshoot.py new file mode 100644 index 0000000..2b2380b --- /dev/null +++ b/cka/lessons_tshoot.py @@ -0,0 +1,551 @@ +"""CKA lessons of the Troubleshooting domain (2.1-2.5). See cka/lessons.py for what a lesson is.""" +from sim.bank import NS, step + +# ---- 2.1 clusters and nodes ------------------------------------------------------------------- +L21 = { + "id": 9201, "objective": "2.1", "workers": 1, + "title": "A node goes NotReady: the kubelet, its configuration, and pressure", + "intro": r"""A node is `Ready` while its **kubelet** reports in. When it stops, the control plane waits under a minute (its node monitor grace period) and marks the node `NotReady`; five minutes later - the default toleration for an unreachable node - its Pods are evicted. So for a `NotReady` node the questions go in this order: + +``` +kubectl get nodes / describe node what the control plane last heard +ssh NODE; sudo -i +systemctl status kubelet is it running, is it enabled? +journalctl -u kubelet -n 50 why it stopped, or why it will not start +/var/lib/kubelet/config.yaml its configuration (KubeletConfiguration) +/etc/kubernetes/kubelet.conf how it reaches the API server +/etc/systemd/system/kubelet.service.d/ its flags (drop-ins), and /var/lib/kubelet/kubeadm-flags.env +systemctl status containerd the runtime it needs +``` + +The kubelet also watches the node's resources. Below a **hard eviction threshold** - `memory.available`, `nodefs.available`, ... - it sets a condition (`MemoryPressure`, `DiskPressure`, `PIDPressure`) and evicts Pods to recover. + +Something has happened to the kubelet on worker `{W1}`.""", + "setup": r""" +mkdir -p "$WS/lesson-2.1" +nx "$W1" 'systemctl disable --now kubelet' /dev/null 2>&1 +# the control plane notices after its grace period; the lesson starts with the node NotReady +for i in $(seq 1 40); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" != True ] && exit 0; sleep 3; done +echo "$W1 is still Ready a minute after its kubelet was stopped" >&2; exit 1 +""", + "steps": [ + step("Which node, and since when", + r"""Find the node that is not `Ready` and write its name to `{WS}/lesson-2.1/node.txt`. + +Then look at `kubectl describe node` for it: under **Conditions** every row says `Kubelet stopped posting node status` - the control plane has not heard from the kubelet, so it knows nothing more. The answer is on the node.""", + check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/node.txt" 2>/dev/null)" "$W1" """, + hint=r"""`kubectl get nodes`. `--no-headers` and `awk '$2 != "Ready" {print $1}'` give just the name.""", + show=r"""kubectl get nodes +kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt +kubectl describe node {W1} | grep -A8 Conditions"""), + step("Bring the kubelet back - for good", + r"""On `{W1}` (`ssh {W1}`, then `sudo -i`), look at the kubelet: `systemctl status kubelet` says `inactive (dead)` and `disabled`. Start it, and enable it so that it survives a reboot - a fix that is gone after the next reboot is not a fix. + +Within a few seconds the node is `Ready` again.""", + check=r""" +nx "$W1" 'systemctl is-active -q kubelet && systemctl is-enabled -q kubelet' /dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["cgroupDriver"])') +[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/cgroup.txt" 2>/dev/null)" "$want" """, + hint=r"""`configz` returns JSON: `| jq -r .kubeletconfig.cgroupDriver`. It must match containerd's `SystemdCgroup` setting, or Pods fail to start.""", + show=r"""ssh -n {W1} 'ps -o args= -C kubelet | tr " " "\n" | grep -- --config; sudo grep -n cgroupDriver /var/lib/kubelet/config.yaml' +kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt +cat {WS}/lesson-2.1/cgroup.txt"""), + step("Set the eviction thresholds", + r"""Look at `evictionHard` in `/var/lib/kubelet/config.yaml` on `{W1}`: kind sets every threshold to `0%`, which means "never evict" - fine for a test cluster, not for a real one. + +On `{W1}`, make the kubelet evict when less than **`100Mi`** of memory or **`10%`** of the node filesystem is left: `memory.available: 100Mi` and `nodefs.available: 10%` under `evictionHard`. Restart the kubelet, and confirm it through `configz`.""", + check=r""" +$K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c ' +import json, sys +e = json.load(sys.stdin)["kubeletconfig"].get("evictionHard") or {} +sys.exit(0 if e.get("memory.available") == "100Mi" and e.get("nodefs.available") == "10%" else 1)' || exit 1 +eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""", + hint=r"""Edit the file with `vim`, or `sed`: change the `nodefs.available` line and add a `memory.available` line inside the `evictionHard:` block (two spaces of indent). A kubelet that does not come back after the restart: `journalctl -u kubelet -n 20` shows the line it could not parse.""", + show=r"""ssh {W1} sudo bash <<'EOF' +f=/var/lib/kubelet/config.yaml +grep -n -A4 '^evictionHard:' $f +sed -i 's/^ nodefs.available: .*/ nodefs.available: 10%/' $f +sed -i '/^evictionHard:/a\ memory.available: 100Mi' $f +grep -n -A5 '^evictionHard:' $f +systemctl restart kubelet +EOF +kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq .kubeletconfig.evictionHard"""), + step("Read the pressure conditions", + r"""The kubelet reports the result of those thresholds as node conditions. Write the status of `{W1}`'s three pressure conditions to `{WS}/lesson-2.1/pressure.txt`, one per line, as `TYPE STATUS` - for example `MemoryPressure False`. + +`True` on any of them means the kubelet is evicting Pods, and the node gets a taint (`node.kubernetes.io/memory-pressure`, ...) so no new Pods land there.""", + check=r""" +want=$(for c in MemoryPressure DiskPressure PIDPressure; do echo "$c $(j node "$W1" "{.status.conditions[?(@.type==\"$c\")].status}")"; done | sort) +eq "$(tr -s ' \t' ' ' < "$WS/lesson-2.1/pressure.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """, + hint=r"""`kubectl describe node {W1}` lists them under Conditions. As a list: `-o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}'`, then keep the three `*Pressure` lines.""", + show=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt +cat {WS}/lesson-2.1/pressure.txt"""), + ], + "outro": r"""You found a `NotReady` node, brought its kubelet back for good, read its configuration from the file and from the running kubelet, set eviction thresholds and read what the kubelet reports about them. + +**For the exam** + +- `NotReady`: the node is the place to look. `systemctl status kubelet`, then `journalctl -u kubelet`. A kubelet that restarts in a loop prints the reason a moment before each exit. +- Fixes stay fixed: `systemctl enable`, and edits in the files, not on the command line. +- Usual faults: kubelet stopped or disabled, a wrong path in `config.yaml` (CA file, static Pod path), a wrong flag in a drop-in or `kubeadm-flags.env` (`systemctl daemon-reload` after editing a unit), containerd down or the wrong runtime endpoint, an expired client certificate. +- `kubectl get --raw /api/v1/nodes/NODE/proxy/configz` is the kubelet's live configuration. +- Eviction: `evictionHard` (and `evictionSoft` with a grace period) in `config.yaml`. Pressure conditions taint the node; evicted Pods show `Failed` with reason `Evicted`. + +Next, prove it against the clock: **drill 2.1**.""", +} + +# ---- 2.2 cluster components ------------------------------------------------------------------- +_BREAK_CP = r""" +mkns ops +mkdir -p "$WS/lesson-2.2" +gone() { local i; for i in $(seq 1 45); do [ -z "$(nx "$CP" "crictl ps --name $1 -q" &2; exit 1; } +$K -n ops create deploy queue --image=busybox:1.36 --replicas=2 -- sleep 86400 >/dev/null +nx "$CP" 'sed -i "s|^\( *\)- kube-scheduler$|\1- kube-schedulerr|" /etc/kubernetes/manifests/kube-scheduler.yaml' &2; exit 1; } +$K -n ops run waiting --image=busybox:1.36 -- sleep 86400 >/dev/null +""" + +L22 = { + "id": 9202, "objective": "2.2", "workers": 1, + "title": "The control plane's own Pods: find what is down and why", + "intro": r"""On a kubeadm cluster the control plane runs as **static Pods**: the kubelet of the control-plane node starts whatever manifests it finds in `/etc/kubernetes/manifests`, and restarts them when a file changes. Each component has its own symptom when it is down: + +``` +kube-apiserver kubectl itself fails: "connection refused" +etcd the API server fails, then kubectl +kube-scheduler new Pods stay Pending, with no node and no event +kube-controller-manager Deployments get no ReplicaSet, ReplicaSets no Pods; nothing self-heals +``` + +When the API server is up, `kubectl -n kube-system get pods` and `describe` show the others. When it is not, go to the node: `crictl ps -a`, `crictl logs ID`, the files under `/var/log/pods/`, and `journalctl -u kubelet`. + +Two things in Namespace `ops` do not work: Deployment `queue` has no Pods, and Pod `waiting` has no node.""", + "setup": NS + _BREAK_CP, + "steps": [ + step("Which components are down", + r"""Find the control-plane components that are not running and write their names to `{WS}/lesson-2.2/down.txt`, one per line (for example `kube-proxy`). + +`kubectl -n kube-system get pods` shows them; on the control-plane node, `sudo crictl ps -a` shows their containers, and `ls /etc/kubernetes/manifests` what the kubelet is told to run.""", + check=r"""eq "$(tr -d ' \r' < "$WS/lesson-2.2/down.txt" 2>/dev/null | grep . | sort | tr '\n' ' ')" "kube-controller-manager kube-scheduler " """, + hint=r"""A Pod that is not `Running` with all its containers ready. The Pod names end in the node's name; the component is the part before it.""", + show=r"""kubectl -n kube-system get pods +ls /etc/kubernetes/manifests +sudo crictl ps -a | grep kube- +printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt"""), + step("Repair the scheduler", + r"""Find out why `kube-scheduler` does not start, and fix its manifest. `kubectl -n kube-system describe pod kube-scheduler-{CP}` shows the error in its events, and so do `sudo crictl ps -a --name kube-scheduler` and `sudo journalctl -u kubelet`. + +The kubelet notices the saved file on its own: no restart, no `kubectl apply`. Once the scheduler runs, Pod `waiting` gets a node.""", + check=r""" +nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml && [ -n "$(crictl ps --name kube-scheduler -q)" ]' /dev/null && grep -q 'readyz check passed' "$WS/lesson-2.2/readyz.txt" """, + hint=r"""`kubectl get --raw` sends a GET to any API path, with your credentials. Quote the path: `?` means something to the shell.""", + show=r"""kubectl get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt | tail -3 +kubectl get --raw '/livez'; echo"""), + ], + "outro": r"""You read the control plane from both sides - `kubectl` and the node's container runtime - and repaired two static Pods by editing their manifests in place. + +**For the exam** + +- Static Pod manifests: `/etc/kubernetes/manifests` (the kubelet's `staticPodPath`). Edit the file; the kubelet restarts the Pod. Keep a copy *outside* that directory - a backup inside it is started too. +- Symptoms: no scheduler, Pods `Pending` with no events; no controller manager, Deployments without ReplicaSets; no API server, `kubectl` refuses to connect - then `crictl ps -a`, `crictl logs`, `/var/log/pods/kube-system_kube-apiserver-*/`. +- Typical faults: a wrong image tag, a typo in the command or a flag, a wrong path to a certificate, a volume mount that does not match a `hostPath`. +- `kubectl get --raw '/readyz?verbose'` and `/livez` show the API server's own checks, etcd's among them. + +Next, prove it against the clock: **drill 2.2**.""", +} + +# ---- 2.3 resource usage ----------------------------------------------------------------------- +L23 = { + "id": 9203, "objective": "2.3", "workers": 1, + "title": "What uses the CPU and memory: kubectl top and the metrics pipeline", + "intro": r"""`kubectl top` shows what Pods and nodes **use** right now. The numbers come from **metrics-server**, which asks every kubelet every 15 seconds and serves the result as an API of its own, `metrics.k8s.io`: + +``` +kubelet (cAdvisor) ──▶ metrics-server ──▶ /apis/metrics.k8s.io ──▶ kubectl top, HorizontalPodAutoscaler +``` + +Without metrics-server there is no `kubectl top` and no autoscaling on CPU or memory. Usage is not the same as **requests**: requests are what the scheduler reserves, and `kubectl describe node` shows them under *Allocated resources*. + +metrics-server runs on this cluster. Namespace `metrics` has a few Pods: one of them burns CPU, one holds a lot of memory, and Pod `duo` has two containers.""", + "setup": NS + r""" +ensure_metrics_server || { echo "metrics-server did not become ready" >&2; exit 1; } +mkns metrics +mkdir -p "$WS/lesson-2.3" +$K -n metrics apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: Pod +metadata: {name: burner} +spec: + containers: + - name: burner + image: busybox:1.36 + command: [sh, -c, 'while true; do :; done'] + resources: {limits: {cpu: 300m, memory: 32Mi}} +--- +apiVersion: v1 +kind: Pod +metadata: {name: hoarder} +spec: + containers: + - name: hoarder + image: busybox:1.36 + command: [sh, -c, 'dd if=/dev/zero of=/cache/blob bs=1M count=90 2>/dev/null; sleep 86400'] + volumeMounts: [{name: cache, mountPath: /cache}] + volumes: [{name: cache, emptyDir: {medium: Memory}}] +--- +apiVersion: v1 +kind: Pod +metadata: {name: idler} +spec: + containers: + - {name: idler, image: busybox:1.36, command: [sleep, '86400']} +--- +apiVersion: v1 +kind: Pod +metadata: {name: duo} +spec: + containers: + - {name: app, image: busybox:1.36, command: [sleep, '86400']} + - name: cache + image: busybox:1.36 + command: [sh, -c, 'dd if=/dev/zero of=/cache/blob bs=1M count=40 2>/dev/null; sleep 86400'] + volumeMounts: [{name: cache, mountPath: /cache}] + volumes: [{name: cache, emptyDir: {medium: Memory}}] +EOF +$K -n metrics wait --for=condition=Ready pod --all --timeout=180s >/dev/null +# until metrics-server has a sample of every Pod +for i in $(seq 1 40); do [ "$($K top pod -n metrics --no-headers 2>/dev/null | grep -c .)" -ge 4 ] && exit 0; sleep 5; done +echo "kubectl top shows no metrics for the lesson's Pods" >&2; exit 1 +""", + "steps": [ + step("The heaviest Pod by memory", + r"""Write the name of the Pod in `metrics` that uses the most memory to `{WS}/lesson-2.3/memory.txt`. `kubectl top` can sort for you.""", + check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/memory.txt" 2>/dev/null)" hoarder""", + hint=r"""`kubectl top pod -n metrics --sort-by=memory`. `--no-headers | head -1 | awk '{print $1}'` keeps just the name.""", + show=r"""kubectl top pod -n metrics --sort-by=memory +kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt"""), + step("The busiest Pod in the cluster", + r"""Across **all** Namespaces, which Pod uses the most CPU? Write it as `NAMESPACE/NAME` to `{WS}/lesson-2.3/cpu.txt`.""", + check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/cpu.txt" 2>/dev/null)" metrics/burner""", + hint=r"""`kubectl top pod -A --sort-by=cpu`; with `-A` the first column is the Namespace.""", + show=r"""kubectl top pod -A --sort-by=cpu | head -5 +kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt"""), + step("Inside one Pod", + r"""`kubectl top pod` adds up a Pod's containers. Which container of Pod `duo` uses the memory? Write its name to `{WS}/lesson-2.3/container.txt`.""", + check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/container.txt" 2>/dev/null)" cache""", + hint=r"""`--containers` splits the Pod into its containers.""", + show=r"""kubectl top pod duo -n metrics --containers +kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt"""), + step("Usage against what is reserved", + r"""`kubectl top node` shows use; `kubectl describe node {W1}`, under **Allocated resources**, shows what Pods have **requested** - which is what decides whether another Pod fits. + +Write the memory that `{W1}` can give to Pods - its **allocatable** memory, as the API reports it (for example `8012345Ki`) - to `{WS}/lesson-2.3/allocatable.txt`.""", + check=r"""want=$(j node "$W1" '{.status.allocatable.memory}'); [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/allocatable.txt" 2>/dev/null)" "$want" """, + hint=r"""Allocatable is capacity minus what the node keeps for itself and for the eviction threshold: `.status.allocatable.memory`.""", + show=r"""kubectl top node +kubectl describe node {W1} | grep -A9 'Allocated resources' +kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt"""), + step("The metrics API itself", + r"""`kubectl top` only reads an API. Read it yourself: save what `/apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder` returns in `{WS}/lesson-2.3/raw.json`. + +If this path answers `NotFound` or `ServiceUnavailable` on some cluster, `kubectl top` fails there too: look at the `metrics-server` Deployment in `kube-system`, and at `kubectl get apiservice v1beta1.metrics.k8s.io`.""", + check=r"""python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); sys.exit(0 if d.get("kind")=="PodMetrics" and d["metadata"]["name"]=="hoarder" and d.get("containers") else 1)' "$WS/lesson-2.3/raw.json" 2>/dev/null""", + hint=r"""`kubectl get --raw PATH`.""", + show=r"""kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json +jq '.containers' {WS}/lesson-2.3/raw.json +kubectl get apiservice v1beta1.metrics.k8s.io"""), + ], + "outro": r"""You ranked Pods by memory and CPU, looked inside a Pod, compared use with what a node can give, and read the metrics API that `kubectl top` uses. + +**For the exam** + +- `kubectl top pod [-A] [-n NS] [-l SELECTOR] --sort-by=cpu|memory [--containers]`, `kubectl top node`. Write exactly what is asked - often just the name. +- No metrics: is metrics-server running, and is `apiservice v1beta1.metrics.k8s.io` `Available`? On kubeadm clusters it often needs `--kubelet-insecure-tls`. +- `top` is usage; `describe node` → *Allocated resources* is requests and limits. The scheduler goes by requests. +- Metrics lag by a few seconds and are averages: a Pod that has just started may not appear yet. + +Next, prove it against the clock: **drill 2.3**.""", +} + +# ---- 2.4 container output streams ------------------------------------------------------------ +L24 = { + "id": 9204, "objective": "2.4", "workers": 1, + "title": "Logs, the previous container, the node's log files, and debug containers", + "intro": r"""Everything a container writes to **stdout** and **stderr** is kept by the node's runtime and served by `kubectl logs`: + +``` +kubectl logs POD [-c CONTAINER] one container (-c when the Pod has several) + --previous the instance before the last restart - where a crash is explained + -l app=api --prefix every Pod of a label, each line marked with its Pod + --since=10m --tail=50 -f a window, the end, follow +``` + +On the node the same output lives in `/var/log/pods/NAMESPACE_POD_UID/CONTAINER/N.log` (N counts restarts), linked from `/var/log/containers/`. `crictl logs` reads it too. + +And when a container has no shell to `exec` into, `kubectl debug` adds an **ephemeral container** to the running Pod - your image, your tools, sharing the Pod's network and, with `--target`, its processes. `kubectl debug node/NAME` does the same for a node. + +Namespace `logs` has Pod `flaky`, which crashed once; Pods `api-1` to `api-3`; and Pod `quiet`, built from an image with no shell at all.""", + "setup": NS + r""" +mkns logs +mkdir -p "$WS/lesson-2.4" +$K -n logs apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: Pod +metadata: {name: flaky} +spec: + volumes: [{name: state, emptyDir: {}}] + containers: + - name: flaky + image: busybox:1.36 + volumeMounts: [{name: state, mountPath: /state}] + command: [sh, -c, 'if [ ! -f /state/booted ]; then touch /state/booted; echo "FATAL config /etc/flaky.conf not found"; exit 1; fi; echo "INFO started on second attempt"; exec sleep 86400'] +--- +apiVersion: v1 +kind: Pod +metadata: {name: quiet} +spec: + containers: + - {name: quiet, image: registry.k8s.io/pause:3.10.2} +EOF +for n in 1 2 3; do + msg='INFO request served'; [ $n = 2 ] && msg='ERROR upstream refused the request' + $K -n logs run api-$n --image=busybox:1.36 --labels=app=api -- sh -c "while true; do echo \"$msg\"; sleep 5; done" >/dev/null +done +$K -n logs wait --for=condition=Ready pod --all --timeout=180s >/dev/null +for i in $(seq 1 30); do [ "$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null && exit 0; sleep 3; done +echo "flaky did not restart" >&2; exit 1 +""", + "steps": [ + step("Why it crashed", + r"""Pod `flaky` is `Running` now, with one restart. The logs of the running container only say it started; the reason for the crash is in the logs of the instance **before** it. + +Save those in `{WS}/lesson-2.4/previous.log`.""", + check=r"""grep -q 'FATAL config /etc/flaky.conf not found' "$WS/lesson-2.4/previous.log" 2>/dev/null && ! grep -q 'second attempt' "$WS/lesson-2.4/previous.log" """, + hint=r"""`kubectl logs --previous` (or `-p`).""", + show=r"""kubectl -n logs logs flaky +kubectl -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log +cat {WS}/lesson-2.4/previous.log"""), + step("The one replica that complains", + r"""Pods `api-1`, `api-2` and `api-3` all carry `app=api`. One of them logs errors. Read all three at once, find it, and write its name to `{WS}/lesson-2.4/erroring.txt`.""", + check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/erroring.txt" 2>/dev/null)" api-2""", + hint=r"""`kubectl logs -l app=api --prefix` puts `[pod/NAME/CONTAINER]` before each line. With a selector, kubectl shows only the last 10 lines of each Pod unless you say `--tail`.""", + show=r"""kubectl -n logs logs -l app=api --prefix --tail=2 +kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt +cat {WS}/lesson-2.4/erroring.txt"""), + step("The log file on the node", + r"""The node keeps one file per container instance. Write the full path of the file that holds the output of `flaky`'s **current** instance to `{WS}/lesson-2.4/logfile.txt`. + +Find the node with `kubectl -n logs get pod flaky -o wide`, then look under `/var/log/pods/` there. The directory name has the Pod's UID in it; the file name is the restart count.""", + check=r""" +p=$(tr -d ' \r\n' < "$WS/lesson-2.4/logfile.txt" 2>/dev/null) +uid=$(j -n logs pod flaky '{.metadata.uid}'); n=$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}'); node=$(j -n logs pod flaky '{.spec.nodeName}') +eq "$p" "/var/log/pods/logs_flaky_$uid/flaky/$n.log" && nx "$node" "grep -q 'second attempt' '$p'" {WS}/lesson-2.4/logfile.txt +ssh -n {W1} "sudo cat $(cat {WS}/lesson-2.4/logfile.txt)"""), + step("Debug a container that has no shell", + r"""`kubectl -n logs exec quiet -- sh` fails: the `pause` image has nothing but its one program. Add an ephemeral container to the running Pod instead: name `dbg`, image `busybox:1.36`, **targeting** container `quiet` so that it sees its processes, running `sleep 3600`. + +Then run `ps` in `dbg` and write the command of the target's process - the first one, PID 1 - to `{WS}/lesson-2.4/target.txt`.""", + check=r""" +eq "$(j -n logs pod quiet '{.spec.ephemeralContainers[0].name} {.spec.ephemeralContainers[0].targetContainerName}')" "dbg quiet" || exit 1 +grep -qx '/pause' <(tr -d ' \r' < "$WS/lesson-2.4/target.txt" 2>/dev/null)""", + hint=r"""`kubectl debug POD --image=IMAGE --target=CONTAINER -c NAME -- COMMAND`. Interactively you would add `-it` and use the shell straight away; without it, `kubectl exec POD -c dbg -- ps` reaches it once it runs. Ephemeral containers cannot be removed - the Pod keeps them until it goes.""", + show=r"""kubectl -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600 +for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done +kubectl -n logs exec quiet -c dbg -- ps +kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt"""), + step("Debug the node itself", + r"""`kubectl debug node/{W1} --image=busybox:1.36` starts a Pod on that node with the node's whole filesystem mounted at `/host` - useful when you cannot `ssh` to a node, but can reach the API. + +Use it to read the node's `/etc/os-release`, and write the value of its `PRETTY_NAME` to `{WS}/lesson-2.4/os.txt`.""", + check=r""" +want=$(nx "$W1" '. /etc/os-release; echo "$PRETTY_NAME"' /dev/null | sed 's/^PRETTY_NAME=//; s/ *$//')" "$want" \ + && [ -n "$($K get pods -A -o name 2>/dev/null | grep node-debugger)" ]""", + hint=r"""Without `-it`, give it a command and read its output with `kubectl logs`: the Pod is called `node-debugger-NODE-xxxxx`, in your current Namespace. Delete it afterwards.""", + show=r"""kubectl debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release +pod=$(kubectl get pods -o name | grep node-debugger | head -1) +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s +kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt +cat {WS}/lesson-2.4/os.txt"""), + ], + "outro": r"""You read a crashed instance's logs, searched several Pods at once, found the log file on the node, and debugged both a Pod without a shell and a node. + +**For the exam** + +- A crash is explained in `kubectl logs --previous`. `describe pod` adds the exit code and the reason (`Error`, `OOMKilled`, `Completed`). +- Several containers: `-c NAME`, or `--all-containers`. Several Pods: `-l SELECTOR --prefix`, and mind the default `--tail=10`. +- Write logs to a file exactly as asked: `> file` for stdout. A container's stderr is in its logs too - `kubectl logs` does not separate them; the node's log file does, with `stdout` / `stderr` on each line. +- `kubectl debug -it POD --image=busybox --target=CONTAINER` for a container without tools; `kubectl debug node/NAME -it --image=busybox`, then `chroot /host`, for a node. + +Next, prove it against the clock: **drill 2.4**.""", +} + +# ---- 2.5 services and networking -------------------------------------------------------------- +L25 = { + "id": 9205, "objective": "2.5", "workers": 1, + "title": "A Service that does not answer: selector, ports, policy, and kube-proxy", + "intro": r"""A request to a Service goes through several hands, and each can drop it: + +``` +client ──DNS──▶ ClusterIP:port ──kube-proxy rules──▶ endpoint PodIP:targetPort ──NetworkPolicy──▶ container +``` + +So check them in order: + +- **Endpoints**: `kubectl get endpointslices -l kubernetes.io/service-name=SVC`. None? The selector matches no ready Pod. +- **Ports**: `targetPort` must be the port the container listens on (a number, or the name of a container port). +- **NetworkPolicy**: once a policy selects a Pod, only what it allows gets in. +- **kube-proxy** turns each Service into rules on every node - iptables chains named `KUBE-SVC-...` here. + +In Namespace `shop`, Deployment `web` (nginx) has a Service `web` in front of it, and nothing gets through. Pods `client` and `stranger` are there to send requests from.""", + "setup": NS + r""" +mkns shop +mkdir -p "$WS/lesson-2.5" +$K -n shop create deploy web --image=nginx:1.27 --replicas=2 --port=80 >/dev/null +$K -n shop apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: Service +metadata: {name: web} +spec: + selector: {app: webapp} + ports: + - {port: 80, targetPort: 8080} +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: {name: lockdown} +spec: + podSelector: {matchLabels: {app: web}} + policyTypes: [Ingress] +EOF +$K -n shop run client --image=busybox:1.36 --labels=app=client -- sleep 86400 >/dev/null +$K -n shop run stranger --image=busybox:1.36 --labels=app=stranger -- sleep 86400 >/dev/null +$K -n shop rollout status deploy web --timeout=180s >/dev/null +$K -n shop wait --for=condition=Ready pod --all --timeout=180s >/dev/null +""", + "steps": [ + step("No endpoints", + r"""Ask the Service from `client`: `kubectl -n shop exec client -- wget -qO- -T3 web`. It times out. Look at the Service's endpoints: there are none, because its selector matches no Pod. + +Fix the **Service** so that it selects the Pods of Deployment `web`.""", + check=r""" +eq "$(j -n shop svc web '{.spec.selector.app}')" web || exit 1 +n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' | grep -c .) +[ "$n" -eq 2 ]""", + hint=r"""Compare `kubectl -n shop get svc web -o wide` (SELECTOR) with `kubectl -n shop get pods --show-labels`. `kubectl set selector` or `kubectl edit svc`.""", + show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web +kubectl -n shop get svc web -o wide +kubectl -n shop get pods --show-labels +kubectl -n shop set selector svc web app=web +kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""), + step("The wrong port", + r"""Now the Service has endpoints, but they point at port `8080` - and nginx listens on `80`. Make the Service send traffic to the port the container serves.""", + check=r""" +eq "$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{.items[0].ports[0].port}')" 80 && eq "$(j -n shop svc web '{.spec.ports[0].port}')" 80""", + hint=r"""`targetPort` is the container's port; `port` is the Service's own. `kubectl -n shop edit svc web`, or a JSON patch on `/spec/ports/0/targetPort`.""", + show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web +kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]' +kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""), + step("Let the right client in", + r"""Still nothing: NetworkPolicy `lockdown` selects the `web` Pods and allows no ingress at all. + +Keep it - do not delete it - and let **`client`** (label `app=client`) reach the `web` Pods on TCP 80. `stranger` must still be refused.""", + check=r""" +[ -n "$(j -n shop netpol lockdown '{.metadata.name}')" ] || exit 1 +ok() { $K -n shop exec "$1" -- wget -qO- -T 3 web >/dev/null 2>&1; } +for i in 1 2 3 4 5; do ok client && break; sleep 3; done; ok client || exit 1 +! ok stranger""", + hint=r"""Add an `ingress` rule to `lockdown`: `from: [{podSelector: {matchLabels: {app: client}}}]` with `ports: [{protocol: TCP, port: 80}]`. A NetworkPolicy's ports are the Pod's, not the Service's.""", + show=r"""kubectl -n shop apply -f - <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: lockdown +spec: + podSelector: + matchLabels: {app: web} + policyTypes: [Ingress] + ingress: + - from: + - podSelector: + matchLabels: {app: client} + ports: + - {protocol: TCP, port: 80} +EOF +kubectl -n shop exec client -- wget -qO- -T3 web | head -4 +kubectl -n shop exec stranger -- wget -qO- -T3 web || echo refused"""), + step("How kube-proxy routes it", + r"""kube-proxy turned Service `web` into rules on every node. Find out how: its mode is in ConfigMap `kube-proxy` in `kube-system`. Write the mode to `{WS}/lesson-2.5/mode.txt`. + +Then, on the control-plane node, find the iptables chain that kube-proxy made for this Service - the `KUBE-SVC-...` chain that its ClusterIP jumps to - and write the chain's name to `{WS}/lesson-2.5/chain.txt`.""", + check=r""" +mode=$($K -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}'); mode=${mode:-iptables} +eq "$(tr -d ' \r\n"' < "$WS/lesson-2.5/mode.txt" 2>/dev/null)" "$mode" || exit 1 +ip=$(j -n shop svc web '{.spec.clusterIP}') +want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" /dev/null)" "$want" """, + hint=r"""`kubectl -n kube-system get cm kube-proxy -o yaml | grep mode`. On the node: `sudo iptables-save -t nat | grep CLUSTER-IP` - the line for the Service jumps (`-j`) to its chain. In `nftables` mode the rules are in `sudo nft list ruleset` instead.""", + show=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt +ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}') +sudo iptables-save -t nat | grep -- "-d $ip/32" +sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt +sudo iptables-save -t nat | grep "^-A $(cat {WS}/lesson-2.5/chain.txt)" """), + ], + "outro": r"""You took a Service from "times out" to answering by fixing its selector, its target port and a NetworkPolicy, and followed it into kube-proxy's rules. + +**For the exam** + +- Endpoints first: `kubectl get endpointslices -l kubernetes.io/service-name=SVC` (or `kubectl get endpoints SVC`). Empty means the selector, or Pods that are not Ready. +- `port` is the Service's, `targetPort` the container's; a named `targetPort` must match a `containerPort` name. +- Test from a Pod: `kubectl exec POD -- wget -qO- -T3 SVC:PORT`, or a throwaway `kubectl run tmp --rm -it --restart=Never --image=busybox:1.36 -- ...`. +- A NetworkPolicy never blocks what it does not select; once one selects a Pod, everything not allowed is dropped - and egress policies need DNS (UDP/TCP 53) allowed. +- kube-proxy: a DaemonSet in `kube-system`, configured by ConfigMap `kube-proxy`; `iptables-save -t nat | grep KUBE-SVC` or `nft list ruleset` on a node. + +Next, prove it against the clock: **drill 2.5**.""", +} + +LESSONS = [L21, L22, L23, L24, L25] diff --git a/cka/lessons_work.py b/cka/lessons_work.py new file mode 100644 index 0000000..d77387c --- /dev/null +++ b/cka/lessons_work.py @@ -0,0 +1,672 @@ +"""CKA lessons of the Workloads and Scheduling domain (3.1-3.5). See cka/lessons.py for what a lesson is.""" +from sim.bank import NS, step + +# ---- 3.1 rolling updates and rollbacks -------------------------------------------------------- +L31 = { + "id": 9301, "objective": "3.1", "workers": 1, + "title": "Rollouts: strategy, a release that fails, undo, and pause", + "intro": r"""Every change to a Deployment's Pod **template** is a new **revision**: a new ReplicaSet, filled while the old one is emptied. Changes elsewhere - `replicas`, the strategy - are not revisions. + +``` +strategy: RollingUpdate maxSurge: extra Pods allowed maxUnavailable: missing Pods allowed +strategy: Recreate all old Pods go first, then the new ones come - a gap, on purpose +kubectl rollout status | history [--revision=N] | undo [--to-revision=N] | pause | resume | restart +``` + +A failed release does not roll itself back: the rollout stops, and with `maxUnavailable: 0` the old Pods keep serving until you act. + +Namespace `rollout` runs Deployment `api`: `nginx:1.26`, 4 replicas.""", + "setup": NS + r""" +mkns rollout +mkdir -p "$WS/lesson-3.1" +$K -n rollout create deploy api --image=nginx:1.26 --replicas=4 >/dev/null +$K -n rollout rollout status deploy api --timeout=180s >/dev/null +""", + "steps": [ + step("Never fewer than four", + r"""Change the update strategy of `api` so that an update first adds **one** Pod and never takes one away before its replacement is ready: `maxSurge: 1`, `maxUnavailable: 0`.""", + check=r"""eq "$(j -n rollout deploy api '{.spec.strategy.type} {.spec.strategy.rollingUpdate.maxSurge} {.spec.strategy.rollingUpdate.maxUnavailable}')" "RollingUpdate 1 0" """, + hint=r"""`kubectl -n rollout edit deploy api`, or `kubectl patch` with `{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}`.""", + show=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' +kubectl -n rollout get deploy api -o jsonpath='{.spec.strategy}'; echo"""), + step("A release with a reason", + r"""Update `api` to `nginx:1.27`, and record why: the annotation `kubernetes.io/change-cause` on the Deployment, set to `upgrade to 1.27`. `kubectl rollout history` shows it beside the revision.""", + check=r""" +eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1 +eq "$(j -n rollout deploy api '{.metadata.annotations.kubernetes\.io/change-cause}')" "upgrade to 1.27" || exit 1 +eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas}')" "4 4" """, + hint=r"""`kubectl set image deploy/api nginx=nginx:1.27` (the container is called `nginx`), then `kubectl annotate deploy api kubernetes.io/change-cause='...'`. The `--record` flag is gone.""", + show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27 +kubectl -n rollout annotate deploy api kubernetes.io/change-cause='upgrade to 1.27' --overwrite +kubectl -n rollout rollout status deploy api --timeout=180s +kubectl -n rollout rollout history deploy api"""), + step("A release that cannot start", + r"""Now a mistake: set the image of `api` to `nginx:1.27-broken`, a tag that does not exist. + +Watch what happens: one new Pod (the surge) fails to pull, and the rollout stops there. Because `maxUnavailable` is `0`, all four `nginx:1.27` Pods keep serving. `kubectl rollout status` would wait for ever (here, until `progressDeadlineSeconds` marks it failed).""", + check=r""" +eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27-broken || exit 1 +[ "$(j -n rollout deploy api '{.status.availableReplicas}')" = 4 ] || exit 1 +$K -n rollout get pods -l app=api --no-headers 2>/dev/null | grep -Eq 'ErrImagePull|ImagePullBackOff' """, + hint=r"""`kubectl set image` again. `kubectl -n rollout get pods` shows the Pod that cannot pull; `kubectl -n rollout get rs` shows the two ReplicaSets side by side.""", + show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27-broken +sleep 10 +kubectl -n rollout get pods +kubectl -n rollout get rs"""), + step("Undo", + r"""Roll `api` back to the revision before - `nginx:1.27` - and let the rollout finish.""", + check=r""" +eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1 +eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """, + hint=r"""`kubectl rollout undo deploy/api`. It does not create a ReplicaSet: it scales the old one back up, and that revision gets a new number.""", + show=r"""kubectl -n rollout rollout undo deploy/api +kubectl -n rollout rollout status deploy api --timeout=180s +kubectl -n rollout rollout history deploy api"""), + step("Back to a revision by number", + r"""Look at the history, then go back to the revision that ran **`nginx:1.26`**, by its number. `kubectl rollout history deploy api --revision=N` shows what a revision's template was.""", + check=r""" +eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.26 || exit 1 +eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """, + hint=r"""Revision 1 is where the Deployment started. `kubectl rollout undo deploy/api --to-revision=1`.""", + show=r"""kubectl -n rollout rollout history deploy api +kubectl -n rollout rollout history deploy api --revision=1 | grep Image +kubectl -n rollout rollout undo deploy/api --to-revision=1 +kubectl -n rollout rollout status deploy api --timeout=180s"""), + step("Two changes, one rollout", + r"""Each `set` command starts a rollout of its own. To make two changes at once, **pause** the Deployment, make them - image `nginx:1.27`, and an environment variable `MODE=prod` in the container - then **resume**: one new revision, one rollout.""", + check=r""" +[ "$(j -n rollout deploy api '{.spec.paused}')" != true ] || exit 1 +eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1 +eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].env[?(@.name=="MODE")].value}')" prod || exit 1 +eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """, + hint=r"""`kubectl rollout pause deploy/api`, `kubectl set image ...`, `kubectl set env deploy/api MODE=prod`, `kubectl rollout resume deploy/api`.""", + show=r"""kubectl -n rollout rollout pause deploy/api +kubectl -n rollout set image deploy/api nginx=nginx:1.27 +kubectl -n rollout set env deploy/api MODE=prod +kubectl -n rollout rollout resume deploy/api +kubectl -n rollout rollout status deploy api --timeout=180s +kubectl -n rollout rollout history deploy api"""), + ], + "outro": r"""You tuned the strategy, released with a recorded cause, watched a broken release stall safely, undid it, went back to a numbered revision, and batched two changes into one rollout. + +**For the exam** + +- `kubectl set image deploy/NAME CONTAINER=IMAGE` - the container's name, not the image's. +- `rollout undo` goes to the previous revision; `--to-revision=N` to any in `rollout history`. The history keeps `revisionHistoryLimit` old ReplicaSets (10 by default). +- `maxSurge` / `maxUnavailable` take numbers or percentages; `maxUnavailable: 0` keeps full capacity during an update. `Recreate` has no such fields. +- `rollout restart` makes new Pods with an unchanged spec (it stamps an annotation into the template). +- A stuck rollout: `kubectl rollout status` waits, `describe deploy` shows `Progressing=False` after `progressDeadlineSeconds`. + +Next, prove it against the clock: **drill 3.1**.""", +} + +# ---- 3.2 ConfigMaps and Secrets --------------------------------------------------------------- +_ENC_CHECK = r""" +nx "$CP" 'ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/conf/db --print-value-only 2>/dev/null | head -c 64' "$WS/lesson-3.2/app.properties" +printf 'DB_USER=ledger\nDB_PASS=t0ps3cret\n' > "$WS/lesson-3.2/db.env" +""", + "steps": [ + step("A ConfigMap from a file and a literal", + r"""Create ConfigMap `app-config` in `conf` with two keys: `app.properties`, holding the file `{WS}/lesson-3.2/app.properties`, and `MODE` with the value `prod`.""", + check=r""" +eq "$(j -n conf cm app-config '{.data.MODE}')" prod && $K -n conf get cm app-config -o jsonpath='{.data.app\.properties}' | grep -q '^cache.size=128$'""", + hint=r"""`kubectl create configmap NAME --from-file=PATH --from-literal=KEY=VALUE`. With `--from-file`, the key is the file's name.""", + show=r"""kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod +kubectl -n conf describe cm app-config"""), + step("A Secret from an env file", + r"""Create Secret `db` (type `Opaque`) in `conf` from `{WS}/lesson-3.2/db.env`, one key per line of the file: `DB_USER` and `DB_PASS`. Then read `DB_PASS` back, decoded.""", + check=r""" +eq "$(j -n conf secret db '{.type}')" Opaque && eq "$(j -n conf secret db '{.data.DB_USER}' | base64 -d)" ledger && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""", + hint=r"""`kubectl create secret generic NAME --from-env-file=PATH`. Reading one key: `-o jsonpath='{.data.DB_PASS}' | base64 -d`.""", + show=r"""kubectl -n conf create secret generic db --from-env-file={WS}/lesson-3.2/db.env +kubectl -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d; echo"""), + step("Consume both", + r"""Run Pod `app` in `conf` (`busybox:1.36`, `sleep 86400`) that gets: + +- every key of Secret `db` as an environment variable (`envFrom`), +- the variable `APP_MODE` from key `MODE` of ConfigMap `app-config`, +- ConfigMap `app-config` as files under `/etc/app`.""", + check=r""" +x() { $K -n conf exec app -- "$@" 2>/dev/null; } +eq "$(x printenv DB_PASS)" t0ps3cret && eq "$(x printenv APP_MODE)" prod && x cat /etc/app/app.properties | grep -q '^log.level=info$'""", + hint=r"""`envFrom: [{secretRef: {name: db}}]`, `env: [{name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: MODE}}}]`, and a `configMap` volume.""", + show=r"""kubectl -n conf apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: app +spec: + containers: + - name: app + image: busybox:1.36 + command: [sleep, '86400'] + envFrom: + - secretRef: {name: db} + env: + - name: APP_MODE + valueFrom: + configMapKeyRef: {name: app-config, key: MODE} + volumeMounts: + - {name: config, mountPath: /etc/app} + volumes: + - name: config + configMap: {name: app-config} +EOF +kubectl -n conf wait --for=condition=Ready pod/app --timeout=120s +kubectl -n conf exec app -- sh -c 'printenv DB_USER APP_MODE; cat /etc/app/app.properties'"""), + step("Change the configuration, live", + r"""Change `cache.size` to `256` in ConfigMap `app-config` - edit the file and re-create the ConfigMap's key from it, or `kubectl edit`. + +Then watch `/etc/app/app.properties` inside Pod `app`: within a minute the kubelet replaces the file. `APP_MODE` would not change, even if you edited `MODE`: the environment is fixed at start.""", + check=r"""$K -n conf exec app -- cat /etc/app/app.properties 2>/dev/null | grep -q '^cache.size=256$'""", + hint=r"""Regenerate and apply: `kubectl create configmap app-config --from-file=... --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f -` - the usual way to update a ConfigMap from files.""", + show=r"""sed -i 's/^cache.size=.*/cache.size=256/' {WS}/lesson-3.2/app.properties +kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f - +sleep 30 +kubectl -n conf exec app -- cat /etc/app/app.properties"""), + step("A TLS Secret", + r"""Make a self-signed certificate for `shop.example.com` (`openssl` is on this host), and store it as Secret `shop-tls` in `conf`, of type `kubernetes.io/tls` - the type an Ingress or Gateway expects. Its keys are `tls.crt` and `tls.key`.""", + check=r""" +eq "$(j -n conf secret shop-tls '{.type}')" kubernetes.io/tls || exit 1 +j -n conf secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'shop.example.com'""", + hint=r"""`openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com`, then `kubectl create secret tls NAME --cert=tls.crt --key=tls.key`.""", + show=r"""cd {WS}/lesson-3.2 +openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null +kubectl -n conf create secret tls shop-tls --cert=tls.crt --key=tls.key +kubectl -n conf get secret shop-tls +cd"""), + step("Encrypt Secrets at rest", + r"""Read Secret `db` straight out of etcd on `{CP}`, and you see `t0ps3cret` in plain text. Make the API server encrypt Secrets before it stores them: + +1. Write an `EncryptionConfiguration` to `/etc/kubernetes/enc/enc.yaml`: for resource `secrets`, provider `aescbc` with one key `key1` (32 random bytes, base64), then `identity: {}` so that what is still unencrypted can be read. +2. Start the API server with `--encryption-provider-config=/etc/kubernetes/enc/enc.yaml`, and mount `/etc/kubernetes/enc` into its Pod (a `hostPath` volume): it runs in a container and sees only what is mounted. +3. Once it is back, rewrite Secret `db`, so that it is stored again - encrypted this time. + +Edit a copy of the manifest outside `/etc/kubernetes/manifests` and copy it back: the kubelet runs every file in that directory, backups too.""", + check=_ENC_CHECK + r""" && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""", + hint=r"""The docs page "Encrypting Confidential Data at Rest" has the file. Stored values of encrypted Secrets start with `k8s:enc:aescbc:v1:key1:`: `sudo ETCDCTL_API=3 etcdctl ... get /registry/secrets/conf/db | od -c | head` (or `hexdump -C`). Rewriting: `kubectl get secret db -o json | kubectl replace -f -`. If the API server does not come back, `sudo crictl ps -a` and the logs under `/var/log/pods/kube-system_kube-apiserver-*` say why.""", + show=r"""sudo mkdir -p /etc/kubernetes/enc +sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null </dev/null 2>&1 && break; sleep 3; done +kubectl -n conf get secret db -o json | kubectl replace -f - +sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \ + get /registry/secrets/conf/db | od -c | head -4"""), + ], + "outro": r"""You built a ConfigMap and Secrets from files, consumed them three ways, updated configuration in a running Pod, and turned on encryption at rest for Secrets. + +**For the exam** + +- `kubectl create configmap|secret generic NAME --from-literal --from-file --from-env-file`; `kubectl create secret tls|docker-registry`. Generate, then apply: `--dry-run=client -o yaml | kubectl apply -f -`. +- `immutable: true` on a ConfigMap or Secret: no more changes, only delete and re-create - and Pods must restart to see the new one. +- Mounted keys follow updates; `subPath` mounts and environment variables do not. +- Encryption at rest: `--encryption-provider-config` and a mount for the file. The **first** provider encrypts; the others are tried when reading. `identity` first means "not encrypted". Existing Secrets stay as they are until rewritten (`kubectl get secrets -A -o json | kubectl replace -f -`). + +Next, prove it against the clock: **drill 3.2**.""", +} + +# ---- 3.3 autoscaling -------------------------------------------------------------------------- +L33 = { + "id": 9303, "objective": "3.3", "workers": 1, + "title": "Autoscaling: the HorizontalPodAutoscaler, and resizing a Pod in place", + "intro": r"""Three kinds of autoscaling exist, and only one is built in: + +``` +HorizontalPodAutoscaler more or fewer Pods, from metrics built in (needs metrics-server) +VerticalPodAutoscaler bigger or smaller requests per Pod an add-on, with its own CRDs +Cluster Autoscaler more or fewer nodes an add-on, per cloud +``` + +An HPA compares each Pod's usage with its **request**: 50% CPU means half of what the Pod requested. A Pod without a request gives it nothing to compare against. The HPA checks every 15 seconds and waits before scaling down (`behavior.scaleDown.stabilizationWindowSeconds`, five minutes unless you say otherwise). + +On v1.37 a running Pod's CPU and memory can also be changed **in place**, through its `resize` subresource, without a new Pod. + +metrics-server runs here. In Namespace `scale`, Deployment `api` (nginx) requests 100m CPU and idles; Deployment `cruncher` requests 50m and burns CPU.""", + "setup": NS + r""" +ensure_metrics_server || { echo "metrics-server did not become ready" >&2; exit 1; } +mkns scale +$K -n scale apply -f - >/dev/null <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: {name: api} +spec: + replicas: 1 + selector: {matchLabels: {app: api}} + template: + metadata: {labels: {app: api}} + spec: + containers: + - name: nginx + image: nginx:1.27 + resources: {requests: {cpu: 100m, memory: 64Mi}} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: cruncher} +spec: + replicas: 1 + selector: {matchLabels: {app: cruncher}} + template: + metadata: {labels: {app: cruncher}} + spec: + containers: + - name: cruncher + image: busybox:1.36 + command: [sh, -c, 'while true; do :; done'] + resources: {requests: {cpu: 50m, memory: 16Mi}, limits: {cpu: 150m, memory: 32Mi}} +EOF +$K -n scale rollout status deploy api --timeout=180s >/dev/null && $K -n scale rollout status deploy cruncher --timeout=180s >/dev/null +""", + "steps": [ + step("An HPA in one command", + r"""Make `api` scale between **2** and **6** replicas, aiming at **50%** CPU. Within a minute the HPA shows a current value - `kubectl -n scale get hpa` - and `api` runs at least 2 replicas.""", + check=r""" +eq "$(j -n scale hpa api '{.spec.minReplicas} {.spec.maxReplicas} {.spec.metrics[0].resource.name} {.spec.metrics[0].resource.target.averageUtilization}')" "2 6 cpu 50" || exit 1 +[ -n "$(j -n scale hpa api '{.status.currentMetrics[0].resource.current.averageUtilization}')" ] && [ "$(j -n scale deploy api '{.status.readyReplicas}')" -ge 2 ] 2>/dev/null""", + hint=r"""`kubectl autoscale deploy api --min=2 --max=6 --cpu=50%` (kubectl before 1.34: `--cpu-percent=50`). `` in the TARGETS column means no metrics yet - or a Pod without a CPU request.""", + show=r"""kubectl -n scale autoscale deploy api --min=2 --max=6 --cpu=50% +sleep 45 +kubectl -n scale get hpa api"""), + step("Scale down calmly", + r"""Give the HPA of `api` a `behavior`: when scaling **down**, wait **120** seconds of stable recommendations (`stabilizationWindowSeconds`), and remove at most **1** Pod per **60** seconds. + +`kubectl autoscale` cannot do that; edit the HPA (`autoscaling/v2`).""", + check=r""" +eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.stabilizationWindowSeconds}')" 120 || exit 1 +eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.policies[0].type} {.spec.behavior.scaleDown.policies[0].value} {.spec.behavior.scaleDown.policies[0].periodSeconds}')" "Pods 1 60" """, + hint=r"""Under `spec`: `behavior: {scaleDown: {stabilizationWindowSeconds: 120, policies: [{type: Pods, value: 1, periodSeconds: 60}]}}`. `kubectl explain hpa.spec.behavior.scaleDown`.""", + show=r"""kubectl -n scale patch hpa api --type merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":120,"policies":[{"type":"Pods","value":1,"periodSeconds":60}]}}}}' +kubectl -n scale describe hpa api | grep -A4 Behavior"""), + step("Watch it scale up", + r"""Write an HPA for `cruncher` as a manifest (`autoscaling/v2`): 1 to **4** replicas at **60%** average CPU. `cruncher` uses about three times its request, so the HPA soon adds Pods: wait until it wants more than one.""", + check=r""" +eq "$(j -n scale hpa cruncher '{.spec.maxReplicas} {.spec.metrics[0].resource.target.averageUtilization}')" "4 60" || exit 1 +[ "$(j -n scale hpa cruncher '{.status.desiredReplicas}')" -gt 1 ] 2>/dev/null""", + hint=r"""`kubectl autoscale ... --dry-run=client -o yaml` writes the manifest for you. `kubectl -n scale get hpa cruncher -w` shows it react; `describe hpa` lists its decisions as events.""", + show=r"""kubectl -n scale apply -f - <<'EOF' +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: cruncher +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: cruncher + minReplicas: 1 + maxReplicas: 4 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 60 +EOF +sleep 60 +kubectl -n scale get hpa cruncher"""), + step("Resize a running Pod", + r"""Run Pod `sizer` in `scale` (`busybox:1.36`, `sleep 86400`) with CPU request `100m` and limit `200m`, memory request and limit `64Mi`. + +Then give it more CPU **without replacing it**: request `150m`, limit `300m`, through the Pod's `resize` subresource. The container keeps running - its restart count stays 0 - and the Pod's status shows the new values once the kubelet has applied them.""", + check=r""" +eq "$(j -n scale pod sizer '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.limits.cpu}')" "150m 300m" || exit 1 +eq "$(j -n scale pod sizer '{.status.containerStatuses[0].resources.limits.cpu} {.status.containerStatuses[0].restartCount}')" "300m 0" """, + hint=r"""`kubectl patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{...}}]}}'`. A plain `kubectl edit` of a Pod's resources is refused; the subresource is the way in. A container's `resizePolicy` decides whether a change needs a restart.""", + show=r"""kubectl -n scale run sizer --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"sizer","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"200m","memory":"64Mi"}}}]}}' +kubectl -n scale wait --for=condition=Ready pod/sizer --timeout=120s +kubectl -n scale patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{"requests":{"cpu":"150m"},"limits":{"cpu":"300m"}}}]}}' +sleep 5 +kubectl -n scale get pod sizer -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}{.status.containerStatuses[0].restartCount}{"\n"}'"""), + ], + "outro": r"""You made HPAs both ways, slowed down their scale-down, watched one scale up, and resized a running Pod without replacing it. + +**For the exam** + +- `kubectl autoscale deploy NAME --min --max --cpu=N%` for the simple case; `autoscaling/v2` YAML for memory, several metrics, or `behavior`. +- The HPA needs metrics-server **and** requests on the target's containers. `kubectl describe hpa` explains ``. +- Utilization is relative to the request; `averageValue` is absolute (`target: {type: AverageValue, averageValue: 200Mi}`). +- In-place resize: `kubectl patch pod NAME --subresource resize ...`; `resizePolicy` (`NotRequired` / `RestartContainer`) per resource. Deployments still replace Pods when their template changes. +- VPA and the Cluster Autoscaler are add-ons: know what they do, and that the VPA should not fight an HPA over the same metric. + +Next, prove it against the clock: **drill 3.3**.""", +} + +# ---- 3.4 robust, self-healing deployments ------------------------------------------------------ +L34 = { + "id": 9304, "objective": "3.4", "workers": 1, + "title": "Self-healing: ReplicaSets, probes, disruption budgets, Jobs", + "intro": r"""Kubernetes heals by comparing what is with what should be, over and over: + +``` +ReplicaSet too few Pods? make more (a Deployment manages them) +livenessProbe container stuck? restart it +readinessProbe not ready? take it out of the Service's endpoints +startupProbe slow to start? hold the other probes until it is up +PodDisruptionBudget how many Pods a voluntary disruption (drain, eviction) may take +Job / CronJob run to completion, retry on failure, on a schedule +``` + +Namespace `heal` runs Deployment `web` (nginx, 3 replicas) and Deployment `api` (nginx, 2 replicas), whose Pods never become ready.""", + "setup": NS + r""" +mkns heal +mkdir -p "$WS/lesson-3.4" +$K -n heal create deploy web --image=nginx:1.27 --replicas=3 >/dev/null +$K -n heal apply -f - >/dev/null <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: {name: api} +spec: + replicas: 2 + selector: {matchLabels: {app: api}} + template: + metadata: {labels: {app: api}} + spec: + containers: + - name: nginx + image: nginx:1.27 + readinessProbe: + httpGet: {path: /healthz, port: 80} + periodSeconds: 5 +EOF +$K -n heal rollout status deploy web --timeout=180s >/dev/null +$K -n heal get pods -l app=web -o jsonpath='{.items[*].metadata.uid}' > "$PRIV/lesson-3.4.uids" +""", + "steps": [ + step("Delete a Pod, get a Pod", + r"""Delete one Pod of Deployment `web`. Its ReplicaSet notices that it has 2 of 3, and makes a new one at once. + +Write the name of that ReplicaSet to `{WS}/lesson-3.4/rs.txt`. `kubectl get pod POD -o yaml` names it under `ownerReferences`.""", + check=r""" +rs=$(tr -d ' \r\n' < "$WS/lesson-3.4/rs.txt" 2>/dev/null); [ -n "$rs" ] || exit 1 +eq "$(j -n heal rs "$rs" '{.metadata.ownerReferences[0].name} {.status.readyReplicas}')" "web 3" || exit 1 +now=$(j -n heal pods -l app=web '{.items[*].metadata.uid}'); for u in $now; do grep -q "$u" "$PRIV/lesson-3.4.uids" || exit 0; done; exit 1""", + hint=r"""`kubectl -n heal delete pod $(kubectl -n heal get pod -l app=web -o name | head -1)`. Pods of a Deployment are named DEPLOYMENT-RSHASH-RANDOM; the middle part is the ReplicaSet's suffix.""", + show=r"""kubectl -n heal delete $(kubectl -n heal get pod -l app=web -o name | head -1) +kubectl -n heal get pods -l app=web +kubectl -n heal get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].name}' > {WS}/lesson-3.4/rs.txt +kubectl -n heal get rs"""), + step("A readiness probe that asks the wrong thing", + r"""The Pods of `api` run, but none is `Ready`: `kubectl -n heal describe pod -l app=api` shows the readiness probe failing with `404`. nginx has no `/healthz`. Make the probe ask for `/` instead, and both Pods become ready.""", + check=r"""eq "$(j -n heal deploy api '{.spec.template.spec.containers[0].readinessProbe.httpGet.path}')" / && eq "$(j -n heal deploy api '{.status.readyReplicas}')" 2""", + hint=r"""A JSON patch on `/spec/template/spec/containers/0/readinessProbe/httpGet/path`, or `kubectl edit deploy api`. The fix is a change of template: a rollout.""", + show=r"""kubectl -n heal describe pod -l app=api | grep -m2 Readiness +kubectl -n heal patch deploy api --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]' +kubectl -n heal rollout status deploy api --timeout=120s"""), + step("A liveness probe that restarts", + r"""Run Pod `fragile` in `heal` (`busybox:1.36`) with the command `sh -c 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400'` and a **liveness** probe that runs `cat /tmp/healthy` every 5 seconds, starting after 5. + +After 20 seconds the file is gone, the probe fails three times, and the kubelet restarts the container - which makes the file again. Wait for the first restart.""", + check=r""" +eq "$(j -n heal pod fragile '{.spec.containers[0].livenessProbe.exec.command[1]}')" /tmp/healthy || exit 1 +[ "$(j -n heal pod fragile '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null""", + hint=r"""`livenessProbe: {exec: {command: [cat, /tmp/healthy]}, initialDelaySeconds: 5, periodSeconds: 5}`. `kubectl -n heal get pod fragile -w` shows RESTARTS go up; `describe` shows `Liveness probe failed`.""", + show=r"""kubectl -n heal apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: fragile +spec: + containers: + - name: fragile + image: busybox:1.36 + command: [sh, -c, 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400'] + livenessProbe: + exec: + command: [cat, /tmp/healthy] + initialDelaySeconds: 5 + periodSeconds: 5 +EOF +sleep 50 +kubectl -n heal get pod fragile"""), + step("A budget for disruptions", + r"""Create PodDisruptionBudget `web-pdb` in `heal`: at least **2** Pods of `app=web` must stay available. + +Delete Pod `fragile` first: it has done its job, and `drain` refuses a Pod that no controller would re-create (unless `--force`, which deletes it for good). + +Then try to drain `{W1}`, where all three `web` Pods run: `kubectl drain {W1} --ignore-daemonsets --timeout=30s`. It evicts what it may and refuses the rest - "would violate the pod's disruption budget". Save its output in `{WS}/lesson-3.4/drain.txt`, then make `{W1}` schedulable again.""", + check=r""" +eq "$(j -n heal pdb web-pdb '{.spec.minAvailable} {.spec.selector.matchLabels.app}')" "2 web" || exit 1 +grep -qi 'disruption budget' "$WS/lesson-3.4/drain.txt" 2>/dev/null && [ "$(j node "$W1" '{.spec.unschedulable}')" != true ]""", + hint=r"""`kubectl create pdb web-pdb --selector=app=web --min-available=2`. Drain prints the refusals on stderr: `2>&1 | tee FILE`. `kubectl uncordon {W1}` afterwards - drain cordons first.""", + show=r"""kubectl -n heal create pdb web-pdb --selector=app=web --min-available=2 +kubectl -n heal get pdb web-pdb +kubectl -n heal delete pod fragile --grace-period=1 +kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 | tee {WS}/lesson-3.4/drain.txt | tail -3 +kubectl uncordon {W1}"""), + step("A Job that runs to completion", + r"""Create Job `batch` in `heal`: `busybox:1.36` running `sh -c 'echo processed; sleep 2'`, **3** completions, **2** at a time, giving up after **2** retries (`backoffLimit`). Wait until it is complete.""", + check=r"""eq "$(j -n heal job batch '{.spec.completions} {.spec.parallelism} {.spec.backoffLimit} {.status.succeeded}')" "3 2 2 3" """, + hint=r"""`kubectl create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'`, then add `completions`, `parallelism` and `backoffLimit` under `spec`. `kubectl wait --for=condition=complete job/batch`.""", + show=r"""kubectl -n heal create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo processed; sleep 2' > {WS}/lesson-3.4/batch.yaml +yq -i '.spec.completions = 3 | .spec.parallelism = 2 | .spec.backoffLimit = 2' {WS}/lesson-3.4/batch.yaml +kubectl -n heal apply -f {WS}/lesson-3.4/batch.yaml +kubectl -n heal wait --for=condition=complete job/batch --timeout=120s +kubectl -n heal get pods -l job-name=batch"""), + step("A CronJob, and a run on demand", + r"""Create CronJob `report` in `heal`: every 30 minutes (`*/30 * * * *`), `busybox:1.36` running `date`, never two runs at once (`concurrencyPolicy: Forbid`), keeping the last **2** successful Jobs. + +Do not wait half an hour: start a Job `report-now` from the CronJob yourself, and let it finish.""", + check=r""" +eq "$(j -n heal cronjob report '{.spec.schedule} {.spec.concurrencyPolicy} {.spec.successfulJobsHistoryLimit}')" "*/30 * * * * Forbid 2" || exit 1 +eq "$(j -n heal job report-now '{.metadata.annotations.cronjob\.kubernetes\.io/instantiate} {.status.succeeded}')" "manual 1" """, + hint=r"""`kubectl create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date`, then the two fields. `kubectl create job report-now --from=cronjob/report`.""", + show=r"""kubectl -n heal create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date > {WS}/lesson-3.4/report.yaml +yq -i '.spec.concurrencyPolicy = "Forbid" | .spec.successfulJobsHistoryLimit = 2' {WS}/lesson-3.4/report.yaml +kubectl -n heal apply -f {WS}/lesson-3.4/report.yaml +kubectl -n heal create job report-now --from=cronjob/report +kubectl -n heal wait --for=condition=complete job/report-now --timeout=120s"""), + ], + "outro": r"""You watched a ReplicaSet replace a Pod, fixed a readiness probe, saw a liveness probe restart a container, stopped a drain with a disruption budget, and ran a Job and a CronJob. + +**For the exam** + +- Readiness decides **traffic** (endpoints); liveness decides **restarts**; a startup probe holds both off until the app has started. Probes: `httpGet`, `tcpSocket`, `exec`, `grpc`. +- A PDB has `minAvailable` **or** `maxUnavailable`. It stops evictions - a drain, or anything else that uses the Eviction API - never a `kubectl delete pod`, and never a node that just dies. +- Jobs: `completions`, `parallelism`, `backoffLimit`, `activeDeadlineSeconds`, `ttlSecondsAfterFinished`; a Job's Pods use `restartPolicy: Never` or `OnFailure`. +- CronJobs: `schedule` in cron syntax (UTC unless `timeZone`), `concurrencyPolicy`, `suspend`, history limits; `kubectl create job NAME --from=cronjob/CJ` to test one. + +Next, prove it against the clock: **drill 3.4**.""", +} + +# ---- 3.5 Pod admission and scheduling --------------------------------------------------------- +L35 = { + "id": 9305, "objective": "3.5", "workers": 2, + "title": "Scheduling: QoS, affinity, spreading, taints, defaults, and nodeName", + "intro": r"""The scheduler puts a Pod on a node that passes every **filter** and scores best: + +``` +requests does it fit? (requests, not limits, and not usage) +nodeSelector / nodeAffinity may it go there? (node labels) +podAffinity / podAntiAffinity near or away from other Pods (their labels, per topologyKey) +taints and tolerations is the node closed to it? cordon = the unschedulable taint +``` + +`nodeName` skips the scheduler entirely. Requests and limits also give each Pod a **QoS class**, which decides who is evicted first when a node runs short: `BestEffort`, then `Burstable`, then `Guaranteed`. A **LimitRange** fills in requests and limits for Pods that bring none. + +This cluster has two workers, `{W1}` and `{W2}`. Namespace `sched` is empty.""", + "setup": NS + r""" +mkns sched +""", + "steps": [ + step("Three QoS classes", + r"""Create three Pods in `sched`, each `busybox:1.36` running `sleep 86400`: + +- `gold`: requests **equal** to limits, for CPU (`100m`) and memory (`64Mi`) → `Guaranteed` +- `silver`: only a memory request of `32Mi` → `Burstable` +- `bronze`: no resources at all → `BestEffort`""", + check=r""" +for p in gold:Guaranteed silver:Burstable bronze:BestEffort; do eq "$(j -n sched pod "${p%%:*}" '{.status.qosClass}')" "${p##*:}" || exit 1; done""", + hint=r"""`kubectl run NAME --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400`, then `resources:` in the container. `.status.qosClass` shows the result.""", + show=r"""kubectl -n sched run bronze --image=busybox:1.36 -- sleep 86400 +kubectl -n sched run silver --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"silver","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"memory":"32Mi"}}}]}}' +kubectl -n sched run gold --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"gold","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"64Mi"}}}]}}' +kubectl -n sched get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass"""), + step("Only on SSD nodes", + r"""Label `{W2}` with `disk=ssd`. Then run Pod `fast` in `sched` (`busybox:1.36`, `sleep 86400`) with a **required** node affinity for `disk In (ssd)`. It lands on `{W2}` - and if no node had the label, it would stay `Pending`.""", + check=r""" +eq "$(j node "$W2" '{.metadata.labels.disk}')" ssd && eq "$(j -n sched pod fast '{.spec.nodeName}')" "$W2" \ + && eq "$(j -n sched pod fast '{.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key}')" disk""", + hint=r"""`kubectl label node {W2} disk=ssd`. The affinity block: `affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[].matchExpressions[]` with `key`, `operator: In`, `values`. The docs page "Assign Pods to Nodes using Node Affinity" has it to copy.""", + show=r"""kubectl label node {W2} disk=ssd +kubectl -n sched apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: fast +spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - {key: disk, operator: In, values: [ssd]} + containers: + - name: fast + image: busybox:1.36 + command: [sleep, '86400'] +EOF +kubectl -n sched get pod fast -o wide"""), + step("One replica per node", + r"""Create Deployment `spread` in `sched` (2 replicas, `busybox:1.36`, `sleep 86400`, label `app=spread`) whose Pods **refuse** to share a node with each other: a required pod anti-affinity on `app=spread`, with `topologyKey: kubernetes.io/hostname`. One Pod lands on each worker; a third replica would stay `Pending`.""", + check=r""" +eq "$(j -n sched deploy spread '{.status.readyReplicas}')" 2 || exit 1 +eq "$(j -n sched deploy spread '{.spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey}')" kubernetes.io/hostname || exit 1 +[ "$($K -n sched get pods -l app=spread -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 2 ]""", + hint=r"""`podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution: [{labelSelector: {matchLabels: {app: spread}}, topologyKey: kubernetes.io/hostname}]` in the Pod template. `topologySpreadConstraints` is the softer, counting way to spread.""", + show=r"""kubectl -n sched apply -f - <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: spread +spec: + replicas: 2 + selector: + matchLabels: {app: spread} + template: + metadata: + labels: {app: spread} + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: {app: spread} + topologyKey: kubernetes.io/hostname + containers: + - name: spread + image: busybox:1.36 + command: [sleep, '86400'] +EOF +kubectl -n sched rollout status deploy spread --timeout=120s +kubectl -n sched get pods -l app=spread -o wide"""), + step("A node for batch work only", + r"""Taint `{W1}` with `dedicated=batch:NoSchedule`: from now on only Pods that tolerate it are scheduled there (Pods already running stay - `NoExecute` would evict them). + +Then run Pod `nightly` in `sched` (`busybox:1.36`, `sleep 86400`) that tolerates the taint **and** is placed on `{W1}` with a `nodeSelector` on `kubernetes.io/hostname`. A toleration only opens the door; it does not lead the Pod there.""", + check=r""" +[ -n "$($K get node "$W1" -o jsonpath='{range .spec.taints[*]}{.key}={.value}:{.effect}{"\n"}{end}' | grep -x 'dedicated=batch:NoSchedule')" ] || exit 1 +eq "$(j -n sched pod nightly '{.spec.nodeName} {.status.phase}')" "$W1 Running" && eq "$(j -n sched pod nightly '{.spec.tolerations[?(@.key=="dedicated")].value}')" batch""", + hint=r"""`kubectl taint node {W1} dedicated=batch:NoSchedule`. The toleration: `{key: dedicated, operator: Equal, value: batch, effect: NoSchedule}`.""", + show=r"""kubectl taint node {W1} dedicated=batch:NoSchedule +kubectl -n sched apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: nightly +spec: + nodeSelector: + kubernetes.io/hostname: {W1} + tolerations: + - {key: dedicated, operator: Equal, value: batch, effect: NoSchedule} + containers: + - name: nightly + image: busybox:1.36 + command: [sleep, '86400'] +EOF +kubectl -n sched get pod nightly -o wide"""), + step("Defaults for Pods that bring none", + r"""Create LimitRange `defaults` in `sched`: containers that say nothing get a **request** of `50m` CPU and `32Mi` memory and a **limit** of `200m` and `128Mi`. + +Then run Pod `plain` (`busybox:1.36`, `sleep 86400`, no resources) and look at what it got. A LimitRange acts when a Pod is created: Pods that already exist keep what they had.""", + check=r""" +eq "$(j -n sched limitrange defaults '{.spec.limits[0].type}')" Container || exit 1 +eq "$(j -n sched pod plain '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.requests.memory} {.spec.containers[0].resources.limits.cpu} {.spec.containers[0].resources.limits.memory}')" "50m 32Mi 200m 128Mi" """, + hint=r"""`spec.limits: [{type: Container, defaultRequest: {cpu: 50m, memory: 32Mi}, default: {cpu: 200m, memory: 128Mi}}]` - `default` is the limit.""", + show=r"""kubectl -n sched apply -f - <<'EOF' +apiVersion: v1 +kind: LimitRange +metadata: + name: defaults +spec: + limits: + - type: Container + defaultRequest: {cpu: 50m, memory: 32Mi} + default: {cpu: 200m, memory: 128Mi} +EOF +kubectl -n sched run plain --image=busybox:1.36 -- sleep 86400 +kubectl -n sched get pod plain -o jsonpath='{.spec.containers[0].resources}'; echo"""), + step("Past the scheduler", + r"""Cordon `{W2}`. Then create Pod `manual` in `sched` (`busybox:1.36`, `sleep 86400`) with **`nodeName: {W2}`**. It runs there anyway: `nodeName` hands the Pod straight to that node's kubelet, and cordoning only tells the *scheduler* to stay away. + +Leave `{W2}` cordoned.""", + check=r"""eq "$(j node "$W2" '{.spec.unschedulable}')" true && eq "$(j -n sched pod manual '{.spec.nodeName} {.status.phase}')" "$W2 Running" """, + hint=r"""`kubectl cordon {W2}`, then a Pod manifest with `spec.nodeName`. Without a running scheduler, this is also how you place a Pod by hand.""", + show=r"""kubectl cordon {W2} +kubectl -n sched run manual --image=busybox:1.36 --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 86400 +kubectl -n sched get pod manual -o wide +kubectl get nodes"""), + ], + "outro": r"""You produced all three QoS classes, steered Pods with node affinity, kept replicas apart, reserved a node with a taint, set defaults with a LimitRange, and bypassed the scheduler. + +**For the exam** + +- `Pending` with `FailedScheduling`: the event lists why each node was rejected - insufficient cpu/memory, a taint not tolerated, affinity not matched, unschedulable. +- Requests decide placement; limits are enforced at run time (CPU is throttled, memory over the limit is `OOMKilled`). +- `required...` affinity filters, `preferred...` only scores. Pod (anti-)affinity needs a `topologyKey`. +- Taint effects: `NoSchedule`, `PreferNoSchedule`, `NoExecute` (evicts running Pods without a toleration). Remove one with a trailing `-`: `kubectl taint node N dedicated=batch:NoSchedule-`. +- ResourceQuota caps a Namespace's total and then **requires** requests/limits on new Pods; a LimitRange supplies them. PriorityClasses decide who is preempted. + +Next, prove it against the clock: **drill 3.5**.""", +} + +LESSONS = [L31, L32, L33, L34, L35] diff --git a/cka/questions.py b/cka/questions.py index e4b6819..1197c2b 100644 --- a/cka/questions.py +++ b/cka/questions.py @@ -21,9 +21,9 @@ TRACK = { "minutes": 120, "pass_pct": 66, # Kubernetes minor of the real exam environment, which picks the kind cluster (shared with CKS). - # LF FAQ / "Important Instructions: CKA and CKAD", checked 2026-10-02: "The CKA environment is - # currently running Kubernetes v1.35". The LF moves each exam on its own schedule. - "k8s": "v1.35", + # LF FAQ / "Important Instructions: CKA and CKAD", checked 2026-10-19: "The CKA environment is + # currently running Kubernetes v1.37". The LF moves each exam on its own schedule. + "k8s": "v1.37", # Worker nodes of a task's cluster unless the task says otherwise ("workers", or WORKERS below). # One, so the control-plane node keeps its NoSchedule taint and workloads land on a worker, as # in the exam; a task that names a worker ($W1, {W2}, ...) gets the three the bank was written for. @@ -34,7 +34,8 @@ TRACK = { # fixed simulator sets (./ksim sim c|d) - excluded from random sampling. Sets A and B were # retired on 2026-09-22; their last (fixed, verified) versions are in git history. "sets": {"c": "Simulator Set C", "d": "Simulator Set D (killer.sh-style)"}, - # CKA_Curriculum_v1.35.pdf (cncf/curriculum) - unchanged since the February 2025 revision + # CKA_Curriculum_v1.37.pdf (cncf/curriculum) - the same document as v1.35, unchanged since the + # February 2025 revision "domains": { "ARCH": "Cluster Architecture, Installation & Configuration (25%)", "WORK": "Workloads & Scheduling (15%)", @@ -97,7 +98,7 @@ Details you need: - "ssh to the control-plane node" is: `{SHELL_CP}` - `etcdctl` and `etcdutl` are installed on that node, in `/usr/local/bin` -Verify the snapshot with `etcdutl snapshot status` (etcd 3.6, which v1.35 runs, removed `etcdctl snapshot status`). +Verify the snapshot with `etcdutl snapshot status` (etcd 3.6 removed `etcdctl snapshot status`; v1.37 runs etcd 3.7). """, "setup": r""" nx "$CP" 'mkdir -p /opt/cka; rm -f /opt/cka/etcd-snapshot.db' @@ -655,11 +656,12 @@ $K -n default delete pod sched-probe --wait=false >/dev/null 2>&1 # ---- 68-85 / 86-102 in the two fixed simulator sets, 103-113 (pool tasks for the objectives the # ---- curriculum map found thin, see curriculum.py) in bank_objectives, and the pool tasks that # ---- bring every domain to 17 in bank_storage (114-124), bank_net (125-128) and bank_tshoot -# ---- (129-131). Numbering is contiguous - keep it that way. -from cka import bank_extra, bank_hard, bank_net, bank_objectives, bank_sim_c, bank_sim_d, bank_storage, bank_tshoot +# ---- (129-131); what a full CKA course teaches beyond that in bank_course (132-147). +# ---- Numbering is contiguous - keep it that way. +from cka import bank_course, bank_extra, bank_hard, bank_net, bank_objectives, bank_sim_c, bank_sim_d, bank_storage, bank_tshoot QUESTIONS = (QUESTIONS + bank_extra.QUESTIONS + bank_hard.QUESTIONS + bank_sim_c.QUESTIONS + bank_sim_d.QUESTIONS + bank_objectives.QUESTIONS - + bank_storage.QUESTIONS + bank_net.QUESTIONS + bank_tshoot.QUESTIONS) + + bank_storage.QUESTIONS + bank_net.QUESTIONS + bank_tshoot.QUESTIONS + bank_course.QUESTIONS) QUESTIONS.sort(key=lambda q: q["id"]) _ids = [q["id"] for q in QUESTIONS] assert len(_ids) == len(set(_ids)), "duplicate question ids" diff --git a/cka/solutions.py b/cka/solutions.py index c951c34..7405f63 100644 --- a/cka/solutions.py +++ b/cka/solutions.py @@ -104,6 +104,9 @@ spec: schedule: {type: string} target: {type: string} EOF +# the new type is served a moment after the CRD is created: a Backup sent at once can fail +# with "no matches for kind Backup" +k wait --for condition=established crd/backups.data.example.com # Step 2: Create a custom resource instance of Backup cat </dev/null; } +sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list +apt_get update +sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null +sudo kubeadm upgrade apply v1.37.1 -y /dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null +sudo systemctl daemon-reload && sudo systemctl restart kubelet +for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done +kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data +ssh {W1} sudo bash <<'EOF' +export DEBIAN_FRONTEND=noninteractive +sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list +apt-get update -qq /dev/null +apt-get install -y -qq kubeadm=1.37.1-1.1 /dev/null +kubeadm upgrade node /dev/null +apt-mark hold kubeadm kubelet kubectl >/dev/null +systemctl daemon-reload && systemctl restart kubelet +EOF +kubectl uncordon {W1} +""", +139: r""" +ssh {W1} sudo bash <<'EOF' +yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml +systemctl restart kubelet +EOF +for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done +""", } # ---- 1-113: the banks carried over from cka-sim. @@ -726,7 +758,9 @@ kubectl -n oslo-secure wait --for=condition=Ready pod/vault-reader --timeout=180 # text the candidate is shown - and a solution that stops working is found the next time it runs. AS_WRITTEN = [1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 40, 42, 43, 44, 47, 48, 49, 50, 52, 53, 54, 55, 56, 58, 62, 63, 64, 68, 69, 72, 73, 74, 75, 78, 80, 82, 83, - 85, 89, 98, 99, 100, 101, 103, 104, 109, 113] + 85, 89, 98, 99, 100, 101, 103, 104, 109, 113, + # bank_course.py: written to run as they stand + 133, 134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 145, 146, 147] def _as_written(): diff --git a/ckad/lessons.py b/ckad/lessons.py new file mode 100644 index 0000000..18bc8ab --- /dev/null +++ b/ckad/lessons.py @@ -0,0 +1,646 @@ +"""CKAD lessons: guided labs, one per curriculum objective, on a cluster of your own and without a +clock. A lesson explains, then walks you through steps that you do on its host; each step has a +check, a hint and the commands that do it. It ends by sending you to the objective's timed drill. +Lessons write nothing to the history or the scores - only a "lesson done" mark on the objective. + +./ksim lesson the lessons of this track, and which you have done +./ksim lesson 2.1 start the lesson of objective 2.1 (id or key) +./ksim lesson check check the step you are on; the next one follows +./ksim lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after + +A lesson's host is named after its id - lesson 9201 runs on `ckad9201` - which never clashes with +a task's. Its files go under {WS}/lesson-/. A step's `show` commands are what the +learner types; the audit runs exactly them, as `candidate`, so what a lesson shows always works. + +Pilot (2026-10-16): the Application Deployment domain, 2.1 to 2.4. +""" +from sim.bank import NS, step + +# ---- 2.1 blue/green and canary --------------------------------------------------------------- +_WEB = r""" +apiVersion: apps/v1 +kind: Deployment +metadata: + name: web-%{v} + labels: {app: web, version: %{v}} +spec: + replicas: 2 + selector: + matchLabels: {app: web, version: %{v}} + template: + metadata: + labels: {app: web, version: %{v}} + spec: + containers: + - name: web + image: busybox:1.36 + command: [sh, -c, 'mkdir -p /www && echo %{v} > /www/index.html && exec httpd -f -p 80 -h /www'] + ports: + - containerPort: 80 +""" + +# a request through Service web from Pod client, retried while endpoints settle +_ASK = r""" +ask() { local i; for i in 1 2 3 4 5 6; do [ "$($K -n shop exec client -- wget -qO- -T 3 web 2>/dev/null)" = "$1" ] && return 0; sleep 2; done; return 1; } +""" + +L21 = { + "id": 9201, "objective": "2.1", "workers": 0, + "title": "Blue/green and canary with plain Deployments", + "intro": r"""A Deployment replaces its Pods by rolling them over. Sometimes you want more control than that: run the new version **beside** the old one, try it, and decide when it gets the traffic. Nothing beyond plain Deployments and a Service is needed, because the **Service's selector** decides which Pods receive requests. + +**Blue/green**: two Deployments, one Service. Changing the Service's selector moves all traffic at once, and changing it back is your rollback: + +``` + ┌─▶ web-blue (blue) +Service web ─selector─┤ + └╌▶ web-green (green) +``` + +**Canary**: let the Service select **both** versions, and set the split with replica counts. With 4 stable Pods and 1 canary Pod, about one request in five reaches the canary. + +In Namespace `shop` you will find Deployment `web-blue` (each of its Pods answers `blue`), Service `web` in front of it, and a Pod `client` to send requests from.""", + "setup": NS + r""" +mkns shop +$K -n shop apply -f - >/dev/null </dev/null +$K -n shop rollout status deploy web-blue --timeout=180s >/dev/null +$K -n shop wait --for=condition=Ready pod/client --timeout=180s >/dev/null +mkdir -p "$WS/lesson-2.1" +""", + "steps": [ + step("Ask the Service who answers", + r"""Before you change anything, see what users get today. The Pod `client` has `wget`, and inside the Namespace a Service is reachable by its name: + +``` +kubectl -n shop exec client -- wget -qO- web +``` + +Save the answer in `{WS}/lesson-2.1/before.txt`.""", + check=r"""tr -d '\r' < "$WS/lesson-2.1/before.txt" 2>/dev/null | grep -qx blue""", + hint=r"""`kubectl exec POD -- COMMAND` runs a command in a running Pod; `wget -qO-` prints the page instead of saving it. Pipe it through `tee FILE`, or redirect with `> FILE`.""", + show=r"""kubectl -n shop exec client -- wget -qO- web | tee {WS}/lesson-2.1/before.txt"""), + step("Start green beside blue", + r"""Create Deployment `web-green` next to `web-blue`: 2 replicas, the same container, except that each Pod answers `green`. Its Pods carry the labels `app=web` and `version=green`. + +Copying blue's manifest is the quickest start: `kubectl -n shop get deploy web-blue -o yaml`. Change the name, the `version` label everywhere and the word `blue` in the command, and leave out `status`, `uid`, `resourceVersion` and `creationTimestamp`. Wait until both Pods are ready. + +The Service still selects `version=blue`, so no user sees green yet: you can try it first.""", + check=r""" +eq "$(j -n shop deploy web-green '{.status.availableReplicas}')" 2 \ + && eq "$(j -n shop deploy web-green '{.spec.template.metadata.labels.app}')" web \ + && eq "$(j -n shop deploy web-green '{.spec.template.metadata.labels.version}')" green || exit 1 +ip=$(j -n shop pods -l app=web,version=green '{.items[0].status.podIP}') +[ -n "$ip" ] && eq "$($K -n shop exec client -- wget -qO- -T 3 "$ip" 2>/dev/null)" green""", + hint=r"""Labels appear three times in a Deployment: `metadata.labels` (optional), `spec.selector.matchLabels` and `spec.template.metadata.labels` - the selector must match the template. `kubectl -n shop rollout status deploy web-green` waits until it is ready, and `kubectl -n shop get pods -l version=green -o wide` shows the Pod IPs to try.""", + show=r"""kubectl -n shop apply -f - <<'EOF' +""" + _WEB.replace("%{v}", "green").strip("\n") + r""" +EOF +kubectl -n shop rollout status deploy web-green +kubectl -n shop get pods -l app=web --show-labels"""), + step("Cut over to green", + r"""Point Service `web` at green. Changing the selector **is** the cutover: within a second every new request goes to green. + +Careful: `kubectl set selector` replaces the **whole** selector, so name both `app` and `version`. Then ask the Service again from `client` - the answer should be `green`.""", + check=_ASK + r""" +eq "$(j -n shop svc web '{.spec.selector.app}')" web && eq "$(j -n shop svc web '{.spec.selector.version}')" green && ask green""", + hint=r"""`kubectl set selector svc web 'app=web,version=green'` - or `kubectl patch svc web -p '{"spec":{"selector":{"version":"green"}}}'`, which merges and keeps `app`. `kubectl -n shop get svc web -o wide` shows the selector.""", + show=r"""kubectl -n shop set selector svc web 'app=web,version=green' +kubectl -n shop get svc web -o wide +kubectl -n shop exec client -- wget -qO- web"""), + step("Roll back in one command", + r"""Say green misbehaves. Rolling back a blue/green release is the same move in reverse: point the Service at blue again. Blue never stopped running, so this is instant - no new rollout, no image to pull. + +Switch back, and ask the Service once more.""", + check=_ASK + r""" +eq "$(j -n shop svc web '{.spec.selector.app}')" web && eq "$(j -n shop svc web '{.spec.selector.version}')" blue && ask blue""", + hint=r"""The command you used for the cutover, with `version=blue`.""", + show=r"""kubectl -n shop set selector svc web 'app=web,version=blue' +kubectl -n shop exec client -- wget -qO- web"""), + step("Canary: one Service, both versions", + r"""Now a canary. Let Service `web` select **every** Pod of the app, blue and green, by dropping `version` from its selector. Requests spread over all ready endpoints, so with 2 blue and 2 green Pods about half of them reach green. + +Ask the Service a few times and watch the answers change.""", + check=r""" +eq "$(j -n shop svc web '{.spec.selector.app}')" web && [ -z "$(j -n shop svc web '{.spec.selector.version}')" ] || exit 1 +for i in 1 2 3 4 5 6; do + n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' 2>/dev/null | grep -c .) + [ "$n" -eq 4 ] && exit 0; sleep 2 +done; exit 1""", + hint=r"""`kubectl set selector svc web 'app=web'` leaves only `app`. `kubectl -n shop get endpointslices -l kubernetes.io/service-name=web` lists the Pods that receive traffic.""", + show=r"""kubectl -n shop set selector svc web 'app=web' +kubectl -n shop get endpointslices -l kubernetes.io/service-name=web +for i in 1 2 3 4 5 6 7 8; do kubectl -n shop exec client -- wget -qO- web; done"""), + step("Weight the canary to 20%", + r"""Half the traffic is too much for a canary. The split follows the replica counts, so run **4** blue Pods and **1** green Pod: about one request in five reaches green. + +When green has proved itself you would scale it up and blue down, then remove blue - that is the whole canary.""", + check=r""" +eq "$(j -n shop deploy web-blue '{.status.availableReplicas}')" 4 && eq "$(j -n shop deploy web-green '{.spec.replicas}')" 1 \ + && eq "$(j -n shop deploy web-green '{.status.availableReplicas}')" 1 \ + && eq "$(j -n shop svc web '{.spec.selector.app}')" web && [ -z "$(j -n shop svc web '{.spec.selector.version}')" ]""", + hint=r"""`kubectl scale deploy NAME --replicas=N`, once for each Deployment.""", + show=r"""kubectl -n shop scale deploy web-blue --replicas=4 +kubectl -n shop scale deploy web-green --replicas=1 +kubectl -n shop rollout status deploy web-blue +kubectl -n shop get deploy"""), + ], + "outro": r"""You did a blue/green cutover, an instant rollback and a weighted canary with nothing but Deployments, labels and one Service. + +**For the exam** + +- The Service's selector is the switch. `kubectl get svc NAME -o wide` shows it; `kubectl get endpointslices -l kubernetes.io/service-name=NAME` shows who receives traffic. +- `kubectl set selector` replaces the whole selector; `kubectl patch` merges into it. +- Keep blue running until a task says otherwise: it is your rollback. +- A canary's share is its Pods divided by all the Pods behind the Service. + +Next, prove it against the clock: **drill 2.1** gives you this objective's tasks with a timer, and no solution until a task is graded.""", +} + +# ---- 2.2 rolling updates and rollbacks ------------------------------------------------------- +L22 = { + "id": 9202, "objective": "2.2", "workers": 0, + "title": "Rolling updates, a bad release, and the way back", + "intro": r"""A Deployment never edits a running Pod. Change its Pod template - a new image, a new environment variable - and it creates a new ReplicaSet, then moves Pods over from the old one a few at a time. Two numbers decide how: + +``` +maxSurge extra Pods during an update +maxUnavailable Pods that may be missing +``` + +Each template change is a **revision**, and you can list revisions and go back to one. + +In Namespace `rollout`, Deployment `api` runs `nginx:1.25` with 4 replicas.""", + "setup": NS + r""" +mkns rollout +$K -n rollout create deploy api --image=nginx:1.25 --replicas=4 >/dev/null +$K -n rollout rollout status deploy api --timeout=180s >/dev/null 2>&1 +mkdir -p "$WS/lesson-2.2" +""", + "steps": [ + step("Read the update strategy", + r"""Find the Deployment's `maxSurge` and `maxUnavailable`, and write them separated by a space to `{WS}/lesson-2.2/strategy.txt` - for example `25% 25%`.""", + check=r"""eq "$(tr -d '\r\n' < "$WS/lesson-2.2/strategy.txt" 2>/dev/null | tr -s ' ')" "25% 25%" """, + hint=r"""Both live under `.spec.strategy.rollingUpdate`. `kubectl explain deploy.spec.strategy.rollingUpdate` describes them, and `-o jsonpath='{...}'` prints only the values.""", + show=r"""kubectl -n rollout get deploy api -o jsonpath='{.spec.strategy.rollingUpdate.maxSurge} {.spec.strategy.rollingUpdate.maxUnavailable}' > {WS}/lesson-2.2/strategy.txt +cat {WS}/lesson-2.2/strategy.txt"""), + step("Never drop below four Pods", + r"""25% of 4 Pods means one Pod may be missing during an update. Make updates add a Pod before they remove one: set `maxUnavailable` to `0` and `maxSurge` to `1`.""", + check=r"""eq "$(j -n rollout deploy api '{.spec.strategy.rollingUpdate.maxUnavailable}')" 0 && eq "$(j -n rollout deploy api '{.spec.strategy.rollingUpdate.maxSurge}')" 1""", + hint=r"""A merge patch with `kubectl patch deploy api -p '{"spec":{"strategy":{...}}}'`, or `kubectl edit deploy api`.""", + show=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' +kubectl -n rollout get deploy api -o jsonpath='{.spec.strategy}'; echo"""), + step("Roll out nginx 1.27, with a reason", + r"""Update the container `nginx` to image `nginx:1.27`, and record why with the annotation `kubernetes.io/change-cause` set to `nginx 1.27`: `kubectl rollout history` shows it in the CHANGE-CAUSE column. Follow the update with `kubectl rollout status` until it finishes.""", + check=r""" +eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 \ + && eq "$(j -n rollout deploy api '{.metadata.annotations.kubernetes\.io/change-cause}')" "nginx 1.27" \ + && eq "$(j -n rollout deploy api '{.status.updatedReplicas}')" 4 && eq "$(j -n rollout deploy api '{.status.availableReplicas}')" 4 \ + && eq "$(j -n rollout deploy api '{.status.replicas}')" 4""", + hint=r"""`kubectl set image deploy/api nginx=nginx:1.27` - the name before `=` is the **container's**. Then `kubectl annotate deploy/api kubernetes.io/change-cause='nginx 1.27'`.""", + show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27 +kubectl -n rollout annotate deploy/api kubernetes.io/change-cause='nginx 1.27' --overwrite +kubectl -n rollout rollout status deploy/api +kubectl -n rollout rollout history deploy/api"""), + step("Ship a bad release", + r"""Now a mistake: update to image `nginx:9.99-nope`, which does not exist. Run `kubectl rollout status` with `--timeout=20s` and look at the Pods: one new Pod cannot pull its image, and because of `maxUnavailable: 0` all four old Pods keep serving. + +A rollout that cannot finish does not take your app down - it stalls.""", + check=r""" +eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:9.99-nope || exit 1 +eq "$(j -n rollout deploy api '{.status.availableReplicas}')" 4 || exit 1 +for i in $(seq 1 20); do + $K -n rollout get pods -o jsonpath='{range .items[*]}{.status.containerStatuses[0].state.waiting.reason}{"\n"}{end}' 2>/dev/null \ + | grep -Eq 'ErrImagePull|ImagePullBackOff' && exit 0 + sleep 3 +done; exit 1""", + hint=r"""`kubectl set image` once more. `kubectl get pods` shows `ErrImagePull` or `ImagePullBackOff`, and `kubectl describe pod NAME` says why.""", + show=r"""kubectl -n rollout set image deploy/api nginx=nginx:9.99-nope +kubectl -n rollout rollout status deploy/api --timeout=20s +kubectl -n rollout get pods"""), + step("Roll back", + r"""Undo the bad release: `kubectl rollout undo` returns to the previous revision, `nginx:1.27`, and the stuck Pod goes away. Wait until the rollout is complete.""", + check=r""" +eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 \ + && eq "$(j -n rollout deploy api '{.status.updatedReplicas}')" 4 && eq "$(j -n rollout deploy api '{.status.availableReplicas}')" 4 \ + && eq "$(j -n rollout deploy api '{.status.replicas}')" 4""", + hint=r"""`kubectl rollout undo deploy/api` goes back one revision; `--to-revision=N` goes back to any one that `kubectl rollout history` lists.""", + show=r"""kubectl -n rollout rollout undo deploy/api +kubectl -n rollout rollout status deploy/api"""), + step("Read the history right", + r"""List the revisions with `kubectl rollout history deploy/api`. The undo did not go back to revision 2: it made the 1.27 template the **newest** revision, and revision 2 is gone from the list. + +Write the current revision number to `{WS}/lesson-2.2/revision.txt`.""", + check=r""" +cur=$(j -n rollout deploy api '{.metadata.annotations.deployment\.kubernetes\.io/revision}') +[ -n "$cur" ] && eq "$(tr -dc '0-9' < "$WS/lesson-2.2/revision.txt" 2>/dev/null)" "$cur" """, + hint=r"""The last line of `kubectl rollout history` - or the Deployment's annotation `deployment.kubernetes.io/revision`.""", + show=r"""kubectl -n rollout rollout history deploy/api +kubectl -n rollout get deploy api -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}' > {WS}/lesson-2.2/revision.txt +cat {WS}/lesson-2.2/revision.txt"""), + ], + "outro": r"""You read and tuned an update strategy, rolled out a new image with a recorded reason, watched a broken release stall without an outage, and rolled it back. + +**For the exam** + +- `kubectl set image deploy/NAME CONTAINER=IMAGE`: the **container** name goes before `=`. +- `kubectl rollout status` waits, `rollout history` lists revisions, `rollout undo [--to-revision=N]` goes back, and `rollout pause` / `resume` gather several changes into one rollout. +- `maxUnavailable: 0` with a `maxSurge` keeps full capacity during an update, and a broken image only stalls it. +- The `kubernetes.io/change-cause` annotation is what `rollout history` shows as CHANGE-CAUSE (`--record` no longer exists). + +Next, prove it against the clock: **drill 2.2**.""", +} + +# ---- 2.3 Helm ----------------------------------------------------------------------------------- +L23 = { + "id": 9203, "objective": "2.3", "workers": 0, + "title": "Helm: install, upgrade, roll back", + "intro": r"""Helm installs an application from a **chart**: templates plus default values. One installation is a **release**, and every change to a release - an upgrade or a rollback - is a new **revision** that you can list and return to. + +``` +chart ── helm install shop ──▶ revision 1 + ── helm upgrade shop ──▶ revision 2, 3 .. +``` + +In the exam a chart usually comes from a repository (`helm repo add`, `helm search repo`). Here it is local, which works the same: a chart directory or a packaged `.tgz` can go anywhere a chart name can. In `{WS}/lesson-2.3` you will find a chart `webapp` 0.1.0 as a directory, `chart/`, and version 0.2.0 packaged as `webapp-0.2.0.tgz`.""", + "setup": r""" +d="$WS/lesson-2.3"; rm -rf "$d"; mkdir -p "$d" +src=$(mktemp -d) +for v in 0.1.0 0.2.0; do + c="$src/$v/webapp"; mkdir -p "$c/templates" + cat > "$c/Chart.yaml" < "$c/values.yaml" <<'EOF' +replicaCount: 1 +image: + repository: nginx + tag: "1.27" +service: + type: ClusterIP + port: 80 +EOF + cat > "$c/templates/deployment.yaml" <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ .Release.Name }}-webapp + labels: {app.kubernetes.io/instance: {{ .Release.Name }}, helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: {app.kubernetes.io/instance: {{ .Release.Name }}} + template: + metadata: + labels: {app.kubernetes.io/instance: {{ .Release.Name }}} + spec: + containers: + - name: web + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + ports: + - containerPort: 80 +EOF + cat > "$c/templates/service.yaml" <<'EOF' +apiVersion: v1 +kind: Service +metadata: + name: {{ .Release.Name }}-webapp +spec: + type: {{ .Values.service.type }} + selector: {app.kubernetes.io/instance: {{ .Release.Name }}} + ports: + - port: {{ .Values.service.port }} + targetPort: 80 +EOF +done +cp -r "$src/0.1.0/webapp" "$d/chart" +helm package "$src/0.2.0/webapp" -d "$d" >/dev/null +rm -rf "$src" +command -v helm >/dev/null || echo "warn: helm is not installed on this machine - the checks need it" +""", + "steps": [ + step("Look inside before you install", + r"""See what a chart would do without touching the cluster. `helm show values` lists its knobs and their defaults; `helm template` renders the manifests Helm would apply. + +Render the chart in `{WS}/lesson-2.3/chart` as release `shop` into `{WS}/lesson-2.3/rendered.yaml`, and read it.""", + check=r"""f="$WS/lesson-2.3/rendered.yaml"; grep -q 'kind: Deployment' "$f" 2>/dev/null && grep -q 'kind: Service' "$f" && grep -q 'name: shop-webapp' "$f" """, + hint=r"""`helm template RELEASE CHART` prints to stdout: redirect it with `> FILE`.""", + show=r"""helm show values {WS}/lesson-2.3/chart +helm template shop {WS}/lesson-2.3/chart > {WS}/lesson-2.3/rendered.yaml +cat {WS}/lesson-2.3/rendered.yaml"""), + step("Install it", + r"""Install the chart as release `shop` into Namespace `charts` - let Helm create the Namespace - with **2** replicas instead of the default 1.""", + check=r""" +helm -n charts status shop -o json 2>/dev/null | python3 -c 'import sys,json; sys.exit(0 if json.load(sys.stdin)["info"]["status"]=="deployed" else 1)' || exit 1 +eq "$(j -n charts deploy shop-webapp '{.status.availableReplicas}')" 2""", + hint=r"""`helm install RELEASE CHART -n NAMESPACE --create-namespace --set replicaCount=2`. `helm list -n charts` shows the release.""", + show=r"""helm -n charts install shop {WS}/lesson-2.3/chart --create-namespace --set replicaCount=2 +helm -n charts list +kubectl -n charts rollout status deploy shop-webapp"""), + step("Upgrade with a values file", + r"""`--set` is fine for one value; for several, a values file. Write `{WS}/lesson-2.3/prod-values.yaml` with `replicaCount: 3` and a Service of type `NodePort`, and upgrade `shop` with it. + +`helm history` then lists a second revision.""", + check=r""" +grep -Eq '^replicaCount: *3' "$WS/lesson-2.3/prod-values.yaml" 2>/dev/null || exit 1 +eq "$(j -n charts deploy shop-webapp '{.status.availableReplicas}')" 3 && eq "$(j -n charts svc shop-webapp '{.spec.type}')" NodePort || exit 1 +[ "$(helm -n charts history shop -o json 2>/dev/null | python3 -c 'import sys,json; print(max(r["revision"] for r in json.load(sys.stdin)))')" -ge 2 ]""", + hint=r"""A values file has the shape of `values.yaml`: `service:` on one line, `type: NodePort` indented under it. Then `helm upgrade shop CHART -n charts -f FILE`.""", + show=r"""cat > {WS}/lesson-2.3/prod-values.yaml <<'EOF' +replicaCount: 3 +service: + type: NodePort +EOF +helm -n charts upgrade shop {WS}/lesson-2.3/chart -f {WS}/lesson-2.3/prod-values.yaml +kubectl -n charts rollout status deploy shop-webapp +helm -n charts history shop"""), + step("A new chart version, with your values", + r"""Upgrade `shop` to chart version 0.2.0, `{WS}/lesson-2.3/webapp-0.2.0.tgz`, **keeping the values you set**. + +Careful: `helm upgrade` without values starts from the chart's defaults again, so your 3 replicas and the NodePort would be gone. `--reuse-values` keeps them (or pass `-f prod-values.yaml` once more).""", + check=r""" +helm -n charts list -o json 2>/dev/null | python3 -c 'import sys,json; r=[x for x in json.load(sys.stdin) if x["name"]=="shop"]; sys.exit(0 if r and r[0]["chart"]=="webapp-0.2.0" else 1)' || exit 1 +eq "$(j -n charts deploy shop-webapp '{.spec.replicas}')" 3 && eq "$(j -n charts svc shop-webapp '{.spec.type}')" NodePort""", + hint=r"""`helm upgrade shop {WS}/lesson-2.3/webapp-0.2.0.tgz -n charts --reuse-values`. `helm list -n charts` shows the chart version, `helm get values shop -n charts` the values in use.""", + show=r"""helm -n charts upgrade shop {WS}/lesson-2.3/webapp-0.2.0.tgz --reuse-values +helm -n charts list +helm -n charts get values shop"""), + step("Roll back", + r"""Go back to revision **2**: chart 0.1.0 with your production values. Look at `helm history` before and after - a rollback is a new revision too, it does not rewind the counter.""", + check=r""" +helm -n charts history shop -o json 2>/dev/null | python3 -c ' +import sys, json +h = json.load(sys.stdin); last = max(h, key=lambda r: r["revision"]) +sys.exit(0 if last["revision"] >= 4 and last["chart"] == "webapp-0.1.0" and last["status"] == "deployed" else 1)' || exit 1 +eq "$(j -n charts deploy shop-webapp '{.spec.replicas}')" 3""", + hint=r"""`helm history shop -n charts`, then `helm rollback shop 2 -n charts`.""", + show=r"""helm -n charts history shop +helm -n charts rollback shop 2 +helm -n charts history shop"""), + step("Clean up", + r"""Uninstall release `shop`. Helm deletes everything the release created - the Deployment and the Service - and its history too, unless you add `--keep-history`. The Namespace stays.""", + check=r"""! helm -n charts status shop >/dev/null 2>&1 && [ -z "$(j -n charts deploy shop-webapp '{.metadata.name}')" ] && [ -z "$(j -n charts svc shop-webapp '{.metadata.name}')" ]""", + hint=r"""`helm uninstall shop -n charts`. `helm list -a -n charts` would still show a release uninstalled with `--keep-history`.""", + show=r"""helm -n charts uninstall shop +kubectl -n charts get all"""), + ], + "outro": r"""You rendered a chart, installed it, upgraded it with a values file and to a new chart version, rolled it back, and removed it. + +**For the exam** + +- `helm repo add NAME URL`, `helm repo update` and `helm search repo` find charts; a directory or a `.tgz` works wherever a chart name does. +- `helm show values` and `helm template` let you look before you install; `helm get values RELEASE` shows what a release runs with. +- `--set` for one value, `-f values.yaml` for several. On `upgrade`, values you do not pass go back to the chart's defaults unless you add `--reuse-values`. +- `helm history` and `helm rollback RELEASE REVISION`: every change, a rollback included, is a new revision. +- Always give the Namespace: `-n NAMESPACE`, and `--create-namespace` on install. + +Next, prove it against the clock: **drill 2.3**.""", +} + +# ---- 2.4 Kustomize ------------------------------------------------------------------------------ +_RENDER = r""" +# rendered NAME: the overlay's objects as JSON, one per line, checked without applying anything +rendered() { + $K kustomize "$WS/lesson-2.4/overlays/$1" 2>/dev/null | $K create --dry-run=client -f - -o json 2>/dev/null | python3 -c ' +import sys, json +t, dec, i = sys.stdin.read(), json.JSONDecoder(), 0 +while True: # one List, or one JSON document per object - kubectl has printed both + while i < len(t) and t[i].isspace(): i += 1 + if i >= len(t): break + o, i = dec.raw_decode(t, i) + for x in (o.get("items", []) if o.get("kind") == "List" else [o]): print(json.dumps(x))' +} +""" + +L24 = { + "id": 9204, "objective": "2.4", "workers": 0, + "title": "Kustomize: one base, two environments", + "intro": r"""Kustomize builds plain YAML from a directory that holds a `kustomization.yaml`. There are no templates and no variables: a **base** holds the manifests, and an **overlay** points at the base and changes it - Namespace, names, labels, images, replicas, or anything else through a patch - without editing it. + +``` +base/ web: Deployment + Service +overlays/staging/ staging-web in staging +overlays/prod/ prod-web in prod, 3 x 1.27 +``` + +`kubectl` has Kustomize built in: `kubectl kustomize DIR` renders, `kubectl apply -k DIR` applies. In `{WS}/lesson-2.4/base` you will find a Deployment `web` (`nginx:1.25`, 1 replica), a Service `web` and their `kustomization.yaml`. The Namespaces `staging` and `prod` exist already.""", + "setup": NS + r""" +mkns staging; mkns prod +d="$WS/lesson-2.4"; rm -rf "$d"; mkdir -p "$d/base" +cat > "$d/base/deployment.yaml" <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: web + labels: {app: web} +spec: + replicas: 1 + selector: + matchLabels: {app: web} + template: + metadata: + labels: {app: web} + spec: + containers: + - name: web + image: nginx:1.25 + ports: + - containerPort: 80 +EOF +cat > "$d/base/service.yaml" <<'EOF' +apiVersion: v1 +kind: Service +metadata: + name: web +spec: + selector: {app: web} + ports: + - port: 80 + targetPort: 80 +EOF +cat > "$d/base/kustomization.yaml" <<'EOF' +resources: +- deployment.yaml +- service.yaml +EOF +( cd "$d/base" && sha256sum deployment.yaml service.yaml kustomization.yaml ) > "$PRIV/lesson-2.4-base.sum" +""", + "steps": [ + step("Render the base", + r"""See what Kustomize makes of the base before you change anything: render `{WS}/lesson-2.4/base` into `{WS}/lesson-2.4/base.yaml`, and read it.""", + check=r"""f="$WS/lesson-2.4/base.yaml"; grep -q 'kind: Deployment' "$f" 2>/dev/null && grep -q 'kind: Service' "$f" && grep -q 'image: nginx:1.25' "$f" """, + hint=r"""`kubectl kustomize DIR` prints the result - the same as `kustomize build DIR`. Redirect it with `> FILE`.""", + show=r"""kubectl kustomize {WS}/lesson-2.4/base > {WS}/lesson-2.4/base.yaml +cat {WS}/lesson-2.4/base.yaml"""), + step("An overlay for staging", + r"""Create `{WS}/lesson-2.4/overlays/staging/kustomization.yaml`. It uses the base `../../base`, puts everything into Namespace `staging`, prefixes every name with `staging-`, and gives every resource the label `env: staging`. + +For the label, use `labels:` with `includeSelectors: true`, which adds it to the selectors too. The older `commonLabels` does the same but is deprecated. Render the overlay to check it.""", + check=_RENDER + r""" +rendered staging | python3 -c ' +import sys, json +items = [json.loads(line) for line in sys.stdin if line.strip()] +by = {(o["kind"], o["metadata"]["name"]): o for o in items} +dep, svc = by.get(("Deployment", "staging-web")), by.get(("Service", "staging-web")) +ok = bool(dep and svc) and all(o["metadata"].get("namespace") == "staging" and o["metadata"].get("labels", {}).get("env") == "staging" for o in (dep, svc)) +sys.exit(0 if ok and dep["spec"]["template"]["metadata"]["labels"].get("env") == "staging" else 1)'""", + hint=r"""``` +resources: +- ../../base +namespace: staging +namePrefix: staging- +labels: +- pairs: + env: staging + includeSelectors: true +```""", + show=r"""mkdir -p {WS}/lesson-2.4/overlays/staging +cat > {WS}/lesson-2.4/overlays/staging/kustomization.yaml <<'EOF' +resources: +- ../../base +namespace: staging +namePrefix: staging- +labels: +- pairs: + env: staging + includeSelectors: true +EOF +kubectl kustomize {WS}/lesson-2.4/overlays/staging"""), + step("Apply staging", + r"""Apply the staging overlay with kubectl's built-in Kustomize, then look at what you got.""", + check=r""" +eq "$(j -n staging deploy staging-web '{.status.availableReplicas}')" 1 && eq "$(j -n staging deploy staging-web '{.metadata.labels.env}')" staging \ + && [ -n "$(j -n staging svc staging-web '{.metadata.name}')" ]""", + hint=r"""`kubectl apply -k DIR` - `-k` instead of `-f`.""", + show=r"""kubectl apply -k {WS}/lesson-2.4/overlays/staging +kubectl -n staging get deploy,svc --show-labels"""), + step("A prod overlay: replicas and image", + r"""Production differs in more than its name. Create the overlay `{WS}/lesson-2.4/overlays/prod`: the same base, Namespace `prod`, prefix `prod-`, label `env: prod`, **3** replicas and image `nginx:1.27`. + +Use Kustomize's own fields for the last two, `replicas:` and `images:` - do not copy the Deployment. Render it to check.""", + check=_RENDER + r""" +rendered prod | python3 -c ' +import sys, json +items = [json.loads(line) for line in sys.stdin if line.strip()] +by = {(o["kind"], o["metadata"]["name"]): o for o in items} +dep = by.get(("Deployment", "prod-web")) +ok = bool(dep) and dep["metadata"].get("namespace") == "prod" and dep["metadata"].get("labels", {}).get("env") == "prod" +ok = ok and dep["spec"].get("replicas") == 3 and dep["spec"]["template"]["spec"]["containers"][0]["image"] == "nginx:1.27" +sys.exit(0 if ok and ("Service", "prod-web") in by else 1)'""", + hint=r"""Besides `resources`, `namespace`, `namePrefix` and `labels`: + +``` +replicas: +- name: web + count: 3 +images: +- name: nginx + newTag: "1.27" +``` + +`replicas` names the Deployment by its base name; `images` names the image, not the container.""", + show=r"""mkdir -p {WS}/lesson-2.4/overlays/prod +cat > {WS}/lesson-2.4/overlays/prod/kustomization.yaml <<'EOF' +resources: +- ../../base +namespace: prod +namePrefix: prod- +labels: +- pairs: + env: prod + includeSelectors: true +replicas: +- name: web + count: 3 +images: +- name: nginx + newTag: "1.27" +EOF +kubectl kustomize {WS}/lesson-2.4/overlays/prod"""), + step("Patch what the fields cannot reach", + r"""For anything else there is a patch. Give the `web` container in prod a memory limit of `128Mi` with a patch file in the prod overlay, listed under `patches:` - then apply the prod overlay.""", + check=r""" +eq "$(j -n prod deploy prod-web '{.status.availableReplicas}')" 3 \ + && eq "$(j -n prod deploy prod-web '{.spec.template.spec.containers[0].image}')" nginx:1.27 \ + && eq "$(j -n prod deploy prod-web '{.spec.template.spec.containers[0].resources.limits.memory}')" 128Mi \ + && eq "$(j -n prod deploy prod-web '{.metadata.labels.env}')" prod""", + hint=r"""A strategic-merge patch looks like a small Deployment that names only what changes - by the **base** name `web`: + +``` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: web +spec: + template: + spec: + containers: + - name: web + resources: + limits: + memory: 128Mi +``` + +and in `kustomization.yaml`: `patches:` with `- path: limits.yaml`.""", + show=r"""cat > {WS}/lesson-2.4/overlays/prod/limits.yaml <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: web +spec: + template: + spec: + containers: + - name: web + resources: + limits: + memory: 128Mi +EOF +cat >> {WS}/lesson-2.4/overlays/prod/kustomization.yaml <<'EOF' +patches: +- path: limits.yaml +EOF +kubectl apply -k {WS}/lesson-2.4/overlays/prod +kubectl -n prod rollout status deploy prod-web"""), + step("One base, two environments", + r"""Look at both environments side by side: `kubectl get deploy -A -l app=web -o wide`. They come from one base that you never edited - the exam checks that too. + +Write the image that **staging** runs to `{WS}/lesson-2.4/staging-image.txt`.""", + check=r""" +( cd "$WS/lesson-2.4/base" && sha256sum -c --quiet "$PRIV/lesson-2.4-base.sum" >/dev/null 2>&1 ) || exit 1 +eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/staging-image.txt" 2>/dev/null)" nginx:1.25 || exit 1 +[ -n "$(j -n staging deploy staging-web '{.metadata.name}')" ] && [ -n "$(j -n prod deploy prod-web '{.metadata.name}')" ]""", + hint=r"""`-o jsonpath='{.spec.template.spec.containers[0].image}'` prints just the image.""", + show=r"""kubectl get deploy -A -l app=web -o wide +kubectl -n staging get deploy staging-web -o jsonpath='{.spec.template.spec.containers[0].image}' > {WS}/lesson-2.4/staging-image.txt +cat {WS}/lesson-2.4/staging-image.txt"""), + ], + "outro": r"""You rendered a base, built two overlays on it with Kustomize's own fields and a patch, and applied both - and the base stayed as it was. + +**For the exam** + +- `kubectl kustomize DIR` renders, `kubectl apply -k DIR` applies, `kubectl delete -k DIR` removes. +- An overlay lists its base under `resources:` and never edits it. +- Built-in fields first - `namespace`, `namePrefix` / `nameSuffix`, `labels`, `images`, `replicas` - and a patch for anything else. A patch names objects by their **base** names. +- `configMapGenerator` and `secretGenerator` make ConfigMaps and Secrets whose names change with their content, so a Deployment that uses one rolls when the data changes - drill 2.4 has one. + +Next, prove it against the clock: **drill 2.4**.""", +} + +LESSONS = [L21, L22, L23, L24] diff --git a/cks/bank_harden.py b/cks/bank_harden.py index b158850..689f95d 100644 --- a/cks/bank_harden.py +++ b/cks/bank_harden.py @@ -41,9 +41,12 @@ print(json.dumps(p) if isinstance(p, (list, dict)) else p)' "$1" 2>/dev/null; } # older than the node's for the three packages, held as kubeadm's documentation asks. APT = r""" K8S_LIST='-o Dir::Etc::sourcelist=/etc/apt/sources.list.d/kubernetes.list -o Dir::Etc::sourceparts=- -o APT::Get::List-Cleanup=0' -# hold_back NODE VERSION: put NODE's kubelet, kubeadm and kubectl on VERSION and restart its kubelet +# hold_back NODE VERSION: put NODE's kubelet, kubeadm and kubectl on VERSION and restart its kubelet. +# pkgs.k8s.io has one repository per minor, so the node's repository line moves to VERSION's minor +# first, and stays there - as on a node installed with that minor. hold_back() { - nx "$1" "apt-get update -qq $K8S_LIST >/dev/null 2>&1 + nx "$1" "sed -i 's#/core:/stable:/v[0-9.]*/deb/#/core:/stable:/v${2%.*}/deb/#' /etc/apt/sources.list.d/kubernetes.list +apt-get update -qq $K8S_LIST >/dev/null 2>&1 DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --allow-downgrades --allow-change-held-packages -o Dpkg::Options::=--force-confold kubelet=$2-1.1 kubeadm=$2-1.1 kubectl=$2-1.1 >/dev/null 2>&1 || exit 1 apt-mark hold kubelet kubeadm kubectl >/dev/null; systemctl daemon-reload; systemctl restart kubelet" "Upgrade worker nodes". kubeadm first, because it rewrites -# the kubelet's configuration for the new version; the kubelet package last, then the restart - -# `k get nodes` shows the version of the running kubelet, not of the package. +# docs: "Upgrading kubeadm clusters" -> "Changing the package repository", then "Upgrade worker +# nodes". {W1} is a minor behind, and pkgs.k8s.io serves one repository per minor: until the line +# names v1.37, apt offers nothing newer than 1.36. kubeadm first, because it rewrites the kubelet's +# configuration for the new version; the kubelet package last, then the restart - `k get nodes` +# shows the version of the running kubelet, not of the package. """, }, { "id": 38, "domain": "HARDEN", "title": "Upgrade the control plane", "workers": 0, "difficulty": "hard", "task": """ -A vulnerability in the API server was fixed in a newer patch release of Kubernetes v1.35. +A vulnerability in the API server was fixed in a newer patch release of Kubernetes v1.37. -Upgrade this cluster — a single control-plane node, `{CP}` — to the **newest v1.35 patch release** +Upgrade this cluster — a single control-plane node, `{CP}` — to the **newest v1.37 patch release** that the node's package repository offers: 1. Upgrade the `kubeadm` package, look at `kubeadm upgrade plan`, and apply the upgrade. @@ -790,7 +797,7 @@ t=$(cat "$PRIV/q38.target"); cur=$(pkgs "$CP" | cut -d' ' -f1) [ "$t" != "$cur" ] || { echo "the repository has nothing newer than $cur: there is no upgrade to do" >&2; exit 1; } """, "checks": [ - C("the API server runs the newest v1.35 patch release", 4, API + r""" + C("the API server runs the newest v1.37 patch release", 4, API + r""" t=$(cat "$PRIV/q38.target"); wait_api 60 || exit 1 eq "$($K version -o json 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["serverVersion"]["gitVersion"])')" "v${t%%-*}" """), C("the node's kubelet does too, and the node is Ready and schedulable", 3, API + r""" @@ -806,13 +813,13 @@ j -n kube-system pod "kube-controller-manager-$CP" '{.spec.containers[0].image}' ], "solution": """ sudo -i -apt-get update && apt-cache madison kubeadm | head -3 # newest first, say 1.35.X-1.1 -apt-mark unhold kubeadm && apt-get install -y kubeadm=1.35.X-1.1 && apt-mark hold kubeadm +apt-get update && apt-cache madison kubeadm | head -3 # newest first, say 1.37.X-1.1 +apt-mark unhold kubeadm && apt-get install -y kubeadm=1.37.X-1.1 && apt-mark hold kubeadm kubeadm version -o short kubeadm upgrade plan # what would change, and to which version -kubeadm upgrade apply v1.35.X # a few minutes; answers y +kubeadm upgrade apply v1.37.X # a few minutes; answers y kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data -apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.35.X-1.1 kubectl=1.35.X-1.1 && apt-mark hold kubelet kubectl +apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.37.X-1.1 kubectl=1.37.X-1.1 && apt-mark hold kubelet kubectl systemctl daemon-reload && systemctl restart kubelet kubectl uncordon {CP} kubectl get nodes; kubectl version @@ -824,7 +831,7 @@ kubectl get nodes; kubectl version }, { "id": 39, "domain": "HARDEN", "title": "Prepare an upgrade", "workers": 2, - "params": lambda r: {"old": r.choice(["1.35.4", "1.35.5", "1.35.6"]), "n": r.choice([1, 2])}, + "params": lambda r: {"old": r.choice(["1.36.3", "1.36.4", "1.36.5"]), "n": r.choice([1, 2])}, "task": """ A CVE affects every kubelet older than the version this cluster's control plane runs. Prepare the upgrade — do **not** upgrade anything yet. @@ -833,8 +840,8 @@ upgrade — do **not** upgrade anything yet. `{WS}/q39/outdated.txt`, one per line. 2. Cordon those nodes, so that no new Pod lands on a vulnerable kubelet. Do not drain them, and leave the other nodes schedulable. -3. Write the newest v1.35 package version of `kubeadm` that the repository on `{CP}` offers - (as apt prints it, for example `1.35.2-1.1`) to `{WS}/q39/target.txt`. +3. Write the newest v1.37 package version of `kubeadm` that the repository on `{CP}` offers + (as apt prints it, for example `1.37.2-1.1`) to `{WS}/q39/target.txt`. 4. Save the output of `kubeadm upgrade plan` on `{CP}` to `{WS}/q39/plan.txt`. """, "setup": API + APT + r""" @@ -867,7 +874,7 @@ sudo kubeadm upgrade plan | tee {WS}/q39/plan.txt # cordon only marks the node unschedulable: running Pods stay. drain = cordon + evict. # `upgrade plan` reads the cluster and changes nothing: it lists the current and the target version # of each component and which kubelets it found. The kubelet may be up to three minor versions -# behind the API server, never ahead - but an old patch release still carries its bugs. +# behind the API server, never ahead - but an old release still carries its bugs. """, }, ] diff --git a/cks/bank_runtime.py b/cks/bank_runtime.py index 645d8ed..6bc569e 100644 --- a/cks/bank_runtime.py +++ b/cks/bank_runtime.py @@ -327,7 +327,7 @@ t = datetime.datetime(2026, 9, 14, 2, 0, 0) ACTOR, IP = "system:serviceaccount:%{ns}:%{sa}", "%{ip}" out = [] -def ev(user, groups, ip, verb, res, ns, name, code, sub=None, group="", agent="kubectl/v1.35.8"): +def ev(user, groups, ip, verb, res, ns, name, code, sub=None, group="", agent="kubectl/v1.37.0"): global t t += datetime.timedelta(seconds=r.randint(1, 9), microseconds=r.randint(0, 999999)) uri = ("/api/v1" if not group else "/apis/" + group + "/v1") + ("/namespaces/" + ns if ns else "") + "/" + res + ("/" + name if name else "") + ("/" + sub if sub else "") diff --git a/cks/curriculum.py b/cks/curriculum.py index aecc815..9e69c33 100644 --- a/cks/curriculum.py +++ b/cks/curriculum.py @@ -245,15 +245,15 @@ k config set-credentials jane --client-certificate=jane.crt --client-key=jane.ke ], "fast": """ kubeadm upgrade plan -apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm='1.35.x-*' && apt-mark hold kubeadm -kubeadm upgrade apply v1.35.x # workers: kubeadm upgrade node +apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm='1.37.x-*' && apt-mark hold kubeadm +kubeadm upgrade apply v1.37.x # workers: kubeadm upgrade node k drain NODE --ignore-daemonsets --delete-emptydir-data -apt-mark unhold kubelet kubectl && apt-get install -y kubelet='1.35.x-*' kubectl='1.35.x-*' && apt-mark hold kubelet kubectl +apt-mark unhold kubelet kubectl && apt-get install -y kubelet='1.37.x-*' kubectl='1.37.x-*' && apt-mark hold kubelet kubectl systemctl daemon-reload && systemctl restart kubelet; k uncordon NODE k get nodes; kubelet --version """, "traps": [ - "The package repository is per minor version (pkgs.k8s.io/core:/stable:/v1.35): a new minor needs the repository line changed first.", + "The package repository is per minor version (pkgs.k8s.io/core:/stable:/v1.37): a new minor needs the repository line changed first.", "`kubectl get nodes` shows the kubelet's version, so a node looks old until its kubelet is restarted.", "Forgetting to uncordon leaves the node unschedulable.", ], diff --git a/cks/questions.py b/cks/questions.py index 39a7493..d769497 100644 --- a/cks/questions.py +++ b/cks/questions.py @@ -15,10 +15,12 @@ TRACK = { "name": "Certified Kubernetes Security Specialist", "minutes": 120, "pass_pct": 67, - # Kubernetes minor of the real exam environment, which picks the kind cluster (shared with CKA). - # LF "Important Instructions: CKS", checked 2026-10-02: "The CKS environment is currently running - # Kubernetes v1.35". (cncf/curriculum still names the PDF "CKS_Curriculum v1.34".) - "k8s": "v1.35", + # Kubernetes minor of the cluster, which picks the kind node image (shared with CKA). Ahead of + # the real exam on purpose: LF "Important Instructions: CKS", checked 2026-10-19, still says "The + # CKS environment is currently running Kubernetes v1.35", but CKA and CKAD are on v1.37 and + # CKS is expected to follow by the time it is taken. (cncf/curriculum still names the PDF + # "CKS_Curriculum v1.34".) + "k8s": "v1.37", # Worker nodes of a task's cluster unless the task says otherwise ("workers", or WORKERS below) "workers": 1, # CKS hosts are the common exam host plus the scanners, gVisor, istioctl and Kubernetes as apt @@ -239,8 +241,9 @@ $K -n %{ns} create deploy %{dep} --image=nginx:1.27 >/dev/null C("%{dep} uses %{dep}-sa, which does mount a token", 3, r""" eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.serviceAccountName}')" %{dep}-sa \ && [ "$(j -n %{ns} sa %{dep}-sa '{.automountServiceAccountToken}')" != false ]"""), - C("%{dep} pods have a projected token mounted", 3, r""" -p=$(j -n %{ns} pod -l app=%{dep} '{.items[0].metadata.name}'); [ -n "$p" ] || exit 1 + C("%{dep} pods run as %{dep}-sa, with a projected token mounted", 3, r""" +# a Pod of the new template: untouched, the Deployment's Pods run as default - with a token +p=$(j -n %{ns} pod -l app=%{dep} '{.items[?(@.spec.serviceAccountName=="%{dep}-sa")].metadata.name}' | awk '{print $1}'); [ -n "$p" ] || exit 1 $K -n %{ns} exec "$p" -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null"""), C("a pod on the default ServiceAccount has no token", 3, r""" cleanup() { $K -n %{ns} delete pod plain --wait=false >/dev/null 2>&1; } diff --git a/cks/solve.py b/cks/solve.py index cdf2daa..f3d650e 100644 --- a/cks/solve.py +++ b/cks/solve.py @@ -482,7 +482,7 @@ kubectl -n %{ns} create rolebinding auditors-view --clusterrole=view --group=aud 37: r""" V=$(kubectl get node {CP} -o jsonpath='{.status.nodeInfo.kubeletVersion}' | tr -d v) kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data --force -ssh -n {W1} "sudo apt-get update -qq; sudo apt-mark unhold kubeadm kubelet kubectl >/dev/null; sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$V-1.1 >/dev/null && sudo kubeadm upgrade node && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$V-1.1 kubectl=$V-1.1 >/dev/null; sudo apt-mark hold kubeadm kubelet kubectl; sudo systemctl daemon-reload; sudo systemctl restart kubelet" +ssh -n {W1} "sudo sed -i 's|/v[0-9.]*/deb/|/v${V%.*}/deb/|' /etc/apt/sources.list.d/kubernetes.list; sudo apt-get update -qq; sudo apt-mark unhold kubeadm kubelet kubectl >/dev/null; sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$V-1.1 >/dev/null && sudo kubeadm upgrade node && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$V-1.1 kubectl=$V-1.1 >/dev/null; sudo apt-mark hold kubeadm kubelet kubectl; sudo systemctl daemon-reload; sudo systemctl restart kubelet" kubectl uncordon {W1} kubectl -n %{ns} rollout status deploy web --timeout=180s sleep 15 diff --git a/images/desktop/Containerfile b/images/desktop/Containerfile index 4ad5bc3..ca3b0cd 100644 --- a/images/desktop/Containerfile +++ b/images/desktop/Containerfile @@ -1,8 +1,20 @@ -# The exam's remote desktop: XFCE with a terminal and Firefox, served to the browser over noVNC. -# It is the "base" machine of a session - no kubeconfig here; every task says which host to ssh to. -FROM docker.io/library/debian:trixie-slim +# The exam's remote desktop: XFCE on Ubuntu 24.04 with a terminal and Firefox, served to the +# browser over noVNC. It is the "base" machine of a session - no kubeconfig here; every task says +# which host to ssh to. +FROM docker.io/library/ubuntu:24.04 ENV DEBIAN_FRONTEND=noninteractive LANG=C.UTF-8 +# Ubuntu's own firefox package only installs the snap, which cannot run in a container: Firefox ESR +# comes from Mozilla's APT repository instead, pinned above Ubuntu's. Ubuntu's image has a user +# `ubuntu` on uid 1000, which is the candidate's. RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates curl \ + && install -d -m 0755 /etc/apt/keyrings \ + && curl -fsSL -o /etc/apt/keyrings/packages.mozilla.org.asc https://packages.mozilla.org/apt/repo-signing-key.gpg \ + && echo 'deb [signed-by=/etc/apt/keyrings/packages.mozilla.org.asc] https://packages.mozilla.org/apt mozilla main' \ + > /etc/apt/sources.list.d/mozilla.list \ + && printf 'Package: *\nPin: origin packages.mozilla.org\nPin-Priority: 1000\n' > /etc/apt/preferences.d/mozilla \ + && userdel -r ubuntu \ + && apt-get update \ && apt-get install -y --no-install-recommends \ xfce4 xfce4-terminal dbus-x11 elementary-xfce-icon-theme greybird-gtk-theme \ fonts-dejavu fonts-liberation \ diff --git a/images/node-cks/Containerfile b/images/node-cks/Containerfile index 6acfcdc..11d1e1c 100644 --- a/images/node-cks/Containerfile +++ b/images/node-cks/Containerfile @@ -1,38 +1,15 @@ # CKS exam host: the common exam host (images/node) plus what the CKS hosts carry - the scanners, -# a sandbox runtime, a service mesh client, and Kubernetes installed as real packages so that an -# upgrade is `apt-get install` and `kubeadm upgrade`, as on the day. Built by `./ksim build cks`. +# a sandbox runtime and a service mesh client. Built by `./ksim build cks`. ARG BASE FROM ${BASE} RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ - python3-minimal bsdextrautils binutils xxd strace file bzip2 gnupg conntrack tcpdump lsof kmod vsftpd \ + python3-minimal binutils strace file bzip2 conntrack tcpdump lsof kmod vsftpd \ && rm -rf /var/lib/apt/lists/* \ # vsftpd is here to be found and removed in a task; it must not run on every host && (systemctl disable vsftpd >/dev/null 2>&1 || true) \ && rm -f /etc/systemd/system/multi-user.target.wants/vsftpd.service -# --- Kubernetes as packages ------------------------------------------------------------------- -# A kind node carries kubelet, kubeadm and kubectl as loose binaries. Installing the same version -# from pkgs.k8s.io puts them under dpkg, so a task can hold a node back on an older patch release -# and the candidate upgrades it the documented way. kind's own kubelet unit and drop-ins live in -# /etc/systemd and keep taking precedence over the package's. -ARG K8S_VERSION -ARG K8S_MINOR -RUN mkdir -p /etc/apt/keyrings \ - && curl -fsSL "https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/Release.key" | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg \ - && echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/ /" > /etc/apt/sources.list.d/kubernetes.list \ - # kubeadm refuses to upgrade a node without the cpuset cgroup controller, which a rootless - # container does not get. The packaged binary is diverted and a wrapper (rootfs/usr/bin/kubeadm) - # passes --ignore-preflight-errors=SystemVerification to `kubeadm upgrade`, so the candidate - # types the same commands as on a real node. The diversion survives package upgrades. - && dpkg-divert --add --rename --divert /usr/bin/kubeadm.real /usr/bin/kubeadm \ - && apt-get update \ - && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends -o Dpkg::Options::=--force-confold \ - "kubelet=${K8S_VERSION}-*" "kubeadm=${K8S_VERSION}-*" "kubectl=${K8S_VERSION}-*" \ - && apt-mark hold kubelet kubeadm kubectl \ - && rm -rf /var/lib/apt/lists/* \ - && test -x /usr/bin/kubeadm.real - # --- CKS tools -------------------------------------------------------------------------------- ARG KUBE_BENCH=0.16.0 ARG TRIVY=0.75.0 @@ -56,9 +33,8 @@ RUN cd /tmp && G=https://github.com \ && chown -R root:root /usr/local/bin /etc/kube-bench COPY rootfs/ / -RUN chmod 755 /usr/bin/kubeadm \ - # gVisor as a second runtime handler ("runsc") beside runc; a RuntimeClass selects it - && printf '\n[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]\n runtime_type = "io.containerd.runsc.v1"\n [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]\n TypeUrl = "io.containerd.runsc.v1.options"\n ConfigPath = "/etc/containerd/runsc.toml"\n' >> /etc/containerd/config.toml \ +# gVisor as a second runtime handler ("runsc") beside runc; a RuntimeClass selects it +RUN printf '\n[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]\n runtime_type = "io.containerd.runsc.v1"\n [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]\n TypeUrl = "io.containerd.runsc.v1.options"\n ConfigPath = "/etc/containerd/runsc.toml"\n' >> /etc/containerd/config.toml \ # trivy reads /etc/trivy.yaml (the Docker Hub cache as a mirror, one cache directory for all # users), for logins and for sudo alike && install -d -m 1777 /var/lib/trivy \ diff --git a/images/node/Containerfile b/images/node/Containerfile index bef2387..d905dda 100644 --- a/images/node/Containerfile +++ b/images/node/Containerfile @@ -1,23 +1,101 @@ -# Exam host: a kind node (kubeadm, kubelet, containerd, static-pod control plane) that you reach -# with `ssh ` as `candidate`, carrying the tools the real exam hosts have. Built once per -# Kubernetes minor by `./ksim build`; every question gets its own cluster from this image. +# Exam host: a kind node (kubeadm, kubelet, containerd, static-pod control plane) on Ubuntu 24.04, +# as the exam's hosts are, that you reach with `ssh ` as `candidate`, carrying the tools the +# real exam hosts have. Built once per Kubernetes minor by `./ksim build`; every question gets its +# own cluster from this image. ARG BASE ARG ETCD_IMAGE FROM ${ETCD_IMAGE} AS etcd +FROM ${BASE} AS kind + +# kind's node image is Debian: kind's images/base/Dockerfile (v0.33.0) on debian:trixie-slim, plus +# Kubernetes. This is the same node on Ubuntu. kind's packages and systemd trimming are done here; +# what does not depend on the distribution is copied from kind's image: its scripts and units, +# containerd, runc, crictl, the CNI plugins and the preloaded images. Kubernetes itself comes from +# pkgs.k8s.io below, as on a node installed with kubeadm - and with it the kubernetes-cni package, +# whose plugins (the same release) take the place of kind's. +FROM docker.io/library/ubuntu:24.04 +# Ubuntu's container image is minimized: no man pages, and `man` a stub. The exam hosts have them +# ("man and man pages for further documentation"), so they come back the way `unminimize` does it; +# docs and translations stay out. Its user `ubuntu` holds uid 1000, which is the candidate's. +RUN printf 'path-exclude=/usr/share/doc/*\npath-include=/usr/share/doc/*/copyright\npath-exclude=/usr/share/locale/*/LC_MESSAGES/*.mo\n' \ + > /etc/dpkg/dpkg.cfg.d/excludes \ + && rm -f /usr/bin/man && dpkg-divert --quiet --remove --rename /usr/bin/man \ + && userdel -r ubuntu \ + && apt-get update \ + && dpkg -S /usr/share/man/ | sed 's|, |\n|g;s|: [^:]*$||' | DEBIAN_FRONTEND=noninteractive xargs apt-get install --reinstall -y -qq \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + systemd \ + conntrack iptables nftables iproute2 ethtool util-linux mount kmod \ + libseccomp2 pigz fuse-overlayfs \ + nfs-common open-iscsi \ + e2fsprogs \ + bash ca-certificates curl jq procps \ + man-db manpages \ + && rm -rf /var/lib/apt/lists/* +COPY --from=kind /kind/ /kind/ +COPY --from=kind /LICENSES/ /LICENSES/ +COPY --from=kind /usr/local/bin/ /usr/local/bin/ +COPY --from=kind /usr/local/sbin/runc /usr/local/sbin/ +COPY --from=kind /opt/cni/ /opt/cni/ +COPY --from=kind /etc/containerd/ /etc/containerd/ +COPY --from=kind /etc/crictl.yaml /etc/ +COPY --from=kind /etc/default/kubelet /etc/default/ +COPY --from=kind /etc/sysctl.d/10-network-magic.conf /etc/sysctl.d/10-network-security.conf /etc/sysctl.d/ +COPY --from=kind /etc/systemd/system/containerd.service /etc/systemd/system/containerd-fuse-overlayfs.service \ + /etc/systemd/system/kubelet.service /etc/systemd/system/kubelet.slice /etc/systemd/system/undo-mount-hacks.service \ + /etc/systemd/system/ +COPY --from=kind /etc/systemd/system/kubelet.service.d/ /etc/systemd/system/kubelet.service.d/ +COPY --from=kind /var/lib/containerd/ /var/lib/containerd/ +RUN find /lib/systemd/system/sysinit.target.wants/ -name "systemd-tmpfiles-setup.service" -delete \ + && rm -f /lib/systemd/system/multi-user.target.wants/* \ + && rm -f /etc/systemd/system/*.wants/* \ + && rm -f /lib/systemd/system/local-fs.target.wants/* \ + && rm -f /lib/systemd/system/sockets.target.wants/*udev* \ + && rm -f /lib/systemd/system/sockets.target.wants/*initctl* \ + && rm -f /lib/systemd/system/basic.target.wants/* \ + && echo "ReadKMsg=no" >> /etc/systemd/journald.conf \ + && ln -sf /usr/lib/systemd/systemd /sbin/init \ + && systemctl enable kubelet.service containerd.service undo-mount-hacks.service \ + && systemctl mask systemd-binfmt.service \ + && mkdir -p /etc/kubernetes/manifests +ENV container=docker +STOPSIGNAL SIGRTMIN+3 +ENTRYPOINT ["/usr/local/bin/entrypoint", "/sbin/init"] -FROM ${BASE} # etcdctl/etcdutl at exactly the version of the cluster's etcd (the etcd image itself is distroless) COPY --from=etcd /usr/local/bin/etcdctl /usr/local/bin/etcdutl /usr/local/bin/ RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ - openssh-server dbus sudo vim nano less jq tmux bash-completion curl wget ca-certificates \ - iputils-ping dnsutils netcat-openbsd psmisc tree openssl git \ + openssh-server dbus sudo vim nano less jq tmux bash-completion curl wget ca-certificates gnupg \ + iputils-ping dnsutils netcat-openbsd psmisc tree openssl git bsdextrautils xxd \ podman netavark aardvark-dns \ && rm -rf /var/lib/apt/lists/* \ - # Debian's podman ships a CNI config. The kubelet would take it for the cluster's pod network + # A podman package may ship a CNI config. The kubelet would take it for the cluster's pod network # until kindnet has written its own, and the first Pods would land on podman's bridge. && rm -f /etc/cni/net.d/87-podman-bridge.conflist \ && touch /etc/containers/nodocker + +# --- Kubernetes as packages ------------------------------------------------------------------- +# kubelet, kubeadm and kubectl of the cluster's version from pkgs.k8s.io, held, with the repository +# of their minor - as on an exam node. An upgrade is then `apt-get install` and `kubeadm upgrade`, +# as on the day, and a task can hold a node back on an older release. kind's kubelet unit and +# drop-ins live in /etc/systemd and keep taking precedence over the package's. +ARG K8S_VERSION +ARG K8S_MINOR +RUN mkdir -p /etc/apt/keyrings \ + && curl -fsSL "https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/Release.key" | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg \ + && echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/ /" > /etc/apt/sources.list.d/kubernetes.list \ + # kubeadm refuses to upgrade a node without the cpuset cgroup controller, which a rootless + # container does not get. The packaged binary is diverted and a wrapper (rootfs/usr/bin/kubeadm) + # passes --ignore-preflight-errors=SystemVerification to `kubeadm upgrade`, so the candidate + # types the same commands as on a real node. The diversion survives package upgrades. + && dpkg-divert --add --rename --divert /usr/bin/kubeadm.real /usr/bin/kubeadm \ + && apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends -o Dpkg::Options::=--force-confold \ + "kubelet=${K8S_VERSION}-*" "kubeadm=${K8S_VERSION}-*" "kubectl=${K8S_VERSION}-*" \ + && apt-mark hold kubelet kubeadm kubectl \ + && rm -rf /var/lib/apt/lists/* \ + && test -x /usr/bin/kubeadm.real ARG HELM_VERSION RUN arch="$(dpkg --print-architecture)" \ && curl -fsSL "https://get.helm.sh/helm-${HELM_VERSION}-linux-${arch}.tar.gz" \ @@ -26,15 +104,18 @@ RUN arch="$(dpkg --print-architecture)" \ && chmod +x /usr/local/bin/yq COPY rootfs/ / # uid 1000, passwordless sudo: the exam gives you `candidate` and expects `sudo -i` for node work -RUN useradd -m -s /bin/bash -u 1000 candidate \ +RUN chmod 755 /usr/bin/kubeadm \ + && useradd -m -s /bin/bash -u 1000 candidate \ && echo 'candidate ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/candidate && chmod 440 /etc/sudoers.d/candidate \ && cp /etc/skel/.vimrc /root/.vimrc \ && echo '. /etc/exam.bashrc' >> /etc/bash.bashrc \ - && systemctl enable ssh \ + # sshd as a service of its own: Ubuntu 24.04 would start it from ssh.socket on the first + # connection, and a host that is up should be listening + && (systemctl disable ssh.socket 2>/dev/null || true) && systemctl enable ssh.service \ # registry mirrors are read from /etc/containerd/certs.d (see docker.io/hosts.toml there) && printf '\n[plugins."io.containerd.grpc.v1.cri".registry]\n config_path = "/etc/containerd/certs.d"\n' >> /etc/containerd/config.toml \ # Installing dbus stamps the image with a D-Bus machine id, and systemd-machine-id-setup (run by # kind's entrypoint on every boot) copies that into /etc/machine-id instead of making a new one: # every node would share one machine id, derive the same eth0 MAC from it, and be unreachable from - # its neighbours. Point it at /etc/machine-id, as a stock Debian does, so each node gets its own. + # its neighbours. Point it at /etc/machine-id, as a stock Ubuntu does, so each node gets its own. && rm -f /var/lib/dbus/machine-id && ln -s /etc/machine-id /var/lib/dbus/machine-id diff --git a/images/node-cks/rootfs/usr/bin/kubeadm b/images/node/rootfs/usr/bin/kubeadm similarity index 82% rename from images/node-cks/rootfs/usr/bin/kubeadm rename to images/node/rootfs/usr/bin/kubeadm index 79470f2..24e349a 100755 --- a/images/node-cks/rootfs/usr/bin/kubeadm +++ b/images/node/rootfs/usr/bin/kubeadm @@ -1,6 +1,6 @@ #!/bin/sh # kubeadm on an exam host. The packaged binary is /usr/bin/kubeadm.real (a dpkg diversion, see -# images/node-cks/Containerfile). `kubeadm upgrade` checks the node first and stops when the cpuset +# images/node/Containerfile). `kubeadm upgrade` checks the node first and stops when the cpuset # cgroup controller is missing - which it always is on a rootless container node, and never on a # real one. This wrapper lets that one check pass and changes nothing else. if [ "$1" = upgrade ]; then diff --git a/images/vm/guest/provision.sh b/images/vm/guest/provision.sh index fd3389f..f2a442d 100644 --- a/images/vm/guest/provision.sh +++ b/images/vm/guest/provision.sh @@ -14,7 +14,7 @@ export DEBIAN_FRONTEND=noninteractive modprobe 9p; modprobe 9pnet_virtio mkdir -p /mnt/build && mount -t 9p -o trans=virtio,version=9p2000.L,ro build /mnt/build -. /mnt/build/env # K8S_VERSION=1.35.8 K8S_MINOR=v1.35 +. /mnt/build/env # K8S_VERSION=1.37.0 K8S_MINOR=v1.37 # --- nothing that works behind the candidate's back systemctl disable --now unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer 2>/dev/null || true diff --git a/ksim b/ksim index a37fe5e..bcfa8da 100755 --- a/ksim +++ b/ksim @@ -22,7 +22,7 @@ from collections.abc import Callable from datetime import datetime from typing import Any -VERSION = "2026.10.15.3" +VERSION = "2026.10.19.1" ROOT = os.path.dirname(os.path.abspath(__file__)) sys.path.insert(0, ROOT) from sim import desktop, envs # noqa: E402 @@ -47,8 +47,9 @@ WEAK_FILE = "" PROGRESS_FILE = "" # study-.json: when each objective was done, your focus, what you marked known OBJECTIVES = [] # the track's curriculum, objective by objective (/curriculum.py), if it has one STUDY_ORDER = [] # domains in the order the path walks them, when that differs from the curriculum's own -K8S = "" # Kubernetes minor the track's real exam runs, e.g. "v1.35": picks the exam host image +K8S = "" # Kubernetes minor the track's real exam runs, e.g. "v1.37": picks the exam host image PENDING = {} # tasks of the bank that are held back, id -> why (PENDING in /questions.py) +LESSONS = [] # guided labs, one per objective (/lessons.py), each run like a task on a host of its own def load_track(track_id): @@ -83,6 +84,12 @@ def load_track(track_id): if e.name != f"{track_id}.curriculum": raise objectives, study_order = [], [] # no objective map for this track: drill by domain only + try: + lessons = [lesson_task(l, objectives) for l in importlib.reload(importlib.import_module(f"{track_id}.lessons")).LESSONS] + except ModuleNotFoundError as e: + if e.name != f"{track_id}.lessons": + raise + lessons = [] for q in questions: q["weight"] = sum(c["pts"] for c in q["checks"]) q.setdefault("difficulty", "normal") @@ -107,10 +114,19 @@ def load_track(track_id): HISTORY_FILE=os.path.join(ROOT, f"history-{track_id}.jsonl"), WEAK_FILE=os.path.join(ROOT, f"weak-spots-{track_id}.md"), PROGRESS_FILE=os.path.join(ROOT, f"study-{track_id}.json"), OBJECTIVES=objectives, STUDY_ORDER=study_order, - K8S=track["k8s"], PENDING=pending, + K8S=track["k8s"], PENDING=pending, LESSONS=lessons, ) +def lesson_task(l, objectives): + """A lesson as the engine runs it: a task-shaped dict, so its host, setup and checks go through + the same code as a task's (host_of, provision, run). Its steps' checks are its checks.""" + o = next((o for o in objectives if o["id"] == l["objective"]), None) + return dict(l, lesson=True, domain=o["domain"] if o else "", difficulty="normal", set=None, + checks=[{"name": s["title"], "pts": 1, "script": s["check"]} for s in l["steps"]], weight=len(l["steps"]), + task=l["intro"], solution="\n\n".join(s["show"] for s in l["steps"] if s["show"])) + + def set_questions(name): return [q for q in QUESTIONS if q.get("set") == name] @@ -128,9 +144,12 @@ PARALLEL = max(1, int(os.environ.get("KSIM_PARALLEL") or max(2, min(8, (os.cpu_c # ---------------------------------------------- a task's own cluster ---- _WORKER_REF = re.compile(r"(?:\$\{?|\{(?:SHELL_)?)W([123])\b") -def node_image(): - """The exam host image of the current track: its Kubernetes minor, in the track's own flavour if it has one.""" - return envs.node_image(K8S, TRACK.get("image")) +def node_image(q=None): + """The exam host image of a task: its track's Kubernetes minor, in the track's own flavour if it + has one - unless the task names a minor ("k8s") or a flavour ("image") of its own, as a task + that upgrades its cluster to the track's minor does.""" + q = q or {} + return envs.node_image(q.get("k8s", K8S), q.get("image", TRACK.get("image"))) def is_vm(q): @@ -142,7 +161,7 @@ def make_host(q): """Create the host of a task: a kind cluster of containers, or a VM. Raises envs.EnvError.""" if is_vm(q): return envs.create_vm(host_of(q), K8S) - return envs.create(host_of(q), node_image(), workers_of(q)) + return envs.create(host_of(q), node_image(q), workers_of(q)) def workers_of(q): @@ -209,7 +228,7 @@ ensure_gateway_crds() { } # ensure_metrics_server -> a working metrics API for `kubectl top`. kind ships none, and kind's # kubelet serving certs have no IP SANs, so it only becomes Ready with --kubelet-insecure-tls. -# Pinned: v0.9.x supports Kubernetes 1.34+, so it serves the v1.35 and v1.37 hosts alike. +# Pinned: v0.9.x supports Kubernetes 1.34+, so it serves the v1.37 hosts. METRICS_SERVER_URL=https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.0/components.yaml ensure_metrics_server() { if ! $K -n kube-system get deploy metrics-server >/dev/null 2>&1; then @@ -221,6 +240,43 @@ ensure_metrics_server() { fi $K -n kube-system rollout status deploy metrics-server --timeout=150s >/dev/null 2>&1 } +# ensure_csi_hostpath -> the CSI hostpath driver with snapshots, and StorageClass csi-hostpath-sc +# (expandable), as the real CKA's clusters have it: what local-path cannot do - expanding a claim, +# VolumeSnapshots, cloning. Pinned to csi-driver-host-path v1.18.0 (its deploy/kubernetes-1.34 +# directory, which kubernetes-1.35 and -latest link to); each sidecar's RBAC at the version of its +# image there, and the snapshot CRDs and controller of external-snapshotter at the snapshotter's. +CSI_GH=https://raw.githubusercontent.com/kubernetes-csi +ensure_csi_hostpath() { + $K get csidriver hostpath.csi.k8s.io >/dev/null 2>&1 && $K get sc csi-hostpath-sc >/dev/null 2>&1 \ + && $K -n default get pod csi-hostpathplugin-0 -o jsonpath='{.status.containerStatuses[*].ready}' 2>/dev/null | grep -qv false && return 0 + local f s=$CSI_GH/external-snapshotter/v8.6.0 + for f in volumesnapshotclasses volumesnapshotcontents volumesnapshots; do + curl -sfL --max-time 60 "$s/client/config/crd/snapshot.storage.k8s.io_$f.yaml" | $K apply -f - >/dev/null 2>&1 || return 1 + done + $K wait --for condition=established --timeout=60s crd/volumesnapshotclasses.snapshot.storage.k8s.io \ + crd/volumesnapshotcontents.snapshot.storage.k8s.io crd/volumesnapshots.snapshot.storage.k8s.io >/dev/null 2>&1 || return 1 + for f in $s/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml $s/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml \ + $CSI_GH/external-provisioner/v6.3.0/deploy/kubernetes/rbac.yaml $CSI_GH/external-attacher/v4.12.0/deploy/kubernetes/rbac.yaml \ + $s/deploy/kubernetes/csi-snapshotter/rbac-csi-snapshotter.yaml $CSI_GH/external-resizer/v2.2.1/deploy/kubernetes/rbac.yaml \ + $CSI_GH/external-health-monitor/v0.18.0/deploy/kubernetes/external-health-monitor-controller/rbac.yaml; do + curl -sfL --max-time 60 "$f" | $K apply -f - >/dev/null 2>&1 || return 1 + done + for f in driverinfo plugin snapshotclass; do + curl -sfL --max-time 60 "$CSI_GH/csi-driver-host-path/v1.18.0/deploy/kubernetes-1.34/hostpath/csi-hostpath-$f.yaml" | $K apply -f - >/dev/null 2>&1 || return 1 + done + $K apply -f - >/dev/null 2>&1 <<'CSI_SC' || return 1 +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: csi-hostpath-sc +provisioner: hostpath.csi.k8s.io +reclaimPolicy: Delete +volumeBindingMode: Immediate +allowVolumeExpansion: true +CSI_SC + $K -n kube-system rollout status deploy snapshot-controller --timeout=180s >/dev/null 2>&1 \ + && $K -n default rollout status sts csi-hostpathplugin --timeout=240s >/dev/null 2>&1 +} """ def _session_tracks(): @@ -290,6 +346,8 @@ def paused_total(st: dict[str, Any] | None) -> float: def remaining(st: dict[str, Any] | None) -> float: if not st: return 0.0 + if st.get("mode") == "lesson": # a lesson teaches: it has no clock + return float("inf") return float(st.get("started_at", 0)) + float(st.get("duration", EXAM_SECONDS)) + paused_total(st) - time.time() def set_paused(st: dict[str, Any], on: bool) -> None: @@ -335,11 +393,11 @@ def session_questions(st: dict[str, Any] | None) -> list[dict[str, Any]]: # so compose_exam's rng.shuffle never reached the candidate and every exam arrived grouped by # domain and difficulty - base tasks first, then hard, then the sim sets. Real exams are not # ordered by difficulty, and practising triage on a sorted list teaches the wrong instinct. - by_id = {q["id"]: q for q in QUESTIONS} + by_id = {q["id"]: q for q in QUESTIONS + LESSONS} return [by_id[i] for i in ids if i in by_id] def q_by_id(n: int) -> dict[str, Any]: - for q in QUESTIONS: + for q in QUESTIONS + LESSONS: # a lesson has a host like a task: ./ksim ssh 9201 reaches it if q["id"] == n: return q # ids run 1..N without gaps today, but print the real ranges rather than "1-" so a @@ -401,19 +459,21 @@ def run(q, script, timeout=180, params=None): # environment points KUBECONFIG at admin.conf, which only root can read and sshd never passes on. AS_CANDIDATE = ["-u", "candidate", "-w", "/home/candidate", "-e", "KUBECONFIG=/home/candidate/.kube/config"] -def solve_on_host(q, params, timeout=600): - """Run a task's scripted solution the way a candidate would type it: as `candidate`, in an - interactive shell on the task's host, so the host's aliases and profile apply.""" +def solve_on_host(q, params, timeout=600, script=None): + """Run a task's scripted solution (or another script: a lesson step's commands) the way a + candidate would type it: as `candidate`, in an interactive shell on the task's host, so the + host's aliases and profile apply.""" into = ["ssh", "candidate@vm", "bash -i"] if is_vm(q) else ["bash", "-i"] try: p = subprocess.run([envs.RUNTIME, "exec", "-i", *([] if is_vm(q) else AS_CANDIDATE), nodes_of(q)["CP"], *into], - input=render(q["solve"], params, q), capture_output=True, text=True, timeout=timeout) + input=render(q["solve"] if script is None else script, params, q), capture_output=True, text=True, + errors="replace", timeout=timeout) # a script may print binary (an etcd value, say) except subprocess.TimeoutExpired: return 1, f"solution script timed out after {timeout}s" return p.returncode, p.stdout[-400:] def k8s_version(): - """The Kubernetes release the track's exam hosts run, e.g. v1.35.8 (from the pinned node image).""" + """The Kubernetes release the track's exam hosts run, e.g. v1.37.0 (from the pinned node image).""" m = re.search(r":(v[\d.]+)", envs.BASE_IMAGES.get(K8S, "")) return m.group(1) if m else K8S @@ -824,6 +884,7 @@ def load_progress(): objectives objective id -> when it was first done focus domains the path walks first, in that order (./ksim focus) known domains and objectives you declared known, which the path leaves out (./ksim skip) + lessons objective id -> when its lesson was first done (every step checked) tasks what the study sessions of earlier versions solved - kept as it was, read by nothing. Results against the clock live in the history; this file only steers the path.""" try: @@ -834,14 +895,16 @@ def load_progress(): oids = {o["id"] for o in OBJECTIVES} return {"tasks": dict(d.get("tasks", {})), "objectives": dict(d.get("objectives", {})), "focus": [x for x in d.get("focus", []) if x in DOMAINS], - "known": [x for x in dict.fromkeys(d.get("known", [])) if x in DOMAINS or x in oids]} + "known": [x for x in dict.fromkeys(d.get("known", [])) if x in DOMAINS or x in oids], + "lessons": dict(d.get("lessons", {}))} def save_progress(prog): with open(PROGRESS_FILE, "w") as f: json.dump({**({"tasks": prog["tasks"]} if prog.get("tasks") else {}), "objectives": {k: prog["objectives"][k] for k in sorted(prog["objectives"])}, **({"focus": prog["focus"]} if prog.get("focus") else {}), - **({"known": prog["known"]} if prog.get("known") else {})}, f, indent=1) + **({"known": prog["known"]} if prog.get("known") else {}), + **({"lessons": {k: prog["lessons"][k] for k in sorted(prog["lessons"])}} if prog.get("lessons") else {})}, f, indent=1) f.write("\n") def load_known(): @@ -861,6 +924,60 @@ def known_task(q, known): objs = objectives_of(q["id"]) return bool(objs) and all(is_known(o, known) for o in objs) +# ------------------------------------------------------------ lessons ---- +# A lesson (/lessons.py) is a guided lab on one objective: an introduction, steps that you do +# on a host of your own - each with a check, a hint and the commands that do it - and a wrap-up that +# sends you to the objective's timed drill. It runs as a session of mode "lesson": no clock, its +# steps checked one at a time and in order, nothing written to the history or the statistics; a +# lesson whose every step was checked is stamped done in study-.json. +def find_lesson(arg): + """A lesson by its objective (2.1, or the objective's key) or by its id (9201); None if none fits.""" + o = find_objective(arg) + return next((l for l in LESSONS if (o and l["objective"] == o["id"]) or str(l["id"]) == str(arg)), None) + +def lesson_of(st): + """The lesson a session is, or None.""" + if not st or st.get("mode") != "lesson": + return None + return next((l for l in LESSONS if l["id"] in st.get("ids", [])), None) + +def lessons_done(): + """objective id -> when its lesson was first done.""" + return load_progress()["lessons"] + +def check_step(l, params, i): + """Run the check of step i on the lesson's host.""" + try: + rc, out = run(l, l["steps"][i]["check"], timeout=240, params=params) + except subprocess.TimeoutExpired: + rc, out = 1, "timed out" + return {"pass": rc == 0, "out": out[-600:], "at": time.time()} + +def open_step(st, l): + """The first step not checked yet, or None when every one is.""" + done = st.get("steps", {}) + return next((i for i in range(len(l["steps"])) if not done.get(str(i), {}).get("pass")), None) + +def lesson_check(st, i): + """Check step i of the lesson in progress and keep the result. Steps count in order - a step + builds on the one before - so only the first open step or one already done can be checked, and + a pass stays a pass when a later step changes what an earlier check looked at. + Returns (result, None) or (None, why not).""" + l = lesson_of(st) + if not l or st.get("ended_at"): + return None, "No lesson in progress." + if not 0 <= i < len(l["steps"]): + return None, f"The lesson has steps 1 to {len(l['steps'])}." + first = open_step(st, l) + if first is not None and i > first: + return None, f"Do step {first + 1} first: steps count in order." + r = check_step(l, session_params(st, l), i) + steps = st.setdefault("steps", {}) + if r["pass"] or not steps.get(str(i), {}).get("pass"): + steps[str(i)] = r + save_state(st) + return r, None + def task_state(qid, agg): """passed = the latest attempt against the clock was a full pass; failed = it was not; new = never attempted.""" @@ -1273,7 +1390,7 @@ def preflight(qs): ("kubectl", "seeds and grades the tasks"), ("ssh-keygen", "makes the key the desktop logs in with")): if not shutil.which(tool): die(f"'{tool}' is not installed - it {why}.") - missing = [i for i in (node_image(), envs.DESKTOP_IMAGE) if not envs.image_exists(i)] + missing = [i for i in dict.fromkeys([node_image(q) for q in qs if not is_vm(q)] + [envs.DESKTOP_IMAGE]) if not envs.image_exists(i)] if missing: die(f"Not built yet: {', '.join(missing)}\n Build the exam host and desktop images (once, a few minutes): ./ksim build") if any(is_vm(q) for q in qs) and not (envs.vm_ready(K8S) and envs.image_exists(envs.VM_IMAGE)): @@ -1296,7 +1413,7 @@ def _log_msg(*args: Any, **kwargs: Any) -> None: def provision(q, params, log: Callable[..., Any] = _log_msg): """One task, start to finish: its cluster, its scenario, and the hand-over to `candidate`. Returns None, or what went wrong.""" - host, tag = host_of(q), f"Q{q['id']:02d} {q['title']}" + host, tag = host_of(q), (f"lesson {q['objective']} {q['title']}" if q.get("lesson") else f"Q{q['id']:02d} {q['title']}") try: make_host(q) except envs.EnvError as e: @@ -1402,25 +1519,54 @@ def begin_session(qs, mode, duration, force, log: Callable[..., Any] = _log_msg, st = {"mode": mode, "track": TRACK_ID, "hard": hard, "ids": [q["id"] for q in qs], "params": params_by_id, "duration": duration, "started_at": time.time(), "ended_at": None, "k8s": k8s_version(), "results": {}, "flags": [], "notes": "", - "objective": objective} # the curriculum objective a drill was started for, if any + "objective": objective} # the curriculum objective a drill or a lesson was started for, if any + lesson = mode == "lesson" + if lesson: + st["steps"] = {} # step index -> its latest check (a pass sticks) save_state(st) log("") log(bold("=" * 64)) - log(bold(f" {TRACK_ID.upper()} {'EXAM' if mode == 'exam' else 'DRILL'}{' · HARD' if hard else ''} - clock started")) + log(bold(f" {TRACK_ID.upper()} {'LESSON' if lesson else 'EXAM' if mode == 'exam' else 'DRILL'}{' · HARD' if hard else ''} - " + + ("no clock: take your time" if lesson else "clock started"))) log(f" Hosts : {hosts[0]}" + (f" .. {len(hosts)} in all" if len(hosts) > 1 else "") + f" (Kubernetes {st['k8s']}, as the real {TRACK_ID.upper()} runs {K8S})") - log(f" Ends : {datetime.fromtimestamp(st['started_at'] + duration).strftime('%H:%M:%S')} ({duration // 60} min)") - log(f" Tasks : {', '.join('Q%02d' % q['id'] for q in qs) if mode == 'drill' else len(qs)} Pass mark: {PASS_PCT}%") + if lesson: + log(f" Lesson : {objective} {qs[0]['title']} - {len(qs[0]['steps'])} steps") + else: + log(f" Ends : {datetime.fromtimestamp(st['started_at'] + duration).strftime('%H:%M:%S')} ({duration // 60} min)") + log(f" Tasks : {', '.join('Q%02d' % q['id'] for q in qs) if mode == 'drill' else len(qs)} Pass mark: {PASS_PCT}%") if problems: log(yellow(f" Seeding had problems on: {', '.join(problems)} - ./ksim reset N rebuilds one task's host")) log(bold("=" * 64)) - log(" ./ksim web the exam in the browser: task pane, remote desktop, clock\n" - " ./ksim list / show N / grade N / timer / end terminal workflow\n" - + f" Every task names its host: ssh {hosts[0]} from the desktop's terminal, then sudo -i when you need root.\n" - + f" Files a task asks for go under {WS_DISPLAY}/ on its host.") + log(" ./ksim web " + ("the lesson in the browser: its steps beside the remote desktop\n" if lesson else + "the exam in the browser: task pane, remote desktop, clock\n") + + (" ./ksim lesson step / check / hint / show, ./ksim end terminal workflow\n" if lesson else + " ./ksim list / show N / grade N / timer / end terminal workflow\n") + + f" {'The lesson runs on' if lesson else 'Every task names its host:'} ssh {hosts[0]} from the desktop's terminal, then sudo -i when you need root.\n" + + f" Files go under {WS_DISPLAY}/ on the host.") return st +def finish_lesson(st): + """End a lesson: nothing is graded or recorded in the history; a lesson whose every step was + checked is stamped done on its objective (the first time). Returns the lines to show.""" + l = lesson_of(st) + st["ended_at"] = time.time() + n, done = len(l["steps"]), sum(1 for i in range(len(l["steps"])) if st.get("steps", {}).get(str(i), {}).get("pass")) + if done == n: + prog = load_progress() + prog["lessons"].setdefault(l["objective"], datetime.now().isoformat(timespec="minutes")) + save_progress(prog) + lines = [f"✓ Lesson {l['objective']} done - {n}/{n} steps checked. Now prove it against the clock: ./ksim drill {l['objective']}"] + else: + lines = [f"Lesson {l['objective']} finished with {done}/{n} steps checked - take it again any time: ./ksim lesson {l['objective']}"] + st["milestones"] = lines + save_state(st) + return lines + def finish_session(st, log: Callable[..., Any] = _log_msg): + if st.get("mode") == "lesson": + finish_lesson(st) + return None qs = session_questions(st) over = remaining(st) <= 0 set_paused(st, False) @@ -1529,6 +1675,163 @@ def cmd_drill(args): begin_session(kept, "drill", (minutes * 60) if minutes else per_task * len(kept), force, hard=hard, objective=objective if len(args) == 1 else None) +def cli_md(text): + """A lesson's Markdown for a terminal: fenced blocks indented, `code` highlighted.""" + out, fence = [], False + for line in text.strip("\n").split("\n"): + if line.strip().startswith("```"): + fence = not fence + continue + out.append(" " + line if fence else paint_code(line)) + return "\n".join(out) + +def print_step(l, params, i, st=None): + s = l["steps"][i] + done = bool(st and st.get("steps", {}).get(str(i), {}).get("pass")) + print(bold(f"Step {i + 1} of {len(l['steps'])}: {render(s['title'], params, l)}") + (green(" ✓ checked") if done else "")) + print(cli_md(render(s["text"], params, l))) + print(dim(" ./ksim lesson check when it is done ./ksim lesson hint / show if you are stuck")) + +def print_outro(l, params): + print(bold("Every step is checked.")); print("-" * 64) + print(cli_md(render(l["outro"], params, l))); print("-" * 64) + print(dim(" ./ksim end finishes the lesson")) + +def cmd_lesson(args): + """Lessons (/lessons.py): guided labs on a host of your own, with no clock. + ./ksim lesson | lesson OBJ | lesson step|check|hint|show [N] | lesson audit [OBJ..]""" + sub = args[0] if args else "" + if sub == "audit": + return lesson_audit(args[1:]) + st = load_state() + live = lesson_of(st) if st and not st.get("ended_at") else None + if sub in ("check", "step", "hint", "show"): + if not live: + die("No lesson in progress. ./ksim lesson lists them; ./ksim lesson 2.1 starts one.") + params = session_params(st, live) + num = [a for a in args[1:] if a.isdigit()] + i = int(num[0]) - 1 if num else open_step(st, live) + if i is None: + return print_outro(live, params) + if not 0 <= i < len(live["steps"]): + die(f"The lesson has steps 1 to {len(live['steps'])}.") + s = live["steps"][i] + if sub == "step": + return print_step(live, params, i, st) + if sub in ("hint", "show"): + print(bold(f"{'Hint' if sub == 'hint' else 'The commands'} - step {i + 1}: {render(s['title'], params, live)}")) + text = render(s[sub], params, live) + print((cli_md(text) if sub == "hint" else text) or dim("Nothing for this step.")) + return + r, why = lesson_check(st, i) + if why: + die(why) + if not r["pass"]: + print(red(f"✗ Step {i + 1} is not done yet: {render(s['title'], params, live)}")) + if r["out"]: + print(dim(" " + r["out"][-300:].replace("\n", "\n "))) + print(dim(" ./ksim lesson hint a hint ./ksim lesson show the commands")) + return + print(green(f"✓ Step {i + 1} done: {render(s['title'], params, live)}")); print() + nxt = open_step(load_state(), live) + return print_outro(live, params) if nxt is None else print_step(live, params, nxt) + if sub: + l = find_lesson(sub) + if not l: + die(f"No lesson for '{sub}'. The {TRACK_ID.upper()} lessons: " + + (", ".join(f"{x['objective']} ({x['title']})" for x in LESSONS) or "none yet") + " (./ksim lesson lists them)") + st = begin_session([l], "lesson", 0, "--force" in args, objective=l["objective"]) + params = session_params(st, l) + print(); print(bold(f"Lesson {l['objective']} - {l['title']}")); print("-" * 64) + print(cli_md(render(l["intro"], params, l))); print("-" * 64) + print_step(l, params, 0, st) + return + if not LESSONS: + print(f"No lessons for {TRACK_ID.upper()} yet.") + return + done = lessons_done() + print(bold(f"{TRACK['name']} - lessons") + dim(" guided labs on a host of your own, with no clock")) + for l in LESSONS: + print(f" {green('✓') if l['objective'] in done else dim('·')} {l['objective']:<5} {l['title']}" + + dim(f" {len(l['steps'])} steps · {DOMAINS.get(l['domain'], l['domain'])}")) + print(dim(f"\n ./ksim lesson {LESSONS[0]['objective']} starts that lesson ./ksim lesson audit proves them on real clusters")) + +def lesson_audit(args): + """Prove lessons on real clusters: each is seeded on a host of its own; then every step must + fail its check, have its commands run as `candidate`, and pass its check - so a lesson can be + done exactly as it shows, and no step is done before you do it.""" + for t in TRACKS: + try: + with open(os.path.join(STATE_DIR, f"state-{t}.json")) as f: + if not json.load(f).get("ended_at"): + die(f"A {t.upper()} session is open, and a new session would delete the audit's hosts. End it first.") + except (OSError, ValueError): + pass + names = [a for a in args if not a.startswith("--")] + ls = list(LESSONS) if not names or names == ["all"] else [find_lesson(n) for n in names] + if not ls or None in ls: + die(f"usage: ./ksim lesson audit [all|OBJECTIVE..] [--keep] (lessons: {', '.join(l['objective'] for l in LESSONS) or 'none'})") + preflight(ls[:PARALLEL]) + envs.ensure_registry() + lock = os.path.join(STATE_DIR, f"audit-{os.getpid()}.lock") # keeps sessions from starting meanwhile + os.makedirs(STATE_DIR, exist_ok=True) + open(lock, "w").close() + import atexit + atexit.register(lambda: os.path.exists(lock) and os.remove(lock)) + print(bold(f"Lesson audit of {len(ls)} {TRACK_ID.upper()} lesson(s), {PARALLEL} at a time - each step must fail, be done by its own commands, then pass.")) + def one(l): + params = make_params(l, random.Random(l["id"])); host = host_of(l); t0 = time.time() + res = {"id": l["id"], "objective": l["objective"], "title": l["title"], "host": host, "steps": len(l["steps"]), "proven": 0} + try: + make_host(l) + except envs.EnvError as e: + res.update(verdict="no cluster", detail=str(e)[-400:], secs=int(time.time() - t0)) + return res + try: + rc, out = run(l, l["setup"], params=params, timeout=300) + except subprocess.TimeoutExpired: + rc, out = 1, "setup timed out after 300s" + envs.hand_over(host, None) + if rc != 0: + res.update(verdict="seed failed", detail=out[-400:]) + else: + for i, s in enumerate(l["steps"]): + if check_step(l, params, i)["pass"]: + res.update(verdict=f"step {i + 1} passes before its commands") + break + _, sout = solve_on_host(l, params, script=s["show"]) + # what the commands created may still be starting: a learner would watch it come up + r, waited = check_step(l, params, i), 0 + while not r["pass"] and waited < 90: + time.sleep(10); waited += 10 + r = check_step(l, params, i) + if not r["pass"]: + res.update(verdict=f"step {i + 1} fails after its commands", detail=(r["out"] + "\n--- the commands printed:\n" + sout)[-600:]) + break + res["proven"] = i + 1 + else: + res["verdict"] = "ok" + if "--keep" not in args: + envs.destroy(host) + res["secs"] = int(time.time() - t0) + return res + results = [] + with concurrent.futures.ThreadPoolExecutor(max_workers=PARALLEL) as pool: + for fut in concurrent.futures.as_completed([pool.submit(one, l) for l in ls]): + res = fut.result(); results.append(res) + col = green if res["verdict"] == "ok" else red + print(f" {col(res['verdict']):<44} {res['objective']:<5}{res['proven']}/{res['steps']} steps {res['secs']:>4}s {res['title']}", flush=True) + if res.get("detail"): + print(dim(" " + res["detail"].replace("\n", "\n ")), flush=True) + results.sort(key=lambda r: r["id"]) + out_dir = os.path.join(STATE_DIR, "audit"); os.makedirs(out_dir, exist_ok=True) + path = os.path.join(out_dir, f"lessons-{TRACK_ID}.json") + with open(path, "w") as f: + json.dump({"k8s": k8s_version(), "at": datetime.now().isoformat(timespec="minutes"), "results": results}, f, indent=1) + bad = [r for r in results if r["verdict"] != "ok"] + print(bold(f"\n{len(results) - len(bad)}/{len(results)} lessons proven step by step.") + (red(f" {len(bad)} need work: " + ", ".join(r["objective"] for r in bad)) if bad else "")) + print(dim(f" report: {os.path.relpath(path, ROOT)}")) + def cmd_sim(args): """A fixed simulator set, full exam rules and clock.""" force = "--force" in args @@ -1589,6 +1892,8 @@ def cmd_grade(args): # meant to rehearse the real thing (no feedback until you submit) must not do by default. if st.get("mode") == "exam" and not st.get("ended_at") and not force: die("No feedback during an exam - the real one gives none. Submit with ./ksim end (--force to peek anyway).") + if st.get("mode") == "lesson": + die("A lesson is checked step by step: ./ksim lesson check") ids = [q["id"] for q in session_questions(st)] if args[0] == "all" else [int(a) for a in args] for n in ids: q = q_by_id(n); r = grade_one(q) @@ -1614,6 +1919,10 @@ def print_score(st): print(f"\n{bold('Score:')} {earned}/{total} = {pct}% -> {verdict} (pass mark {PASS_PCT}%)") def status_line(st): + l = lesson_of(st) + if l: + n = len(l["steps"]); done = n if open_step(st, l) is None else open_step(st, l) + return green("lesson - no clock") + dim(f" [{done}/{n} steps checked]") r = remaining(st) if st.get("paused_at"): return yellow(f"⏸ PAUSED - {fmt(r)} remaining") + dim(" ./ksim resume") @@ -1629,10 +1938,15 @@ def cmd_status(args): print(status_line(st)) print(dim(f"{sum(1 for v in st['results'].values() if v['pass'])} passed / {len(st['results'])} graded / {len(session_questions(st))} in session")) +def no_clock(st): + if st.get("mode") == "lesson": + die("A lesson has no clock - take your time. (./ksim lesson step shows where you are)") + def cmd_timer(args): st = require_started() if st.get("ended_at"): die("Session already ended.") + no_clock(st) try: while True: st = load_state(); r = remaining(st) @@ -1652,10 +1966,12 @@ def cmd_pause(args): st = require_started() if st.get("ended_at"): die("Session already ended.") + no_clock(st) set_paused(st, True); print(status_line(st)) def cmd_resume(args): st = require_started() + no_clock(st) set_paused(st, False); print(status_line(st)) def cmd_restart(args): @@ -1664,6 +1980,7 @@ def cmd_restart(args): st = require_started() if st.get("ended_at"): die("Session already ended - nothing to restart. Start a new one: ./ksim drill, ./ksim start, ./ksim next") + no_clock(st) restart_clock(st) print(status_line(st) + dim(" clock restarted - the hosts and your work are as they were")) @@ -1671,6 +1988,11 @@ def cmd_end(args): st = require_started() if st.get("ended_at"): die("Session already ended and recorded.") + if st.get("mode") == "lesson": # nothing to grade: the steps were checked as you went + for line in finish_lesson(st): + print(green(line) if line.startswith("✓") else line) + print(dim("A lesson writes nothing to the history or the scores. Its host stays up until the next session.")) + return down = hosts_down(st) if down: die(down_message(down)) @@ -1823,7 +2145,12 @@ class Api(http.server.BaseHTTPRequestHandler): ulog = lambda line: Api.log_lines.append(ansi.sub("", line)) def work(): try: - if body.get("sim") in SETS: + if body.get("lesson"): + l = find_lesson(body["lesson"]) + if not l: + die(f"No lesson for {body['lesson']}.") + begin_session([l], "lesson", 0, True, log=ulog, other_ok=False, objective=l["objective"]) + elif body.get("sim") in SETS: begin_session(set_questions(body["sim"]), "exam", EXAM_SECONDS, True, log=ulog, hard=True, other_ok=False) elif mode == "exam": begin_session(compose_exam(hard, rng), "exam", EXAM_SECONDS, True, log=ulog, hard=hard, other_ok=False) @@ -1862,9 +2189,15 @@ class Api(http.server.BaseHTTPRequestHandler): return self._json({"seeding": True}) if not st: return self._json({"error": "no session"}, 400) + if st.get("mode") == "lesson" and p.path in ("/api/pause", "/api/restart", "/api/grade"): + return self._json({"error": "A lesson has no clock and is checked step by step."}, 400) if p.path == "/api/flag": qid = int(body["id"]); flags = set(st.get("flags", [])) flags.symmetric_difference_update({qid}); st["flags"] = sorted(flags); save_state(st) + elif p.path == "/api/lesson/check": # one step of the lesson in progress + _, why = lesson_check(st, int(body.get("step", -1))) + if why: + return self._json({"error": why}, 400) elif p.path == "/api/pause": set_paused(st, bool(body.get("paused", True))) elif p.path == "/api/restart": @@ -1888,7 +2221,7 @@ class Api(http.server.BaseHTTPRequestHandler): st["results"][str(q["id"])] = grade_one(q, session_params(st, q)); save_state(st) elif p.path == "/api/end": if not st.get("ended_at"): - down = hosts_down(st) # grading hosts that are not running would record zeros + down = hosts_down(st) if st.get("mode") != "lesson" else [] # grading hosts that are not running would record zeros if down: return self._json({"error": down_message(down)}, 503) finish_session(st, log=lambda *_: None) @@ -1929,16 +2262,31 @@ class Api(http.server.BaseHTTPRequestHandler): "objective": st.get("objective"), "milestones": st.get("milestones", []), "aborted": bool(st.get("aborted")), "results": st["results"], "score": {"earned": earned, "total": total, "pct": pct, "pass_pct": PASS_PCT}, "domains": self.domain_scores(st), "server_time": time.time()} + l = lesson_of(st) + if l: + payload["session"]["steps"] = st.get("steps", {}) + payload["lesson"] = self.lesson_payload(l, session_params(st, l)) return payload + def lesson_payload(self, l: dict[str, Any], params: dict[str, Any]) -> dict[str, Any]: + """A lesson as the page shows it: introduction, steps (with their hint and commands, which the + page keeps behind a click), wrap-up, its host and the objective's drill.""" + r = lambda t: render(t, params, l).strip("\n") + return {"id": l["id"], "objective": l["objective"], "title": l["title"], "host": host_of(l), "nodes": 1 + workers_of(l), + "intro": r(l["intro"]), "outro": r(l["outro"]), + "steps": [{"title": r(s["title"]), "text": r(s["text"]), "hint": r(s["hint"]), "show": r(s["show"])} for s in l["steps"]]} + def path_payload(self) -> dict[str, Any]: """The curriculum as the UI shows it: every objective with its notes, tasks and state, how many are done, and the next step on the path. Empty for a track without one.""" if not OBJECTIVES: return {"objectives": [], "focus": [], "known": load_known(), "path": None} - agg, known = stats(), load_known() + agg, known, done = stats(), load_known(), lessons_done() + lesson = {l["objective"]: l for l in LESSONS} objs = [{"id": o["id"], "key": o["key"], "domain": o["domain"], "text": o["text"], "state": objective_state(o, agg, known), "know": o["know"], "fast": o["fast"].strip("\n"), "traps": o["traps"], "docs": o["docs"], + "lesson": ({"id": lesson[o["id"]]["id"], "title": lesson[o["id"]]["title"], "steps": len(lesson[o["id"]]["steps"]), + "done": o["id"] in done} if o["id"] in lesson else None), "tasks": [{"id": q["id"], "title": q["title"], "set": q.get("set"), "state": task_state(q["id"], agg)} for q in objective_tasks(o, True)]} for o in OBJECTIVES] step = next_step(agg); full, fixed = mock_scores() @@ -2014,13 +2362,17 @@ def cmd_build(args): for t in tracks: if t not in TRACKS: die(f"usage: ./ksim build [{'|'.join(TRACKS)}|all] [--vm] (default: the current track, {TRACK_ID})") - minors, variants = [], [] + minors, variants, vm_minors = [], [], [] for t in tracks: load_track(t) - if K8S not in minors: - minors.append(K8S) - if TRACK.get("image") and (K8S, TRACK["image"]) not in variants: - variants.append((K8S, TRACK["image"])) + if any(is_vm(q) for q in QUESTIONS) and K8S not in vm_minors: + vm_minors.append(K8S) + for q in [{}] + [q for q in QUESTIONS + LESSONS if not is_vm(q)]: # the track's own, then any a task names + k8s, variant = q.get("k8s", K8S), q.get("image", TRACK.get("image")) + if k8s not in minors: + minors.append(k8s) + if variant and (k8s, variant) not in variants: + variants.append((k8s, variant)) print(bold(f"Building images for {', '.join(t.upper() for t in tracks)} - the first build downloads a few hundred MB...")) try: for k8s in minors: @@ -2029,7 +2381,7 @@ def cmd_build(args): envs.build_variant_image(k8s, variant) envs.build_desktop_image() if "--vm" in args: # the VM exam host, for tasks that need a kernel of their own - for k8s in minors: + for k8s in vm_minors: envs.build_vm_image(k8s) except envs.EnvError as e: die(str(e)) @@ -2226,6 +2578,10 @@ def cmd_help(args): " as passed, and a mock still draws from it (skip clear / unskip X bring it back)\n" " notes [OBJ|DOMAIN|all] what to know, fastest commands, traps and docs for an objective (3.2 or probes)\n" " drill OBJ an objective's tasks against the clock, those not yet passed first\n") + objective_help += "" if not LESSONS else ( + f" lesson [OBJ] the track's lessons ({len(LESSONS)}), or start one: a guided lab on a host of your own, no clock\n" + " lesson step|check|hint|show [N] the step you are on, check it, a hint, the commands; ./ksim end finishes\n" + " lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after them\n") print(f"""usage: ./ksim [--track cka|ckad|cks] (current track: {TRACK_ID}, {len(QUESTIONS)} tasks) Every task runs on its own cluster at the exam's Kubernetes version ({TRACK_ID}: {k8s_version()}), reached with `ssh ` from the remote desktop. @@ -2267,7 +2623,7 @@ COMMANDS = {"start": cmd_start, "drill": cmd_drill, "list": cmd_list, "show": cm "reset": cmd_reset, "web": cmd_web, "abort": cmd_abort, "progress": cmd_progress, "solution": cmd_solution, "sim": cmd_sim, "version": cmd_version, "help": cmd_help, "build": cmd_build, "hosts": cmd_hosts, "ssh": cmd_ssh, "down": cmd_down, "audit": cmd_audit, "coverage": cmd_coverage, "objectives": cmd_objectives, "curriculum": cmd_objectives, "notes": cmd_notes, "next": cmd_next, "focus": cmd_focus, - "skip": cmd_skip, "known": cmd_skip, "unskip": cmd_unskip} + "skip": cmd_skip, "known": cmd_skip, "unskip": cmd_unskip, "lesson": cmd_lesson, "lessons": cmd_lesson} if __name__ == "__main__": cmd = ARGV[0] if ARGV else "help" diff --git a/sim/bank.py b/sim/bank.py index b03690b..901cade 100644 --- a/sim/bank.py +++ b/sim/bank.py @@ -19,6 +19,12 @@ mkns() { def C(name, pts, script): return {"name": name, "pts": pts, "script": script} +def step(title, text, check, hint="", show=""): + """One step of a lesson (/lessons.py): what to do (Markdown), the check that proves it + (bash, exit 0 = done, run like a task's checks), a hint, and the commands that do it - shown on + request, and run by ./ksim lesson audit to prove the step can be done as shown.""" + return {"title": title, "text": text, "check": check, "hint": hint, "show": show} + def ns_pick(r, base): """Namespace name with a random suffix so the same task never lands on the same namespace twice.""" return f"{base}-{r.choice(['alpha', 'bravo', 'delta', 'echo', 'kilo', 'nova', 'omega', 'zulu'])}" diff --git a/sim/envs.py b/sim/envs.py index 7d01dc1..dd91102 100644 --- a/sim/envs.py +++ b/sim/envs.py @@ -26,10 +26,11 @@ COURSE = "/opt/course" # a task's files, as its host sees them # cka0037-control-plane, cka0037-worker, ... Only names of this shape are ever deleted. HOST_RE = re.compile(r"^(cka|ckad|cks)\d{4}$") -# The kind node image each exam host image is built from, one per Kubernetes minor an exam runs. -# Pinned by digest: kind only guarantees an image built for its own release (these are v0.33.0's). +# The kind node image each exam host image is built from, one per Kubernetes minor an exam runs - +# and v1.36, for the tasks that upgrade their cluster to v1.37 ("k8s": "v1.36"). Pinned by digest: +# kind only guarantees an image built for its own release (these are v0.33.0's). BASE_IMAGES = { - "v1.35": "docker.io/kindest/node:v1.35.8@sha256:07b2536e30b803ed61d1677a79df6115f798ce64c80f9e22f6ed45afd09323c0", + "v1.36": "docker.io/kindest/node:v1.36.4@sha256:099e049362a1526b2db71494e1947aae99bd16290d7c895f2b7ea312e3cbfaed", "v1.37": "docker.io/kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5", } HELM_VERSION = "v4.3.0" @@ -83,17 +84,22 @@ def image_exists(name): def build_node_image(k8s, log=print): + """The exam host image of a Kubernetes minor (images/node): Ubuntu, with kind's runtime and + preloaded images from the pinned kind node image, and Kubernetes of that release as packages.""" base = BASE_IMAGES.get(k8s) - if not base: + m = re.search(r":v([\d.]+)@", base or "") + if not m: raise EnvError(f"no kind node image is pinned for Kubernetes {k8s} (BASE_IMAGES in sim/envs.py)") - log(f" {node_image(k8s)}: from {base.split('@')[0]}") + log(f" {node_image(k8s)}: Ubuntu 24.04, Kubernetes {m.group(1)} as packages, kind's runtime from {base.split('@')[0]}") # etcdctl/etcdutl come out of the etcd image this kubeadm would deploy, so they match the server rc, out = sh([RUNTIME, "run", "--rm", "--entrypoint", "sh", base, "-c", 'kubeadm config images list --kubernetes-version "$(cat /kind/version)" 2>/dev/null | grep /etcd:']) - etcd = out.strip().splitlines()[-1] if rc == 0 and out.strip() else "" + # (the line that names it: when the image is pulled first, podman's progress comes before it) + etcd = next((l.strip() for l in reversed(out.splitlines()) if "/etcd:" in l), "") if rc == 0 else "" if not etcd: raise EnvError(f"could not read the etcd image of {base}:\n{out[-400:]}") rc, out = sh([RUNTIME, "build", "-t", node_image(k8s), "--build-arg", f"BASE={base}", "--build-arg", f"ETCD_IMAGE={etcd}", + "--build-arg", f"K8S_VERSION={m.group(1)}", "--build-arg", f"K8S_MINOR={k8s}", "--build-arg", f"HELM_VERSION={HELM_VERSION}", os.path.join(ROOT, "images", "node")], timeout=1800) if rc != 0: raise EnvError(f"building {node_image(k8s)} failed:\n{out[-1500:]}") @@ -105,14 +111,11 @@ def build_variant_image(k8s, variant, log=print): ctx = os.path.join(ROOT, "images", f"node-{variant}") if not os.path.isdir(ctx): raise EnvError(f"no image directory for the '{variant}' exam host (images/node-{variant})") - rc, out = sh([RUNTIME, "run", "--rm", "--entrypoint", "cat", node_image(k8s), "/kind/version"]) - version = out.strip().splitlines()[-1].lstrip("v") if rc == 0 and out.strip() else "" - if not re.match(r"^\d+\.\d+\.\d+$", version): - raise EnvError(f"could not read the Kubernetes version of {node_image(k8s)}:\n{out[-300:]}") + if not image_exists(node_image(k8s)): + raise EnvError(f"{node_image(k8s)} is not built yet; the '{variant}' exam host is built on top of it") name = node_image(k8s, variant) - log(f" {name}: from {node_image(k8s)} (Kubernetes {version} as packages, {variant.upper()} tools)") - rc, out = sh([RUNTIME, "build", "-t", name, "--build-arg", f"BASE={node_image(k8s)}", - "--build-arg", f"K8S_VERSION={version}", "--build-arg", f"K8S_MINOR={k8s}", ctx], timeout=3600) + log(f" {name}: from {node_image(k8s)}, with the {variant.upper()} tools") + rc, out = sh([RUNTIME, "build", "-t", name, "--build-arg", f"BASE={node_image(k8s)}", ctx], timeout=3600) if rc != 0: raise EnvError(f"building {name} failed:\n{out[-1500:]}") log(f" {name}: built") diff --git a/web/index.html b/web/index.html index 6a8fdba..a1ef6fb 100644 --- a/web/index.html +++ b/web/index.html @@ -236,7 +236,8 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' color: #fff; border-bottom: 1px solid var(--head-line); } .xtitlebox { display: flex; align-items: center; gap: 15px; min-width: 0; } .xtitle { font-size: 1.25rem; font-weight: 700; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; } - .timer { padding: .25rem .5rem; background: rgba(255,255,255,.1); border-radius: .25rem; color: #fff; font-weight: 500; font-variant-numeric: tabular-nums; transition: background-color .3s, color .3s; } + .timer { padding: .25rem .5rem; background: rgba(255,255,255,.1); border-radius: .25rem; color: #fff; font-weight: 500; font-variant-numeric: tabular-nums; transition: background-color .3s, color .3s; + white-space: nowrap; flex-shrink: 0; } .timer.warning { background: rgba(220,53,69,.25); color: var(--yellow); animation: pulse 1.4s ease-in-out infinite; } .timer.paused { color: #bbb; } @keyframes pulse { 0%, 100% { background: rgba(220,53,69,.25); } 50% { background: rgba(220,53,69,.5); } } @@ -276,6 +277,20 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' .qcontent hr { border: 0; border-top: 1px solid var(--line); margin: 1rem 0; } .qtitle { font-size: 18px; margin: 0 0 .6rem; color: var(--ink); } .qbody p { margin: 0 0 12px; white-space: pre-wrap; } + .qbody ul, .notes-md ul { margin: 0 0 12px; padding-left: 20px; } .qbody li { margin: 3px 0; } + /* a fenced block in a task or a lesson: kept as written, and a click copies it */ + pre.code { margin: 0 0 12px; padding: 10px 12px; font-size: 12.5px; line-height: 1.5; background: var(--pre-bg); border: 1px solid var(--line); + border-radius: 6px; color: var(--text); overflow-x: auto; white-space: pre; cursor: copy; } + pre.code:hover { border-color: var(--link); } + pre.code.copied { border-color: var(--ok-text); } + /* lessons */ + .lstate { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; margin: 16px 0 4px; } + .lstate .ok { color: var(--ok-text); font-weight: 600; } .lstate .no { color: var(--bad-text); font-weight: 600; } + .lstate .lock { color: var(--muted); font-size: 13px; } + .lmark { font: 600 11px var(--sans); text-transform: uppercase; letter-spacing: .04em; padding: 1px 7px; border-radius: 50px; + border: 1px solid var(--line); color: var(--muted); margin-right: 6px; white-space: nowrap; } + .lmark.done { border-color: var(--ok-text); color: var(--ok-text); } + .stepsum { margin: 0 0 14px; color: var(--muted); font-size: 13px; text-transform: uppercase; letter-spacing: .05em; } .checkrow { display: flex; gap: 10px; align-items: center; flex-wrap: wrap; margin-top: 16px; font-size: 13px; } .checks { margin-top: 14px; background: var(--surface-2); border: 1px solid var(--line); border-radius: 6px; padding: 12px 14px; font-size: 14px; } .checks div { padding: 3px 0; } @@ -321,7 +336,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' .score-box { background: var(--score-bg); color: #fff; font-size: 22px; font-weight: 700; padding: 8px 18px; border-radius: 4px; font-variant-numeric: tabular-nums; } .pct { font-size: 30px; font-weight: 800; color: var(--res-head); } .rank-badge { padding: 8px 16px; border-radius: 4px; font-weight: 700; color: #fff; } - .rank-high { background: var(--ok); } .rank-low { background: var(--bad); } + .rank-high { background: var(--ok); } .rank-low { background: var(--bad); } .rank-mid { background: var(--grey); } .bars { margin: 0 0 18px; } .bar { display: grid; grid-template-columns: 76px 1fr 64px; gap: 10px; align-items: center; margin: 6px 0; font-size: 14px; } .bar .track { height: 8px; background: var(--track); border-radius: 5px; overflow: hidden; } @@ -387,7 +402,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '

Your path

One objective at a time against the clock, then a mock exam, then the fixed simulator sets.

-
+

Drill a domain

@@ -450,6 +465,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
+
Notepad - kept with the attempt, not graded
@@ -470,10 +486,10 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
- - + + - +
@@ -482,7 +498,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '

Results

-

Points scored

0 / 0
+

Points scored

0 / 0
@@ -498,6 +514,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
+ @@ -562,7 +579,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '