CHANGELOG.md

@@ -4,64 +4,6 @@ Notable changes to kubestronaut-sim, newest first. A version is the `VERSION` st
 (year.month.day.n); `./ksim help` and the top bar of the web page show the one you are running.
 Every entry names the commit it came in.
 
-## 2026.10.27.2 — 2026-10-27
-
-Everything below came in `3b72021`.
-
-### Changed
-
-- **The 105 CKA solutions rewritten in 2026.10.27.1 are proven**: `./ksim audit` ran each on a
-  host of its own on v1.37, as the solution shows it, every edit by hand done by its script. All
-  105 seed cleanly, score nothing untouched and reach full marks. The README's audit record says
-  so. No task's setup or checks have changed since the audit of 2026-10-19, nor have the other 42
-  solutions.
-
-## 2026.10.27.1 — 2026-10-27
-
-Everything below came in `e761d53`.
-
-### Changed
-
-- **Every CKA reference solution shows the exam way**, as the lessons do: a generator and the few
-  lines to add in `vim`, a manifest pasted from the docs page it names, `kubectl edit`, `ssh NODE`,
-  or a value you read and `echo` - no heredocs, `sed`, JSON patches, `kubectl wait` or loops. 105
-  of the 147 were rewritten; the other 42 were commands you type already. The page and `./ksim
-  solution N` show each thing to type as a block of its own, which a click copies.
-- A solution in parts (`cmd`, `hand`) is its own audit script: `./ksim audit` runs what it shows,
-  and for each edit by hand the script beside it, so the solution you read is the one proven. Their
-  separate scripts in `cka/solve.py` are gone.
-
-### Added
-
-- `./ksim solution lint [N|DOMAIN..]`: what the reference solutions have you type, against the
-  list `./ksim lesson lint` uses.
-
-## 2026.10.26.4 — 2026-10-26
-
-Everything below came in `226ef19`.
-
-### Changed
-
-- **Every CKA lesson step shows the exam way** (*Show me how*, `./ksim lesson show`): `k`, a
-  generator and the few lines to add in `vim`, a manifest pasted from the docs page it names,
-  `kubectl edit`, `ssh NODE` and `sudo -i`, or a value you read and `echo` - never a script. Each
-  command is a block of its own, which a click copies. The scripts the audit needs (`yq`, `sed`,
-  waits) are kept out of sight, one per part done by hand, so `./ksim lesson audit` still proves
-  every step: 135 of 135 on v1.37.
-
-### Added
-
-- `./ksim lesson lint`: what each step has you type, against what you would not type in the exam
-  (`sed`, `awk`, `yq`, heredocs, loops, `kubectl wait`, `--overrides`, JSON patches, `jsonpath`
-  into a file, `kubectl` in full). The lesson audit runs it first.
-- A task's or a lesson's host whose setup failed when the session started is marked: the page
-  says so above it, with **Rebuild this host**, and `./ksim lesson` names `./ksim reset N`. Before,
-  only the log of the preparing window said so.
-
-### Fixed
-
-- The page shows `*italics*` in lessons and tasks instead of the stars.
-
 ## 2026.10.26.3 — 2026-10-26
 
 Everything below came in `95f9d1b`.

README.md

@@ -155,8 +155,7 @@ host name such as `ckad0013` only resolves in the desktop's terminal: from your
 
 **Lessons.** A lesson teaches one objective as a guided lab on a host of its own: an introduction,
 then steps you do in the desktop's terminal, each with a **Check step** button, a hint and **Show me
-how** (the exam way: what you type, what you edit in vim), and a wrap-up that sends you to
+the commands**, and a wrap-up that sends you to the objective's timed drill. Lessons are the one
-the objective's timed drill. Lessons are the one
 untimed part of the simulator — they teach, the drills test — and they write nothing to the
 history, the statistics or the weak list; a lesson whose every step was checked is marked done on
 its objective (`lesson ✓` in the objective list). Steps count in order. Start one from an
@@ -200,7 +199,7 @@ Each track is a directory: `cka/`, `ckad/`, `cks/`.
 | `bank_*.py` | more tasks; `bank_sim_*` are the fixed simulator sets |
 | `curriculum.py` | the exam's objectives in order, each with study notes and the tasks that practise it |
 | `solutions.py` | reference solutions (CKA keeps them apart; the others inline them) |
-| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved - for the tasks whose reference solution is not in parts |
+| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved |
 | `lessons.py` | guided labs, one per objective (CKA: all, one `lessons_*.py` per domain · CKAD: 2.1–2.4): introduction, steps with their check, hint and commands, wrap-up |
 
 | Track | Kubernetes | Bank today | Target (17–20 pool tasks per domain) |
@@ -213,12 +212,9 @@ Each track is a directory: `cka/`, `ckad/`, `cks/`.
 
 **CKA, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 147 tasks seed cleanly on a host of
 their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when
-their scripted solution is run on the host as `candidate`. Q132 really upgrades its cluster from
+their scripted solution is run on the host as `candidate` — for 80 of them that script is the
-v1.36.4 to v1.37.1, worker included. **On 2026-10-27** the 105 solutions rewritten in parts that
+reference solution exactly as a drill shows it once the task is graded. Q132 really upgrades its
-day were audited again: each task seeded cleanly, scored nothing untouched, and reached full marks
+cluster from v1.36.4 to v1.37.1, worker included.
-from what its solution shows, every edit by hand done by its script. No task's setup or checks have
-changed since 2026-10-19. The other 42 solutions are commands with a script in `solve.py`, which
-for 11 of them is the solution exactly as written.
 
 **CKAD, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 105 tasks seed cleanly on a host of
 their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when
@@ -245,8 +241,7 @@ v1.36), CKAD Q1 and CKS Q60, Q61 and Q64 passed again on the rebuilt images.
  "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
  "setup": "bash that seeds the scenario on the task's own cluster",
  "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
- "solution": [cmd("k ... --dry-run=client -o yaml > f.yaml"),   # or one string, a script as it stands
+ "solution": "...",
-              hand("vim f.yaml", "what to add, Markdown", run="the same, scripted: yq, a heredoc")],
  "workers": 2,                                              # optional: worker nodes of its cluster
  "k8s": "v1.36"}                                            # optional: a minor of its own (an upgrade task)
 ```
@@ -291,35 +286,22 @@ and the untouched grade, which catches a check that only passes once the seeded
 until a stopped kubelet shows as NotReady; `--keep` leaves the host up to look at
 (`KUBECONFIG=.state/envs/<host>/kubeconfig`, `./ksim down` removes it).
 
-A reference solution in parts - `cmd()` and `hand()`, as a lesson step's `show` - is its own
+In `cka/solve.py`, `AS_WRITTEN` lists the tasks whose reference solution is a script as it stands:
-script: the audit runs what is typed as it stands, and for each part done by hand its `run`, so the
+for those the audit runs exactly the text the candidate is shown, so a solution that stops working
-solution the candidate reads after a drill is the one that is proven, and one that stops working
+with a new Kubernetes release fails the next audit instead of misleading you after a drill.
-with a new Kubernetes release fails the next audit. 105 of the 147 CKA solutions are written so
-(the page and `./ksim solution N` show each thing to type as a block of its own, which a click
-copies, with the edit to make in `vim` beneath it); the other 42 are commands typed as they stand,
-each with its script in `solve.py`. `./ksim solution lint` reads what they have you type, against
-the same list of things you would not type in the exam as `./ksim lesson lint`. In `cka/solve.py`,
-`AS_WRITTEN` lists the few tasks whose reference solution is still one script, which the audit
-runs as it stands.
 
 **Writing a lesson.** A lesson in `<track>/lessons.py` is a dict with `id` (9000 plus the objective,
 `9201` for 2.1 — its host is `ckad9201`, which no task's can be), `objective`, `title`, `workers`,
 `intro`, `setup` (bash, as a task's), `steps` and `outro` - and, like a task, `k8s` or `image` if
 it needs a host of its own kind (lesson 4.4 runs on v1.36 to upgrade it). Each step comes from
 `step(title, text, check, hint, show)` in `sim/bank.py`: `check` runs like a task's check, and
-`show` is how you would do the step in the exam, as parts. `cmd(text)` is what you type as it
+`show` is what the learner would type. `./ksim lesson audit` seeds each lesson on its own host and,
-stands. `hand(text, how, run)` is what you do by hand: what you type to begin (`vim role.yaml`,
+step by step, requires the check to fail, runs `show` there as `candidate`, and requires the check
-`k edit cm app-config`, `ssh NODE` and `sudo -i`, `echo NAME > file`), what to do there, and a script
+to pass — so a lesson can always be done exactly as it shows, and no step is done before you do it.
-that does the same unattended. `./ksim lesson audit` seeds each lesson on its own host and, step by
+Work on another node is shown as `ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain
-step, requires the check to fail, runs the commands (and each hand part's script) there as
+`ssh NODE` would read the rest of the commands as its own input. **As audited on 2026-10-19**
-`candidate`, and requires the check to pass, so a lesson can always be done as it shows, and no
+(v1.37, Ubuntu hosts): CKA's 27 lessons, 135 of 135 steps, and CKAD 2.1–2.4, 24 of 24 steps,
-step is done before you do it. `./ksim lesson lint` reads what every step has you type and refuses
+proven.
-scripts: no `sed`, `awk`, `yq`, heredocs, loops, `kubectl wait`, `--overrides`, JSON patches or
-`jsonpath` into a file, and `k` rather than `kubectl`. In a script, a command on another node is
-`ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain `ssh NODE` would read the rest of
-the script as its own input. **As audited on 2026-10-26** (v1.37, Ubuntu hosts): CKA's 27 lessons,
-done the exam way, 135 of 135 steps proven, and none has you type a script. CKAD 2.1–2.4 (24
-steps, audited on 2026-10-19) still show their commands as scripts.
 
 ## What is different from the real exam
 

cka/bank_course.py

@@ -12,8 +12,6 @@ operator with broken RBAC do its job, and 146-147 grow, snapshot, restore and cl
 the CSI hostpath driver's class `csi-hostpath-sc`, as the real exam's clusters have it
 (ensure_csi_hostpath).
 """
-
-from sim.bank import cmd, hand
 from cka.common import C, CAN, NET, NS, ONE, backup_operator, ns_pick, name_pick
 
 _ETCD = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
@@ -61,65 +59,38 @@ for n in "$CP" "$W1"; do
 done
 eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" v1.37.1 && eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" 2"""),
  ],
- "solution": [hand(r"""sudo vim /etc/apt/sources.list.d/kubernetes.list""",
+ "solution": """
-                   r"""The docs page *Upgrading kubeadm clusters* starts here: pkgs.k8s.io has one repository per minor, so 1.37 packages exist for apt only once the line says `/v1.37/` instead of `/v1.36/`.""",
+# the control plane - this host
-                   run=r"""apt_get() { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq -y -o Dpkg::Options::=--force-confold "$@" </dev/null >/dev/null; }
+sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list   # one repository per minor
-sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list"""),
+sudo apt-get update
-              hand(r"""sudo apt-get update
 apt-cache madison kubeadm | head -3
-sudo apt-mark unhold kubeadm
+sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm
-sudo apt-get install -y kubeadm=1.37.1-1.1
+sudo kubeadm upgrade plan
-sudo apt-mark hold kubeadm""",
+sudo kubeadm upgrade apply v1.37.1 -y                                       # a few minutes
-                   r"""The control plane - this host - first, and kubeadm first: it drives the upgrade.""",
+k drain {CP} --ignore-daemonsets --delete-emptydir-data
-                   run=r"""apt_get update
+sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl
-sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null"""),
+sudo systemctl daemon-reload && sudo systemctl restart kubelet
-              hand(r"""sudo kubeadm upgrade plan
+k uncordon {CP}
-sudo kubeadm upgrade apply v1.37.1""",
+# the worker
-                   r"""Answer `y`; it takes a few minutes, and the API server restarts on the new version.""",
+k drain {W1} --ignore-daemonsets --delete-emptydir-data
-                   run=r"""sudo kubeadm upgrade apply v1.37.1 -y </dev/null | tail -2"""),
+{SHELL_W1}
-              cmd(r"""k drain {CP} --ignore-daemonsets --delete-emptydir-data"""),
-              hand(r"""sudo apt-mark unhold kubelet kubectl
-sudo apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
-sudo apt-mark hold kubelet kubectl
-sudo systemctl daemon-reload
-sudo systemctl restart kubelet""",
-                   r"""Then the node's kubelet and kubectl, held again.""",
-                   run=r"""sudo apt-mark unhold kubelet kubectl >/dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null
-sudo systemctl daemon-reload && sudo systemctl restart kubelet"""),
-              hand(r"""k uncordon {CP}
-k get nodes""",
-                   r"""Again until `k` answers - the kubelet is restarting. `{CP}` shows `v1.37.1`.""",
-                   run=r"""for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done"""),
-              cmd(r"""k drain {W1} --ignore-daemonsets --delete-emptydir-data"""),
-              hand(r"""{SHELL_W1}
 sudo -i
-vim /etc/apt/sources.list.d/kubernetes.list""",
+sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
-                   r"""`/v1.36/` to `/v1.37/` here too.""",
+apt-get update
-                   run=r""""""),
-              hand(r"""apt-get update
 apt-mark unhold kubeadm kubelet kubectl
 apt-get install -y kubeadm=1.37.1-1.1
-kubeadm upgrade node
+kubeadm upgrade node                                                        # a worker: node, not apply
 apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
 apt-mark hold kubeadm kubelet kubectl
-systemctl daemon-reload
-systemctl restart kubelet
-exit
-exit""",
-                   r"""A worker runs `kubeadm upgrade node`, not `apply`: it hosts no control plane, only its kubelet's configuration to update.""",
-                   run=r"""ssh {W1} sudo bash <<'EOF'
-export DEBIAN_FRONTEND=noninteractive
-sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
-apt-get update -qq </dev/null
-apt-mark unhold kubeadm kubelet kubectl >/dev/null
-apt-get install -y -qq kubeadm=1.37.1-1.1 </dev/null >/dev/null
-kubeadm upgrade node </dev/null | tail -1
-apt-get install -y -qq kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 </dev/null >/dev/null
-apt-mark hold kubeadm kubelet kubectl >/dev/null
 systemctl daemon-reload && systemctl restart kubelet
-EOF"""),
+exit; exit
-              cmd(r"""k uncordon {W1}
+k uncordon {W1}
-k get nodes""", r"""Both `Ready`, both `v1.37.1` - `VERSION` is each kubelet's.""")],
+k get nodes
+# docs: "Upgrading kubeadm clusters". pkgs.k8s.io has one repository per minor, so 1.37 packages
+# appear only once the line names v1.37. kubeadm first - it drives the upgrade and rewrites the
+# kubelet's configuration; kubelet and kubectl after `upgrade apply`, then the restart: `k get
+# nodes` shows each kubelet's running version. Workers run `kubeadm upgrade node`.
+""",
 },
 # ------------------------------------------------------------------ 133 ----
 {
@@ -225,16 +196,19 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.ephemeralContainers[?(@.name=="debugger")].t
   && eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q134.uid") 0" """),
   C("mode.txt holds the value the process runs with", 3, ONE + r"""eq "$(one "$WS/q134/mode.txt")" %{mode}"""),
  ],
- "solution": [cmd(r"""k -n %{ns} exec %{pod} -- sh
+ "solution": """
-k -n %{ns} get cm %{pod}-env -o yaml""", r"""No shell in the image. And the ConfigMap says `MODE=open` - now; the environment was read once, when the container started."""),
+k -n %{ns} exec %{pod} -- sh                        # fails: no shell in the image
-              cmd(r"""k -n %{ns} debug %{pod} --image=busybox:1.36 --target=app -c debugger --profile=general -- sleep 3600""", r"""`--target` puts the debug container into `app`'s process namespace; the `general` profile gives it `SYS_PTRACE`, which reading another process's environment needs. Interactively: `k debug -it ... -- sh`."""),
+k -n %{ns} get cm %{pod}-env -o yaml                # MODE=open - but that is the ConfigMap NOW
-              hand(r"""k -n %{ns} exec %{pod} -c debugger -- ps""",
+k -n %{ns} debug %{pod} --image=busybox:1.36 --target=app -c debugger --profile=general -- sleep 3600
-                   r"""Again until it answers: PID 1 is `/pause`, `app`'s process.""",
+for i in $(seq 1 30); do k -n %{ns} exec %{pod} -c debugger -- true 2>/dev/null && break; sleep 2; done
-                   run=r"""for i in $(seq 1 30); do kubectl -n %{ns} exec %{pod} -c debugger -- true 2>/dev/null && break; sleep 2; done"""),
+k -n %{ns} exec %{pod} -c debugger -- ps                   # PID 1 is /pause, the app's process
-              cmd(r"""k -n %{ns} exec %{pod} -c debugger -- cat /proc/1/environ | tr '\0' '\n'""", r"""`MODE=%{mode}`: what the process runs with."""),
+k -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\\0" "\\n" < /proc/1/environ' | grep '^MODE='
-              hand(r"""echo %{mode} > {WS}/q134/mode.txt""",
+k -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\\0" "\\n" < /proc/1/environ' | sed -n 's/^MODE=//p' > {WS}/q134/mode.txt
-                   r"""With the value after `MODE=`.""",
+# Environment variables from a ConfigMap are read once, when the container starts: editing the
-                   run=r"""kubectl -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\0" "\n" < /proc/1/environ' | sed -n 's/^MODE=//p' > {WS}/q134/mode.txt""")],
+# ConfigMap changes nothing for a running process. --target puts the debug container into app's
+# process namespace, so /proc/1 is app's process; the `general` profile gives it SYS_PTRACE,
+# which reading another user's /proc/PID/environ needs. Interactively: `k debug -it ...  -- sh`.
+""",
 },
 # ------------------------------------------------------------------ 135 ----
 {
@@ -279,19 +253,19 @@ tok = [v for v in s.get("volumes", []) if any("serviceAccountToken" in src for s
 sys.exit(1 if tok else 0)' || exit 1
 done"""),
  ],
- "solution": [cmd(r"""k -n %{ns} create serviceaccount pipeline
+ "solution": """
+k -n %{ns} create serviceaccount pipeline
 k -n %{ns} create role pipeline --verb=get,list,create,update,patch --resource=deployments.apps
 k -n %{ns} create rolebinding pipeline --role=pipeline --serviceaccount=%{ns}:pipeline
-k auth can-i delete deployments -n %{ns} --as=system:serviceaccount:%{ns}:pipeline""", r"""`no`."""),
+k auth can-i delete deployments -n %{ns} --as=system:serviceaccount:%{ns}:pipeline     # no
-              cmd(r"""k -n %{ns} create token pipeline --duration=%{h}h > {WS}/q135/pipeline.token""", r"""The TokenRequest API issues a short-lived, bound token; nothing is stored in a Secret."""),
+k -n %{ns} create token pipeline --duration=%{h}h > {WS}/q135/pipeline.token
-              hand(r"""k -n %{ns} edit deploy frontend""",
+k -n %{ns} patch deploy frontend -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":false}}}}'
-                   r"""In the Pod template's `spec:`, beside `containers:` and indented like it, then `:wq`:
+k -n %{ns} rollout status deploy frontend
-```
+# `create token` asks the TokenRequest API for a short-lived, bound token - nothing is stored in a
-      automountServiceAccountToken: false
+# Secret. automountServiceAccountToken: false on the Pod template (or on the ServiceAccount; the
-```
+# Pod's setting wins) leaves out the projected volume kube-api-access-*. To try a token alone, use
-It leaves out the projected volume `kube-api-access-*`. (On the ServiceAccount it works too; the Pod's setting wins.)""",
+# no kubeconfig: a client certificate in yours would be checked first.
-                   run=r"""kubectl -n %{ns} patch deploy frontend -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":false}}}}'"""),
+""",
-              cmd(r"""k -n %{ns} rollout status deploy frontend""")],
 },
 # ------------------------------------------------------------------ 136 ----
 {
@@ -324,20 +298,22 @@ eq "$(x theme)" dark && eq "$(x locale)" en_GB && eq "$(x db/password)" pw-%{mem
 x() { $K -n %{ns} exec %{pod} -- cat "/etc/portal/$1" 2>/dev/null; }
 x labels | grep -qx 'app="%{pod}"' && eq "$(x mem-limit)" %{mem}"""),
  ],
- "solution": [cmd(r"""k -n %{ns} run %{pod} --image=busybox:1.36 --labels=app=%{pod} --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
+ "solution": """
-              hand(r"""vim pod.yaml""",
+k -n %{ns} apply -f - <<'EOF'
-                   r"""Replace the container's `resources: {}` with the limit and the mount:
+apiVersion: v1
-```
+kind: Pod
+metadata:
+  name: %{pod}
+  labels: {app: %{pod}}
+spec:
+  containers:
+  - name: portal
+    image: busybox:1.36
+    command: [sleep, '86400']
     resources:
-      limits:
+      limits: {memory: %{mem}Mi}
-        memory: %{mem}Mi
     volumeMounts:
-    - name: portal
+    - {name: portal, mountPath: /etc/portal, readOnly: true}
-      mountPath: /etc/portal
-      readOnly: true
-```
-and add to the Pod's `spec:`, right below `spec:` (the docs page *Projected Volumes* has an example with all three kinds):
-```
   volumes:
   - name: portal
     projected:
@@ -347,33 +323,20 @@ and add to the Pod's `spec:`, right below `spec:` (the docs page *Projected Volu
       - secret:
           name: portal-db
           items:
-          - key: password
+          - {key: password, path: db/password}
-            path: db/password
       - downwardAPI:
           items:
           - path: labels
-            fieldRef:
+            fieldRef: {fieldPath: metadata.labels}
-              fieldPath: metadata.labels
           - path: mem-limit
-            resourceFieldRef:
+            resourceFieldRef: {containerName: portal, resource: limits.memory, divisor: 1Mi}
-              containerName: %{pod}
+EOF
-              resource: limits.memory
+k -n %{ns} wait --for=condition=Ready pod/%{pod}
-              divisor: 1Mi
+k -n %{ns} exec %{pod} -- sh -c 'ls -R /etc/portal; cat /etc/portal/labels /etc/portal/mem-limit'
-```
+# `items` picks keys and renames them: only `password` from the Secret, as db/password. A
-`items` picks keys and renames them; in a volume a `resourceFieldRef` needs the container's name, and the divisor turns bytes into MiB.""",
+# resourceFieldRef needs the container's name in a volume (in an env var it is implied), and the
-                   run=r"""yq -i '.spec.containers[0].resources = {"limits": {"memory": "%{mem}Mi"}}
+# divisor turns bytes into MiB.
-  | .spec.containers[0].volumeMounts = [{"name": "portal", "mountPath": "/etc/portal", "readOnly": true}]
+""",
-  | .spec.volumes = [{"name": "portal", "projected": {"sources": [
-      {"configMap": {"name": "portal-config"}},
-      {"secret": {"name": "portal-db", "items": [{"key": "password", "path": "db/password"}]}},
-      {"downwardAPI": {"items": [{"path": "labels", "fieldRef": {"fieldPath": "metadata.labels"}},
-                                 {"path": "mem-limit", "resourceFieldRef": {"containerName": "%{pod}", "resource": "limits.memory", "divisor": "1Mi"}}]}}]}}]' pod.yaml"""),
-              cmd(r"""k apply -f pod.yaml"""),
-              hand(r"""k -n %{ns} get pod %{pod}""",
-                   r"""Again until it is `Running`.""",
-                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=120s"""),
-              cmd(r"""k -n %{ns} exec %{pod} -- ls -R /etc/portal
-k -n %{ns} exec %{pod} -- cat /etc/portal/labels /etc/portal/mem-limit""")],
 },
 # ------------------------------------------------------------------ 137 ----
 {
@@ -441,19 +404,18 @@ eq "$(j -n %{ns} deploy critical '{.status.availableReplicas} {.status.updatedRe
 eq "$(j -n %{ns} deploy critical '{.spec.template.spec.containers[0].resources.requests.cpu} {.spec.template.spec.containers[0].resources.limits.cpu} {.spec.template.spec.containers[0].resources.requests.memory} {.spec.template.spec.containers[0].resources.limits.memory}')" "200m 200m 128Mi 128Mi" || exit 1
 for q in $($K -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'); do [ "$q" = Guaranteed ] || exit 1; done"""),
  ],
- "solution": [cmd(r"""k -n %{ns} get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass""", r"""Guaranteed: every container has requests equal to limits, CPU and memory. BestEffort: no container has any. Everything else is Burstable - `delta` too, as one of its containers sets nothing."""),
+ "solution": """
-              hand(r"""vim {WS}/q137/qos.txt""",
+k -n %{ns} get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
-                   r"""```
+k -n %{ns} get pods alpha beta delta gamma -o jsonpath='{range .items[*]}{.metadata.name} {.status.qosClass}{"\\n"}{end}' > {WS}/q137/qos.txt
-alpha Guaranteed
+echo gamma > {WS}/q137/first.txt
-beta Burstable
+k -n %{ns} set resources deploy critical --requests=cpu=200m,memory=128Mi --limits=cpu=200m,memory=128Mi
-gamma BestEffort
-delta Burstable
-```""",
-                   run=r"""printf 'alpha Guaranteed\nbeta Burstable\ngamma BestEffort\ndelta Burstable\n' > {WS}/q137/qos.txt"""),
-              cmd(r"""echo gamma > {WS}/q137/first.txt""", r"""Under memory pressure BestEffort goes first, then Burstable Pods using more than they requested, Guaranteed last."""),
-              cmd(r"""k -n %{ns} set resources deploy critical --requests=cpu=200m,memory=128Mi --limits=cpu=200m,memory=128Mi
 k -n %{ns} rollout status deploy critical
-k -n %{ns} get pods -l app=critical -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass""")],
+k -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'; echo
+# Guaranteed: every container has requests equal to limits for both CPU and memory. BestEffort: no
+# container has any. Everything else is Burstable - delta too, because one container sets nothing.
+# Under memory pressure the kubelet evicts BestEffort first, then Burstable Pods using more than
+# they requested; Guaranteed Pods last.
+""",
 },
 # ------------------------------------------------------------------ 138 ----
 {
@@ -490,21 +452,15 @@ eq "$(j -n %{ns} pod %{pod} '{.status.containerStatuses[0].resources.requests.cp
 eq "$(j -n %{ns} pod %{pod} '{.spec.containers[0].resources.limits.cpu}')" %{lim} \
   && eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q138.uid") 0" """),
  ],
- "solution": [cmd(r"""k -n %{ns} get pod %{pod} -o yaml | grep -A6 resources:"""),
+ "solution": """
-              hand(r"""k -n %{ns} edit pod %{pod} --subresource resize""",
+k -n %{ns} get pod %{pod} -o jsonpath='{.spec.containers[0].resources}'; echo
-                   r"""Through the `resize` subresource - a plain `k edit` of a Pod's resources is refused. Set the CPU, leave the memory, then `:wq`:
+k -n %{ns} patch pod %{pod} --subresource resize --patch '{"spec":{"containers":[{"name":"encoder","resources":{"requests":{"cpu":"%{req}"},"limits":{"cpu":"%{lim}"}}}]}}'
-```
+k -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].resources}{"\\n"}{.status.containerStatuses[0].restartCount}{"\\n"}'
-    resources:
+# A running Pod's CPU and memory can be changed in place, through the `resize` subresource - a
-      limits:
+# plain `k edit` or `k apply` of a Pod's resources is refused. The spec
-        cpu: %{lim}
+# says what you asked for; status.containerStatuses[].resources what the kubelet has applied.
-        memory: 48Mi
+# A container's resizePolicy decides whether a resource's change needs a restart (CPU: no).
-      requests:
+""",
-        cpu: %{req}
-        memory: 48Mi
-```""",
-                   run=r"""kubectl -n %{ns} patch pod %{pod} --subresource resize --patch '{"spec":{"containers":[{"name":"encoder","resources":{"requests":{"cpu":"%{req}"},"limits":{"cpu":"%{lim}"}}}]}}'"""),
-              cmd(r"""k -n %{ns} get pod %{pod} -o yaml | grep -A6 resources:
-k -n %{ns} get pod %{pod}""", r"""The spec says what you asked for, `status.containerStatuses[].resources` what the kubelet applied; `RESTARTS 0` - a container's `resizePolicy` decides, and CPU needs no restart.""")],
 },
 # ------------------------------------------------------------------ 139 ----
 {
@@ -539,33 +495,24 @@ sys.exit(0 if s == "%{soft}Mi" and secs == 90 else 1)'"""),
 nx "$W1" 'grep -q "%{soft}Mi" /var/lib/kubelet/config.yaml && grep -q "%{mem}Mi" /var/lib/kubelet/config.yaml && grep -q "^staticPodPath:" /var/lib/kubelet/config.yaml && systemctl is-active -q kubelet' </dev/null || exit 1
 eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True"""),
  ],
- "solution": [hand(r"""{SHELL_W1}
+ "solution": """
+{SHELL_W1}
 sudo -i
-grep -n -A4 ^eviction /var/lib/kubelet/config.yaml
+grep -n -A4 '^eviction' /var/lib/kubelet/config.yaml       # kind sets the hard thresholds to 0%
-vim /var/lib/kubelet/config.yaml""",
+yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%"
-                   r"""Under `evictionHard:` set the two hard thresholds, and add the soft one with its grace period at the top level - a soft threshold without a grace period for the same signal, and the kubelet refuses to start:
+  | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
-```
-evictionHard:
-  memory.available: "%{mem}Mi"
-  nodefs.available: "%{fs}%"
-evictionSoft:
-  memory.available: "%{soft}Mi"
-evictionSoftGracePeriod:
-  memory.available: "1m30s"
-```
-Keep the other `evictionHard` entries as they are.""",
-                   run=r"""ssh {W1} sudo bash <<'EOF'
-yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
 systemctl restart kubelet
-EOF"""),
-              hand(r"""systemctl restart kubelet
 journalctl -u kubelet -n 5 --no-pager
-exit
+exit; exit
-exit""",
+k get --raw /api/v1/nodes/{W1}/proxy/configz | jq '.kubeletconfig | {evictionHard, evictionSoft, evictionSoftGracePeriod}'
-                   r"""No complaint about the configuration; then back on the host.""",
+# A soft threshold needs a grace period for the same signal, or the kubelet refuses to start. By
-                   run=r"""for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done"""),
+# hand: under evictionHard add `memory.available: "%{mem}Mi"` and change nodefs.available; then
-              cmd(r"""k get node {W1}
+#   evictionSoft:
-k get --raw /api/v1/nodes/{W1}/proxy/configz | tr ',' '\n' | grep -i eviction""", r"""`Ready`; `configz` shows the configuration the running kubelet uses.""")],
+#     memory.available: "%{soft}Mi"
+#   evictionSoftGracePeriod:
+#     memory.available: "1m30s"
+# configz shows what the running kubelet uses.
+""",
 },
 # ------------------------------------------------------------------ 140 ----
 {
@@ -601,24 +548,19 @@ ch=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A
 n=$(nx "$CP" "iptables-save -t nat | grep -- '^-A $ch ' | grep -c -- '-j KUBE-SEP-'" </dev/null)
 eq "$n" %{n} && eq "$(one "$WS/q140/endpoints.txt")" "$n" """),
  ],
- "solution": [cmd(r"""k -n kube-system get cm kube-proxy -o yaml | grep mode""", r"""`mode: iptables` - an empty mode means the default, iptables too."""),
+ "solution": """
-              hand(r"""echo iptables > {WS}/q140/mode.txt""",
+k -n kube-system get cm kube-proxy -o jsonpath='{.data.config\\.conf}' | grep '^mode'       # iptables
-                   r"""The mode it printed.""",
+echo iptables > {WS}/q140/mode.txt
-                   run=r"""m=$(kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"')
+ip=$(k -n %{ns} get svc %{svc} -o jsonpath='{.spec.clusterIP}')
-echo ${m:-iptables} > {WS}/q140/mode.txt"""),
+sudo iptables-save -t nat | grep -- "-d $ip/32"                 # KUBE-SERVICES ... -j KUBE-SVC-XXXX
-              cmd(r"""k -n %{ns} get svc %{svc}""", r"""Note its `CLUSTER-IP`."""),
+chain=$(sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1)
-              hand(r"""sudo iptables-save -t nat | grep 10.96.12.34""",
+echo $chain > {WS}/q140/chain.txt
-                   r"""With that IP: `KUBE-SERVICES` matches the ClusterIP and port and jumps (`-j`) to the Service's own `KUBE-SVC-...` chain.""",
+sudo iptables-save -t nat | grep -- "^-A $chain "               # one -j KUBE-SEP-... per endpoint
-                   run=r""""""),
+sudo iptables-save -t nat | grep -- "^-A $chain " | grep -c -- '-j KUBE-SEP-' > {WS}/q140/endpoints.txt
-              hand(r"""echo KUBE-SVC-ABCDEFGHIJKLMNOP > {WS}/q140/chain.txt
+# KUBE-SERVICES matches ClusterIP:port and jumps to the Service's KUBE-SVC chain; that picks an
-sudo iptables-save -t nat | grep -- '-A KUBE-SVC-ABCDEFGHIJKLMNOP '""",
+# endpoint chain at random (statistic mode random), and each KUBE-SEP chain DNATs to one Pod.
-                   r"""With the chain's name. Its rules pick one endpoint chain at random (`statistic mode random`); each `KUBE-SEP-...` DNATs to one Pod - count them. In nftables mode: `sudo nft list ruleset | grep -A5 %{svc}`.""",
+# One KUBE-SEP per ready endpoint. In nftables mode: `sudo nft list ruleset | grep -A5 %{svc}`.
-                   run=r"""ip=$(kubectl -n %{ns} get svc %{svc} -o jsonpath='{.spec.clusterIP}')
+""",
-sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/q140/chain.txt"""),
-              hand(r"""echo 3 > {WS}/q140/endpoints.txt""",
-                   r"""With the number of `-j KUBE-SEP-` rules: one per ready endpoint.""",
-                   run=r"""chain=$(cat {WS}/q140/chain.txt)
-sudo iptables-save -t nat | grep -- "^-A $chain " | grep -c -- '-j KUBE-SEP-' > {WS}/q140/endpoints.txt""")],
 },
 # ------------------------------------------------------------------ 141 ----
 {
@@ -651,65 +593,36 @@ eq "$(j -n %{ns} endpointslice billing-1 '{.metadata.labels.kubernetes\.io/servi
   C("wget billing from client returns the API's answer", 3, r"""
 for i in 1 2 3 4 5; do [ "$($K -n %{ns} exec client -- wget -qO- -T 3 billing 2>/dev/null)" = billing-api-%{port} ] && exit 0; sleep 3; done; exit 1"""),
  ],
- "solution": [cmd(r"""cat {WS}/q141/billing.txt""", r"""`ADDRESS:PORT`."""),
+ "solution": """
-              hand(r"""vim billing.yaml""",
+cat {WS}/q141/billing.txt                                       # ADDRESS:PORT
-                   r"""A Service with no selector - no controller writes its endpoints - and the EndpointSlice you maintain, tied to it by the label `kubernetes.io/service-name` (the docs page *Service* has both under *Services without selectors*). With the address and port from the file:
+addr=$(cut -d: -f1 {WS}/q141/billing.txt); port=$(cut -d: -f2 {WS}/q141/billing.txt)
-```
+k -n %{ns} apply -f - <<EOF
 apiVersion: v1
 kind: Service
 metadata:
   name: billing
-  namespace: %{ns}
 spec:
   ports:
-  - port: 80
+  - {port: 80, protocol: TCP}
-    protocol: TCP
 ---
 apiVersion: discovery.k8s.io/v1
 kind: EndpointSlice
 metadata:
   name: billing-1
-  namespace: %{ns}
   labels:
     kubernetes.io/service-name: billing
 addressType: IPv4
 endpoints:
-- addresses:
+- addresses: ["$addr"]
-  - 172.18.0.3
 ports:
-- port: %{port}
+- {port: $port, protocol: TCP}
-  protocol: TCP
+EOF
-```
+k -n %{ns} exec client -- wget -qO- -T3 billing
-The slice's port is where traffic goes - the Service has no `targetPort` to speak of - and its name must match the Service port's: here both unnamed.""",
+# Without a selector, no controller writes endpoints for the Service: the label
-                   run=r"""addr=$(cut -d: -f1 {WS}/q141/billing.txt); port=$(cut -d: -f2 {WS}/q141/billing.txt)
+# kubernetes.io/service-name ties a slice to it. The slice's port is where traffic goes (the
-cat > billing.yaml <<EOF
+# Service's targetPort plays no part), and its name must match the Service port's - here both
-apiVersion: v1
+# unnamed. ExternalName would need a DNS name; an address needs this.
-kind: Service
+""",
-metadata:
-  name: billing
-  namespace: %{ns}
-spec:
-  ports:
-  - port: 80
-    protocol: TCP
----
-apiVersion: discovery.k8s.io/v1
-kind: EndpointSlice
-metadata:
-  name: billing-1
-  namespace: %{ns}
-  labels:
-    kubernetes.io/service-name: billing
-addressType: IPv4
-endpoints:
-- addresses:
-  - "$addr"
-ports:
-- port: $port
-  protocol: TCP
-EOF"""),
-              cmd(r"""k apply -f billing.yaml
-k -n %{ns} exec client -- wget -qO- -T3 billing""", r"""`billing-api-%{port}`. (`ExternalName` would need a DNS name; an address needs this.)""")],
 },
 # ------------------------------------------------------------------ 142 ----
 {
@@ -747,24 +660,8 @@ $K -n %{ns} create secret generic ksim-probe --from-literal=x=y >/dev/null 2>&1
 nx "$CP" '""" + _ETCD + r""" get /registry/secrets/%{ns}/ksim-probe --print-value-only 2>/dev/null | head -c 64' </dev/null | grep -aq '^k8s:enc:aescbc:v1:'; r=$?
 $K -n %{ns} delete secret ksim-probe --ignore-not-found >/dev/null 2>&1; exit $r"""),
  ],
- "solution": [cmd(r"""head -c 32 /dev/urandom | base64""", r"""The host is the control-plane node. Copy the output: that is the key."""),
+ "solution": """
-              hand(r"""sudo mkdir -p /etc/kubernetes/enc
+sudo mkdir -p /etc/kubernetes/enc
-sudo vim /etc/kubernetes/enc/enc.yaml""",
-                   r"""Paste the example from the docs page *Encrypting Confidential Data at Rest* (search for `aescbc`) and cut it down to:
-```
-apiVersion: apiserver.config.k8s.io/v1
-kind: EncryptionConfiguration
-resources:
-- resources: [secrets]
-  providers:
-  - aescbc:
-      keys:
-      - name: key1
-        secret: <your key>
-  - identity: {}
-```
-The first provider encrypts; `identity` after it still reads what was stored before.""",
-                   run=r"""sudo mkdir -p /etc/kubernetes/enc
 sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
 apiVersion: apiserver.config.k8s.io/v1
 kind: EncryptionConfiguration
@@ -776,31 +673,23 @@ resources:
       - name: key1
         secret: $(head -c 32 /dev/urandom | base64)
   - identity: {}
-EOF"""),
+EOF
-              cmd(r"""sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/""", r"""Edit a copy: the kubelet acts on every save in `manifests/`, half-done edits too."""),
+sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak     # a backup OUTSIDE manifests/
-              hand(r"""sudo vim ~/kube-apiserver.yaml""",
+sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
-                   r"""Add three lines, each beside its kind (`yy` and `p` on an existing `k8s-certs` mount or volume, then change it):
+sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
-```
-    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
-```
-in the `command:` list,
-```
-    - {name: enc, mountPath: /etc/kubernetes/enc, readOnly: true}
-```
-under `volumeMounts:`, and
-```
-  - {name: enc, hostPath: {path: /etc/kubernetes/enc, type: DirectoryOrCreate}}
-```
-under `volumes:` - the API server runs in a container, and sees only what is mounted.""",
-                   run=r"""sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
   | .spec.containers[0].volumeMounts += [{"name": "enc", "mountPath": "/etc/kubernetes/enc", "readOnly": true}]
-  | .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' ~/kube-apiserver.yaml"""),
+  | .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' /tmp/kube-apiserver.yaml
-              cmd(r"""sudo cp ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
+sudo cp /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
-              hand(r"""k -n kube-system get pods""",
+sleep 20; for i in $(seq 1 40); do k get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
-                   r"""Again until it answers: the API server is back in a minute or so.""",
+k -n %{ns} get secret %{sec} -o json | k replace -f -                 # stored again: encrypted now
-                   run=r"""sleep 20; for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
+sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \\
-              cmd(r"""k -n %{ns} get secret %{sec} -o yaml | k replace -f -""", r"""Written again, the Secret is stored encrypted; every Secret at once: `k get secrets -A -o json | k replace -f -`."""),
+  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \\
-              cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/%{ns}/%{sec} | head -c 80""", r"""`k8s:enc:aescbc:v1:key1:` and no plain text.""")],
+  get /registry/secrets/%{ns}/%{sec} | od -c | head -3                 # k 8 s : e n c : a e s c b c : v 1 : k e y 1
+# By hand: in kube-apiserver.yaml add the flag to the command, a volumeMount for /etc/kubernetes/enc
+# and a hostPath volume of the same path - the API server runs in a container and sees only what is
+# mounted. The FIRST provider encrypts; `identity` after it reads what was stored before. Every
+# Secret: `k get secrets -A -o json | k replace -f -`.
+""",
 },
 # ------------------------------------------------------------------ 143 ----
 {
@@ -841,39 +730,20 @@ sys.exit(0 if ok else 1)'"""),
 eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas} {.status.updatedReplicas} {.status.replicas}')" "3 3 3" || exit 1
 [ "$($K -n %{ns} get pods -l app=%{dep} --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 3 ]"""),
  ],
- "solution": [cmd(r"""k label node {W2} {W3} tier=edge"""),
+ "solution": """
-              hand(r"""k -n %{ns} edit deploy %{dep}""",
+k label node {W2} {W3} tier=edge
-                   r"""In the Pod template's `spec:`, above `containers:` (the docs page *Assigning Pods to Nodes* has both kinds):
+k -n %{ns} patch deploy %{dep} --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}},"template":{"spec":{"affinity":{
-```
-      affinity:
-        podAntiAffinity:
-          requiredDuringSchedulingIgnoredDuringExecution:
-          - labelSelector:
-              matchLabels:
-                app: %{dep}
-            topologyKey: kubernetes.io/hostname
-        nodeAffinity:
-          preferredDuringSchedulingIgnoredDuringExecution:
-          - weight: 100
-            preference:
-              matchExpressions:
-              - key: tier
-                operator: In
-                values:
-                - edge
-```
-and in the same edit, under `strategy:`, let the rollout replace one Pod at a time:
-```
-    rollingUpdate:
-      maxSurge: 0
-      maxUnavailable: 1
-```
-With 3 workers and one Pod each, a surge Pod has nowhere to go, and 25% of 3 unavailable rounds down to 0: the default rollout would wait for ever. Then `:wq`.""",
-                   run=r"""kubectl -n %{ns} patch deploy %{dep} --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}},"template":{"spec":{"affinity":{
   "podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app":"%{dep}"}},"topologyKey":"kubernetes.io/hostname"}]},
-  "nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"tier","operator":"In","values":["edge"]}]}}]}}}}}}'"""),
+  "nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"tier","operator":"In","values":["edge"]}]}}]}}}}}}'
-              cmd(r"""k -n %{ns} rollout status deploy %{dep}
+k -n %{ns} rollout status deploy %{dep}
-k -n %{ns} get pods -o wide""", r"""Three nodes, one Pod each. The anti-affinity's selector names the Pods to keep away from - its own - and `topologyKey` per what: one per hostname, one per node.""")],
+k -n %{ns} get pods -o wide
+# The anti-affinity's labelSelector picks the Pods to stay away from - its own, by its template's
+# labels - and topologyKey says per what: one per hostname = one per node. With 3 workers a 4th
+# replica would stay Pending - and so does the rolling update's surge Pod, since every node already
+# has one of the old Pods (same labels). The default maxUnavailable of 25% is 0 Pods for 3
+# replicas, so the rollout would wait for ever: let it replace one Pod at a time instead
+# (maxSurge 0, maxUnavailable 1), in the same change.
+""",
 },
 # ------------------------------------------------------------------ 144 ----
 {
@@ -917,18 +787,15 @@ eq "$(j -n %{ns} pod %{sts}-0 '{.spec.containers[0].image}') $(j -n %{ns} pod %{
 eq "$(for i in 0 1; do j -n %{ns} pod %{sts}-$i '{.metadata.uid}'; echo; done)" "$(cat "$PRIV/q144.uids")" \
   && eq "$(j -n %{ns} sts %{sts} '{.spec.template.spec.containers[0].image}')" nginx:1.27"""),
  ],
- "solution": [hand(r"""k -n %{ns} edit sts %{sts}""",
+ "solution": """
-                   r"""The partition first - a StatefulSet updates from the highest ordinal down, and without one the new image would reach all three within moments. Under `updateStrategy:`, then `:wq`:
+k -n %{ns} patch sts %{sts} -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'
-```
+k -n %{ns} set image sts/%{sts} nginx=nginx:1.27
-  updateStrategy:
-    rollingUpdate:
-      partition: 2
-    type: RollingUpdate
-```""",
-                   run=r"""kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'"""),
-              cmd(r"""k -n %{ns} set image sts %{sts} nginx=nginx:1.27
 k -n %{ns} rollout status sts %{sts}
-k -n %{ns} get pods -l app=%{sts} -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image""", r"""Ordinals at or above the partition get the new template: `%{sts}-2` alone. The next steps of the canary: partition 1, then 0.""")],
+k -n %{ns} get pods -l app=%{sts} -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image
+# The partition FIRST: a StatefulSet updates from the highest ordinal down, and with no partition
+# the new image would reach -1 and -0 within moments. Pods with an ordinal >= partition get the new
+# template; the others keep theirs. Next step of the canary: partition 1, then 0.
+""",
 },
 # ------------------------------------------------------------------ 145 ----
 {
@@ -958,53 +825,32 @@ eq "$(j crd backups.data.example.com '{.status.conditions[?(@.type=="Established
   C("Backup %{name} is Scheduled: the controller may update its status", 3, r"""
 eq "$(j -n %{ns} backup %{name} '{.status.phase}')" Scheduled && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller"""),
  ],
- "solution": [cmd(r"""grep -e ^kind: -e '^  name:' {WS}/q145/operator.yaml
+ "solution": """
-k apply -f {WS}/q145/operator.yaml"""),
+grep -E '^kind:|^  name:' {WS}/q145/operator.yaml
-              hand(r"""k get crd backups.data.example.com
+k apply -f {WS}/q145/operator.yaml
-k -n backup-operator get pods""",
+k wait --for condition=established crd/backups.data.example.com
-                   r"""Again until the controller is `Running`.""",
+k -n backup-operator rollout status deploy controller
-                   run=r"""kubectl wait --for condition=established crd/backups.data.example.com --timeout=60s
+k create ns %{ns}
-kubectl -n backup-operator rollout status deploy controller --timeout=180s"""),
+k -n %{ns} apply -f - <<'EOF'
-              cmd(r"""k create ns %{ns}"""),
-              hand(r"""vim backup.yaml""",
-                   r"""```
 apiVersion: data.example.com/v1
 kind: Backup
 metadata:
   name: %{name}
-  namespace: %{ns}
 spec:
   schedule: "30 1 * * *"
-```""",
+EOF
-                   run=r"""cat > backup.yaml <<'EOF'
+sleep 10
-apiVersion: data.example.com/v1
+k -n %{ns} get cm backup-%{name}                                  # made - but:
-kind: Backup
+k -n %{ns} get backup %{name} -o jsonpath='{.status}'; echo       # no phase
-metadata:
+k -n backup-operator logs deploy/controller | tail -3             # ... cannot patch resource "backups/status" ...
-  name: %{name}
+k auth can-i patch backups.data.example.com --subresource=status --as=system:serviceaccount:backup-operator:controller -A
-  namespace: %{ns}
+k patch clusterrole backup-operator --type json -p '[{"op":"add","path":"/rules/0/resources/-","value":"backups/status"}]'
-spec:
+sleep 10
-  schedule: "30 1 * * *"
+k -n %{ns} get backup %{name} -o jsonpath='{.status.phase}'; echo
-EOF"""),
+# The CRD has a status subresource, and a subresource is a resource of its own for RBAC:
-              cmd(r"""k apply -f backup.yaml"""),
+# `backups/status`. The controller could read Backups and write ConfigMaps, but not set a phase.
-              hand(r"""k -n %{ns} get cm backup-%{name}
+# The log of an operator's controller is where such a gap shows - here as `forbidden`.
-k -n %{ns} get backup %{name} -o yaml | grep -A1 status
+""",
-k -n backup-operator logs deploy/controller | tail -3""",
-                   r"""The ConfigMap is made, but no `phase`: the log says `cannot patch resource "backups/status"`. The CRD has a status subresource, and for RBAC a subresource is a resource of its own.""",
-                   run=r"""sleep 10"""),
-              cmd(r"""k auth can-i patch backups.data.example.com --subresource=status --as=system:serviceaccount:backup-operator:controller -A""", r"""`no`."""),
-              hand(r"""k edit clusterrole backup-operator""",
-                   r"""Add `backups/status` to the first rule's resources, then `:wq` - the one thing wrong:
-```
-- apiGroups:
-  - data.example.com
-  resources:
-  - backups
-  - backups/status
-```""",
-                   run=r"""kubectl patch clusterrole backup-operator --type json -p '[{"op":"add","path":"/rules/0/resources/-","value":"backups/status"}]'"""),
-              hand(r"""k -n %{ns} get backup %{name} -o yaml | grep -A1 status""",
-                   r"""Again until `phase: Scheduled` - the controller tries every 5 seconds.""",
-                   run=r"""for i in $(seq 1 20); do [ "$(kubectl -n %{ns} get backup %{name} -o jsonpath='{.status.phase}')" = Scheduled ] && break; sleep 3; done""")],
 },
 # ------------------------------------------------------------------ 146 ----
 {
@@ -1035,66 +881,41 @@ eq "$(j -n %{ns} pod web-server "{.spec.volumes[?(@.name==\"$v\")].persistentVol
 eq "$(j -n %{ns} pvc pv-volume '{.spec.resources.requests.storage} {.status.capacity.storage}')" "%{size} %{size}" || exit 1
 eq "$(j -n %{ns} pod web-server '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """),
  ],
- "solution": [cmd(r"""k get sc csi-hostpath-sc""", r"""`ALLOWVOLUMEEXPANSION true`: a claim of this class may grow."""),
+ "solution": """
-              hand(r"""vim pvc.yaml""",
+k get sc csi-hostpath-sc                         # ALLOWVOLUMEEXPANSION: true - a claim of this class may grow
-                   r"""```
+k -n %{ns} apply -f - <<'EOF'
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: pv-volume
-  namespace: %{ns}
 spec:
   storageClassName: csi-hostpath-sc
-  accessModes:
+  accessModes: [ReadWriteOnce]
-  - ReadWriteOnce
   resources:
-    requests:
+    requests: {storage: 10Mi}
-      storage: 10Mi
+---
-```""",
-                   run=r"""cat > pvc.yaml <<'EOF'
 apiVersion: v1
-kind: PersistentVolumeClaim
+kind: Pod
 metadata:
-  name: pv-volume
+  name: web-server
-  namespace: %{ns}
 spec:
-  storageClassName: csi-hostpath-sc
+  containers:
-  accessModes:
+  - name: web-server
-  - ReadWriteOnce
+    image: nginx:1.27
-  resources:
-    requests:
-      storage: 10Mi
-EOF"""),
-              cmd(r"""k apply -f pvc.yaml
-k -n %{ns} run web-server --image=nginx:1.27 --dry-run=client -o yaml > pod.yaml"""),
-              hand(r"""vim pod.yaml""",
-                   r"""Add to the container, below `name: web-server`:
-```
     volumeMounts:
-    - name: html
+    - {name: html, mountPath: /usr/share/nginx/html}
-      mountPath: /usr/share/nginx/html
-```
-and to the Pod's `spec:`, right below `spec:`:
-```
   volumes:
   - name: html
-    persistentVolumeClaim:
+    persistentVolumeClaim: {claimName: pv-volume}
-      claimName: pv-volume
+EOF
-```""",
+k -n %{ns} wait --for=condition=Ready pod/web-server
-                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "html", "mountPath": "/usr/share/nginx/html"}]
+k -n %{ns} patch pvc pv-volume -p '{"spec":{"resources":{"requests":{"storage":"%{size}"}}}}'
-  | .spec.volumes = [{"name": "html", "persistentVolumeClaim": {"claimName": "pv-volume"}}]' pod.yaml"""),
+k -n %{ns} get pvc pv-volume                     # CAPACITY follows once the driver has resized it
-              cmd(r"""k apply -f pod.yaml"""),
+# `k edit pvc pv-volume` works just as well. A claim only grows, and only when its class has
-              hand(r"""k -n %{ns} get pod web-server""",
+# allowVolumeExpansion: true and its driver can expand; spec.resources.requests is what you ask
-                   r"""Again until it is `Running`.""",
+# for, status.capacity what you have. Filesystem volumes of this driver grow online - the Pod
-                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/web-server --timeout=180s"""),
+# keeps running.
-              hand(r"""k -n %{ns} edit pvc pv-volume""",
+""",
-                   r"""Under `spec.resources.requests`, then `:wq` - a claim only grows:
-```
-      storage: %{size}
-```""",
-                   run=r"""kubectl -n %{ns} patch pvc pv-volume -p '{"spec":{"resources":{"requests":{"storage":"%{size}"}}}}'"""),
-              cmd(r"""k -n %{ns} get pvc pv-volume
-k -n %{ns} get pod web-server""", r"""Again until `CAPACITY %{size}`: `spec.resources.requests` is what you asked for, `status.capacity` what the driver has done. This driver grows filesystems online - the Pod keeps running.""")],
 },
 # ------------------------------------------------------------------ 147 ----
 {
@@ -1142,132 +963,71 @@ x() { $K -n %{ns} exec auditor -- cat "$1/ledger.txt" 2>/dev/null; }
 eq "$(x /restore)" entries-%{n} && eq "$(x /copy)" entries-%{n} \
   && eq "$(j -n %{ns} pod auditor '{.spec.volumes[*].persistentVolumeClaim.claimName}' | tr ' ' '\n' | sort | tr '\n' ' ')" "ledger-copy ledger-restore " """),
  ],
- "solution": [cmd(r"""k get volumesnapshotclass""", r"""`csi-hostpath-snapclass`, driver `hostpath.csi.k8s.io`."""),
+ "solution": """
-              hand(r"""vim snap.yaml""",
+k get volumesnapshotclass                        # csi-hostpath-snapclass, driver hostpath.csi.k8s.io
-                   r"""The docs page *Volume Snapshots* has the VolumeSnapshot:
+k -n %{ns} apply -f - <<'EOF'
-```
 apiVersion: snapshot.storage.k8s.io/v1
 kind: VolumeSnapshot
 metadata:
   name: ledger-snap
-  namespace: %{ns}
 spec:
   volumeSnapshotClassName: csi-hostpath-snapclass
   source:
     persistentVolumeClaimName: ledger-data
-```""",
+EOF
-                   run=r"""cat > snap.yaml <<'EOF'
+k -n %{ns} wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/ledger-snap --timeout=120s
-apiVersion: snapshot.storage.k8s.io/v1
+k -n %{ns} apply -f - <<'EOF'
-kind: VolumeSnapshot
-metadata:
-  name: ledger-snap
-  namespace: %{ns}
-spec:
-  volumeSnapshotClassName: csi-hostpath-snapclass
-  source:
-    persistentVolumeClaimName: ledger-data
-EOF"""),
-              cmd(r"""k apply -f snap.yaml"""),
-              hand(r"""k -n %{ns} get volumesnapshot ledger-snap""",
-                   r"""Again until `READYTOUSE true`.""",
-                   run=r"""kubectl -n %{ns} wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/ledger-snap --timeout=120s"""),
-              hand(r"""vim claims.yaml""",
-                   r"""`dataSource` makes a claim from a snapshot (*Volume Snapshots*) or from another claim of the same class and namespace - a clone (*CSI Volume Cloning*; no `apiGroup`, claims are core). Both at least as big as the source:
-```
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: ledger-restore
-  namespace: %{ns}
 spec:
   storageClassName: csi-hostpath-sc
+  accessModes: [ReadWriteOnce]
+  resources:
+    requests: {storage: 50Mi}
   dataSource:
     apiGroup: snapshot.storage.k8s.io
     kind: VolumeSnapshot
     name: ledger-snap
-  accessModes:
-  - ReadWriteOnce
-  resources:
-    requests:
-      storage: 50Mi
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: ledger-copy
-  namespace: %{ns}
 spec:
   storageClassName: csi-hostpath-sc
+  accessModes: [ReadWriteOnce]
+  resources:
+    requests: {storage: 50Mi}
   dataSource:
     kind: PersistentVolumeClaim
     name: ledger-data
-  accessModes:
-  - ReadWriteOnce
-  resources:
-    requests:
-      storage: 50Mi
-```""",
-                   run=r"""cat > claims.yaml <<'EOF'
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: ledger-restore
-  namespace: %{ns}
-spec:
-  storageClassName: csi-hostpath-sc
-  dataSource:
-    apiGroup: snapshot.storage.k8s.io
-    kind: VolumeSnapshot
-    name: ledger-snap
-  accessModes:
-  - ReadWriteOnce
-  resources:
-    requests:
-      storage: 50Mi
 ---
 apiVersion: v1
-kind: PersistentVolumeClaim
+kind: Pod
 metadata:
-  name: ledger-copy
+  name: auditor
-  namespace: %{ns}
 spec:
-  storageClassName: csi-hostpath-sc
+  containers:
-  dataSource:
+  - name: auditor
-    kind: PersistentVolumeClaim
+    image: busybox:1.36
-    name: ledger-data
+    command: [sleep, '86400']
-  accessModes:
-  - ReadWriteOnce
-  resources:
-    requests:
-      storage: 50Mi
-EOF"""),
-              cmd(r"""k apply -f claims.yaml
-k -n %{ns} run auditor --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
-              hand(r"""vim pod.yaml""",
-                   r"""Add to the container, below `name: auditor`:
-```
     volumeMounts:
-    - name: restore
+    - {name: restore, mountPath: /restore}
-      mountPath: /restore
+    - {name: copy, mountPath: /copy}
-    - name: copy
-      mountPath: /copy
-```
-and to the Pod's `spec:`, right below `spec:`:
-```
   volumes:
   - name: restore
-    persistentVolumeClaim:
+    persistentVolumeClaim: {claimName: ledger-restore}
-      claimName: ledger-restore
   - name: copy
-    persistentVolumeClaim:
+    persistentVolumeClaim: {claimName: ledger-copy}
-      claimName: ledger-copy
+EOF
-```""",
+k -n %{ns} wait --for=condition=Ready pod/auditor --timeout=180s
-                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "restore", "mountPath": "/restore"}, {"name": "copy", "mountPath": "/copy"}]
+k -n %{ns} exec auditor -- cat /restore/ledger.txt /copy/ledger.txt
-  | .spec.volumes = [{"name": "restore", "persistentVolumeClaim": {"claimName": "ledger-restore"}}, {"name": "copy", "persistentVolumeClaim": {"claimName": "ledger-copy"}}]' pod.yaml"""),
+# A snapshot is taken by the CSI driver (VolumeSnapshotClass -> driver); `dataSource` makes a new
-              cmd(r"""k apply -f pod.yaml"""),
+# claim from it - apiGroup snapshot.storage.k8s.io - or from another claim of the same class and
-              hand(r"""k -n %{ns} get pvc,pod""",
+# Namespace, a clone (no apiGroup: claims are core). Both need a CSI driver that can do it, and
-                   r"""Again until both claims are `Bound` and `auditor` `Running`.""",
+# the new claim must be at least as big as the source.
-                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/auditor --timeout=180s"""),
+""",
-              cmd(r"""k -n %{ns} exec auditor -- cat /restore/ledger.txt /copy/ledger.txt""", r"""`entries-%{n}`, twice.""")],
 },
 ]

cka/bank_hard.py

@@ -186,9 +186,7 @@ $K delete ns %{ns} --ignore-not-found --wait=false >/dev/null
 },
 {
  "id": 49, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults",
- # low, short: the plural's stem and the shortName, which the solution spells out
+ "params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])},
- "params": lambda r: (lambda p: dict(p, low=p["kind"].lower(), short=p["kind"][:2].lower()))(
-     {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])}),
  "task": CTX + """
 Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the kind in lowercase plus
 `s`, shortName = the first two letters of the kind in lowercase) with schema:

cka/bank_net.py

@@ -5,8 +5,6 @@ Written for the per-task hosts: a task may rewrite the cluster's CoreDNS configu
 resolution and hostPorts are tested for real; Ingress and Gateway objects are graded as objects,
 as no controller for them runs on the hosts.
 """
-
-from sim.bank import cmd, hand
 from cka.common import C, DNS, NS, ONE, PODS, PY, ns_pick, name_pick
 
 QUESTIONS = [
@@ -155,102 +153,43 @@ parents == [("%{gw}", "%{ns}")] and s.get("hostnames") == ["%{host}.example.com"
 out=$($K -n %{ns} get ingress %{host} 2>&1); case "$out" in *NotFound*) ;; *) exit 1 ;; esac
 eq "$(j -n %{ns} secret %{host}-tls '{.metadata.uid}')" "$(cat "$PRIV/q126.uid")" """),
  ],
- "solution": [cmd(r"""k -n %{ns} get ingress %{host} -o yaml""", r"""The host, the TLS Secret `%{host}-tls`, `/api` to `api:8080` and `/` to `web:80`."""),
+ "solution": """
-              hand(r"""vim gw.yaml""",
+k -n %{ns} get ingress %{host} -o yaml     # host, the TLS Secret (%{host}-tls), /api -> api:8080, / -> web:80
-                   r"""What the Ingress keeps in one object, the Gateway API splits: the listener - port, hostname, certificate - goes in the Gateway, the routing in the HTTPRoute. The docs page *Gateway API* has both; `k explain gateway.spec.listeners.tls` the TLS fields:
+cat <<EOF | k apply -f -
-```
 apiVersion: gateway.networking.k8s.io/v1
 kind: Gateway
-metadata:
+metadata: {name: %{gw}, namespace: %{ns}}
-  name: %{gw}
-  namespace: %{ns}
 spec:
   gatewayClassName: %{gc}
   listeners:
-  - name: https
+    - name: https
-    protocol: HTTPS
+      protocol: HTTPS
-    port: 443
+      port: 443
-    hostname: %{host}.example.com
+      hostname: %{host}.example.com
-    tls:
+      tls:
-      mode: Terminate
+        mode: Terminate
-      certificateRefs:
+        certificateRefs: [{kind: Secret, name: %{host}-tls}]
-      - kind: Secret
-        name: %{host}-tls
 ---
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
-metadata:
+metadata: {name: %{host}, namespace: %{ns}}
-  name: %{host}
-  namespace: %{ns}
 spec:
-  parentRefs:
+  parentRefs: [{name: %{gw}}]
-  - name: %{gw}
+  hostnames: [%{host}.example.com]
-  hostnames:
-  - %{host}.example.com
   rules:
-  - matches:
+    - matches: [{path: {type: PathPrefix, value: /api}}]
-    - path:
+      backendRefs: [{name: api, port: 8080}]
-        type: PathPrefix
+    - matches: [{path: {type: PathPrefix, value: /}}]
-        value: /api
+      backendRefs: [{name: web, port: 80}]
-    backendRefs:
+EOF
-    - name: api
-      port: 8080
-  - matches:
-    - path:
-        type: PathPrefix
-        value: /
-    backendRefs:
-    - name: web
-      port: 80
-```""",
-                   run=r"""cat > gw.yaml <<'EOF'
-apiVersion: gateway.networking.k8s.io/v1
-kind: Gateway
-metadata:
-  name: %{gw}
-  namespace: %{ns}
-spec:
-  gatewayClassName: %{gc}
-  listeners:
-  - name: https
-    protocol: HTTPS
-    port: 443
-    hostname: %{host}.example.com
-    tls:
-      mode: Terminate
-      certificateRefs:
-      - kind: Secret
-        name: %{host}-tls
----
-apiVersion: gateway.networking.k8s.io/v1
-kind: HTTPRoute
-metadata:
-  name: %{host}
-  namespace: %{ns}
-spec:
-  parentRefs:
-  - name: %{gw}
-  hostnames:
-  - %{host}.example.com
-  rules:
-  - matches:
-    - path:
-        type: PathPrefix
-        value: /api
-    backendRefs:
-    - name: api
-      port: 8080
-  - matches:
-    - path:
-        type: PathPrefix
-        value: /
-    backendRefs:
-    - name: web
-      port: 80
-EOF"""),
-              cmd(r"""k apply -f gw.yaml
 k -n %{ns} get gateway,httproute
-k -n %{ns} delete ingress %{host}""")],
+k -n %{ns} delete ingress %{host}
+# What an Ingress keeps in one object, the Gateway API splits by role: the Gateway is the
+# listener - port, hostname, certificate - and belongs to whoever runs the edge; the HTTPRoute is
+# the routing and belongs to the application. spec.tls of the Ingress becomes tls.certificateRefs
+# on the listener, each path becomes a rule with a PathPrefix match, and a Service backend becomes
+# a backendRef. `k explain gateway.spec.listeners.tls` and the Gateway API docs are allowed.
+""",
 },
 # ------------------------------------------------------------------ 127 ----
 {
@@ -460,37 +399,23 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.hostNetwork}')" true || exit 1
 pip=$(j -n %{ns} pod %{pod} '{.status.podIP}'); hip=$(j -n %{ns} pod %{pod} '{.status.hostIP}')
 [ -n "$pip" ] && eq "$pip" "$hip" && eq "$(tr -s ' \t\r\n' ' ' < "$WS/q128/ips.txt" 2>/dev/null | sed 's/^ //; s/ $//')" "$pip $hip" """),
  ],
- "solution": [hand(r"""k -n %{ns} edit ds %{ds}""",
+ "solution": """
-                   r"""Give the container's port a `hostPort`, then `:wq`:
+k -n %{ns} edit ds %{ds}                   # under the container:
-```
+#   ports: [{containerPort: 8080, hostPort: %{hp}}]
-        ports:
+k -n %{ns} rollout status ds %{ds}
-        - containerPort: 8080
+k get nodes -o wide                        # INTERNAL-IP
-          hostPort: %{hp}
+curl -s http://<a worker's INTERNAL-IP>:%{hp}/            # agent on <that node>
-          protocol: TCP
+
-```
+k -n %{ns} logs %{pod} --tail=2            # bad address %{svc}
-`hostPort` maps one port of the node to the Pod on it (the CNI's `portmap` plugin writes the rule); `hostNetwork` would put the whole Pod in the node's network.""",
+k -n %{ns} get pod %{pod} -o jsonpath='{.spec.dnsPolicy}'; echo       # ClusterFirst - which a hostNetwork Pod does not get
-                   run=r"""kubectl -n %{ns} patch ds %{ds} --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/ports/0/hostPort", "value": %{hp}}]'"""),
+vi {WS}/q128/collector.yaml                # spec.dnsPolicy: ClusterFirstWithHostNet
-              cmd(r"""k -n %{ns} rollout status ds %{ds}
+k -n %{ns} delete pod %{pod}; k apply -f {WS}/q128/collector.yaml
-k get nodes -o wide""", r"""Note a worker's `INTERNAL-IP`."""),
+k -n %{ns} logs %{pod} --tail=2            # resolved %{svc}
-              hand(r"""curl -s http://172.18.0.3:%{hp}/""",
+k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt
-                   r"""With that IP: `agent on <that node>`.""",
+# hostPort maps one port of the node to one Pod on it (the CNI's portmap plugin writes the rule);
-                   run=r""""""),
+# hostNetwork puts the whole Pod into the node's network namespace - its IP IS the node's IP, and
-              cmd(r"""k -n %{ns} logs %{pod} --tail=2
+# it uses the node's /etc/resolv.conf, which knows nothing of cluster.local, unless dnsPolicy is
-k -n %{ns} get pod %{pod} -o yaml | grep -e hostNetwork -e dnsPolicy""", r"""`bad address`: a `hostNetwork` Pod uses the node's `/etc/resolv.conf`, which knows nothing of `cluster.local` - unless its `dnsPolicy` is `ClusterFirstWithHostNet`."""),
+# ClusterFirstWithHostNet. dnsPolicy cannot be changed on a running Pod: recreate it.
-              hand(r"""vim {WS}/q128/collector.yaml""",
+""",
-                   r"""In the Pod's `spec:`, below `hostNetwork: true`:
-```
-  dnsPolicy: ClusterFirstWithHostNet
-```""",
-                   run=r"""yq -i '.spec.dnsPolicy = "ClusterFirstWithHostNet"' {WS}/q128/collector.yaml"""),
-              cmd(r"""k replace --force -f {WS}/q128/collector.yaml""", r"""A running Pod's `dnsPolicy` cannot change: `--force` deletes it and creates it again."""),
-              hand(r"""k -n %{ns} logs %{pod} --tail=2
-k -n %{ns} get pod %{pod} -o wide""",
-                   r"""Again until `resolved %{svc}`. Its IP is its node's: note it.""",
-                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=180s
-sleep 8"""),
-              hand(r"""echo "172.18.0.3 172.18.0.3" > {WS}/q128/ips.txt""",
-                   r"""The Pod's IP and its node's - the same address.""",
-                   run=r"""kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt""")],
 },
 ]

cka/bank_objectives.py

@@ -20,7 +20,7 @@ kubeadm, etcd, the runtime. All normal difficulty, all in the random pool; nothi
 node or a control-plane component (109 cordons the control-plane node while you work on it and has
 you uncordon it; the session baseline uncordons it if you do not).
 """
-from sim.bank import C, NS, ns_pick, name_pick, cmd, hand
+from sim.bank import C, NS, ns_pick, name_pick
 from ckad import bank_extra as _ckad_extra, bank_more as _ckad_more
 
 
@@ -30,34 +30,18 @@ def _port(bank, src_id, new_id, domain, **over):
     old, new = f"q{src_id}", f"q{new_id}"
     q = dict(src, id=new_id, domain=domain, **over)
     for k in ("task", "setup", "solution"):
-        if k not in over:   # a solution of its own (in parts) is the CKA one already
+        q[k] = src[k].replace(old, new)
-            q[k] = src[k].replace(old, new)
     q["checks"] = [dict(c, script=c["script"].replace(old, new), name=c["name"].replace(old, new)) for c in src["checks"]]
     return q
 
 
-# 103's own solution: the CKAD task's, done as in the exam
-_SOL103 = [hand(r"""k -n %{ns} top pod""",
-                r"""Again until it shows all three: metrics-server needs about 30 seconds after the Pods start. Without it, `Metrics API not available`.""",
-                run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} top pod --no-headers 2>/dev/null | wc -l)" -ge 3 ] && break; sleep 5; done"""),
-           cmd(r"""k -n %{ns} top pod --sort-by=cpu
-k -n %{ns} top pod --sort-by=memory""", r"""The first line of each is the heaviest."""),
-           hand(r"""echo %{cpu} > {WS}/q103/cpu.txt
-echo %{mem} > {WS}/q103/memory.txt""",
-                r"""With the names on those first lines.""",
-                run=r"""kubectl -n %{ns} top pod --sort-by=cpu --no-headers | head -1 | awk '{print $1}' > {WS}/q103/cpu.txt
-kubectl -n %{ns} top pod --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/q103/memory.txt"""),
-           cmd(r"""echo 'kubectl top node' > {WS}/q103/nodes.sh
-echo 'kubectl top pod -A --sort-by=memory' > {WS}/q103/pods.sh""", r"""`k top` reads the metrics API (`metrics.k8s.io`) that metrics-server serves; `--containers` breaks a Pod down per container.""")]
-
-
 ETCDCTL = ("etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
            "--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key")
 
 QUESTIONS = [
 # ---------------------------------------------------------------- TSHOOT ----
 # needs a working metrics API, which Q23 breaks on purpose: never in one session with it
-_port(_ckad_extra, 23, 103, "TSHOOT", breaks="metrics-server", solution=_SOL103),
+_port(_ckad_extra, 23, 103, "TSHOOT", breaks="metrics-server"),
 _port(_ckad_more, 64, 104, "TSHOOT"),
 # ------------------------------------------------------------------ WORK ----
 {
@@ -199,44 +183,32 @@ cg=$(nx "$W3" "awk '/^cgroupDriver:/{print \$2}' /var/lib/kubelet/config.yaml" <
 want=$(printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s' "$fwd" "$swap" "$fso" "$cg")
 eq "$(tr -d ' \t\r' < "$WS/q107/node.txt" 2>/dev/null | grep .)" "$want" """),
  ],
- "solution": [hand(r"""{SHELL_W3}
+ "solution": """
+{SHELL_W3}
 sudo -i
-vim /etc/modules-load.d/%{name}.conf""",
+printf 'overlay\\nbr_netfilter\\n' > /etc/modules-load.d/%{name}.conf
-                   r"""The docs page *Container Runtimes* has both files under *Install and configure prerequisites*. One module per line:
+cat > /etc/sysctl.d/%{name}.conf <<EOF
-```
-overlay
-br_netfilter
-```""",
-                   run=r"""ssh -n {W3} 'printf "overlay\nbr_netfilter\n" | sudo tee /etc/modules-load.d/%{name}.conf >/dev/null'"""),
-              hand(r"""vim /etc/sysctl.d/%{name}.conf""",
-                   r"""```
 net.bridge.bridge-nf-call-iptables  = 1
 net.bridge.bridge-nf-call-ip6tables = 1
 net.ipv4.ip_forward                 = 1
-```
+EOF
-On a real machine they would now be loaded and applied without a reboot: `modprobe overlay`, `modprobe br_netfilter`, `sysctl --system`. `sysctl -w` and `modprobe` alone are gone after a reboot; the files make them stick.""",
+# on a real machine you now load and apply them without a reboot:
-                   run=r"""ssh -n {W3} 'printf "net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\n" | sudo tee /etc/sysctl.d/%{name}.conf >/dev/null'"""),
+#   modprobe overlay && modprobe br_netfilter && sysctl --system
-              hand(r"""sysctl -n net.ipv4.ip_forward
+sysctl -n net.ipv4.ip_forward                              # 1
-cat /proc/swaps
+cat /proc/swaps                                            # any line under the header = swap is on  (swapon --show)
-grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml
+grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml
-exit
+exit; exit
-exit""",
+cat > {WS}/q107/node.txt <<EOF
-                   r"""Any line below the header of `/proc/swaps` means swap is on. No `failSwapOn` line means the default, `true`.""",
+ip_forward=1
-                   run=r""""""),
+swap=<on|off>
-              hand(r"""vim {WS}/q107/node.txt""",
+fail-swap-on=<true|false>
-                   r"""What the node printed:
+cgroup-driver=<systemd|cgroupfs>
-```
+EOF
-ip_forward=<0 or 1>
+# `sysctl -w` and `modprobe` alone are gone after a reboot - the files are what makes them stick.
-swap=<on or off>
+# kubeadm's preflight refuses a node with swap on unless the kubelet is told to tolerate it
-fail-swap-on=<true or false>
+# (failSwapOn: false), and the kubelet's cgroup driver has to match the container runtime's
-cgroup-driver=<systemd or cgroupfs>
+# (SystemdCgroup = true in /etc/containerd/config.toml goes with cgroupDriver: systemd).
-```
+""",
-kubeadm's preflight refuses a node with swap on unless the kubelet tolerates it (`failSwapOn: false`); and the kubelet's cgroup driver has to match the runtime's (`SystemdCgroup = true` in containerd goes with `cgroupDriver: systemd`).""",
-                   run=r"""IPF=$(ssh -n {W3} 'sudo sysctl -n net.ipv4.ip_forward')
-SW=off; [ "$(ssh -n {W3} 'sed 1d /proc/swaps | grep -c .')" -gt 0 ] && SW=on
-FS=$(ssh -n {W3} "sudo awk '/^failSwapOn:/ {print \$2}' /var/lib/kubelet/config.yaml"); [ -n "$FS" ] || FS=true
-CG=$(ssh -n {W3} "sudo awk '/^cgroupDriver:/ {print \$2}' /var/lib/kubelet/config.yaml")
-printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s\n' "$IPF" "$SW" "$FS" "$CG" > {WS}/q107/node.txt""")],
 },
 {
  "id": 108, "domain": "ARCH", "title": "The container runtime behind a Pod",
@@ -291,35 +263,29 @@ cid=$($K -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].cont
 log=$(nx "$W2" "crictl inspect -o go-template --template '{{.status.logPath}}' $cid 2>/dev/null" </dev/null | tr -d '[:space:]')
 [ -n "$cid" ] && [ -n "$log" ] && tr -d ' \t\r' < "$WS/q108/runtime.txt" 2>/dev/null | grep -qx "log=$log" """),
  ],
- "solution": [cmd(r"""k -n %{ns} get pod %{pod} -o wide""", r"""On `{W2}`."""),
+ "solution": """
-              hand(r"""{SHELL_W2}
+k -n %{ns} get pod %{pod} -o wide                       # on {W2}
+{SHELL_W2}
 sudo -i
-crictl version
+crictl version                                          # RuntimeName: containerd   RuntimeVersion: v2.x.y
-grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
+grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml       # unix:///run/containerd/containerd.sock
-crictl ps --no-trunc --name app""",
+crictl pods --name %{pod}                               # the Pod sandbox
-                   r"""`RuntimeName` and `RuntimeVersion`; the CRI socket the kubelet uses - crictl speaks the same protocol, which is why it works even when the API server does not; and the container's full ID (`--no-trunc`), the line whose `POD` is `%{pod}`.""",
+crictl ps --no-trunc --name app --pod $(crictl pods --name %{pod} -q)     # full container ID
-                   run=r""""""),
+crictl inspect <id> | grep -E '"pid"|logPath'           # .info.pid and .status.logPath
-              hand(r"""crictl inspect 0123abcd... | grep -e '"pid"' -e logPath
+#   or:  crictl inspect -o go-template --template '{{.info.pid}} {{.status.logPath}}' <id>
-exit
+exit; exit
-exit""",
+cat > {WS}/q108/runtime.txt <<EOF
-                   r"""With that ID: `.info.pid` and `.status.logPath`. The log is the file `kubectl logs` reads, under `/var/log/pods`.""",
-                   run=r""""""),
-              hand(r"""vim {WS}/q108/runtime.txt""",
-                   r"""The values you read:
-```
 runtime=containerd <version>
 socket=unix:///run/containerd/containerd.sock
-container-id=<the 64 hex characters>
+container-id=<64 hex characters>
-pid=<the number>
+pid=<number>
 log=/var/log/pods/%{ns}_%{pod}_<pod uid>/app/0.log
-```""",
+EOF
-                   run=r"""V=$(ssh -n {W2} 'sudo crictl version 2>/dev/null')
+# The kubelet drives the runtime over the CRI socket, and crictl speaks the same protocol - which
-RN=$(echo "$V" | awk '/^RuntimeName:/ {print $2}'); RV=$(echo "$V" | awk '/^RuntimeVersion:/ {print $2}')
+# is why it still works when the API server does not. `kubectl get pod -o
-SOCK=$(ssh -n {W2} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
+# jsonpath='{.status.containerStatuses[0].containerID}'` shows the same ID with a containerd://
-CID=$(kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#^[a-z-]*://##')
+# prefix; /var/log/pods/... is the file `kubectl logs` reads.
-PID=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.info.pid}}' $CID 2>/dev/null")
+""",
-LOG=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.status.logPath}}' $CID 2>/dev/null")
-printf 'runtime=%s %s\nsocket=%s\ncontainer-id=%s\npid=%s\nlog=%s\n' "$RN" "$RV" "$SOCK" "$CID" "$PID" "$LOG" > {WS}/q108/runtime.txt""")],
 },
 {
  # drains the control-plane node: CoreDNS moves off it, so never beside the tasks that break DNS
@@ -502,33 +468,29 @@ c=$(j -n kube-system lease kube-controller-manager '{.spec.holderIdentity}' | cu
 f=$(tr -d ' \t\r' < "$WS/q111/ha.txt" 2>/dev/null)
 [ -n "$s" ] && [ -n "$c" ] && echo "$f" | grep -qx "scheduler-leader=$s" && echo "$f" | grep -qx "controller-manager-leader=$c" """),
  ],
- "solution": [cmd(r"""k get nodes -l node-role.kubernetes.io/control-plane
+ "solution": """
-k -n kube-system get pods -l component=etcd -o wide
+k get nodes -l node-role.kubernetes.io/control-plane              # how many
-k -n kube-system get lease kube-scheduler kube-controller-manager""", r"""An etcd static Pod on the control-plane node: stacked. A Lease's `HOLDER` is `<node>_<uuid>`."""),
+k -n kube-system get pods -l component=etcd -o wide               # an etcd static Pod on the control-plane node = stacked
-              hand(r"""sudo -i
+k -n kube-system get lease kube-scheduler kube-controller-manager # HOLDER is <node>_<uuid>
-export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
+sudo -i                # the task's host is the control-plane node: no hop needed
-etcdctl member list -w table
+export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \\
-etcdctl endpoint status -w table
+       ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
-exit""",
+etcdctl member list -w table                                      # one line per member: ID, NAME, peer and client URLs
-                   r"""The host is the control-plane node. One line per member; `IS LEADER` in the status (`--cluster` asks every member).""",
+etcdctl endpoint status -w table                                  # IS LEADER (add --cluster to ask every member)
-                   run=r""""""),
+exit
-              hand(r"""vim {WS}/q111/ha.txt""",
+cat > {WS}/q111/ha.txt <<EOF
-                   r"""```
 control-plane-nodes=1
 etcd-topology=stacked
 etcd-members=1
 etcd-leader={CP}
 scheduler-leader={CP}
 controller-manager-leader={CP}
-```
+EOF
-In an HA control plane every API server serves at once, but one scheduler and one controller manager act, through the leader Lease. etcd elects its own leader and needs a majority to stay writable: three members survive the loss of one, five of two.""",
+# In an HA control plane every kube-apiserver serves at once, but only one kube-scheduler and one
-                   run=r"""E='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
+# kube-controller-manager act: they elect a leader through a Lease in kube-system, and the others
-N=$(kubectl get nodes -l node-role.kubernetes.io/control-plane --no-headers | wc -l)
+# wait. etcd elects its own leader and needs a majority of members to stay writable - three
-M=$($E member list | wc -l)
+# members survive the loss of one, five of two, and an even number adds nothing.
-L=$($E member list | awk -F', ' 'NR==1 {print $3}')
+""",
-S=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
-C=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
-printf 'control-plane-nodes=%s\netcd-topology=stacked\netcd-members=%s\netcd-leader=%s\nscheduler-leader=%s\ncontroller-manager-leader=%s\n' "$N" "$M" "$L" "$S" "$C" > {WS}/q111/ha.txt""")],
 },
 {
  "id": 112, "domain": "ARCH", "title": "CNI, CSI and CRI: what this cluster plugs in",
@@ -570,35 +532,30 @@ prov=$($K get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclas
 f=$(tr -d ' \t\r' < "$WS/q112/interfaces.txt" 2>/dev/null)
 echo "$f" | grep -qx "csi-drivers=$n" && echo "$f" | grep -qx "default-provisioner=$prov" """),
  ],
- "solution": [hand(r"""{SHELL_W1}
+ "solution": """
+{SHELL_W1}
 sudo -i
-ls /etc/cni/net.d
+ls /etc/cni/net.d/                                    # 10-kindnet.conflist - the kubelet's runtime takes the first file
-cat /etc/cni/net.d/10-kindnet.conflist
+cat /etc/cni/net.d/10-kindnet.conflist                # "plugins": [{"type": "ptp", "ipam": {"type": "host-local"}}, {"type": "portmap"}]
+ls /opt/cni/bin                                       # the plugin binaries those types name
 grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
-exit
+exit; exit
-exit""",
+k get csidrivers                                      # No resources found = 0
-                   r"""The runtime takes the first file of `/etc/cni/net.d`. Its `plugins` chain, in order - each `type` names a binary in `/opt/cni/bin` - and the `ipam` type inside one of them.""",
+k get sc                                              # the (default) class and its PROVISIONER
-                   run=r""""""),
+cat > {WS}/q112/interfaces.txt <<EOF
-              cmd(r"""k get csidrivers
-k get sc""", r"""`No resources found`: no CSI driver - the local-path provisioner is not one. The `(default)` class's `PROVISIONER`."""),
-              hand(r"""vim {WS}/q112/interfaces.txt""",
-                   r"""The values you read - here:
-```
 cni-config=/etc/cni/net.d/10-kindnet.conflist
 cni-plugins=ptp,portmap
 cni-ipam=host-local
 cri-socket=unix:///run/containerd/containerd.sock
 csi-drivers=0
 default-provisioner=rancher.io/local-path
-```
+EOF
-`ptp` wires each Pod's veth pair, `host-local` hands out addresses from the node's Pod CIDR, `portmap` serves hostPorts.""",
+# CRI: the kubelet talks to the runtime over that socket (crictl uses the same one). CNI: the
-                   run=r"""CONF=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
+# runtime calls the plugins of the chain in order for every Pod sandbox - here ptp wires the veth
-PL=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[].type] | join(",")')
+# pair, host-local hands out an address from the node's Pod CIDR, portmap serves hostPorts.
-IPAM=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[] | select(.ipam) | .ipam.type][0]')
+# CSI: a storage driver registers a CSIDriver object and a StorageClass names it as provisioner;
-SOCK=$(ssh -n {W1} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
+# this cluster's local-path provisioner is not a CSI driver, hence none.
-N=$(kubectl get csidrivers --no-headers 2>/dev/null | wc -l)
+""",
-PROV=$(kubectl get sc -o json | jq -r '.items[] | select(.metadata.annotations["storageclass.kubernetes.io/is-default-class"] == "true") | .provisioner')
-printf 'cni-config=%s\ncni-plugins=%s\ncni-ipam=%s\ncri-socket=%s\ncsi-drivers=%s\ndefault-provisioner=%s\n' "$CONF" "$PL" "$IPAM" "$SOCK" "$N" "$PROV" > {WS}/q112/interfaces.txt""")],
 },
 # ------------------------------------------------------------------- NET ----
 {
@@ -666,18 +623,19 @@ eq "$(tr -s ' \t' ' ' < "$WS/q113/pod-cidrs.txt" 2>/dev/null | tr -d '\r' | sed
   C("outside.txt names %{edge}, why.txt names hostNetwork", 2, r"""
 eq "$(tr -d '[:space:]' < "$WS/q113/outside.txt" 2>/dev/null | sed 's#^pod/##')" %{edge} && grep -qi 'hostNetwork' "$WS/q113/why.txt" 2>/dev/null"""),
  ],
- "solution": [cmd(r"""k -n %{ns} get pods -o wide
+ "solution": """
-k get nodes -o wide""", r"""The `IP` and `NODE` of every Pod. `%{edge}`'s IP is its node's own `INTERNAL-IP`, not one from a Pod CIDR."""),
+k -n %{ns} get pods -o wide                           # IP and NODE of every Pod
-              hand(r"""echo "10.244.1.5 {W1}" > {WS}/q113/web.txt""",
+k -n %{ns} get pod %{web} -o jsonpath='{.status.podIP} {.spec.nodeName}' > {WS}/q113/web.txt
-                   r"""With `%{web}`'s IP and node.""",
+k -n %{ns} exec %{client} -- wget -qO- -T3 http://$(cut -d' ' -f1 {WS}/q113/web.txt) > {WS}/q113/direct.html
-                   run=r"""kubectl -n %{ns} get pod %{web} -o jsonpath='{.status.podIP} {.spec.nodeName}' > {WS}/q113/web.txt"""),
+k get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\\n"}{end}' > {WS}/q113/pod-cidrs.txt
-              hand(r"""k -n %{ns} exec %{client} -- wget -qO- -T3 http://10.244.1.5 > {WS}/q113/direct.html""",
+#   %{edge}'s IP is its node's own address (compare with `k get nodes -o wide`), not one from 10.244.x.0/24
-                   r"""With that IP: Pod to Pod, across nodes, no Service and no NAT - the network model the CNI makes true.""",
-                   run=r"""kubectl -n %{ns} exec %{client} -- wget -qO- -T3 http://$(kubectl -n %{ns} get pod %{web} -o jsonpath='{.status.podIP}') > {WS}/q113/direct.html"""),
-              cmd(r"""k get nodes -o custom-columns=NAME:.metadata.name,CIDR:.spec.podCIDR --no-headers > {WS}/q113/pod-cidrs.txt
-cat {WS}/q113/pod-cidrs.txt
-k -n %{ns} get pod %{edge} -o yaml | grep hostNetwork
 echo %{edge} > {WS}/q113/outside.txt
-echo hostNetwork > {WS}/q113/why.txt""", r"""A `hostNetwork` Pod lives in its node's network namespace and shares the node's address and ports.""")],
+k -n %{ns} get pod %{edge} -o yaml | grep hostNetwork
+echo hostNetwork > {WS}/q113/why.txt
+# Every Pod gets its own address from its node's Pod CIDR, and any Pod reaches any other Pod on
+# any node at that address without NAT - that is the whole network model, and the CNI plugin is
+# what makes it true. A hostNetwork Pod opts out: it lives in the node's network namespace and
+# shares its address and ports. Pod addresses change with every new Pod, which is why Services exist.
+""",
 },
 ]

cka/bank_sim_c.py

@@ -12,7 +12,7 @@ Two scenarios are deliberately adapted so they cannot wreck the practice cluster
 
 Run with:  ./ksim sim c
 """
-from sim.bank import C, NS, cmd, hand
+from sim.bank import C, NS
 
 Q = []
 
@@ -62,87 +62,42 @@ eq "$(j -n geo-fence pod geofence-reader '{.status.phase}')" Running \
  && eq "$(j -n geo-fence pod geofence-reader '{.spec.volumes[0].persistentVolumeClaim.claimName}')" geofence-pvc \
  && j -n geo-fence pod geofence-reader '{.spec.containers[0].volumeMounts[*].mountPath}' | grep -q /geo-data"""),
     ],
-    solution=[hand(r"""vim pv.yaml""",
+    solution="""
-                   r"""The PersistentVolume and PersistentVolumeClaim from the docs page *Configure a Pod to Use a PersistentVolume for Storage*, `---` between them. `volumeName` pins the claim to this PV: without it the binder takes any PV whose size, access mode and class fit.
+cat <<EOF | k apply -f -
-```
 apiVersion: v1
 kind: PersistentVolume
-metadata:
+metadata: {name: geofence-pv}
-  name: geofence-pv
 spec:
+  capacity: {storage: 250Mi}
+  accessModes: [ReadWriteOnce]
+  hostPath: {path: /data/geofence}
   storageClassName: manual
-  capacity:
-    storage: 250Mi
-  accessModes:
-  - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
-  hostPath:
-    path: /data/geofence
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata:
+metadata: {name: geofence-pvc, namespace: geo-fence}
-  name: geofence-pvc
-  namespace: geo-fence
 spec:
+  accessModes: [ReadWriteOnce]
+  resources: {requests: {storage: 250Mi}}
   storageClassName: manual
   volumeName: geofence-pv
-  accessModes:
-  - ReadWriteOnce
-  resources:
-    requests:
-      storage: 250Mi
-```""",
-                   run=r"""cat > pv.yaml <<'EOF'
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: geofence-pv
-spec:
-  storageClassName: manual
-  capacity:
-    storage: 250Mi
-  accessModes:
-  - ReadWriteOnce
-  persistentVolumeReclaimPolicy: Retain
-  hostPath:
-    path: /data/geofence
 ---
 apiVersion: v1
-kind: PersistentVolumeClaim
+kind: Pod
-metadata:
+metadata: {name: geofence-reader, namespace: geo-fence}
-  name: geofence-pvc
-  namespace: geo-fence
 spec:
-  storageClassName: manual
+  volumes: [{name: geo, persistentVolumeClaim: {claimName: geofence-pvc}}]
-  volumeName: geofence-pv
+  containers:
-  accessModes:
+    - name: reader
-  - ReadWriteOnce
+      image: busybox:1.36
-  resources:
+      command: [sh, -c, "ls /geo-data && sleep 3600"]
-    requests:
+      volumeMounts: [{name: geo, mountPath: /geo-data}]
-      storage: 250Mi
+EOF
-EOF"""),
-              cmd(r"""k apply -f pv.yaml
 k -n geo-fence get pvc
-k -n geo-fence run geofence-reader --image=busybox:1.36 --dry-run=client -o yaml --command -- sh -c "ls /geo-data && sleep 3600" > pod.yaml""", r"""`Bound` to `geofence-pv`."""),
+# volumeName pins the claim to one PV by name. Without it the binder picks ANY PV whose capacity,
-              hand(r"""vim pod.yaml""",
+# accessModes and storageClassName match - which on a busy cluster is rarely the one you meant.
-                   r"""Add to the container, below `name: geofence-reader`:
+""",
-```
-    volumeMounts:
-    - name: geo
-      mountPath: /geo-data
-```
-and to the Pod's `spec:`, right below `spec:`:
-```
-  volumes:
-  - name: geo
-    persistentVolumeClaim:
-      claimName: geofence-pvc
-```""",
-                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "geo", "mountPath": "/geo-data"}]
-  | .spec.volumes = [{"name": "geo", "persistentVolumeClaim": {"claimName": "geofence-pvc"}}]' pod.yaml"""),
-              cmd(r"""k apply -f pod.yaml
-k -n geo-fence get pod geofence-reader""")],
 )
 
 # ------------------------------------------------------------------ 69 ----
@@ -305,19 +260,15 @@ want=$(j -n dispatch-hub svc dispatch-service '{.spec.clusterIP}')
 got=$(tr -d '[:space:]' < "$WS/q71/service-ip.txt" 2>/dev/null)
 [ -n "$want" ] && eq "$got" "$want" """),
     ],
-    solution=[cmd(r"""k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
+    solution="""
-k -n dispatch-hub get svc,endpointslices""", r"""`expose` copies the Deployment's selector, which is what makes the endpoints appear. Note the Service's `CLUSTER-IP`."""),
+k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
-              hand(r"""k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt
+k -n dispatch-hub get endpointslices -l kubernetes.io/service-name=dispatch-service   # must not be empty
-cat {WS}/q71/test-result.txt""",
+k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- \\
-                   r"""nginx's welcome page, fetched from a Pod in the cluster.""",
+  wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt
-                   run=r"""kubectl -n dispatch-hub rollout status deploy dispatch-api --timeout=240s
+k -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt
-for i in 1 2 3 4 5; do
+# `expose` copies the Deployment's selector for you, which is what makes the endpoints appear.
-  kubectl -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt </dev/null 2>/dev/null
+# Write the IP with jsonpath, not by eyeballing `get svc` - the grader compares it literally.
-  grep -qi nginx {WS}/q71/test-result.txt && break; kubectl -n dispatch-hub delete pod probe --ignore-not-found >/dev/null 2>&1; sleep 3
+""",
-done"""),
-              hand(r"""echo 10.96.123.45 > {WS}/q71/service-ip.txt""",
-                   r"""With the `CLUSTER-IP` that `get svc` showed - the address alone.""",
-                   run=r"""kubectl -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt""")],
 )
 
 # ------------------------------------------------------------------ 72 ----
@@ -407,102 +358,54 @@ j -n ride-history deploy ride-archive '{.spec.template.spec.volumes[*].persisten
 j -n ride-history deploy ride-archive '{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' | grep -qw ride-archive-pvc-new || exit 1
 for i in $(seq 1 20); do [ "$(j -n ride-history deploy ride-archive '{.status.availableReplicas}')" -ge 1 ] 2>/dev/null && exit 0; sleep 5; done; exit 1"""),
     ],
-    solution=[cmd(r"""k -n ride-history get pvc
+    solution="""
-k get pv ride-archive-pv""", r"""`RECLAIM POLICY Delete` on the old PV: losing the claim loses the data."""),
+k -n ride-history get pvc,pv                    # note reclaimPolicy Delete on the old PV
-              hand(r"""vim new.yaml""",
+cat <<EOF | k apply -f -
-                   r"""The new PV and its claim, pinned to it by name (the docs page *Configure a Pod to Use a PersistentVolume for Storage* has both):
-```
 apiVersion: v1
 kind: PersistentVolume
-metadata:
+metadata: {name: ride-archive-pv-new}
-  name: ride-archive-pv-new
 spec:
+  capacity: {storage: 500Mi}
+  accessModes: [ReadWriteOnce]
+  hostPath: {path: /data/ride-archive-new}
   storageClassName: manual
-  capacity:
-    storage: 500Mi
-  accessModes:
-  - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
-  hostPath:
-    path: /data/ride-archive-new
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata:
+metadata: {name: ride-archive-pvc-new, namespace: ride-history}
-  name: ride-archive-pvc-new
-  namespace: ride-history
 spec:
+  accessModes: [ReadWriteOnce]
+  resources: {requests: {storage: 500Mi}}
   storageClassName: manual
   volumeName: ride-archive-pv-new
-  accessModes:
+EOF
-  - ReadWriteOnce
+k -n ride-history scale deploy ride-archive --replicas=0     # release the RWO claim first
-  resources:
+cat <<EOF | k apply -f -
-    requests:
-      storage: 500Mi
-```""",
-                   run=r"""cat > new.yaml <<'EOF'
 apiVersion: v1
-kind: PersistentVolume
+kind: Pod
-metadata:
+metadata: {name: data-migrator, namespace: ride-history}
-  name: ride-archive-pv-new
 spec:
-  storageClassName: manual
+  restartPolicy: Never
-  capacity:
-    storage: 500Mi
-  accessModes:
-  - ReadWriteOnce
-  persistentVolumeReclaimPolicy: Retain
-  hostPath:
-    path: /data/ride-archive-new
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: ride-archive-pvc-new
-  namespace: ride-history
-spec:
-  storageClassName: manual
-  volumeName: ride-archive-pv-new
-  accessModes:
-  - ReadWriteOnce
-  resources:
-    requests:
-      storage: 500Mi
-EOF"""),
-              cmd(r"""k apply -f new.yaml
-k -n ride-history scale deploy ride-archive --replicas=0
-k -n ride-history run data-migrator --image=busybox:1.36 --restart=Never --dry-run=client -o yaml -- sh -c "cp -a /old-data/. /new-data/ && echo migration-complete && sleep 3600" > m.yaml""", r"""Scale to 0 first: the running Pod holds the old claim. `cp -a /old-data/. /new-data/` copies the contents, dotfiles too, rather than the directory itself."""),
-              hand(r"""vim m.yaml""",
-                   r"""Both claims: add to the container, below `name: data-migrator`,
-```
-    volumeMounts:
-    - name: old
-      mountPath: /old-data
-    - name: new
-      mountPath: /new-data
-```
-and to the Pod's `spec:`, right below `spec:`:
-```
   volumes:
-  - name: old
+    - {name: old, persistentVolumeClaim: {claimName: ride-archive-pvc}}
-    persistentVolumeClaim:
+    - {name: new, persistentVolumeClaim: {claimName: ride-archive-pvc-new}}
-      claimName: ride-archive-pvc
+  containers:
-  - name: new
+    - name: migrator
-    persistentVolumeClaim:
+      image: busybox:1.36
-      claimName: ride-archive-pvc-new
+      command: [sh, -c, "cp -a /old-data/. /new-data/ && echo migration-complete && sleep 3600"]
-```""",
+      volumeMounts:
-                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "old", "mountPath": "/old-data"}, {"name": "new", "mountPath": "/new-data"}]
+        - {name: old, mountPath: /old-data}
-  | .spec.volumes = [{"name": "old", "persistentVolumeClaim": {"claimName": "ride-archive-pvc"}}, {"name": "new", "persistentVolumeClaim": {"claimName": "ride-archive-pvc-new"}}]' m.yaml"""),
+        - {name: new, mountPath: /new-data}
-              cmd(r"""k apply -f m.yaml"""),
+EOF
-              hand(r"""k -n ride-history logs data-migrator""",
+k -n ride-history logs data-migrator             # migration-complete
-                   r"""Again until it prints `migration-complete`.""",
+k -n ride-history patch deploy ride-archive --type=json \\
-                   run=r"""for i in $(seq 1 60); do kubectl -n ride-history logs data-migrator 2>/dev/null | grep -q migration-complete && break; sleep 2; done"""),
+  -p '[{"op":"replace","path":"/spec/template/spec/volumes/0/persistentVolumeClaim/claimName","value":"ride-archive-pvc-new"}]'
-              cmd(r"""k -n ride-history delete pod data-migrator --now"""),
+k -n ride-history scale deploy ride-archive --replicas=1
-              hand(r"""k -n ride-history edit deploy ride-archive""",
+# ReadWriteOnce is per-node, but a second pod on the same node still cannot mount a claim that a
-                   r"""In the Pod template's `volumes:`, change `claimName: ride-archive-pvc` to `claimName: ride-archive-pvc-new`, then `:wq`.""",
+# running pod holds - hence the scale to 0 first. `cp -a /old/. /new/` copies the CONTENTS
-                   run=r"""kubectl -n ride-history patch deploy ride-archive --type=json -p '[{"op":"replace","path":"/spec/template/spec/volumes/0/persistentVolumeClaim/claimName","value":"ride-archive-pvc-new"}]'"""),
+# (including dotfiles) rather than nesting the directory inside the target.
-              cmd(r"""k -n ride-history scale deploy ride-archive --replicas=1
+""",
-k -n ride-history get pods""")],
 )
 
 # ------------------------------------------------------------------ 73 ----
@@ -542,21 +445,17 @@ for i in $(seq 1 12); do
   sleep 3
 done; exit 1"""),
     ],
-    solution=[cmd(r"""k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend
+    solution="""
-k -n eta-service get pods --show-labels
+k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend    # empty
-k -n eta-service describe svc eta-backend""", r"""No endpoints. The Pods are `app=eta-calculator`, the Service selects `app=eta-service`; and it sends to `TargetPort 9090`, where nginx listens on 80. Empty endpoints is almost always the selector; endpoints that refuse connections, the target port."""),
+k -n eta-service get deploy eta-calculator -o jsonpath='{.spec.template.metadata.labels}'  # app=eta-calculator
-              hand(r"""k -n eta-service edit svc eta-backend""",
+k -n eta-service get svc eta-backend -o jsonpath='{.spec.selector}'                        # app=eta-service  <- fault 1
-                   r"""Fix both, then `:wq`:
+k -n eta-service patch svc eta-backend -p '{"spec":{"selector":{"app":"eta-calculator"}}}'
-```
+k -n eta-service patch svc eta-backend -p '{"spec":{"ports":[{"port":80,"targetPort":80,"protocol":"TCP"}]}}'
-  ports:
+k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend
-  - port: 80
+# Empty endpoints is almost always the selector; a populated endpoint list that still refuses
-    protocol: TCP
+# connections is almost always targetPort. Check the endpoints first - it splits the two cases
-    targetPort: 80
+# in one command.
-  selector:
+""",
-    app: eta-calculator
-```""",
-                   run=r"""kubectl -n eta-service patch svc eta-backend -p '{"spec":{"selector":{"app":"eta-calculator"},"ports":[{"port":80,"targetPort":80,"protocol":"TCP"}]}}'"""),
-              cmd(r"""k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend""")],
 )
 
 # ------------------------------------------------------------------ 74 ----
@@ -622,91 +521,36 @@ sys.exit(0 if ok else 1)'"""),
         C("the HTTPRoute is attached to fleet-gateway", 1, r"""
 j -n fleet-ops httproute fleet-route '{.spec.parentRefs[*].name}' | grep -qw fleet-gateway"""),
     ],
-    solution=[hand(r"""vim gw.yaml""",
+    solution="""
-                   r"""The docs page *Gateway API* has a Gateway and an HTTPRoute to start from:
+cat <<EOF | k apply -f -
-```
 apiVersion: gateway.networking.k8s.io/v1
 kind: Gateway
-metadata:
+metadata: {name: fleet-gateway, namespace: fleet-ops}
-  name: fleet-gateway
-  namespace: fleet-ops
 spec:
   gatewayClassName: fleet-gateway-class
   listeners:
-  - name: http
+    - name: http
-    protocol: HTTP
+      protocol: HTTP
-    port: 80
+      port: 80
-    hostname: fleet.example.com
+      hostname: fleet.example.com
 ---
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
-metadata:
+metadata: {name: fleet-route, namespace: fleet-ops}
-  name: fleet-route
-  namespace: fleet-ops
 spec:
-  parentRefs:
+  parentRefs: [{name: fleet-gateway}]
-  - name: fleet-gateway
+  hostnames: [fleet.example.com]
-  hostnames:
-  - fleet.example.com
   rules:
-  - matches:
+    - matches:
-    - headers:
+        - headers: [{type: Exact, name: X-Fleet-Version, value: canary}]
-      - name: X-Fleet-Version
+      backendRefs: [{name: fleet-dashboard-v2, port: 80}]
-        value: canary
+    - matches:
-    backendRefs:
+        - path: {type: PathPrefix, value: /}
-    - name: fleet-dashboard-v2
+      backendRefs: [{name: fleet-dashboard-v1, port: 80}]
-      port: 80
+EOF
-  - matches:
+# Order matters: HTTPRoute picks the most specific match, and a bare PathPrefix "/" matches
-    - path:
+# everything - so the header rule must be able to win. Listing it first also keeps it readable.
-        type: PathPrefix
+""",
-        value: /
-    backendRefs:
-    - name: fleet-dashboard-v1
-      port: 80
-```
-The more specific match wins, so the header rule takes canary requests though `/` matches everything.""",
-                   run=r"""cat > gw.yaml <<'EOF'
-apiVersion: gateway.networking.k8s.io/v1
-kind: Gateway
-metadata:
-  name: fleet-gateway
-  namespace: fleet-ops
-spec:
-  gatewayClassName: fleet-gateway-class
-  listeners:
-  - name: http
-    protocol: HTTP
-    port: 80
-    hostname: fleet.example.com
----
-apiVersion: gateway.networking.k8s.io/v1
-kind: HTTPRoute
-metadata:
-  name: fleet-route
-  namespace: fleet-ops
-spec:
-  parentRefs:
-  - name: fleet-gateway
-  hostnames:
-  - fleet.example.com
-  rules:
-  - matches:
-    - headers:
-      - name: X-Fleet-Version
-        value: canary
-    backendRefs:
-    - name: fleet-dashboard-v2
-      port: 80
-  - matches:
-    - path:
-        type: PathPrefix
-        value: /
-    backendRefs:
-    - name: fleet-dashboard-v1
-      port: 80
-EOF"""),
-              cmd(r"""k apply -f gw.yaml
-k -n fleet-ops get gateway,httproute""")],
 )
 
 # ------------------------------------------------------------------ 75 ----
@@ -870,49 +714,29 @@ eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \
  && $n -o jsonpath='{.spec.policyTypes}' | grep -q Egress \
  && [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ]"""),
     ],
-    solution=[cmd(r"""k -n kube-system get ds
+    solution="""
-echo kindnet > {WS}/q77/cni.txt""", r"""`kindnet` runs on every node: it is the CNI. `kube-proxy` is not."""),
+k -n kube-system get ds                       # kindnet  -> {WS}/q77/cni.txt
-              cmd(r"""k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv
+# Don't answer from memory - prove it. Two pods, one deny policy, one wget:
-k -n zero-trust expose pod srv --port 80
+k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv; k -n zero-trust expose pod srv --port 80
-k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600""", r"""Prove it rather than remember it: two Pods, a policy that denies everything, and a request between them."""),
+k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600
-              hand(r"""k -n zero-trust get pods""",
+k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null && echo reachable
-                   r"""Again until both are `Running`.""",
+cat <<EOF | k apply -f -
-                   run=r"""kubectl -n zero-trust wait --for=condition=Ready pod/srv pod/cli --timeout=180s"""),
-              cmd(r"""k -n zero-trust exec cli -- wget -qO- -T3 http://srv""", r"""nginx's page: no policy yet."""),
-              hand(r"""vim deny.yaml""",
-                   r"""The default deny for both directions from the docs page *Network Policies*: every Pod, both policy types, no rules.
-```
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
-metadata:
+metadata: {name: default-deny-all, namespace: zero-trust}
-  name: default-deny-all
-  namespace: zero-trust
 spec:
   podSelector: {}
-  policyTypes:
+  policyTypes: [Ingress, Egress]
-  - Ingress
+EOF
-  - Egress
+k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null || echo BLOCKED   # -> BLOCKED here
-```""",
+echo yes > {WS}/q77/enforced.txt
-                   run=r"""cat > deny.yaml <<'EOF'
+k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv                # leave only the policy
-apiVersion: networking.k8s.io/v1
+# podSelector {} means every pod in the namespace; naming both policyTypes with no ingress/egress
-kind: NetworkPolicy
+# blocks listed means "deny everything". Recent kindnet builds (this cluster's included) embed a
-metadata:
+# policy engine - look for `nft list table inet kindnet-network-policies` on a node. Flannel stores
-  name: default-deny-all
+# NetworkPolicies and enforces nothing; Calico and Cilium enforce. A policy that "works" on a
-  namespace: zero-trust
+# non-enforcing CNI is the classic hardening-review trap.
-spec:
+""",
-  podSelector: {}
-  policyTypes:
-  - Ingress
-  - Egress
-EOF"""),
-              cmd(r"""k apply -f deny.yaml
-k -n zero-trust exec cli -- wget -qO- -T3 http://srv""", r"""`download timed out`: the CNI enforces it. (Flannel would store the policy and enforce nothing; Calico and Cilium enforce. Recent kindnet has a policy engine of its own.)"""),
-              hand(r"""echo yes > {WS}/q77/enforced.txt""",
-                   r"""What the test showed.""",
-                   run=r"""sleep 6
-if kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null 2>&1; then echo no; else echo yes; fi > {WS}/q77/enforced.txt"""),
-              cmd(r"""k -n zero-trust delete pod srv cli --now
-k -n zero-trust delete svc srv""", r"""Leave only the policy.""")],
 )
 
 # ------------------------------------------------------------------ 78 ----
@@ -943,16 +767,23 @@ for i in $(seq 1 20); do [ "$(j -n rider-support deploy support-portal '{.status
 eq "$(j -n rider-support deploy support-portal '{.status.readyReplicas}')" 2 \
  && eq "$(j -n rider-support pod -l app=support-portal '{.items[0].spec.priority}')" 1000000"""),
     ],
-    solution=[cmd(r"""k create priorityclass rider-support-priority --value=1000000 --description='Priority for rider support services'""", r"""Not the global default unless `--global-default`."""),
+    solution="""
-              hand(r"""k -n rider-support edit deploy support-portal""",
+cat <<EOF | k apply -f -
-                   r"""In the Pod template's `spec:`, beside `containers:` and indented like it, then `:wq`:
+apiVersion: scheduling.k8s.io/v1
-```
+kind: PriorityClass
-      priorityClassName: rider-support-priority
+metadata: {name: rider-support-priority}
-```
+value: 1000000
-The value is copied into each Pod when it is created, so only new Pods carry it - the edit rolls them.""",
+globalDefault: false
-                   run=r"""kubectl -n rider-support patch deploy support-portal -p '{"spec":{"template":{"spec":{"priorityClassName":"rider-support-priority"}}}}'"""),
+description: Priority for rider support services
-              cmd(r"""k -n rider-support rollout status deploy support-portal
+EOF
-k -n rider-support get pods -o custom-columns=NAME:.metadata.name,PRIORITY:.spec.priority""")],
+k -n rider-support patch deploy support-portal \\
+  -p '{"spec":{"template":{"spec":{"priorityClassName":"rider-support-priority"}}}}'
+k -n rider-support rollout status deploy support-portal
+k -n rider-support get pod -l app=support-portal -o jsonpath='{.items[0].spec.priority}'
+# The admission controller copies the class's value into pod.spec.priority at creation time, so
+# existing pods keep their old priority until they are recreated - patching the template does
+# that for you. globalDefault: true would apply the class to every pod without one, cluster-wide.
+""",
 )
 
 # ------------------------------------------------------------------ 79 ----
@@ -1061,35 +892,31 @@ un=$(grep -n 'uncordon' "$f" | head -1 | cut -d: -f1)
 grep -Eq 'kubeadm[[:space:]]+upgrade[[:space:]]+node' "$WS/q80/worker.txt" 2>/dev/null \
  && ! grep -q 'upgrade apply' "$WS/q80/worker.txt" """),
     ],
-    solution=[cmd(r"""k version
+    solution="""
+k version -o json | jq -r .serverVersion.gitVersion > {WS}/q80/current.txt
 k get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion --no-headers > {WS}/q80/kubelets.txt
-cat {WS}/q80/kubelets.txt"""),
+cat > {WS}/q80/steps.sh <<'EOF'
-              hand(r"""echo v1.37.0 > {WS}/q80/current.txt""",
+sed -i 's|/v1.37/|/v1.38/|' /etc/apt/sources.list.d/kubernetes.list
-                   r"""With the `Server Version` that `k version` printed.""",
-                   run=r"""kubectl version -o json | jq -r .serverVersion.gitVersion > {WS}/q80/current.txt"""),
-              hand(r"""vim {WS}/q80/steps.sh""",
-                   r"""The order of the docs page *Upgrading kubeadm clusters*: kubeadm first, which upgrades the control plane; then the node's kubelet and kubectl, drained; uncordon last. The next minor is 1.38 - first point `/etc/apt/sources.list.d/kubernetes.list` at the v1.38 repository, and take its newest patch (`apt-cache madison kubeadm`):
-```
 apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.38.0-1.1 && apt-mark hold kubeadm
 kubeadm upgrade plan
-kubeadm upgrade apply v1.38.0
+kubeadm upgrade apply v1.38.0 -y
-kubectl drain {CP} --ignore-daemonsets
+kubectl drain <control-plane> --ignore-daemonsets
-apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
+apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
 systemctl daemon-reload
 systemctl restart kubelet
-kubectl uncordon {CP}
+kubectl uncordon <control-plane>
-```""",
+EOF
-                   run=r"""cat > {WS}/q80/steps.sh <<'EOF'
+echo 'kubeadm upgrade node' > {WS}/q80/worker.txt
-apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.38.0-1.1 && apt-mark hold kubeadm
+# The cluster runs v1.37, so the next minor is 1.38 - and pkgs.k8s.io serves one apt repo per
-kubeadm upgrade plan
+# minor, so the repo line has to move to v1.38 before 1.38 packages exist for apt at all. (1.38
-kubeadm upgrade apply v1.38.0
+# is due in December 2026; until then that repo is empty, which is why this is only a plan. Take
-kubectl drain {CP} --ignore-daemonsets
+# the newest 1.38 patch on offer then: `apt-cache madison kubeadm`.)
-apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
+# kubeadm goes first because it drives the whole thing; the control plane is upgraded by
-systemctl daemon-reload
+# `upgrade apply`, and only then do the node binaries (kubelet/kubectl) follow, ending with
-systemctl restart kubelet
+# uncordon. Workers run `kubeadm upgrade node` - they host no control-plane static pods, so
-kubectl uncordon {CP}
+# there is nothing for `upgrade apply` to do there. The kubelet may trail the API server by up
-EOF"""),
+# to three minor versions, but must never be ahead of it.
-              cmd(r"""echo 'kubeadm upgrade node' > {WS}/q80/worker.txt""", r"""A worker hosts no control plane, so there is nothing for `upgrade apply` to do there; `upgrade node` updates its kubelet configuration.""")],
+""",
 )
 
 # ------------------------------------------------------------------ 81 ----
@@ -1150,44 +977,35 @@ done; exit 1"""),
         C("fare-calculator pods are Running and Ready", 3, r"""
 for i in $(seq 1 24); do [ "$(j -n fare-engine deploy fare-calculator '{.status.readyReplicas}')" -ge 1 ] 2>/dev/null && exit 0; sleep 5; done; exit 1"""),
     ],
-    solution=[cmd(r"""k -n fare-engine get pvc,pods
+    solution="""
-k -n fare-engine describe pvc fare-data-pvc | tail -3
+k -n fare-engine get pvc                       # Pending
-k get sc""", r"""`storageclass.storage.k8s.io "fast-storage" not found`: the class here is `standard`."""),
+k -n fare-engine describe pvc fare-data-pvc    # storageclass.storage.k8s.io "fast-storage" not found
-              cmd(r"""k -n fare-engine delete pvc fare-data-pvc""", r"""A claim's `storageClassName` cannot change: make it again."""),
+k get sc                                       # on kind the class is called `standard`
-              hand(r"""vim pvc.yaml""",
+# A PVC's storageClassName is immutable, so recreate it with the name your cluster really has:
-                   r"""```
+k -n fare-engine delete pvc fare-data-pvc
+cat <<EOF | k apply -f -
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata:
+metadata: {name: fare-data-pvc, namespace: fare-engine}
-  name: fare-data-pvc
-  namespace: fare-engine
 spec:
+  accessModes: [ReadWriteOnce]
+  resources: {requests: {storage: 100Mi}}
   storageClassName: standard
-  accessModes:
+EOF
-  - ReadWriteOnce
+k -n fare-engine describe pod -l app=fare-calculator   # 0/4 nodes match node affinity disk in [nvme]
-  resources:
+k get nodes --show-labels | grep -c 'disk=nvme'        # 0
-    requests:
+# Either label a node to satisfy the rule, or drop the rule. Labelling is the smaller change:
-      storage: 100Mi
+k label node <a-worker> disk=nvme
-```""",
+k -n fare-engine rollout status deploy fare-calculator
-                   run=r"""cat > pvc.yaml <<'EOF'
+# Two independent blockers stack up silently: the PVC never binds, AND nothing would schedule
-apiVersion: v1
+# even if it did. `describe pod` reports only the scheduling failure, so check the PVC too
-kind: PersistentVolumeClaim
+# whenever a pod that mounts one is stuck Pending.
-metadata:
+# Never assume the class name - `standard` here, `local-path` on a stock local-path-provisioner
-  name: fare-data-pvc
+# install, something else on a cloud cluster. `k get sc` first, every time; a PVC naming a class
-  namespace: fare-engine
+# that does not exist sits Pending forever with no other symptom.
-spec:
+# `standard` is WaitForFirstConsumer, so the PVC only binds once the pod can actually be placed -
-  storageClassName: standard
+# which is why fixing the affinity is what finally unblocks the storage too.
-  accessModes:
+""",
-  - ReadWriteOnce
-  resources:
-    requests:
-      storage: 100Mi
-EOF"""),
-              cmd(r"""k apply -f pvc.yaml
-k -n fare-engine describe pod -l app=fare-calculator | tail -3
-k get nodes -L disk""", r"""`didn't match Pod's node affinity/selector`: it wants `disk=nvme`, which no node has. `describe pod` names only this one; the claim's fault you had to find yourself."""),
-              cmd(r"""k label node {W1} disk=nvme
-k -n fare-engine rollout status deploy fare-calculator""", r"""Labelling a node is the smaller change than dropping the rule. `standard` binds only once the Pod is placed (`WaitForFirstConsumer`), so the claim goes `Bound` now too.""")],
 )
 
 # ------------------------------------------------------------------ 82 ----
@@ -1236,20 +1054,30 @@ ok=any(p.get("type")=="Pods" and int(p.get("value",0))==2 and int(p.get("periodS
        for p in b.get("policies",[]))
 sys.exit(0 if ok else 1)'"""),
     ],
-    solution=[cmd(r"""k -n route-planner autoscale deploy route-planner --name=route-planner-hpa --min=1 --max=6 --memory=70% --dry-run=client -o yaml > hpa.yaml""", r"""`autoscale` writes an `autoscaling/v2` HPA with the memory target; it has no flag for `behavior`."""),
+    solution="""
-              hand(r"""vim hpa.yaml""",
+cat <<EOF | k apply -f -
-                   r"""Add to its `spec:`, beside `maxReplicas:` (the docs page *Horizontal Pod Autoscaling* has the block under *Configurable scaling behavior*):
+apiVersion: autoscaling/v2
-```
+kind: HorizontalPodAutoscaler
+metadata: {name: route-planner-hpa, namespace: route-planner}
+spec:
+  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: route-planner}
+  minReplicas: 1
+  maxReplicas: 6
+  metrics:
+    - type: Resource
+      resource:
+        name: memory
+        target: {type: Utilization, averageUtilization: 70}
   behavior:
     scaleUp:
       policies:
-      - type: Pods
+        - {type: Pods, value: 2, periodSeconds: 60}
-        value: 2
+EOF
-        periodSeconds: 60
+k -n route-planner get hpa route-planner-hpa
-```""",
+# `kubectl autoscale` can set a memory target (--memory=70%) but has no flag for a behavior block,
-                   run=r"""yq -i '.spec.behavior.scaleUp.policies = [{"type": "Pods", "value": 2, "periodSeconds": 60}]' hpa.yaml"""),
+# so memory + behavior means writing the v2 YAML by hand (or autoscale, then kubectl edit). Utilisation is a percentage OF THE REQUEST, so the Deployment must
-              cmd(r"""k apply -f hpa.yaml
+# declare a memory request or the HPA reports <unknown> and never scales.
-k -n route-planner get hpa route-planner-hpa""", r"""Utilisation is a percentage of the request: the Deployment has a memory request, so the target reads a number, not `<unknown>`.""")],
+""",
 )
 
 # ------------------------------------------------------------------ 83 ----
@@ -1303,81 +1131,26 @@ ok=any(not r.get("to") and {p.get("protocol") for p in r.get("ports",[]) if str(
        for r in eg)
 sys.exit(0 if ok else 1)'"""),
     ],
-    solution=[hand(r"""vim netpol.yaml""",
+    solution="""
-                   r"""Paste the example from the docs page *Network Policies* and change it to:
+cat <<EOF | k apply -f -
-```
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
-metadata:
+metadata: {name: safety-api-policy, namespace: safety-monitor}
-  name: safety-api-policy
-  namespace: safety-monitor
 spec:
-  podSelector:
+  podSelector: {matchLabels: {app: safety-api}}
-    matchLabels:
+  policyTypes: [Ingress, Egress]
-      app: safety-api
-  policyTypes:
-  - Ingress
-  - Egress
   ingress:
-  - from:
+    - from: [{podSelector: {matchLabels: {app: safety-dashboard}}}]
-    - podSelector:
+      ports: [{protocol: TCP, port: 8443}]
-        matchLabels:
-          app: safety-dashboard
-    ports:
-    - protocol: TCP
-      port: 8443
   egress:
-  - to:
+    - to: [{podSelector: {matchLabels: {app: safety-db}}}]
-    - podSelector:
+      ports: [{protocol: TCP, port: 5432}]
-        matchLabels:
+    - ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
-          app: safety-db
+EOF
-    ports:
+# The DNS rule deliberately has no `to:` - CoreDNS lives in kube-system, and an egress rule with
-    - protocol: TCP
+# only `ports` allows that port to anywhere. Naming Egress in policyTypes denies all outbound
-      port: 5432
+# traffic by default, so forgetting DNS is what silently breaks every name lookup in the pod.
-  - ports:
+""",
-    - protocol: UDP
-      port: 53
-    - protocol: TCP
-      port: 53
-```
-The DNS rule has no `to:` - only ports, so port 53 anywhere. With `Egress` in `policyTypes` everything else outbound is denied, and forgetting DNS breaks every name lookup in the Pod.""",
-                   run=r"""cat > netpol.yaml <<'EOF'
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: safety-api-policy
-  namespace: safety-monitor
-spec:
-  podSelector:
-    matchLabels:
-      app: safety-api
-  policyTypes:
-  - Ingress
-  - Egress
-  ingress:
-  - from:
-    - podSelector:
-        matchLabels:
-          app: safety-dashboard
-    ports:
-    - protocol: TCP
-      port: 8443
-  egress:
-  - to:
-    - podSelector:
-        matchLabels:
-          app: safety-db
-    ports:
-    - protocol: TCP
-      port: 5432
-  - ports:
-    - protocol: UDP
-      port: 53
-    - protocol: TCP
-      port: 53
-EOF"""),
-              cmd(r"""k apply -f netpol.yaml
-k -n safety-monitor describe netpol safety-api-policy""")],
 )
 
 # ------------------------------------------------------------------ 84 ----

cka/bank_sim_d.py

@@ -16,7 +16,7 @@ task was seeded, solved and graded on an isolated kind cluster before it went in
 
 Run with:  ./ksim sim d
 """
-from sim.bank import C, NS, cmd, hand
+from sim.bank import C, NS
 
 Q = []
 # Tasks written for the pre-2025 exam opened by switching kubectl context. Every task now has a host
@@ -73,27 +73,25 @@ if [ "$own" = ReplicaSet ]; then want="deployment $(j -n kube-system rs "$name"
 else want="$(echo "$own" | tr A-Z a-z) $name"; fi
 eq "$(sed -n 's/^ *dns *: *//p' "$f" | head -1 | tr -s ' ' | tr A-Z a-z | sed 's/ *$//')" "$want" """),
     ],
-    solution=[cmd(r"""systemctl status kubelet | head -3
+    solution="""
-sudo ls /etc/kubernetes/manifests""", r"""The host is the control-plane node. The kubelet is a systemd service - a process; etcd and the three control-plane components are manifests in the static Pod path."""),
+sudo -i                # the task's host is the control-plane node: no hop needed
-              cmd(r"""k -n kube-system get pods -o wide
+systemctl status kubelet | head -3          # a systemd service: a plain process on the node
-k -n kube-system get deploy,ds""", r"""Static Pods carry the node's name as a suffix. What answers DNS is the `coredns` Deployment, though its Service is called `kube-dns`; `kube-proxy` and `kindnet` are DaemonSets."""),
+ls /etc/kubernetes/manifests                # etcd, kube-apiserver, kube-controller-manager, kube-scheduler
-              hand(r"""vim {WS}/q86/controlplane.txt""",
+exit
-                   r"""```
+k -n kube-system get pod -o wide            # static pods carry the node name as a suffix...
+k -n kube-system get pod kube-apiserver-{CP} -o jsonpath='{.metadata.ownerReferences[0].kind}{"\\n"}'   # ...and are owned by the Node
+k -n kube-system get deploy,ds              # coredns is a Deployment (kube-proxy, kindnet are DaemonSets)
+cat > {WS}/q86/controlplane.txt <<'EOF'
 kubelet: process
 kube-apiserver: static-pod
 kube-scheduler: static-pod
 kube-controller-manager: static-pod
 etcd: static-pod
 dns: deployment coredns
-```""",
+EOF
-                   run=r"""cat > {WS}/q86/controlplane.txt <<'EOF'
+# The Service is called kube-dns for historical reasons; what actually answers is the coredns
-kubelet: process
+# Deployment. Follow the chain svc selector -> pod -> ownerReferences when unsure.
-kube-apiserver: static-pod
+""",
-kube-scheduler: static-pod
-kube-controller-manager: static-pod
-etcd: static-pod
-dns: deployment coredns
-EOF""")],
 )
 
 # ------------------------------------------------------------------ 87 ----
@@ -127,27 +125,24 @@ cfg=$(nx "$W1" 'ls /etc/cni/net.d/*.conflist /etc/cni/net.d/*.conf 2>/dev/null |
         C("static Pod name suffix on {W3} (line 5)", 1, r"""
 v=$(sed -n 's/^ *5 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]'); [ "$v" = "-$W3" ] || [ "$v" = "$W3" ]"""),
     ],
-    solution=[cmd(r"""k get nodes
+    solution="""
-k -n kube-system describe pod -l component=kube-apiserver | grep service-cluster-ip-range
+k get nodes                                   # ROLES shows control-plane; count the rest
-k -n kube-system get ds""", r"""`ROLES control-plane` on one node, the rest workers; the API server's `--service-cluster-ip-range`; and the CNI's DaemonSet, `kindnet`."""),
+k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
-              hand(r"""ssh {W1} ls /etc/cni/net.d""",
+k -n kube-system get ds                       # kindnet
-                   r"""The CNI's configuration on a node: `10-kindnet.conflist`.""",
+{SHELL_W1}
-                   run=r""""""),
+sudo -i
-              hand(r"""vim {WS}/q87/cluster-info.txt""",
+ls /etc/cni/net.d/                            # 10-kindnet.conflist
-                   r"""The values you read - here:
+exit; exit
-```
+cat > {WS}/q87/cluster-info.txt <<'EOF'
 1: 1
 2: 3
 3: 10.96.0.0/16
 4: kindnet, /etc/cni/net.d/10-kindnet.conflist
 5: -{W3}
-```
+EOF
-A static Pod's mirror is named `<pod>-<node>`, e.g. `kube-apiserver-{CP}`.""",
+# A static Pod's mirror is named <pod>-<node name>, e.g. kube-apiserver-{CP}. Values come from THIS
-                   run=r"""C=$(kubectl get nodes --no-headers -l node-role.kubernetes.io/control-plane | wc -l)
+# cluster - check each one instead of trusting the ones shown here.
-W=$(kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' | wc -l)
+""",
-CIDR=$(kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -o 'service-cluster-ip-range=[^ ]*' | head -1 | cut -d= -f2)
-F=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
-printf '1: %s\n2: %s\n3: %s\n4: kindnet, %s\n5: -{W3}\n' "$C" "$W" "$CIDR" "$F" > {WS}/q87/cluster-info.txt""")],
 )
 
 # ------------------------------------------------------------------ 88 ----
@@ -223,19 +218,18 @@ grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-n|--namespa
 timeout 60 bash "$f" 2>/dev/null | grep -q etcd"""),
         C("no aliases in either script", 1, r"""[ -s "$WS/q89/pods-by-age.sh" ] && [ -s "$WS/q89/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q89/pods-by-age.sh" "$WS/q89/pods-by-uid.sh" """),
     ],
-    solution=[hand(r"""vim {WS}/q89/pods-by-age.sh""",
+    solution="""
-                   r"""`kubectl` in full: a script runs in a fresh shell, where the `k` alias does not exist.
+cat > {WS}/q89/pods-by-age.sh <<'EOF'
-```
 kubectl get pods -A --sort-by=.metadata.creationTimestamp
-```""",
+EOF
-                   run=r"""echo 'kubectl get pods -A --sort-by=.metadata.creationTimestamp' > {WS}/q89/pods-by-age.sh"""),
+cat > {WS}/q89/pods-by-uid.sh <<'EOF'
-              hand(r"""vim {WS}/q89/pods-by-uid.sh""",
-                   r"""```
 kubectl -n kube-system get pods --sort-by=.metadata.uid
-```""",
+EOF
-                   run=r"""echo 'kubectl -n kube-system get pods --sort-by=.metadata.uid' > {WS}/q89/pods-by-uid.sh"""),
+chmod +x {WS}/q89/*.sh
-              cmd(r"""chmod +x {WS}/q89/*.sh
+bash {WS}/q89/pods-by-age.sh | head -3        # the control-plane pods come first - they are oldest
-{WS}/q89/pods-by-age.sh | head -3""", r"""The control-plane Pods first: they are the oldest. `--sort-by` takes a JSONPath into each item, ascending only.""")],
+# --sort-by takes a JSONPath into each item; ascending only, so "newest first" is `| tac` or
+# sorting by a field that grows the other way.
+""",
 )
 
 # ------------------------------------------------------------------ 90 ----
@@ -279,35 +273,31 @@ for f in pod-deleted.log container-killed.log; do
 done"""),
         C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q90/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""),
     ],
-    solution=[hand(r"""vim {WS}/q90/cluster-events.sh""",
+    solution="""
-                   r"""```
+cat > {WS}/q90/cluster-events.sh <<'EOF'
 kubectl get events -A --sort-by=.metadata.creationTimestamp
-```""",
+EOF
-                   run=r"""echo 'kubectl get events -A --sort-by=.metadata.creationTimestamp' > {WS}/q90/cluster-events.sh"""),
+chmod +x {WS}/q90/cluster-events.sh
-              cmd(r"""chmod +x {WS}/q90/cluster-events.sh
+
-k -n kube-system get pods -l k8s-app=kube-proxy -o wide""", r"""Note the name of the one on `{W2}`."""),
+k -n kube-system get pod -l k8s-app=kube-proxy -o wide              # the one on {W2}
-              hand(r"""k -n kube-system delete pod kube-proxy-xxxxx""",
+k -n kube-system delete pod <kube-proxy-on-{W2}>
-                   r"""With that name. The DaemonSet replaces it with a new Pod at once.""",
+k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
-                   run=r"""P=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].metadata.name}')
+#   Killing (old pod), SuccessfulCreate (DaemonSet), Scheduled, Pulled, Created, Started (new pod)
-kubectl -n kube-system delete pod $P
+
-for i in $(seq 1 40); do
+{SHELL_W2}
-  [ "$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].status.phase}' 2>/dev/null)" = Running ] && break; sleep 3
-done
-sleep 5"""),
-              cmd(r"""k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
-cat {WS}/q90/pod-deleted.log""", r"""`Killing` the old Pod, `SuccessfulCreate` by the DaemonSet, then `Scheduled`, `Pulled`, `Created`, `Started` for the new one."""),
-              hand(r"""{SHELL_W2}
 sudo -i
-crictl ps --name kube-proxy
+crictl ps --name kube-proxy                 # container id
-crictl stop 1a2b3c4d5e6f
+crictl stop <id>
-exit
+exit; exit
-exit""",
+k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
-                   r"""With the container ID `crictl ps` printed. Only the container goes: the kubelet starts a new one in the same Pod.""",
+#   Pulled, Created, Started - same Pod, new container
-                   run=r"""ssh -n {W2} 'sudo crictl stop $(sudo crictl ps --name kube-proxy -q)'
+k -n kube-system get pod -l k8s-app=kube-proxy -o wide              # RESTARTS 1 on {W2}
-sleep 25"""),
+echo container > {WS}/q90/restarts.txt
-              cmd(r"""k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
+# Deleting the Pod makes the DaemonSet create a brand-new Pod (RESTARTS 0). Killing only the
-k -n kube-system get pods -l k8s-app=kube-proxy -o wide
+# container makes the kubelet start a new container inside the SAME Pod - that is a restart.
-echo container > {WS}/q90/restarts.txt""", r"""`RESTARTS 1` on `{W2}`: a new container in the same Pod is a restart; a new Pod starts again at 0.""")],
+# Do not `crictl rm` it straight away: on v1.35 the kubelet counts the restart from the exited
+# container it sees, and if rm wins the race it starts a fresh attempt 0 and RESTARTS stays 0.
+""",
 )
 
 # ------------------------------------------------------------------ 91 ----
@@ -343,15 +333,15 @@ got=$(awk '{print $1}' "$f" | sed 's/\..*//' | grep . | sort -u)
 best=$(for ns in census-north census-south census-east census-west; do echo "$ns $($K -n $ns get role --no-headers 2>/dev/null | wc -l)"; done | sort -k2 -n | tail -1)
 eq "$(tr -s ' ' < "$WS/q91/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """),
     ],
-    solution=[cmd(r"""k api-resources --namespaced=true -o name > {WS}/q91/namespaced.txt
+    solution="""
-head -3 {WS}/q91/namespaced.txt""", r"""`-o name` prints the plural names alone (`deployments.apps`: name and group); `--namespaced=false` would give the cluster-scoped half."""),
+k api-resources --namespaced=true -o name > /tmp/r.txt       # deployments.apps, ... (group suffix)
-              cmd(r"""k -n census-north get role --no-headers | wc -l
+k api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q91/namespaced.txt
-k -n census-south get role --no-headers | wc -l
+for ns in census-north census-south census-east census-west; do echo "$ns $(k -n $ns get role --no-headers | wc -l)"; done
-k -n census-east get role --no-headers | wc -l
+#   pick the largest, e.g.  echo "census-east 8" > {WS}/q91/most-roles.txt
-k -n census-west get role --no-headers | wc -l""", r"""Or all at once: `k get role -A | grep census`."""),
+# --namespaced=false gives the cluster-scoped half (nodes, pv, storageclasses, clusterroles...).
-              hand(r"""echo "census-east 9" > {WS}/q91/most-roles.txt""",
+# `k get role -A --no-headers | awk '{print $1}' | sort | uniq -c | sort -n` counts every
-                   r"""With the namespace that has the most, and its count.""",
+# namespace at once.
-                   run=r"""for ns in census-north census-south census-east census-west; do echo "$ns $(kubectl -n $ns get role --no-headers | wc -l)"; done | sort -k2 -n | tail -1 > {WS}/q91/most-roles.txt""")],
+""",
 )
 
 # ------------------------------------------------------------------ 92 ----
@@ -490,27 +480,20 @@ ip=$(j -n default pod "ops-beacon-$CP" '{.status.podIP}'); [ -n "$ip" ] || exit
 j -n default endpointslices -l kubernetes.io/service-name=ops-beacon-svc '{.items[*].endpoints[*].addresses[*]}' | tr ' ' '\n' | grep -qx "$ip" """),
         C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q94/nodeport.txt" 2>/dev/null)" "$np" """),
     ],
-    solution=[cmd(r"""k -n default run ops-beacon --image=nginx:1.27 --labels=app=ops-beacon --dry-run=client -o yaml > beacon.yaml"""),
+    solution="""
-              hand(r"""vim beacon.yaml""",
+k run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon $do > beacon.yaml
-                   r"""Replace the container's `resources: {}` with:
+# add to the container:  resources: {requests: {cpu: 10m, memory: 20Mi}}
-```
+sudo -i                # the task's host is the control-plane node: no hop needed
-    resources:
+grep staticPodPath /var/lib/kubelet/config.yaml          # /etc/kubernetes/manifests
-      requests:
+vi /etc/kubernetes/manifests/ops-beacon.yaml             # paste beacon.yaml
-        cpu: 10m
+exit
-        memory: 20Mi
+k -n default get pod ops-beacon-{CP} -o wide             # mirror pod, named <pod>-<node>
-```""",
+k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
-                   run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "10m", "memory": "20Mi"}}' beacon.yaml"""),
+k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc   # the mirror pod's IP
-              cmd(r"""sudo grep staticPodPath /var/lib/kubelet/config.yaml
+k -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt
-sudo cp beacon.yaml /etc/kubernetes/manifests/ops-beacon.yaml""", r"""The host is the control-plane node. Its taint keeps the scheduler away, not the kubelet: static Pods run regardless."""),
+# The control-plane taint only keeps the SCHEDULER from placing pods there; the kubelet runs its
-              hand(r"""k -n default get pod ops-beacon-{CP} -o wide""",
+# static pods regardless. `expose pod` copies the pod's labels into the selector.
-                   r"""Again until the mirror Pod is `Running`.""",
+""",
-                   run=r"""for i in $(seq 1 60); do [ "$(kubectl -n default get pod ops-beacon-{CP} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done"""),
-              cmd(r"""k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
-k -n default get svc ops-beacon-svc
-k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc""", r"""`expose pod` makes the selector from the Pod's labels: the mirror Pod's IP is the endpoint. Note the node port, after `80:`."""),
-              hand(r"""echo 31234 > {WS}/q94/nodeport.txt""",
-                   r"""With the node port `get svc` showed.""",
-                   run=r"""kubectl -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt""")],
 )
 
 # ------------------------------------------------------------------ 95 ----
@@ -774,107 +757,40 @@ for i in 0 1; do eq "$($K -n lima-data exec ledger-db-$i -- cat /data/id.txt 2>/
 want=$($K -n lima-data get pvc --no-headers -o custom-columns=N:.metadata.name 2>/dev/null | grep '^data-ledger-db-' | sort)
 [ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q99/pvcs.txt" 2>/dev/null)" = "$want" ]"""),
     ],
-    solution=[hand(r"""vim sts.yaml""",
+    solution="""
-                   r"""The docs page *StatefulSets* has a headless Service and a StatefulSet with `volumeClaimTemplates` together; change them to:
+cat <<EOF | k apply -f -
-```
 apiVersion: v1
 kind: Service
-metadata:
+metadata: {name: ledger-db, namespace: lima-data}
-  name: ledger-db
+spec: {clusterIP: None, selector: {app: ledger-db}, ports: [{port: 5432}]}
-  namespace: lima-data
-spec:
-  clusterIP: None
-  selector:
-    app: ledger-db
-  ports:
-  - port: 5432
 ---
 apiVersion: apps/v1
 kind: StatefulSet
-metadata:
+metadata: {name: ledger-db, namespace: lima-data}
-  name: ledger-db
-  namespace: lima-data
 spec:
   serviceName: ledger-db
   replicas: 3
-  selector:
+  selector: {matchLabels: {app: ledger-db}}
-    matchLabels:
-      app: ledger-db
   template:
-    metadata:
+    metadata: {labels: {app: ledger-db}}
-      labels:
-        app: ledger-db
     spec:
       containers:
-      - name: db
+        - name: db
-        image: busybox:1.36
+          image: busybox:1.36
-        command: ["sh", "-c", "hostname > /data/id.txt && sleep 86400"]
+          command: [sh, -c, "hostname > /data/id.txt && sleep 86400"]
-        volumeMounts:
+          volumeMounts: [{name: data, mountPath: /data}]
-        - name: data
-          mountPath: /data
   volumeClaimTemplates:
-  - metadata:
+    - metadata: {name: data}
-      name: data
+      spec: {accessModes: [ReadWriteOnce], storageClassName: standard, resources: {requests: {storage: 100Mi}}}
-    spec:
+EOF
-      accessModes:
-      - ReadWriteOnce
-      storageClassName: standard
-      resources:
-        requests:
-          storage: 100Mi
-```""",
-                   run=r"""cat > sts.yaml <<'EOF'
-apiVersion: v1
-kind: Service
-metadata:
-  name: ledger-db
-  namespace: lima-data
-spec:
-  clusterIP: None
-  selector:
-    app: ledger-db
-  ports:
-  - port: 5432
----
-apiVersion: apps/v1
-kind: StatefulSet
-metadata:
-  name: ledger-db
-  namespace: lima-data
-spec:
-  serviceName: ledger-db
-  replicas: 3
-  selector:
-    matchLabels:
-      app: ledger-db
-  template:
-    metadata:
-      labels:
-        app: ledger-db
-    spec:
-      containers:
-      - name: db
-        image: busybox:1.36
-        command: ["sh", "-c", "hostname > /data/id.txt && sleep 86400"]
-        volumeMounts:
-        - name: data
-          mountPath: /data
-  volumeClaimTemplates:
-  - metadata:
-      name: data
-    spec:
-      accessModes:
-      - ReadWriteOnce
-      storageClassName: standard
-      resources:
-        requests:
-          storage: 100Mi
-EOF"""),
-              cmd(r"""k apply -f sts.yaml
 k -n lima-data rollout status sts ledger-db
-k -n lima-data scale sts ledger-db --replicas=2"""),
+k -n lima-data scale sts ledger-db --replicas=2
-              cmd(r"""k -n lima-data get pvc -o custom-columns=NAME:.metadata.name --no-headers | sort > {WS}/q99/pvcs.txt
+k -n lima-data get pvc -o custom-columns=N:.metadata.name --no-headers | grep '^data-ledger-db-' | sort > {WS}/q99/pvcs.txt
-cat {WS}/q99/pvcs.txt""", r"""All three, `data-ledger-db-0` to `-2`: claims are named `<template>-<statefulset>-<ordinal>` and outlive their Pods, so scaling up again gives `ledger-db-2` its old data back.""")],
+cat {WS}/q99/pvcs.txt           # all THREE: data-ledger-db-0, -1, -2
+# Claims are named <template>-<statefulset>-<ordinal> and outlive their pods: scaling down (or
+# deleting the StatefulSet) keeps them unless spec.persistentVolumeClaimRetentionPolicy says
+# otherwise, so scaling back up re-attaches ledger-db-2 to its old data.
+""",
 )
 
 # ------------------------------------------------------------------ 100 ----
@@ -933,63 +849,28 @@ $K -n oslo-db get netpol db-ingress >/dev/null 2>&1 || exit 1
 ! $K -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1 \
  && ! $K -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1"""),
     ],
-    solution=[cmd(r"""k get ns --show-labels | grep -e oslo-api -e lima-api
+    solution="""
-k -n oslo-api get pods --show-labels""", r"""`env=prod` and `env=dev`; `api` is `role=backend`, `debug` is not."""),
+k get ns --show-labels | grep -E 'oslo-api|lima-api'    # env=prod / env=dev
-              hand(r"""vim netpol.yaml""",
+cat <<EOF | k apply -f -
-                   r"""One `from` entry holding both selectors - both must match:
-```
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
-metadata:
+metadata: {name: db-ingress, namespace: oslo-db}
-  name: db-ingress
-  namespace: oslo-db
 spec:
-  podSelector:
+  podSelector: {matchLabels: {app: db}}
-    matchLabels:
+  policyTypes: [Ingress]
-      app: db
-  policyTypes:
-  - Ingress
   ingress:
-  - from:
+    - from:
-    - namespaceSelector:
+        - namespaceSelector: {matchLabels: {env: prod}}
-        matchLabels:
+          podSelector: {matchLabels: {role: backend}}      # SAME list item = AND
-          env: prod
+      ports: [{protocol: TCP, port: 80}]
-      podSelector:
+EOF
-        matchLabels:
+k -n oslo-api exec deploy/api   -- wget -qO- -T3 http://db.oslo-db | head -1     # <!DOCTYPE html>
-          role: backend
+k -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db               # download timed out
-    ports:
+k -n lima-api exec deploy/api   -- wget -qO- -T3 http://db.oslo-db               # download timed out
-    - protocol: TCP
+# The trap is one dash: "- namespaceSelector: ...\\n  - podSelector: ..." (two list items) means OR -
-      port: 80
+# any pod in a prod namespace, OR a backend pod in oslo-db's own namespace. One item with both
-```
+# selectors means AND.
-The trap is one dash: `- podSelector:` would make it a second entry, and the rule an OR - any Pod in a prod namespace, or any backend Pod in `oslo-db`.""",
+""",
-                   run=r"""cat > netpol.yaml <<'EOF'
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: db-ingress
-  namespace: oslo-db
-spec:
-  podSelector:
-    matchLabels:
-      app: db
-  policyTypes:
-  - Ingress
-  ingress:
-  - from:
-    - namespaceSelector:
-        matchLabels:
-          env: prod
-      podSelector:
-        matchLabels:
-          role: backend
-    ports:
-    - protocol: TCP
-      port: 80
-EOF"""),
-              cmd(r"""k apply -f netpol.yaml
-k -n oslo-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db
-k -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db
-k -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db""", r"""nginx's page, then `download timed out` twice.""")],
 )
 
 # ------------------------------------------------------------------ 101 ----
@@ -1038,18 +919,18 @@ for i in $(seq 1 15); do
 done; exit 1"""),
         C("fix.txt names dnsPolicy and its new value", 1, r"""grep -Eqi '^ *dnsPolicy *= *ClusterFirst *$' "$WS/q101/fix.txt" 2>/dev/null"""),
     ],
-    solution=[cmd(r"""k -n lima-client logs deploy/reporter --tail=3
+    solution="""
-k -n lima-client get endpointslices
+k -n lima-client logs deploy/reporter --tail=3            # wget: bad address 'ledger:8080'
-k -n lima-client get deploy reporter -o yaml | grep dnsPolicy""", r"""`bad address 'ledger:8080'`, though the Service has endpoints. `dnsPolicy: Default` uses the *node's* resolver, not cluster DNS - the default policy is `ClusterFirst`, despite the name."""),
+k -n lima-client get endpointslices                       # the Service has endpoints - not the problem
-              hand(r"""k -n lima-client edit deploy reporter""",
+k -n lima-client get deploy reporter -o yaml | grep -n dns
-                   r"""In the Pod template's `spec:`, then `:wq`:
+#   dnsPolicy: Default   <- "Default" means: use the NODE's resolver, NOT cluster DNS
-```
+k -n lima-client exec deploy/reporter -- cat /etc/resolv.conf   # the podman/host resolver, no cluster search domains
-      dnsPolicy: ClusterFirst
+k -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'
-```""",
+k -n lima-client logs deploy/reporter -f                  # "ledger ok"
-                   run=r"""kubectl -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'"""),
+echo dnsPolicy=ClusterFirst > {WS}/q101/fix.txt
-              cmd(r"""echo dnsPolicy=ClusterFirst > {WS}/q101/fix.txt
+# The trap is the name: the DEFAULT dnsPolicy is ClusterFirst, while the policy literally called
-k -n lima-client rollout status deploy reporter
+# "Default" inherits the node's /etc/resolv.conf. ClusterFirstWithHostNet is for hostNetwork pods.
-k -n lima-client logs deploy/reporter --tail=3""", r"""`ledger ok`.""")],
+""",
 )
 
 # ------------------------------------------------------------------ 102 ----
@@ -1096,16 +977,22 @@ eq "$(j -n oslo-files deploy uploader '{.status.availableReplicas}')" 1 \
  && j -n oslo-files deploy uploader '{.spec.template.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /files"""),
         C("reason.txt names the unsupported access mode", 2, r"""grep -Eqi 'ReadWriteMany|RWX|access ?mode' "$WS/q102/reason.txt" 2>/dev/null"""),
     ],
-    solution=[cmd(r"""k -n oslo-files get pod,pvc
+    solution="""
-k -n oslo-files describe pvc shared-files | tail -4""", r"""`ProvisioningFailed ... NodePath only supports ReadWriteOnce and ReadWriteOncePod access modes`: the claim asks for `ReadWriteMany`, which local-path cannot give."""),
+k -n oslo-files get pod                          # Pending
-              cmd(r"""echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
+k -n oslo-files describe pod -l app=uploader | tail -4    # unbound immediate PersistentVolumeClaims / waiting
-k -n oslo-files get pvc shared-files -o yaml > pvc.yaml"""),
+k -n oslo-files describe pvc shared-files | tail -4
-              hand(r"""vim pvc.yaml""",
+#   ProvisioningFailed ... NodePath only supports ReadWriteOnce and ReadWriteOncePod (1.22+) access modes
-                   r"""Under `accessModes:`, change `ReadWriteMany` to `ReadWriteOnce` - name, size and class stay.""",
+# accessModes is immutable on a PVC, so recreate it. The Pending pod was never scheduled, so it
-                   run=r"""yq -i '.spec.accessModes = ["ReadWriteOnce"]' pvc.yaml"""),
+# does not hold the claim (pvc-protection only counts scheduled pods) and the delete goes through.
-              cmd(r"""k replace --force -f pvc.yaml
+k -n oslo-files get pvc shared-files -o yaml > pvc.yaml     # change ReadWriteMany -> ReadWriteOnce, strip status/uid
-k -n oslo-files rollout restart deploy uploader
+k -n oslo-files delete pvc shared-files
-k -n oslo-files get pvc,pod""", r"""Access modes cannot change on a claim: `--force` deletes it and creates it again. The Pending Pod was never scheduled, so it does not hold the claim. Again until it is `Bound` and the Pod `Running`.""")],
+k apply -f pvc.yaml
+k -n oslo-files rollout restart deploy uploader            # or just wait - the pending pod retries
+k -n oslo-files get pvc,pod
+echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
+# RWX needs a filesystem many nodes can mount at once (NFS, CephFS, a cloud file service).
+# ReadWriteOnce is per NODE, not per pod - ReadWriteOncePod is the per-pod one.
+""",
 )
 
 QUESTIONS = Q

cka/bank_storage.py

@@ -5,8 +5,6 @@ served by the local-path provisioner in Namespace local-path-storage), so a task
 provisioner or change the default class. Volumes are tested for real - a claim that binds, a file
 that is still on the node, a Pod that reads it.
 """
-
-from sim.bank import cmd, hand
 from cka.common import C, NS, PY, STOR, ns_pick, name_pick
 
 # what a setup records about the objects a task must leave alone
@@ -52,85 +50,44 @@ gone -n %{ns} pvc %{pvc} || exit 1
 node=$(pvnode "$pv"); path=$(pvpath "$pv"); [ -n "$node" ] && [ -n "$path" ] || exit 1
 eq "$(nx "$node" "cat $path/keep.txt" </dev/null 2>/dev/null | tr -d '\r')" "%{text}" """),
  ],
- "solution": [cmd(r"""k get sc standard -o yaml | grep provisioner""", r"""`rancher.io/local-path`."""),
+ "solution": """
-              hand(r"""vim sc.yaml""",
+cat <<EOF | k apply -f -
-                   r"""The class, from the docs page *Storage Classes* - no `is-default-class` annotation, so not the default - and the claim:
-```
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
-metadata:
+metadata: {name: %{sc}}
-  name: %{sc}
+provisioner: rancher.io/local-path          # k get sc standard -o yaml shows it
-provisioner: rancher.io/local-path
 reclaimPolicy: Retain
 volumeBindingMode: WaitForFirstConsumer
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata:
+metadata: {name: %{pvc}, namespace: %{ns}}
-  name: %{pvc}
-  namespace: %{ns}
 spec:
   storageClassName: %{sc}
-  accessModes:
+  accessModes: [ReadWriteOnce]
-  - ReadWriteOnce
+  resources: {requests: {storage: 300Mi}}
-  resources:
-    requests:
-      storage: 300Mi
-```""",
-                   run=r"""cat > sc.yaml <<'EOF'
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata:
-  name: %{sc}
-provisioner: rancher.io/local-path
-reclaimPolicy: Retain
-volumeBindingMode: WaitForFirstConsumer
 ---
 apiVersion: v1
-kind: PersistentVolumeClaim
+kind: Pod
-metadata:
+metadata: {name: %{pod}, namespace: %{ns}}
-  name: %{pvc}
-  namespace: %{ns}
 spec:
-  storageClassName: %{sc}
+  volumes: [{name: data, persistentVolumeClaim: {claimName: %{pvc}}}]
-  accessModes:
+  containers:
-  - ReadWriteOnce
+    - name: writer
-  resources:
+      image: busybox:1.36
-    requests:
+      command: [sh, -c, "echo %{text} > /data/keep.txt; sleep 86400"]
-      storage: 300Mi
+      volumeMounts: [{name: data, mountPath: /data}]
-EOF"""),
+EOF
-              cmd(r"""k apply -f sc.yaml
+k -n %{ns} get pvc %{pvc}                 # Pending until the Pod is scheduled: WaitForFirstConsumer
-k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sh -c "echo %{text} > /data/keep.txt; sleep 86400" > pod.yaml""", r"""The claim stays `Pending` until a Pod uses it: `WaitForFirstConsumer`."""),
+k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt
-              hand(r"""vim pod.yaml""",
+k -n %{ns} delete pod %{pod}; k -n %{ns} delete pvc %{pvc}
-                   r"""Add to the container, below `name: %{pod}`:
+k get pv $(cat {WS}/q114/pv.txt)          # STATUS Released, RECLAIM POLICY Retain
-```
+echo Released > {WS}/q114/phase.txt
-    volumeMounts:
+# A class's reclaimPolicy is copied into every volume it provisions. Delete (the default) removes
-    - name: data
+# the volume and its data with the claim; Retain leaves a Released volume that no new claim can
-      mountPath: /data
+# bind until an administrator clears spec.claimRef. A StorageClass cannot be edited afterwards,
-```
+# except for a few fields - to change the policy of one volume, patch the PV.
-and to the Pod's `spec:`, right below `spec:`:
+""",
-```
-  volumes:
-  - name: data
-    persistentVolumeClaim:
-      claimName: %{pvc}
-```""",
-                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}]
-  | .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "%{pvc}"}}]' pod.yaml"""),
-              cmd(r"""k apply -f pod.yaml"""),
-              hand(r"""k -n %{ns} get pod,pvc""",
-                   r"""Again until the Pod is `Running` and the claim `Bound`; note its `VOLUME`.""",
-                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
-              hand(r"""echo pvc-0b1c2d3e-... > {WS}/q114/pv.txt""",
-                   r"""With the volume's name.""",
-                   run=r"""kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt"""),
-              cmd(r"""k -n %{ns} delete pod %{pod} --now
-k -n %{ns} delete pvc %{pvc}"""),
-              hand(r"""k get pv
-echo Released > {WS}/q114/phase.txt""",
-                   r"""`STATUS Released`, `RECLAIM POLICY Retain`: the class's policy is copied into every volume it provisions. A Released volume binds no new claim until its `claimRef` is cleared.""",
-                   run=r"""sleep 4
-kubectl get pv $(cat {WS}/q114/pv.txt) -o jsonpath='{.status.phase}' > {WS}/q114/phase.txt""")],
 },
 # ------------------------------------------------------------------ 115 ----
 {
@@ -176,43 +133,39 @@ eq "$(one "$WS/q115/claim.txt" | sed 's#.*/##')" %{pod}-scratch || exit 1
 eq "$(j -n %{ns} pvc %{pod}-scratch '{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}')" Pod/%{pod} || exit 1
 eq "$(one "$WS/q115/owner.txt" | tr A-Z a-z)" pod/%{pod}"""),
  ],
- "solution": [cmd(r"""k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
+ "solution": """
-              hand(r"""vim pod.yaml""",
+cat <<EOF | k apply -f -
-                   r"""Add to the container, below `name: %{pod}`:
+apiVersion: v1
-```
+kind: Pod
-    volumeMounts:
+metadata: {name: %{pod}, namespace: %{ns}}
-    - name: scratch
+spec:
-      mountPath: /scratch
-    - name: cache
-      mountPath: /cache
-```
-and to the Pod's `spec:`, right below `spec:` (the docs page *Ephemeral Volumes* has the generic ephemeral volume):
-```
   volumes:
-  - name: scratch
+    - name: scratch
-    ephemeral:
+      ephemeral:
-      volumeClaimTemplate:
+        volumeClaimTemplate:
-        spec:
+          spec:
-          storageClassName: standard
+            storageClassName: standard
-          accessModes:
+            accessModes: [ReadWriteOnce]
-          - ReadWriteOnce
+            resources: {requests: {storage: 100Mi}}
-          resources:
+    - name: cache
-            requests:
+      emptyDir: {medium: Memory, sizeLimit: 32Mi}
-              storage: 100Mi
+  containers:
-  - name: cache
+    - name: encoder
-    emptyDir:
+      image: busybox:1.36
-      medium: Memory
+      command: [sleep, "86400"]
-      sizeLimit: 32Mi
+      volumeMounts:
-```""",
+        - {name: scratch, mountPath: /scratch}
-                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "scratch", "mountPath": "/scratch"}, {"name": "cache", "mountPath": "/cache"}]
+        - {name: cache, mountPath: /cache}
-  | .spec.volumes = [{"name": "scratch", "ephemeral": {"volumeClaimTemplate": {"spec": {"storageClassName": "standard", "accessModes": ["ReadWriteOnce"], "resources": {"requests": {"storage": "100Mi"}}}}}}, {"name": "cache", "emptyDir": {"medium": "Memory", "sizeLimit": "32Mi"}}]' pod.yaml"""),
+EOF
-              cmd(r"""k apply -f pod.yaml"""),
+k -n %{ns} get pvc                                   # %{pod}-scratch: <pod name>-<volume name>
-              hand(r"""k -n %{ns} get pod,pvc""",
+echo %{pod}-scratch > {WS}/q115/claim.txt
-                   r"""Again until the Pod is `Running`. The claim is `<pod>-<volume>`: `%{pod}-scratch`.""",
+k -n %{ns} get pvc %{pod}-scratch -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}'
-                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
+echo Pod/%{pod} > {WS}/q115/owner.txt
-              cmd(r"""echo %{pod}-scratch > {WS}/q115/claim.txt
+# A generic ephemeral volume is a claim written into the Pod: Kubernetes creates the PVC, the Pod
-k -n %{ns} get pvc %{pod}-scratch -o yaml | grep -A5 ownerReferences
+# owns it, and deleting the Pod deletes claim and volume. Unlike emptyDir it can come from any
-echo Pod/%{pod} > {WS}/q115/owner.txt""", r"""The Pod owns it: delete the Pod, and the claim and its volume go too. An emptyDir in `Memory` is a tmpfs, counted as the container's memory.""")],
+# StorageClass - size, speed, snapshots - and unlike a normal claim it never outlives the Pod.
+# An emptyDir with medium Memory is a tmpfs: what is written there counts as the container's memory.
+""",
 },
 # ------------------------------------------------------------------ 116 ----
 {
@@ -281,20 +234,18 @@ gone -n %{ns} pvc data-%{sts}-1 && gone -n %{ns} pvc data-%{sts}-2 && eq "$(j -n
   C("claims.txt lists what is left: data-%{sts}-0", 1, r"""
 eq "$(tr -d ' \t\r' < "$WS/q116/claims.txt" 2>/dev/null | sed 's#.*/##' | grep . | sort | tr '\n' ' ')" "data-%{sts}-0 " """),
  ],
- "solution": [cmd(r"""k -n %{ns} get sts %{sts} -o yaml | grep -A2 persistentVolumeClaimRetentionPolicy""", r"""`Retain` for both: by default a StatefulSet never deletes a claim."""),
+ "solution": """
-              hand(r"""k -n %{ns} edit sts %{sts}""",
+k -n %{ns} get sts %{sts} -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}'      # whenDeleted Retain, whenScaled Retain
-                   r"""Set it before scaling - it applies to the Pods being removed - then `:wq`:
+k -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'
-```
+k -n %{ns} scale sts %{sts} --replicas=1
-  persistentVolumeClaimRetentionPolicy:
+k -n %{ns} get pvc -w                     # data-%{sts}-2 and data-%{sts}-1 disappear with their Pods
-    whenDeleted: Retain
+k -n %{ns} get pvc -o name | cut -d/ -f2 > {WS}/q116/claims.txt
-    whenScaled: Delete
+# By default a StatefulSet never deletes a claim: scale down and up again and each replica finds
-```""",
+# its old data. persistentVolumeClaimRetentionPolicy changes that per event - whenScaled for the
-                   run=r"""kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'"""),
+# replicas a scale-down removes, whenDeleted for all of them when the StatefulSet goes. Set the
-              cmd(r"""k -n %{ns} scale sts %{sts} --replicas=1"""),
+# policy BEFORE scaling: it is applied to the Pods being removed, not to claims orphaned earlier
-              hand(r"""k -n %{ns} get pvc""",
+# (those you delete by hand). It is one of the few StatefulSet fields that can be changed in place.
-                   r"""Again until `data-%{sts}-2` and `data-%{sts}-1` have gone with their Pods.""",
+""",
-                   run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc --no-headers | wc -l)" = 1 ] && break; sleep 3; done"""),
-              cmd(r"""echo data-%{sts}-0 > {WS}/q116/claims.txt""")],
 },
 # ------------------------------------------------------------------ 117 ----
 {
@@ -341,76 +292,34 @@ eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}/{.spec.storageClassName}/{.spec.
 [ -n "$(j pv %{pv} '{.metadata.name}')" ] || exit 1
 eq "$(j -n %{oth} pvc any-volume '{.metadata.uid}')" "$(cat "$PRIV/q117.uid")" && eq "$(j -n %{oth} pvc any-volume '{.status.phase}')" Pending"""),
  ],
- "solution": [cmd(r"""k -n %{oth} get pvc any-volume""", r"""`Pending`, for any 1Gi `manual` volume: an Available one would go to it."""),
+ "solution": """
-              hand(r"""vim pv.yaml""",
+cat <<EOF | k apply -f -
-                   r"""A `claimRef` on the PV reserves it for one claim, so the waiting one cannot take it:
-```
 apiVersion: v1
 kind: PersistentVolume
-metadata:
+metadata: {name: %{pv}}
-  name: %{pv}
 spec:
+  capacity: {storage: 1Gi}
+  accessModes: [ReadWriteOnce]
   storageClassName: manual
-  capacity:
-    storage: 1Gi
-  accessModes:
-  - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
-  hostPath:
+  hostPath: {path: /mnt/%{pv}}
-    path: /mnt/%{pv}
+  claimRef: {namespace: %{ns}, name: %{pvc}}      # the reservation: no other claim can bind this volume
-  claimRef:
-    namespace: %{ns}
-    name: %{pvc}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata:
+metadata: {name: %{pvc}, namespace: %{ns}}
-  name: %{pvc}
-  namespace: %{ns}
 spec:
   storageClassName: manual
-  accessModes:
+  accessModes: [ReadWriteOnce]
-  - ReadWriteOnce
+  resources: {requests: {storage: 1Gi}}
-  resources:
+EOF
-    requests:
+k get pv %{pv}; k -n %{ns} get pvc %{pvc}; k -n %{oth} get pvc any-volume      # Bound, Bound, Pending
-      storage: 1Gi
+# Without claimRef the volume would be Available, and the control plane hands an Available volume
-```
+# to whichever matching claim it looks at first - here the one that has been waiting. That cannot
-`volumeName` on a claim is the other half: it asks for one volume, but keeps no other claim away from it.""",
+# be undone cleanly: a claim whose volume is taken away ends up Lost. claimRef on the PV reserves
-                   run=r"""cat > pv.yaml <<'EOF'
+# it; volumeName on a claim is the other half (it asks for one volume but does not keep others
-apiVersion: v1
+# away from it). Use both when it has to be this pair and no other.
-kind: PersistentVolume
+""",
-metadata:
-  name: %{pv}
-spec:
-  storageClassName: manual
-  capacity:
-    storage: 1Gi
-  accessModes:
-  - ReadWriteOnce
-  persistentVolumeReclaimPolicy: Retain
-  hostPath:
-    path: /mnt/%{pv}
-  claimRef:
-    namespace: %{ns}
-    name: %{pvc}
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: %{pvc}
-  namespace: %{ns}
-spec:
-  storageClassName: manual
-  accessModes:
-  - ReadWriteOnce
-  resources:
-    requests:
-      storage: 1Gi
-EOF"""),
-              cmd(r"""k apply -f pv.yaml
-k get pv %{pv}
-k -n %{ns} get pvc %{pvc}
-k -n %{oth} get pvc any-volume""", r"""`Bound`, `Bound`, still `Pending`.""")],
 },
 # ------------------------------------------------------------------ 118 ----
 {
@@ -496,53 +405,30 @@ done; exit 1"""),
 eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}')" %{pv} || exit 1
 for i in $(seq 1 10); do gone pv "$(cat "$PRIV/q118.wrong")" && exit 0; sleep 3; done; exit 1"""),
  ],
- "solution": [cmd(r"""k -n %{ns} get pvc %{pvc}
+ "solution": """
-k get pv %{pv}""", r"""Bound to a `pvc-...` volume of class `standard`, while `%{pv}` is `Available` with no class: the manifest names no class, so the claim got the default one, and a new empty volume."""),
+k -n %{ns} get pvc %{pvc}                  # Bound to pvc-..., STORAGECLASS standard - not to %{pv}
-              hand(r"""echo pvc-0b1c2d3e-... > {WS}/q118/wrong-pv.txt""",
+k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt
-                   r"""With the claim's `VOLUME`.""",
+k get pv %{pv}                             # Available, STORAGECLASS empty
-                   run=r"""kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt"""),
+# The manifest names no class, so the claim was given the cluster's default one (standard) and a
-              cmd(r"""k -n %{ns} scale deploy %{dep} --replicas=0""", r"""A claim's class and volume cannot be edited, and a claim in use is not deleted (pvc-protection): stop the Pod first."""),
+# brand-new, empty volume was provisioned for it. A claim's class and volume cannot be edited:
-              hand(r"""k -n %{ns} get pods""",
+k -n %{ns} scale deploy %{dep} --replicas=0          # a claim in use is not deleted (pvc-protection)
-                   r"""Again until the Pod has gone.""",
+k -n %{ns} delete pvc %{pvc}                         # its volume goes with it: reclaim policy Delete
-                   run=r"""kubectl -n %{ns} wait --for=delete pod -l app=%{dep} --timeout=120s"""),
+cat <<EOF | k apply -f -
-              cmd(r"""k -n %{ns} delete pvc %{pvc}""", r"""Its volume goes with it: reclaim policy `Delete`."""),
-              hand(r"""vim pvc.yaml""",
-                   r"""`storageClassName: ""` - no class: never the default, never provisioned - and the volume by name:
-```
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata:
+metadata: {name: %{pvc}, namespace: %{ns}}
-  name: %{pvc}
-  namespace: %{ns}
 spec:
-  storageClassName: ""
+  storageClassName: ""            # "no class": never the default, never provisioned
   volumeName: %{pv}
-  accessModes:
+  accessModes: [ReadWriteOnce]
-  - ReadWriteOnce
+  resources: {requests: {storage: 500Mi}}
-  resources:
+EOF
-    requests:
-      storage: 500Mi
-```
-`volumeName` alone would not do: the default class is still filled in, and it does not match the class-less volume.""",
-                   run=r"""cat > pvc.yaml <<'EOF'
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: %{pvc}
-  namespace: %{ns}
-spec:
-  storageClassName: ""
-  volumeName: %{pv}
-  accessModes:
-  - ReadWriteOnce
-  resources:
-    requests:
-      storage: 500Mi
-EOF"""),
-              cmd(r"""k apply -f pvc.yaml
 k -n %{ns} scale deploy %{dep} --replicas=1
-k -n %{ns} rollout status deploy %{dep}
+k -n %{ns} logs deploy/%{dep}              # ledger %{tok} balanced
-k -n %{ns} logs deploy/%{dep}""", r"""`ledger %{tok} balanced`.""")],
+# storageClassName has three states: a name; "" = only volumes without a class, no provisioning;
+# and absent = "give me the default class". volumeName alone would not have helped: the default
+# class is still filled in, and it does not match the class-less volume.
+""",
 },
 # ------------------------------------------------------------------ 119 ----
 {
@@ -606,120 +492,34 @@ eq "$(j pv %{{p}}-{u} '{{.metadata.uid}}')" "$(sed -n {n}p "$PRIV/q119.uids")" |
 eq "$(j pv %{{p}}-{u} '{{.spec.capacity.storage}}/{{.spec.accessModes[*]}}/{{.spec.storageClassName}}/{{.metadata.labels.use}}')" {size}/ReadWriteOnce/manual/{u}""")
   for n, (u, size) in enumerate([("reports", "1Gi"), ("archive", "2Gi"), ("media", "1Gi")], 1)
  ],
- "solution": [cmd(r"""k -n %{ns} get pvc
+ "solution": """
-k -n %{ns} describe pvc reports archive media | grep -e ^Name: -e Warning -e Normal""", r"""`media`: `storageclass.storage.k8s.io "manuel" not found` - the class, a typo. The other two only say no volume fits: compare claim and volume yourself."""),
+k -n %{ns} get pvc; k get pv --show-labels
-              cmd(r"""k -n %{ns} get pvc -o custom-columns=NAME:.metadata.name,SIZE:.spec.resources.requests.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
+k -n %{ns} describe pvc reports archive media | grep -E '^Name:|Warning|Normal'
-k get pv -o custom-columns=NAME:.metadata.name,SIZE:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName""", r"""`reports` asks for 5Gi of a 1Gi volume; `archive` for `ReadWriteMany`, its volume offers `ReadWriteOnce`. A claim binds only when the class matches, the volume is large enough, it has every access mode asked for, and the selector matches."""),
+#   media:   storageclass.storage.k8s.io "manuel" not found           -> class (a typo for manual)
-              hand(r"""vim {WS}/q119/reasons.txt""",
+#   reports, archive: no persistent volumes available for this claim   -> compare claim and volume:
-                   r"""```
+k -n %{ns} get pvc -o custom-columns=NAME:.metadata.name,SIZE:.spec.resources.requests.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
-reports capacity
+k get pv -o custom-columns=NAME:.metadata.name,SIZE:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
-archive accessmode
+#   reports asks for 5Gi, its volume has 1Gi                          -> capacity
-media class
+#   archive asks for ReadWriteMany, its volume offers ReadWriteOnce   -> accessmode
-```""",
+printf 'reports capacity\\narchive accessmode\\nmedia class\\n' > {WS}/q119/reasons.txt
-                   run=r"""printf 'reports capacity\narchive accessmode\nmedia class\n' > {WS}/q119/reasons.txt"""),
+# size (downwards), access modes and class of a claim cannot be edited: recreate the three
-              cmd(r"""k -n %{ns} delete pvc reports archive media""", r"""None of the three fields can be edited on a claim: recreate them."""),
+k -n %{ns} delete pvc reports archive media
-              hand(r"""vim pvcs.yaml""",
+for c in "reports 1Gi" "archive 2Gi" "media 1Gi"; do set -- $c; cat <<EOF | k apply -f -
-                   r"""The three claims again, each keeping its selector:
-```
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata:
+metadata: {name: $1, namespace: %{ns}}
-  name: reports
-  namespace: %{ns}
 spec:
   storageClassName: manual
-  accessModes:
+  accessModes: [ReadWriteOnce]
-  - ReadWriteOnce
+  selector: {matchLabels: {use: $1}}
-  selector:
+  resources: {requests: {storage: $2}}
-    matchLabels:
+EOF
-      use: reports
+done
-  resources:
+k -n %{ns} get pvc                         # all Bound, each to its own volume
-    requests:
+# A claim binds a volume only if ALL of these hold: same class, the volume is at least as large,
-      storage: 1Gi
+# it offers every access mode the claim asks for, and it matches the claim's selector. The event
----
+# only names the class problem - for the rest you compare the two objects yourself.
-apiVersion: v1
+""",
-kind: PersistentVolumeClaim
-metadata:
-  name: archive
-  namespace: %{ns}
-spec:
-  storageClassName: manual
-  accessModes:
-  - ReadWriteOnce
-  selector:
-    matchLabels:
-      use: archive
-  resources:
-    requests:
-      storage: 2Gi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: media
-  namespace: %{ns}
-spec:
-  storageClassName: manual
-  accessModes:
-  - ReadWriteOnce
-  selector:
-    matchLabels:
-      use: media
-  resources:
-    requests:
-      storage: 1Gi
-```""",
-                   run=r"""cat > pvcs.yaml <<'EOF'
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: reports
-  namespace: %{ns}
-spec:
-  storageClassName: manual
-  accessModes:
-  - ReadWriteOnce
-  selector:
-    matchLabels:
-      use: reports
-  resources:
-    requests:
-      storage: 1Gi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: archive
-  namespace: %{ns}
-spec:
-  storageClassName: manual
-  accessModes:
-  - ReadWriteOnce
-  selector:
-    matchLabels:
-      use: archive
-  resources:
-    requests:
-      storage: 2Gi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: media
-  namespace: %{ns}
-spec:
-  storageClassName: manual
-  accessModes:
-  - ReadWriteOnce
-  selector:
-    matchLabels:
-      use: media
-  resources:
-    requests:
-      storage: 1Gi
-EOF"""),
-              cmd(r"""k apply -f pvcs.yaml
-k -n %{ns} get pvc""", r"""All three `Bound`, each to its own volume.""")],
 },
 # ------------------------------------------------------------------ 120 ----
 {
@@ -805,20 +605,24 @@ done; exit 1"""),
 eq "$(j pv %{pv} '{.metadata.uid}')" "$(sed -n 1p "$PRIV/q120.uids")" && eq "$(j -n %{ns} pvc %{dep}-data '{.metadata.uid}')" "$(sed -n 2p "$PRIV/q120.uids")" \
  && eq "$(pvnode %{pv})/$(pvpath %{pv})" "$W1//mnt/%{pv}" """),
  ],
- "solution": [hand(r"""k -n %{ns} describe pod -l app=%{dep} | grep FailedScheduling > {WS}/q120/event.txt
+ "solution": """
-cat {WS}/q120/event.txt""",
+k -n %{ns} describe pod -l app=%{dep} | tail -5
-                   r"""`1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) didn't match Pod's node affinity/selector`: the scheduler ANDs every constraint - taints, the Pod's selector, and the node affinity of each volume it mounts.""",
+#   0/3 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s)
-                   run=r"""for i in $(seq 1 40); do
+#   didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s). ...
-  kubectl -n %{ns} describe pod -l app=%{dep} | grep FailedScheduling > {WS}/q120/event.txt && break; sleep 3
+k -n %{ns} get events --field-selector reason=FailedScheduling -o jsonpath='{.items[-1:].message}' > {WS}/q120/event.txt
-done"""),
+k get pv %{pv} -o jsonpath='{.spec.nodeAffinity}'; echo       # the volume is on {W1}
-              cmd(r"""k describe pv %{pv} | grep -A3 'Node Affinity'
+k -n %{ns} get deploy %{dep} -o jsonpath='{.spec.template.spec.nodeSelector}'; echo    # disk=ssd
-k -n %{ns} get deploy %{dep} -o yaml | grep -A1 nodeSelector
+k get nodes -L disk                                           # only {W2} carries disk=ssd
-k get nodes -L disk""", r"""The volume is on `{W1}`; the Pod must go to `disk=ssd`, which only `{W2}` has. The volume cannot move, so the Pod has to."""),
+# The volume cannot move, so the Pod has to: drop the nodeSelector (the volume's own node
-              hand(r"""k -n %{ns} edit deploy %{dep}""",
+# affinity already sends the Pod to the right node) ...
-                   r"""Delete the two lines of the `nodeSelector` (`nodeSelector:` and `disk: ssd`), then `:wq` - the volume's node affinity already sends the Pod to the right node. (If the selector has a reason to exist, `k label node {W1} disk=ssd` instead.)""",
+k -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
-                   run=r"""kubectl -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'"""),
+# ... or, if the selector has a reason to exist, give {W1} the label:  k label node {W1} disk=ssd
-              cmd(r"""k -n %{ns} rollout status deploy %{dep}
+k -n %{ns} rollout status deploy %{dep}; k -n %{ns} logs deploy/%{dep}
-k -n %{ns} logs deploy/%{dep}""", r"""`orders %{tok} open`.""")],
+# The scheduler ANDs every constraint: taints, the Pod's nodeSelector/affinity, and the node
+# affinity of each volume it mounts. "didn't match PersistentVolume's node affinity" (before v1.35:
+# "had volume node affinity conflict") always means: the Pod is not allowed onto the one node its
+# volume lives on.
+""",
 },
 # ------------------------------------------------------------------ 121 ----
 {
@@ -1078,15 +882,19 @@ running %{ns} %{pod} 60"""),
   C("class.txt holds %{sc}", 1, STOR + r"""
 eq "$(j -n %{ns} pvc %{pvc} '{.spec.storageClassName}')" %{sc} && eq "$(one "$WS/q123/class.txt")" %{sc}"""),
  ],
- "solution": [cmd(r"""k get sc
+ "solution": """
-k -n %{ns} describe pvc %{pvc} | tail -2""", r"""No `(default)` anywhere, and the claim waits: `no storage class is set`."""),
+k get sc                                   # no "(default)" behind any name
-              cmd(r"""k annotate sc %{sc} storageclass.kubernetes.io/is-default-class=true
+k -n %{ns} describe pvc %{pvc} | tail -2   # no persistent volumes available for this claim and no storage class is set
-k get sc""", r"""`%{sc} (default)`. A claim without a class gets the default when it is created - or, if there is none, as soon as one appears: no need to recreate it."""),
+k patch sc %{sc} -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
-              hand(r"""k -n %{ns} get pvc,pod""",
+k get sc                                   # %{sc} (default)
-                   r"""Again until the claim shows `STORAGECLASS %{sc}` and `Bound`, and the Pod `Running`.""",
+k -n %{ns} get pvc %{pvc} -w               # within seconds: STORAGECLASS %{sc}, then Bound once the Pod is scheduled
-                   run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.status.phase}')" = Bound ] && break; sleep 3; done
+k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.storageClassName}' > {WS}/q123/class.txt
-kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
+# A claim without storageClassName gets the default class when it is created. If there is none
-              cmd(r"""echo %{sc} > {WS}/q123/class.txt""", r"""Marking `standard` would have bound it too - to the wrong class. With two defaults, new claims take the newest; never leave two.""")],
+# it stays Pending with the field unset - and is given the default as soon as one appears: the
+# claim does not have to be recreated. Marking `standard` instead would have bound it as well,
+# to the wrong class. With two defaults, new claims get the most recently created one - never
+# leave two.
+""",
 },
 # ------------------------------------------------------------------ 124 ----
 {
@@ -1129,116 +937,53 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.nodeName}')" "$W2" || exit 1
 [ -z "$(j -n %{ns} pod %{pod} '{.spec.nodeSelector}{.spec.affinity}')" ] || exit 1
 eq "$($K -n %{ns} exec %{pod} -- cat /disk/disk.id 2>/dev/null | tr -d '\r')" "disk %{tok}" """),
  ],
- "solution": [hand(r"""vim local.yaml""",
+ "solution": """
-                   r"""The docs page *Storage Classes* (under *Local*) has the class, *Persistent Volumes* a `local` volume with its required node affinity:
+cat <<EOF | k apply -f -
-```
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
-metadata:
+metadata: {name: %{sc}}
-  name: %{sc}
 provisioner: kubernetes.io/no-provisioner
 volumeBindingMode: WaitForFirstConsumer
 ---
 apiVersion: v1
 kind: PersistentVolume
-metadata:
+metadata: {name: %{pv}}
-  name: %{pv}
 spec:
-  capacity:
+  capacity: {storage: 1Gi}
-    storage: 1Gi
+  accessModes: [ReadWriteOnce]
-  accessModes:
-  - ReadWriteOnce
-  persistentVolumeReclaimPolicy: Retain
   storageClassName: %{sc}
-  local:
+  persistentVolumeReclaimPolicy: Retain
-    path: /mnt/disks/%{pv}
+  local: {path: /mnt/disks/%{pv}}
-  nodeAffinity:
+  nodeAffinity:                              # required for a local volume
     required:
       nodeSelectorTerms:
-      - matchExpressions:
+        - matchExpressions: [{key: kubernetes.io/hostname, operator: In, values: [{W2}]}]
-        - key: kubernetes.io/hostname
-          operator: In
-          values:
-          - {W2}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata:
+metadata: {name: %{pvc}, namespace: %{ns}}
-  name: %{pvc}
-  namespace: %{ns}
 spec:
   storageClassName: %{sc}
-  accessModes:
+  accessModes: [ReadWriteOnce]
-  - ReadWriteOnce
+  resources: {requests: {storage: 1Gi}}
-  resources:
-    requests:
-      storage: 1Gi
-```""",
-                   run=r"""cat > local.yaml <<'EOF'
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata:
-  name: %{sc}
-provisioner: kubernetes.io/no-provisioner
-volumeBindingMode: WaitForFirstConsumer
 ---
 apiVersion: v1
-kind: PersistentVolume
+kind: Pod
-metadata:
+metadata: {name: %{pod}, namespace: %{ns}}
-  name: %{pv}
 spec:
-  capacity:
+  volumes: [{name: disk, persistentVolumeClaim: {claimName: %{pvc}}}]
-    storage: 1Gi
+  containers:
-  accessModes:
+    - name: prom
-  - ReadWriteOnce
+      image: busybox:1.36
-  persistentVolumeReclaimPolicy: Retain
+      command: [sleep, "86400"]
-  storageClassName: %{sc}
+      volumeMounts: [{name: disk, mountPath: /disk}]
-  local:
+EOF
-    path: /mnt/disks/%{pv}
+k -n %{ns} get pod %{pod} -o wide          # on {W2}
-  nodeAffinity:
+k -n %{ns} exec %{pod} -- cat /disk/disk.id
-    required:
+# hostPath is a directory on whatever node the Pod happens to land on; a `local` volume knows
-      nodeSelectorTerms:
+# which node it is on, and the scheduler follows it. WaitForFirstConsumer delays binding until a
-      - matchExpressions:
+# Pod needs the claim, so that the Pod's other constraints are weighed together with the volume's.
-        - key: kubernetes.io/hostname
+# The page to search for: "Storage Classes" -> Local.
-          operator: In
+""",
-          values:
-          - {W2}
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: %{pvc}
-  namespace: %{ns}
-spec:
-  storageClassName: %{sc}
-  accessModes:
-  - ReadWriteOnce
-  resources:
-    requests:
-      storage: 1Gi
-EOF"""),
-              cmd(r"""k apply -f local.yaml
-k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
-              hand(r"""vim pod.yaml""",
-                   r"""Add to the container, below `name: %{pod}`:
-```
-    volumeMounts:
-    - name: disk
-      mountPath: /disk
-```
-and to the Pod's `spec:`, right below `spec:`:
-```
-  volumes:
-  - name: disk
-    persistentVolumeClaim:
-      claimName: %{pvc}
-```""",
-                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "disk", "mountPath": "/disk"}]
-  | .spec.volumes = [{"name": "disk", "persistentVolumeClaim": {"claimName": "%{pvc}"}}]' pod.yaml"""),
-              cmd(r"""k apply -f pod.yaml"""),
-              hand(r"""k -n %{ns} get pod %{pod} -o wide""",
-                   r"""Again until `Running` - on `{W2}`, sent there by the volume alone. A `hostPath` is a directory on whatever node the Pod lands on; a `local` volume knows its node, and the scheduler follows it.""",
-                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
-              cmd(r"""k -n %{ns} exec %{pod} -- cat /disk/disk.id""")],
 },
 ]

cka/lessons.py

@@ -10,14 +10,11 @@ Lessons write nothing to the history or the scores - only a "lesson done" mark o
 ./ksim lesson audit [OBJ..]   prove lessons: every step fails before its commands and passes after
 
 A lesson's host is named after its id - lesson 9101 runs on `cka9101` - which never clashes with a
-task's. Its files go under {WS}/lesson-<objective>/. A step's `show` is how you would do it in the
+task's. Its files go under {WS}/lesson-<objective>/. A step's `show` commands are what the learner
-exam: cmd() for what you type as it stands, hand() for what you do by hand - `vim` a generated
+types; the audit runs exactly them, as `candidate` on the host, so what a lesson shows always
-skeleton or a manifest pasted from the docs, `kubectl edit`, `ssh NODE` and `sudo -i`, a value you
+works. Work on another node is shown as `ssh NODE sudo bash <<'EOF' ... EOF`: the same commands
-read and `echo` - each with a script that does the same unattended. The audit runs the commands
+you would type after `ssh NODE` and `sudo -i`, in a form that runs unattended. A single command
-as `candidate` on the host and, for each hand() part, its script; `./ksim lesson lint` keeps
+there is `ssh -n NODE 'COMMAND'`: without -n, ssh would read the rest of the commands as its input.
-scripts (sed, awk, yq, heredocs, loops, JSON patches) out of what you type. In a script, a command
-on another node is `ssh NODE sudo bash <<'EOF' ... EOF` or `ssh -n NODE 'COMMAND'`: without -n,
-ssh would read the rest of the script as its input.
 
 One file per domain, as with the task banks. Besides the curriculum's competencies, the lessons
 cover what a full CKA course teaches under each of them (2026-10-19: the topics of KubeDojo's CKA

cka/lessons_arch.py

@@ -1,7 +1,7 @@
 """CKA lessons of the Cluster Architecture, Installation and Configuration domain (4.1-4.8). See
 cka/lessons.py for what a lesson is."""
 from cka.common import backup_operator
-from sim.bank import CAN, NS, PY, step, cmd, hand
+from sim.bank import CAN, NS, PY, step
 
 _ETCDCTL = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
             "--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key")
@@ -38,20 +38,23 @@ S=system:serviceaccount:team:deployer
 can --as=$S -- create deployments.apps -n team && can --as=$S -- list pods -n team && cannot --as=$S -- delete pods -n team \
   && cannot --as=$S -- create deployments.apps -n default""",
              hint=r"""`kubectl create role deploy-edit --verb=... --resource=deployments.apps` - a Role with two resources and different verbs needs two rules, so write YAML or create it and `kubectl edit`. `kubectl create rolebinding NAME --role=ROLE --serviceaccount=team:deployer`.""",
-             show=[cmd(r"""k -n team create sa deployer
+             show=r"""kubectl -n team create serviceaccount deployer
-k -n team create role deploy-edit --verb=get,list,watch,create,update,patch,delete --resource=deployments.apps --dry-run=client -o yaml > role.yaml"""),
+kubectl -n team apply -f - <<'EOF'
-                   hand(r"""vim role.yaml""",
+apiVersion: rbac.authorization.k8s.io/v1
-                        r"""`kubectl create role` gives every resource the same verbs: add the second rule, for Pods, at the end of `rules:`:
+kind: Role
-```
+metadata:
+  name: deploy-edit
+rules:
+- apiGroups: [apps]
+  resources: [deployments]
+  verbs: [get, list, watch, create, update, patch, delete]
 - apiGroups: [""]
   resources: [pods]
   verbs: [get, list]
-```""",
+EOF
-                        run=r"""yq -i '.rules += [{"apiGroups": [""], "resources": ["pods"], "verbs": ["get", "list"]}]' role.yaml"""),
+kubectl -n team create rolebinding deployer --role=deploy-edit --serviceaccount=team:deployer
-                   cmd(r"""k apply -f role.yaml
+kubectl auth can-i create deployments -n team --as=system:serviceaccount:team:deployer
-k -n team create rolebinding deployer --role=deploy-edit --serviceaccount=team:deployer
+kubectl auth can-i delete pods -n team --as=system:serviceaccount:team:deployer"""),
-k auth can-i create deploy -n team --as=system:serviceaccount:team:deployer
-k auth can-i delete po -n team --as=system:serviceaccount:team:deployer""", r"""`yes`, then `no`.""")]),
         step("One ClusterRole, one Namespace",
              r"""Create ClusterRole `pod-reader` (`get`, `list`, `watch` on pods). Give it to the user **`jane`** in Namespace `team` only - with a **RoleBinding** `jane-reads-pods`, not a ClusterRoleBinding.
 
@@ -61,10 +64,10 @@ eq "$(j clusterrole pod-reader '{.rules[0].verbs}')" '["get","list","watch"]' ||
 eq "$(j -n team rolebinding jane-reads-pods '{.roleRef.kind}/{.roleRef.name}')" ClusterRole/pod-reader || exit 1
 can --as=jane -- list pods -n team && cannot --as=jane -- list pods -n default && cannot --as=jane -- list pods -A""",
              hint=r"""`kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods`, then `kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane`. Users exist only in certificates and tokens: nothing to create for `jane`.""",
-             show=[cmd(r"""k create clusterrole pod-reader --verb=get,list,watch --resource=pods
+             show=r"""kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods
-k -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane
+kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane
-k auth can-i list po -n team --as=jane
+kubectl auth can-i list pods -n team --as=jane
-k auth can-i list po -n default --as=jane""", r"""`yes` in `team`, `no` elsewhere.""")]),
+kubectl auth can-i list pods -n default --as=jane"""),
         step("A token for the ServiceAccount",
              r"""Issue a token for `deployer` that is valid for **one hour**, and save it in `{WS}/lesson-4.1/deployer.token`. Then use it - as the **only** credential: your kubeconfig carries the admin's client certificate, and the API server checks a certificate before a token, so `kubectl --token=...` alone would still be you. With no kubeconfig, just the server, the CA and the token, `kubectl auth whoami` says `system:serviceaccount:team:deployer`, `get deployments` works in `team` and `get secrets` is forbidden.""",
              check=r"""
@@ -76,41 +79,44 @@ import base64, json, sys
 s = sys.stdin.read().strip(); c = json.loads(base64.b64decode(s + "=" * (-len(s) % 4)))
 sys.exit(0 if c["exp"] - c["iat"] == 3600 else 1)'""",
              hint=r"""`kubectl -n team create token deployer --duration=1h`. Then `kubectl --kubeconfig=/dev/null --server=URL --certificate-authority=/etc/kubernetes/pki/ca.crt --token=TOKEN ...`; the URL is in your kubeconfig. The token is a JWT: its middle part, base64-decoded, holds `sub`, `iat` and `exp`.""",
-             show=[cmd(r"""k -n team create token deployer --duration=1h > {WS}/lesson-4.1/deployer.token
+             show=r"""kubectl -n team create token deployer --duration=1h > {WS}/lesson-4.1/deployer.token
-k config view --minify | grep server"""),
+server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
-                   hand(r"""k --kubeconfig=/dev/null --server=SERVER --certificate-authority=/etc/kubernetes/pki/ca.crt --token=TOKEN auth whoami""",
+as_deployer() { kubectl --kubeconfig=/dev/null --server=$server --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$(cat {WS}/lesson-4.1/deployer.token) "$@"; }
-                        r"""`SERVER`: the address above; `TOKEN`: the file's content (`cat` it, copy it). With `-n team get deploy` instead of `auth whoami` it works; `get secrets` is forbidden.""",
+as_deployer auth whoami
-                        run=r"""server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
+as_deployer -n team get deployments
-kubectl --kubeconfig=/dev/null --server=$server --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$(cat {WS}/lesson-4.1/deployer.token) auth whoami""")]),
+as_deployer -n team get secrets || true"""),
         step("A Pod without a token",
              r"""Most Pods never talk to the API server, and do not need a token in their filesystem. Create ServiceAccount `quiet` in `team` with `automountServiceAccountToken: false`, and run Pod `notoken` (`busybox:1.36`, `sleep 86400`) as that ServiceAccount. Inside it, `/var/run/secrets/kubernetes.io/serviceaccount` does not exist.""",
              check=r"""
 eq "$(j -n team sa quiet '{.automountServiceAccountToken}')" false && eq "$(j -n team pod notoken '{.spec.serviceAccountName} {.status.phase}')" "quiet Running" || exit 1
 ! $K -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount >/dev/null 2>&1""",
              hint=r"""The field sits at the top level of the ServiceAccount (and can be set on a Pod's `spec` too, which wins). The Pod names its ServiceAccount with `spec.serviceAccountName`.""",
-             show=[cmd(r"""k -n team create sa quiet --dry-run=client -o yaml > quiet.yaml"""),
+             show=r"""kubectl -n team apply -f - <<'EOF'
-                   hand(r"""vim quiet.yaml""",
+apiVersion: v1
-                        r"""Add, at the top level (no indent):
+kind: ServiceAccount
-```
+metadata:
+  name: quiet
 automountServiceAccountToken: false
-```""",
+---
-                        run=r"""yq -i '.automountServiceAccountToken = false' quiet.yaml"""),
+apiVersion: v1
-                   cmd(r"""k apply -f quiet.yaml
+kind: Pod
-k -n team run notoken --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > notoken.yaml"""),
+metadata:
-                   hand(r"""vim notoken.yaml""",
+  name: notoken
-                        r"""Add to the Pod's `spec:`, above `containers:`:
+spec:
-```
   serviceAccountName: quiet
-```""",
+  containers:
-                        run=r"""yq -i '.spec.serviceAccountName = "quiet"' notoken.yaml"""),
+  - name: app
-                   cmd(r"""k -n team apply -f notoken.yaml
+    image: busybox:1.36
-k -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount""", r"""Once it runs: "No such file or directory" - no token.""")]),
+    command: [sleep, '86400']
+EOF
+kubectl -n team wait --for=condition=Ready pod/notoken --timeout=120s
+kubectl -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount || echo 'no token mounted'"""),
         step("Everything one subject may do",
              r"""Write the complete list of what `deployer` may do in `team` to `{WS}/lesson-4.1/deployer-can.txt` - `kubectl auth can-i` prints it with `--list`. It includes what every authenticated subject gets from the built-in ClusterRoles, such as creating `selfsubjectreviews`.""",
              check=r"""f="$WS/lesson-4.1/deployer-can.txt"; grep -q '^deployments.apps' "$f" 2>/dev/null && grep -q '^pods ' "$f" && grep -qi 'verbs' "$f" """,
              hint=r"""`kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer`.""",
-             show=[cmd(r"""k auth can-i --list -n team --as=system:serviceaccount:team:deployer > {WS}/lesson-4.1/deployer-can.txt
+             show=r"""kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer > {WS}/lesson-4.1/deployer-can.txt
-cat {WS}/lesson-4.1/deployer-can.txt""")]),
+cat {WS}/lesson-4.1/deployer-can.txt"""),
     ],
     "outro": r"""You gave a ServiceAccount a Role, reused a ClusterRole in one Namespace, issued and used a token, ran a Pod without one, and listed a subject's permissions.
 
@@ -152,18 +158,10 @@ nx "$W1" 'rm -f /etc/modules-load.d/k8s.conf /etc/sysctl.d/k8s.conf' </dev/null
              r"""On `{W1}`, make `overlay` and `br_netfilter` load at every boot: list them, one per line, in `/etc/modules-load.d/k8s.conf`.""",
              check=r"""eq "$(nx "$W1" 'grep -v "^#" /etc/modules-load.d/k8s.conf 2>/dev/null | tr -d " " | grep . | sort | tr "\n" " "' </dev/null)" "br_netfilter overlay " """,
              hint=r"""`printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/k8s.conf`. On a real machine you would also `modprobe` both now, and `lsmod | grep br_netfilter` would show it.""",
-             show=[hand(r"""ssh {W1}
+             show=r"""ssh {W1} sudo bash <<'EOF'
-sudo -i
-vim /etc/modules-load.d/k8s.conf""",
-                        r"""Two lines (the docs page *Container Runtimes* has them, under *Install and configure prerequisites*):
-```
-overlay
-br_netfilter
-```
-then `exit` twice, back to the control-plane node.""",
-                        run=r"""ssh {W1} sudo bash <<'EOF'
 printf 'overlay\nbr_netfilter\n' > /etc/modules-load.d/k8s.conf
-EOF""")]),
+cat /etc/modules-load.d/k8s.conf
+EOF"""),
         step("Kernel parameters, for good",
              r"""On `{W1}`, put the three parameters Kubernetes needs in `/etc/sysctl.d/k8s.conf` - `net.bridge.bridge-nf-call-iptables = 1`, `net.bridge.bridge-nf-call-ip6tables = 1`, `net.ipv4.ip_forward = 1` - and apply every sysctl file now with `sysctl --system`.""",
              check=r"""
@@ -171,29 +169,15 @@ f=$(nx "$W1" 'cat /etc/sysctl.d/k8s.conf 2>/dev/null' </dev/null | tr -d ' ')
 for p in net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1; do echo "$f" | grep -qx "$p" || exit 1; done
 eq "$(nx "$W1" 'sysctl -n net.ipv4.ip_forward' </dev/null)" 1""",
              hint=r"""A here-document into `sudo tee`, then `sudo sysctl --system` - it prints every file it reads, yours among them. (The bridge parameters only exist once `br_netfilter` is loaded.)""",
-             show=[hand(r"""ssh {W1}
+             show=r"""ssh {W1} sudo bash <<'EOF'
-sudo -i
-vim /etc/sysctl.d/k8s.conf""",
-                        r"""Three lines (the same docs page has them):
-```
-net.bridge.bridge-nf-call-iptables  = 1
-net.bridge.bridge-nf-call-ip6tables = 1
-net.ipv4.ip_forward                 = 1
-```
-then apply every sysctl file and look, still on `{W1}`:
-```
-sysctl --system
-sysctl net.ipv4.ip_forward
-```
-and `exit` twice.""",
-                        run=r"""ssh {W1} sudo bash <<'EOF'
 cat > /etc/sysctl.d/k8s.conf <<'CONF'
 net.bridge.bridge-nf-call-iptables  = 1
 net.bridge.bridge-nf-call-ip6tables = 1
 net.ipv4.ip_forward                 = 1
 CONF
-sysctl --system >/dev/null 2>&1
+sysctl --system 2>/dev/null | grep -A3 k8s.conf
-EOF""")]),
+sysctl net.ipv4.ip_forward
+EOF"""),
         step("Swap and the cgroup driver",
              r"""Two more things kubeadm's preflight checks. Write to `{WS}/lesson-4.2/node.txt`, in this format:
 
@@ -212,30 +196,17 @@ sc=$(nx "$W1" 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml |
 want=$(printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s' "$swap" "${fso% *}" "${fso#* }" "$sc")
 eq "$(tr -d ' \r' < "$WS/lesson-4.2/node.txt" 2>/dev/null | grep .)" "$want" """,
              hint=r"""`cat /proc/swaps` (or `swapon --show`): any line below the header is swap. `grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml` - a field that is not there has its default (`failSwapOn` is `true`). `grep SystemdCgroup /etc/containerd/config.toml`; no such line means `false`.""",
-             show=[hand(r"""ssh {W1}
+             show=r"""ssh {W1} sudo bash <<'EOF'
-sudo -i
 cat /proc/swaps
-grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml
+grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml
-grep SystemdCgroup /etc/containerd/config.toml""",
+grep -n SystemdCgroup /etc/containerd/config.toml || echo 'SystemdCgroup not set'
-                        r"""- swap: `on` if `/proc/swaps` lists a device below its header line, `off` if it lists none;
+EOF
-- `failSwapOn`: `true` when the file does not set it - that is its default;
+swap=$(ssh -n {W1} '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off')
-- `cgroupDriver`, and `SystemdCgroup` (`false` when it is not set).
-
-Then `exit` twice.""",
-                        run=r"""ssh -n {W1} 'sudo cat /proc/swaps; sudo grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml; sudo grep SystemdCgroup /etc/containerd/config.toml' """),
-                   hand(r"""vim {WS}/lesson-4.2/node.txt""",
-                        r"""The four lines, each `VALUE` what you found (`on` or `off`, `true` or `false`, `systemd` or `cgroupfs`):
-```
-swap=VALUE
-failSwapOn=VALUE
-kubelet-cgroup=VALUE
-containerd-systemd-cgroup=VALUE
-```""",
-                        run=r"""swap=$(ssh -n {W1} '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off')
 fso=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.failSwapOn)
 cg=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver)
 sc=$(ssh -n {W1} 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml | head -1 | tr -d " " | cut -d= -f2')
-printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s\n' "$swap" "$fso" "$cg" "${sc:-false}" > {WS}/lesson-4.2/node.txt""")]),
+printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s\n' "$swap" "$fso" "$cg" "${sc:-false}" > {WS}/lesson-4.2/node.txt
+cat {WS}/lesson-4.2/node.txt"""),
         step("The packages, as kubeadm wants them",
              r"""On an exam node, kubeadm, kubelet and kubectl come from the repository of **one** minor at `pkgs.k8s.io`, and are **held** so that an `apt upgrade` never moves them by accident.
 
@@ -244,10 +215,9 @@ Write to `{WS}/lesson-4.2/packages.txt`: the line of `{W1}`'s Kubernetes reposit
 want=$(nx "$W1" 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' </dev/null | grep .)
 [ -n "$want" ] && eq "$(grep . "$WS/lesson-4.2/packages.txt" 2>/dev/null | tr -d '\r')" "$want" """,
              hint=r"""`cat /etc/apt/sources.list.d/kubernetes.list` and `apt-mark showhold`. `apt-cache madison kubeadm` lists the versions that repository offers; `dpkg -l | grep kube` what is installed.""",
-             show=[hand(r"""ssh {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt""",
+             show=r"""ssh -n {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt
-                        r"""The repository line, then the three held packages.""",
+cat {WS}/lesson-4.2/packages.txt
-                        run=r"""ssh -n {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt"""),
+ssh -n {W1} 'dpkg -l | grep -E "kube(adm|let|ctl)"'"""),
-                   cmd(r"""cat {WS}/lesson-4.2/packages.txt""")]),
     ],
     "outro": r"""You set up modules and sysctls that survive a reboot, checked swap and the cgroup drivers, and read how Kubernetes is installed on a node.
 
@@ -289,10 +259,9 @@ nx "$W1" 'rm -f /etc/kubernetes/manifests/node-web.yaml' </dev/null
 want=$($K -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}')
 eq "$(tr -s ' \t' ' ' < "$WS/lesson-4.3/networks.txt" 2>/dev/null | tr -d '\r\n')" "$want" """,
              hint=r"""`kubectl -n kube-system get cm kubeadm-config -o yaml` - look under `networking:`. `kubeadm config print init-defaults` shows every field such a file can have.""",
-             show=[cmd(r"""k -n kube-system get cm kubeadm-config -o yaml | grep -i subnet"""),
+             show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep -A4 networking
-                   hand(r"""echo 'PODSUBNET SERVICESUBNET' > {WS}/lesson-4.3/networks.txt""",
+kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}' > {WS}/lesson-4.3/networks.txt
-                        r"""The two values, `podSubnet` first.""",
+cat {WS}/lesson-4.3/networks.txt"""),
-                        run=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}' > {WS}/lesson-4.3/networks.txt""")]),
         step("A join command for a new worker",
              r"""Create a new bootstrap token that expires in **2 hours**, and save the complete `kubeadm join` command for a worker in `{WS}/lesson-4.3/join.sh`, on one line. Run it on the control-plane node.""",
              check=r"""
@@ -301,9 +270,9 @@ id=$(grep -o -- '--token [a-z0-9]*\.[a-z0-9]*' "$f" | awk '{print $2}' | cut -d.
 exp=$(j -n kube-system secret "bootstrap-token-$id" '{.data.expiration}' | base64 -d); [ -n "$exp" ] || exit 1
 left=$(( $(date -d "$exp" +%s) - $(date +%s) )); [ "$left" -gt 6600 ] && [ "$left" -le 7300 ]""",
              hint=r"""`sudo kubeadm token create --ttl 2h --print-join-command`. `kubeadm token list` shows the tokens and when they expire; each is a Secret `bootstrap-token-ID` in `kube-system`.""",
-             show=[cmd(r"""sudo kubeadm token create --ttl 2h --print-join-command > {WS}/lesson-4.3/join.sh
+             show=r"""sudo kubeadm token create --ttl 2h --print-join-command > {WS}/lesson-4.3/join.sh
 cat {WS}/lesson-4.3/join.sh
-sudo kubeadm token list""")]),
+sudo kubeadm token list"""),
         step("Renew the API server's certificate",
              r"""Check the certificates' expiry dates, then **renew** the API server's serving certificate - only that one - and make the API server use it. A running process keeps the certificate it read at start, so the static Pod has to be restarted: move its manifest out of `/etc/kubernetes/manifests`, wait until the container is gone, and move it back.
 
@@ -316,32 +285,24 @@ file=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -seri
 served=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial' </dev/null)
 [ -n "$file" ] && eq "$file" "$served" """,
              hint=r"""`sudo kubeadm certs check-expiration`, `sudo kubeadm certs renew apiserver`. Restart: `sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/`, wait for `sudo crictl ps --name kube-apiserver` to be empty, move it back. Kubernetes renews the certificates itself on every `kubeadm upgrade`.""",
-             show=[cmd(r"""sudo kubeadm certs check-expiration
+             show=r"""sudo kubeadm certs check-expiration | head -8
 sudo kubeadm certs renew apiserver
-sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml ~/"""),
+sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
-                   hand(r"""sudo crictl ps --name kube-apiserver""",
+for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name kube-apiserver -q)" ] && break; sleep 2; done
-                        r"""Again until it lists nothing: the old API server is gone.""",
+sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
-                        run=r"""for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name kube-apiserver -q)" ] && break; sleep 2; done"""),
+for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
-                   cmd(r"""sudo mv ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
+sudo kubeadm certs check-expiration | grep '^apiserver '"""),
-                   hand(r"""k get no""",
-                        r"""Again until it answers: the API server is back with the new certificate.""",
-                        run=r"""for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
-                   cmd(r"""sudo kubeadm certs check-expiration | grep apiserver""")]),
         step("A static Pod on the worker",
              r"""Every kubelet runs static Pods, not only the control plane's. On `{W1}`, find the kubelet's `staticPodPath` and put a manifest there for Pod `node-web` (`nginx:1.27`).
 
 The API server shows it as a **mirror Pod** named after the node: `node-web-{W1}`. Deleting the mirror Pod with kubectl does nothing lasting - the kubelet puts it back; only removing the file does.""",
              check=r"""eq "$(j -n default pod "node-web-$W1" '{.status.phase} {.metadata.annotations.kubernetes\.io/config\.source}')" "Running file" """,
              hint=r"""`grep staticPodPath /var/lib/kubelet/config.yaml` on the node. `kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml` gives you the manifest; write it with `sudo tee`.""",
-             show=[cmd(r"""k run node-web --image=nginx:1.27 --dry-run=client -o yaml > node-web.yaml"""),
+             show=r"""kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml > {WS}/lesson-4.3/node-web.yaml
-                   hand(r"""scp node-web.yaml {W1}:
+ssh -n {W1} 'grep staticPodPath /var/lib/kubelet/config.yaml'
-ssh {W1}
+ssh {W1} 'sudo tee /etc/kubernetes/manifests/node-web.yaml >/dev/null' < {WS}/lesson-4.3/node-web.yaml
-sudo -i
+sleep 15
-grep staticPodPath /var/lib/kubelet/config.yaml
+kubectl get pods -o wide"""),
-cp ~candidate/node-web.yaml /etc/kubernetes/manifests/""",
-                        r"""The manifest goes into the `staticPodPath` directory; then `exit` twice.""",
-                        run=r"""ssh {W1} 'sudo tee /etc/kubernetes/manifests/node-web.yaml >/dev/null' < node-web.yaml"""),
-                   cmd(r"""k get po -o wide""", r"""Again in a few seconds: `node-web-{W1}`, the mirror Pod.""")]),
     ],
     "outro": r"""You read the cluster's kubeadm configuration, made a join command, renewed a certificate and made the API server load it, and ran a static Pod on a worker.
 
@@ -397,14 +358,15 @@ etcd only accepts clients with a certificate: give `etcdctl` the endpoint and th
 nx "$CP" 'etcdutl snapshot status /opt/backup/etcd.db >/dev/null 2>&1' </dev/null && grep -qi 'hash' "$WS/lesson-4.4/snapshot.txt" 2>/dev/null \
   && grep -qi 'total key' "$WS/lesson-4.4/snapshot.txt" """,
              hint=r"""`etcdctl --endpoints=https://127.0.0.1:2379 --cacert=.../ca.crt --cert=.../server.crt --key=.../server.key snapshot save FILE`, as root. Since etcd 3.6, `status` and `restore` are `etcdutl` only.""",
-             show=[cmd(r"""sudo grep -e listen-client-urls -e cert-file -e key-file -e trusted-ca-file /etc/kubernetes/manifests/etcd.yaml""", r"""The endpoint, and the three files etcdctl needs: `--trusted-ca-file` is its `--cacert`, `--cert-file` its `--cert`, `--key-file` its `--key`."""),
+             show=r"""sudo grep -E -- '--(listen-client-urls|cert-file|key-file|trusted-ca-file)' /etc/kubernetes/manifests/etcd.yaml
-                   cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup/etcd.db
+sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
-sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/snapshot.txt""")]),
+  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup/etcd.db
+sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/snapshot.txt"""),
         step("Lose something",
              r"""Now the mistake a backup is for: delete ConfigMap `marker` in `life`.""",
              check=r"""[ -z "$(j -n life cm marker '{.metadata.name}')" ] && nx "$CP" 'test -s /opt/backup/etcd.db' </dev/null""",
              hint=r"""`kubectl -n life delete configmap marker`.""",
-             show=[cmd(r"""k -n life delete cm marker""")]),
+             show=r"""kubectl -n life delete configmap marker"""),
         step("Restore the snapshot",
              r"""Restore `/opt/backup/etcd.db` so that `marker` is back. Do it in place, so that etcd keeps its data directory `/var/lib/etcd`:
 
@@ -415,16 +377,13 @@ sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/
 Restoring into a **new** directory and pointing the manifest's `hostPath` at it works too, and is what many tasks ask for - but then the `dataDir` in kubeadm's configuration still names the old one, and the next `kubeadm upgrade` would write it back into the manifest.""",
              check=r"""eq "$(j -n life cm marker '{.data.state}')" before-backup && nx "$CP" 'test -d /var/lib/etcd/member' </dev/null""",
              hint=r"""`sudo etcdutl snapshot restore /opt/backup/etcd.db --data-dir /var/lib/etcd` - the target directory must not exist yet. `sudo crictl ps --name etcd` shows whether etcd is running.""",
-             show=[cmd(r"""sudo mv /etc/kubernetes/manifests/etcd.yaml ~/"""),
+             show=r"""sudo mv /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml
-                   hand(r"""sudo crictl ps --name etcd""",
+for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name '^etcd$' -q)" ] && break; sleep 2; done
-                        r"""Again until it lists nothing: etcd has stopped. (kubectl fails from now on until etcd is back.)""",
+sudo mv /var/lib/etcd /var/lib/etcd.before-restore
-                        run=r"""for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name '^etcd$' -q)" ] && break; sleep 2; done"""),
-                   cmd(r"""sudo mv /var/lib/etcd /var/lib/etcd.bak
 sudo etcdutl snapshot restore /opt/backup/etcd.db --data-dir /var/lib/etcd
-sudo mv ~/etcd.yaml /etc/kubernetes/manifests/"""),
+sudo mv /root/etcd.yaml /etc/kubernetes/manifests/etcd.yaml
-                   hand(r"""k -n life get cm marker""",
+for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
-                        r"""Again until it answers - a minute or two: etcd, then the API server come back, and `marker` with them.""",
+kubectl -n life get configmap marker -o yaml | grep state"""),
-                        run=r"""for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done""")]),
         step("Plan the upgrade",
              r"""On the control-plane node, point the Kubernetes repository at **v1.37**, install the **newest 1.37 kubeadm** the repository offers (unhold it first, hold it again after), and save the output of `kubeadm upgrade plan` in `{WS}/lesson-4.4/plan.txt`.
 
@@ -433,19 +392,13 @@ The plan shows two choices: the newest 1.36 patch, and the 1.37 release your kub
 t=$(newest); [ -n "$t" ] || exit 1
 eq "$(ver "$CP" kubeadm)" "$t" && held "$CP" && grep -q "kubeadm upgrade apply v${t%%-*}" "$WS/lesson-4.4/plan.txt" 2>/dev/null""",
              hint=r"""`sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list`, `sudo apt-get update`, `apt-cache madison kubeadm` (newest first), `sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.X-1.1 && sudo apt-mark hold kubeadm`.""",
-             show=[hand(r"""sudo vim /etc/apt/sources.list.d/kubernetes.list""",
+             show=r"""sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
-                        r"""In the URL, change `v1.36` to `v1.37`.""",
+sudo apt-get update -qq
-                        run=r"""sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list"""),
+apt-cache madison kubeadm | head -3
-                   cmd(r"""sudo apt-get update
+v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
-apt-cache madison kubeadm""", r"""The newest `1.37.x-1.1` (at the top) is what to install. The docs page *Upgrading kubeadm clusters* has the commands that follow."""),
+sudo apt-mark unhold kubeadm && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$v && sudo apt-mark hold kubeadm
-                   hand(r"""sudo apt-mark unhold kubeadm
+kubeadm version -o short
-sudo apt-get install -y kubeadm=VERSION
+sudo kubeadm upgrade plan | tee {WS}/lesson-4.4/plan.txt | tail -12"""),
-sudo apt-mark hold kubeadm""",
-                        r"""`VERSION`: that one, for example `1.37.1-1.1`.""",
-                        run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
-sudo apt-mark unhold kubeadm && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$v && sudo apt-mark hold kubeadm"""),
-                   cmd(r"""kubeadm version
-sudo kubeadm upgrade plan | tee {WS}/lesson-4.4/plan.txt""")]),
         step("Upgrade the control plane",
              r"""Apply the upgrade to that 1.37 release (`kubeadm upgrade apply`, a few minutes - it upgrades etcd, the static Pods, CoreDNS and kube-proxy). Then drain the node, upgrade `kubelet` and `kubectl` to the same version, reload systemd, restart the kubelet, and uncordon.
 
@@ -456,23 +409,13 @@ eq "$($K version -o json 2>/dev/null | python3 -c 'import json,sys; print(json.l
 eq "$(j node "$CP" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$CP")" "$t $t $t " && held "$CP" \
   && [ "$(j node "$CP" '{.spec.unschedulable}')" != true ]""",
              hint=r"""`sudo kubeadm upgrade apply v1.37.X -y`; `kubectl drain {CP} --ignore-daemonsets`; `sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=... kubectl=... && sudo apt-mark hold kubelet kubectl`; `sudo systemctl daemon-reload && sudo systemctl restart kubelet`; `kubectl uncordon {CP}`.""",
-             show=[hand(r"""sudo kubeadm upgrade apply vVERSION""",
+             show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
-                        r"""`vVERSION`: the one the plan names, for example `v1.37.1`. Answer `y`; it takes a few minutes.""",
+sudo kubeadm upgrade apply v${v%%-*} -y | tail -3
-                        run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
+kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
-sudo kubeadm upgrade apply v${v%%-*} -y | tail -3"""),
+sudo apt-mark unhold kubelet kubectl && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$v kubectl=$v && sudo apt-mark hold kubelet kubectl
-                   cmd(r"""k drain {CP} --ignore-daemonsets --delete-emptydir-data"""),
+sudo systemctl daemon-reload && sudo systemctl restart kubelet
-                   hand(r"""sudo apt-mark unhold kubelet kubectl
+for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done
-sudo apt-get install -y kubelet=VERSION kubectl=VERSION
+kubectl get nodes"""),
-sudo apt-mark hold kubelet kubectl""",
-                        r"""`VERSION` as for kubeadm, for example `1.37.1-1.1`.""",
-                        run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
-sudo apt-mark unhold kubelet kubectl && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$v kubectl=$v && sudo apt-mark hold kubelet kubectl"""),
-                   cmd(r"""sudo systemctl daemon-reload
-sudo systemctl restart kubelet"""),
-                   hand(r"""k uncordon {CP}""",
-                        r"""Again if it fails at first: the kubelet takes a moment to come back.""",
-                        run=r"""for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done"""),
-                   cmd(r"""k get no""")]),
         step("Upgrade the worker",
              r"""Now `{W1}`: drain it from the control-plane node; on `{W1}`, the repository line, kubeadm, **`kubeadm upgrade node`** (a worker has no control-plane Pods to replace - it only refreshes the kubelet's configuration), kubelet and kubectl, daemon-reload, restart; then uncordon it.
 
@@ -482,20 +425,8 @@ t=$(newest); v="v${t%%-*}"
 eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$W1")" "$t $t $t " && held "$W1" \
   && [ "$(j node "$W1" '{.spec.unschedulable}')" != true ] && eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
              hint=r"""The same packages, from the same repository line, as on the control plane - only `kubeadm upgrade node` instead of `apply`. Drain and uncordon are kubectl commands: run them on the control-plane node.""",
-             show=[cmd(r"""k drain {W1} --ignore-daemonsets --delete-emptydir-data"""),
+             show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
-                   hand(r"""ssh {W1}
+kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data
-sudo -i
-vim /etc/apt/sources.list.d/kubernetes.list
-apt-get update
-apt-mark unhold kubeadm kubelet kubectl
-apt-get install -y kubeadm=VERSION
-kubeadm upgrade node
-apt-get install -y kubelet=VERSION kubectl=VERSION
-apt-mark hold kubeadm kubelet kubectl
-systemctl daemon-reload
-systemctl restart kubelet""",
-                        r"""In the repository line `v1.36` becomes `v1.37`; `VERSION` as on the control-plane node. Then `exit` twice.""",
-                        run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
 ssh {W1} sudo bash -s $v <<'EOF'
 sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
 apt-get update -qq
@@ -505,9 +436,9 @@ kubeadm upgrade node | tail -2
 DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$1 kubectl=$1
 apt-mark hold kubeadm kubelet kubectl
 systemctl daemon-reload && systemctl restart kubelet
-EOF"""),
+EOF
-                   cmd(r"""k uncordon {W1}
+kubectl uncordon {W1}
-k get no""")]),
+kubectl get nodes"""),
     ],
     "outro": r"""You backed etcd up, restored it after a loss, and took a whole cluster from v1.36 to v1.37: control plane first, then the worker, with the packages held at every step.
 
@@ -549,11 +480,9 @@ $K -n kube-system delete secret kubeadm-certs --ignore-not-found >/dev/null 2>&1
 want=$($K -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}')
 [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-4.5/endpoint.txt" 2>/dev/null)" "$want" """,
              hint=r"""`controlPlaneEndpoint:` in the `ClusterConfiguration` of ConfigMap `kubeadm-config`.""",
-             show=[cmd(r"""k -n kube-system get cm kubeadm-config -o yaml | grep controlPlaneEndpoint"""),
+             show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}' > {WS}/lesson-4.5/endpoint.txt
-                   hand(r"""echo ENDPOINT > {WS}/lesson-4.5/endpoint.txt""",
+cat {WS}/lesson-4.5/endpoint.txt
-                        r"""`ENDPOINT`: that value, `host:port`.""",
+sudo grep server: /etc/kubernetes/kubelet.conf"""),
-                        run=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}' > {WS}/lesson-4.5/endpoint.txt"""),
-                   cmd(r"""sudo grep server: /etc/kubernetes/kubelet.conf""")]),
         step("etcd's members and its leader",
              r"""Ask etcd itself. Write to `{WS}/lesson-4.5/etcd.txt`:
 
@@ -569,14 +498,13 @@ import json, sys; m = json.load(sys.stdin)
 st = json.loads(sys.argv[1])[0]["Status"]; print(next(x["name"] for x in m["members"] if x["ID"] == st["leader"]))' "$out")
 eq "$(tr -d ' \r' < "$WS/lesson-4.5/etcd.txt" 2>/dev/null | grep .)" "$(printf 'members=%s\nleader=%s' "$n" "$lead")" """,
              hint=r"""`etcdctl member list -w table` and `etcdctl endpoint status -w table` (the IS LEADER column), with the endpoint and certificates as for a snapshot. `export ETCDCTL_ENDPOINTS=... ETCDCTL_CACERT=...` saves typing.""",
-             show=[cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list -w table""", r"""For the next one, press ↑ and change the last words."""),
+             show=r"""sudo bash <<'EOF'
-                   cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table""", r"""`IS LEADER` says which member leads."""),
+export ETCDCTL_API=3 ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \
-                   hand(r"""vim {WS}/lesson-4.5/etcd.txt""",
+       ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
-                        r"""```
+etcdctl member list -w table
-members=1
+etcdctl endpoint status -w table
-leader={CP}
+EOF
-```""",
+printf 'members=1\nleader={CP}\n' > {WS}/lesson-4.5/etcd.txt"""),
-                        run=r"""printf 'members=1\nleader={CP}\n' > {WS}/lesson-4.5/etcd.txt""")]),
         step("Who leads the scheduler and the controller manager",
              r"""Only one kube-scheduler and one kube-controller-manager act at a time; the others wait for the **Lease** in `kube-system` to expire. Write the node of each holder to `{WS}/lesson-4.5/leaders.txt`:
 
@@ -588,28 +516,20 @@ controller-manager=<node>
 s=$(j -n kube-system lease kube-scheduler '{.spec.holderIdentity}'); c=$(j -n kube-system lease kube-controller-manager '{.spec.holderIdentity}')
 eq "$(tr -d ' \r' < "$WS/lesson-4.5/leaders.txt" 2>/dev/null | grep .)" "$(printf 'scheduler=%s\ncontroller-manager=%s' "${s%%_*}" "${c%%_*}")" """,
              hint=r"""`kubectl -n kube-system get lease kube-scheduler kube-controller-manager` - HOLDER is `NODE_UUID`.""",
-             show=[cmd(r"""k -n kube-system get lease kube-scheduler kube-controller-manager""", r"""`HOLDER` is the node's name, then `_` and an ID: keep only the node's name."""),
+             show=r"""kubectl -n kube-system get lease kube-scheduler kube-controller-manager
-                   hand(r"""vim {WS}/lesson-4.5/leaders.txt""",
+s=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}')
-                        r"""```
-scheduler=NODE
-controller-manager=NODE
-```""",
-                        run=r"""s=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}')
 c=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}')
-printf 'scheduler=%s\ncontroller-manager=%s\n' "${s%%_*}" "${c%%_*}" > {WS}/lesson-4.5/leaders.txt""")]),
+printf 'scheduler=%s\ncontroller-manager=%s\n' "${s%%_*}" "${c%%_*}" > {WS}/lesson-4.5/leaders.txt"""),
         step("What a second control-plane node needs",
              r"""Upload the control-plane certificates for a joining node, and save the complete `kubeadm join` command a new **control-plane** node would run - new token, CA hash, `--control-plane` and the certificate key - to `{WS}/lesson-4.5/join-cp.sh`, on one line. Do not run it anywhere.""",
              check=r"""
 f="$WS/lesson-4.5/join-cp.sh"; grep -q '^kubeadm join .*--token .*--discovery-token-ca-cert-hash sha256:.*--control-plane' "$f" 2>/dev/null || exit 1
 key=$(grep -o -- '--certificate-key [0-9a-f]\{64\}' "$f" | awk '{print $2}'); [ -n "$key" ] && [ -n "$(j -n kube-system secret kubeadm-certs '{.metadata.name}')" ]""",
              hint=r"""`sudo kubeadm init phase upload-certs --upload-certs` prints the key as its last line; `sudo kubeadm token create --print-join-command --certificate-key KEY` adds `--control-plane --certificate-key` itself.""",
-             show=[cmd(r"""sudo kubeadm init phase upload-certs --upload-certs""", r"""The last line is the certificate key."""),
+             show=r"""key=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
-                   hand(r"""sudo kubeadm token create --print-join-command --certificate-key KEY > {WS}/lesson-4.5/join-cp.sh""",
+sudo kubeadm token create --print-join-command --certificate-key $key > {WS}/lesson-4.5/join-cp.sh
-                        r"""`KEY`: that line.""",
+cat {WS}/lesson-4.5/join-cp.sh
-                        run=r"""key=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
+kubectl -n kube-system get secret kubeadm-certs"""),
-sudo kubeadm token create --print-join-command --certificate-key $key > {WS}/lesson-4.5/join-cp.sh"""),
-                   cmd(r"""cat {WS}/lesson-4.5/join-cp.sh
-k -n kube-system get secret kubeadm-certs""")]),
     ],
     "outro": r"""You read the endpoint that makes HA possible, asked etcd for its members and leader, found the leaders of the scheduler and controller manager, and prepared a control-plane join.
 
@@ -717,23 +637,23 @@ A chart is at `{WS}/lesson-4.6/chart` (values `replicaCount`, `image.tag`, `serv
              r"""Render the chart as release `web` with `replicaCount=2`, without installing anything, and save the manifests in `{WS}/lesson-4.6/rendered.yaml`. That is how you check what a set of values does.""",
              check=r"""f="$WS/lesson-4.6/rendered.yaml"; grep -q 'name: web-shop' "$f" 2>/dev/null && grep -q 'replicas: 2' "$f" && [ -z "$(j -n apps deploy web-shop '{.metadata.name}')" ]""",
              hint=r"""`helm template RELEASE CHART --set KEY=VALUE`.""",
-             show=[cmd(r"""helm template web {WS}/lesson-4.6/chart --set replicaCount=2 > {WS}/lesson-4.6/rendered.yaml
+             show=r"""helm template web {WS}/lesson-4.6/chart --set replicaCount=2 > {WS}/lesson-4.6/rendered.yaml
-grep -e kind: -e replicas: -e image: {WS}/lesson-4.6/rendered.yaml""")]),
+grep -E 'kind:|replicas:|image:' {WS}/lesson-4.6/rendered.yaml"""),
         step("Install a release",
              r"""Install the chart as release `web` in Namespace `apps` (create it), with `replicaCount=2`.""",
              check=r"""eq "$(helm -n apps status web -o json 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["info"]["status"], d["version"])')" "deployed 1" && eq "$(j -n apps deploy web-shop '{.status.readyReplicas}')" 2""",
              hint=r"""`helm install web CHART -n apps --create-namespace --set replicaCount=2`; `helm list -n apps`.""",
-             show=[cmd(r"""helm install web {WS}/lesson-4.6/chart -n apps --create-namespace --set replicaCount=2
+             show=r"""helm install web {WS}/lesson-4.6/chart -n apps --create-namespace --set replicaCount=2
 helm list -n apps
-k -n apps rollout status deploy web-shop""")]),
+kubectl -n apps rollout status deploy web-shop --timeout=120s"""),
         step("Upgrade it, keeping what you set",
              r"""Upgrade release `web` so that its Service becomes a `NodePort` - and keep `replicaCount=2`. A plain `helm upgrade --set service.type=NodePort` would quietly put the replica count back to the chart's default.""",
              check=r"""eq "$(j -n apps svc web-shop '{.spec.type}')" NodePort && eq "$(j -n apps deploy web-shop '{.spec.replicas}')" 2 \
   && [ "$(helm -n apps history web -o json 2>/dev/null | python3 -c 'import json,sys; print(len(json.load(sys.stdin)))')" -ge 2 ]""",
              hint=r"""`--reuse-values` keeps the values of the last release and adds yours; `helm get values web -n apps` shows what is set.""",
-             show=[cmd(r"""helm upgrade web {WS}/lesson-4.6/chart -n apps --reuse-values --set service.type=NodePort
+             show=r"""helm upgrade web {WS}/lesson-4.6/chart -n apps --reuse-values --set service.type=NodePort
 helm get values web -n apps
-k -n apps get svc web-shop""")]),
+kubectl -n apps get svc web-shop"""),
         step("Roll the release back",
              r"""The NodePort was a mistake: roll release `web` back to **revision 1**. A rollback is itself a new revision.""",
              check=r"""
@@ -742,18 +662,16 @@ helm -n apps history web -o json 2>/dev/null | python3 -c '
 import json, sys; h = json.load(sys.stdin)
 sys.exit(0 if len(h) >= 3 and h[-1]["status"] == "deployed" and "Rollback to 1" in h[-1]["description"] else 1)'""",
              hint=r"""`helm rollback web 1 -n apps`, then `helm history web -n apps`.""",
-             show=[cmd(r"""helm rollback web 1 -n apps
+             show=r"""helm rollback web 1 -n apps
 helm history web -n apps
-k -n apps get svc web-shop""")]),
+kubectl -n apps get svc web-shop"""),
         step("An overlay for production",
              r"""Create a Kustomize overlay at `{WS}/lesson-4.6/kustomize/prod` on top of the base, which: puts everything in Namespace `prod`, prefixes names with `prod-`, runs **3** replicas, and uses image `nginx:1.27`. Apply it. Leave the base as it is.""",
              check=r"""eq "$(j -n prod deploy prod-api '{.spec.replicas} {.spec.template.spec.containers[0].image} {.status.readyReplicas}')" "3 nginx:1.27 3" \
   && grep -q 'image: nginx:1.26' "$WS/lesson-4.6/kustomize/base/deployment.yaml" """,
              hint=r"""A `kustomization.yaml` with `resources: [../base]`, `namespace: prod`, `namePrefix: prod-`, `replicas: [{name: api, count: 3}]`, `images: [{name: nginx, newTag: "1.27"}]`. The Namespace itself must exist (or be one of the resources). `kubectl kustomize DIR` shows the result first.""",
-             show=[cmd(r"""mkdir -p {WS}/lesson-4.6/kustomize/prod"""),
+             show=r"""mkdir -p {WS}/lesson-4.6/kustomize/prod
-                   hand(r"""vim {WS}/lesson-4.6/kustomize/prod/kustomization.yaml""",
+cat > {WS}/lesson-4.6/kustomize/prod/kustomization.yaml <<'EOF'
-                        r"""(the docs page *Declarative Management of Kubernetes Objects Using Kustomize* has each field)
-```
 resources:
 - ../base
 namespace: prod
@@ -764,23 +682,11 @@ replicas:
 images:
 - name: nginx
   newTag: "1.27"
-```""",
+EOF
-                        run=r"""cat > {WS}/lesson-4.6/kustomize/prod/kustomization.yaml <<'EOF'
+kubectl kustomize {WS}/lesson-4.6/kustomize/prod
-resources:
+kubectl create namespace prod
-- ../base
+kubectl apply -k {WS}/lesson-4.6/kustomize/prod
-namespace: prod
+kubectl -n prod rollout status deploy prod-api --timeout=120s"""),
-namePrefix: prod-
-replicas:
-- name: api
-  count: 3
-images:
-- name: nginx
-  newTag: "1.27"
-EOF"""),
-                   cmd(r"""k kustomize {WS}/lesson-4.6/kustomize/prod
-k create ns prod
-k apply -k {WS}/lesson-4.6/kustomize/prod
-k -n prod rollout status deploy prod-api""")]),
     ],
     "outro": r"""You rendered a chart, installed, upgraded and rolled back a release, and built a production overlay with Kustomize without touching its base.
 
@@ -828,18 +734,11 @@ f=$(nx "$W1" 'ls /etc/cni/net.d/*.conf* 2>/dev/null | sort | head -1' </dev/null
 p=$(nx "$W1" "cat $f" </dev/null | python3 -c 'import json,sys; c=json.load(sys.stdin); print(",".join(x["type"] for x in c.get("plugins", [c])))')
 eq "$(tr -d ' \r' < "$WS/lesson-4.7/cni.txt" 2>/dev/null | grep .)" "$(printf 'config=%s\nplugins=%s' "$f" "$p")" """,
              hint=r"""`ls /etc/cni/net.d/` and `cat` the file: a `.conflist` has a `"plugins"` list, each with a `"type"` - the binary under `/opt/cni/bin` that is run.""",
-             show=[hand(r"""ssh {W1} ls /etc/cni/net.d/
+             show=r"""ssh -n {W1} 'ls /etc/cni/net.d/; cat /etc/cni/net.d/10-kindnet.conflist; ls /opt/cni/bin'
-ssh {W1} cat /etc/cni/net.d/10-kindnet.conflist""",
+f=$(ssh -n {W1} 'ls /etc/cni/net.d/*.conf* | sort | head -1')
-                        r"""The runtime uses the first file by name; the `type` of each entry of `plugins`, in order, is the chain.""",
-                        run=r"""ssh -n {W1} 'ls /etc/cni/net.d/; cat /etc/cni/net.d/10-kindnet.conflist' """),
-                   hand(r"""vim {WS}/lesson-4.7/cni.txt""",
-                        r"""```
-config=/etc/cni/net.d/10-kindnet.conflist
-plugins=TYPE,TYPE,...
-```""",
-                        run=r"""f=$(ssh -n {W1} 'ls /etc/cni/net.d/*.conf* | sort | head -1')
 p=$(ssh -n {W1} "cat $f" | jq -r '[.plugins[].type] | join(",")')
-printf 'config=%s\nplugins=%s\n' "$f" "$p" > {WS}/lesson-4.7/cni.txt""")]),
+printf 'config=%s\nplugins=%s\n' "$f" "$p" > {WS}/lesson-4.7/cni.txt
+cat {WS}/lesson-4.7/cni.txt"""),
         step("The runtime, through CRI",
              r"""Write to `{WS}/lesson-4.7/cri.txt`:
 
@@ -852,63 +751,33 @@ e=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" | python3 -c 'import json,sys
 r=$(j node "$W1" '{.status.nodeInfo.containerRuntimeVersion}')
 eq "$(tr -d ' \r' < "$WS/lesson-4.7/cri.txt" 2>/dev/null | grep .)" "$(printf 'endpoint=%s\nruntime=%s' "$e" "$r")" """,
              hint=r"""`containerRuntimeEndpoint` in the kubelet's configuration (`/var/lib/kubelet/config.yaml`, or `configz`); `kubectl get node -o wide` shows CONTAINER-RUNTIME.""",
-             show=[hand(r"""ssh {W1} grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml""",
+             show=r"""e=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.containerRuntimeEndpoint)
-                        r"""The endpoint the kubelet talks CRI to.""",
-                        run=r"""ssh -n {W1} grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml"""),
-                   cmd(r"""k get no -o wide""", r"""`CONTAINER-RUNTIME` is the runtime and its version."""),
-                   hand(r"""vim {WS}/lesson-4.7/cri.txt""",
-                        r"""```
-endpoint=ENDPOINT
-runtime=RUNTIME
-```""",
-                        run=r"""e=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.containerRuntimeEndpoint)
 r=$(kubectl get node {W1} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}')
-printf 'endpoint=%s\nruntime=%s\n' "$e" "$r" > {WS}/lesson-4.7/cri.txt""")]),
+printf 'endpoint=%s\nruntime=%s\n' "$e" "$r" > {WS}/lesson-4.7/cri.txt
+cat {WS}/lesson-4.7/cri.txt"""),
         step("A Pod's container, from the runtime's side",
              r"""Pod `probe` in `plug` runs on `{W1}`. Ask the runtime about it there, with `crictl`: find its container, and write the container's **process ID on the node** to `{WS}/lesson-4.7/pid.txt`.""",
              check=r"""
 want=$(nx "$W1" 'crictl inspect $(crictl ps --name probe -q | head -1) 2>/dev/null' </dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["info"]["pid"])')
 [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-4.7/pid.txt" 2>/dev/null)" "$want" """,
              hint=r"""`sudo crictl pods --name probe`, `sudo crictl ps --name probe`, `sudo crictl inspect ID` - the PID is under `info.pid`. `crictl` talks to the same socket as the kubelet.""",
-             show=[hand(r"""ssh {W1}
+             show=r"""ssh -n {W1} 'sudo crictl pods --name probe; sudo crictl ps --name probe'
-sudo -i
+ssh -n {W1} 'sudo crictl inspect $(sudo crictl ps --name probe -q | head -1)' | jq .info.pid > {WS}/lesson-4.7/pid.txt
-crictl ps --name probe
+cat {WS}/lesson-4.7/pid.txt"""),
-crictl inspect CONTAINER | grep -m1 '"pid"' """,
-                        r"""`CONTAINER`: the ID `crictl ps` printed. Then `exit` twice.""",
-                        run=r"""ssh -n {W1} 'sudo crictl pods --name probe; sudo crictl ps --name probe' """),
-                   hand(r"""echo PID > {WS}/lesson-4.7/pid.txt""",
-                        r"""`PID`: that number.""",
-                        run=r"""ssh -n {W1} 'sudo crictl inspect $(sudo crictl ps --name probe -q | head -1)' | jq .info.pid > {WS}/lesson-4.7/pid.txt""")]),
         step("A RuntimeClass",
              r"""Create RuntimeClass `plain` for containerd's handler `runc`, and run Pod `classy` in `plug` (`busybox:1.36`, `sleep 86400`) with that runtime class. A Pod that names a handler the node does not have stays in `ContainerCreating`.""",
              check=r"""eq "$(j runtimeclass plain '{.handler}')" runc && eq "$(j -n plug pod classy '{.spec.runtimeClassName} {.status.phase}')" "plain Running" """,
              hint=r"""A RuntimeClass is `node.k8s.io/v1` with a top-level `handler:` and no `spec`. The Pod: `spec.runtimeClassName: plain`.""",
-             show=[hand(r"""vim rc.yaml""",
+             show=r"""kubectl apply -f - <<'EOF'
-                        r"""No `kubectl create` for a RuntimeClass: paste the example from the docs page *Runtime Class*, and make it:
-```
 apiVersion: node.k8s.io/v1
 kind: RuntimeClass
 metadata:
   name: plain
 handler: runc
-```""",
+EOF
-                        run=r"""cat > rc.yaml <<'EOF'
+kubectl -n plug run classy --image=busybox:1.36 --overrides='{"spec":{"runtimeClassName":"plain"}}' -- sleep 86400
-apiVersion: node.k8s.io/v1
+kubectl -n plug wait --for=condition=Ready pod/classy --timeout=120s
-kind: RuntimeClass
+kubectl get runtimeclass"""),
-metadata:
-  name: plain
-handler: runc
-EOF"""),
-                   cmd(r"""k apply -f rc.yaml
-k -n plug run classy --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > classy.yaml"""),
-                   hand(r"""vim classy.yaml""",
-                        r"""Add to the Pod's `spec:`, above `containers:`:
-```
-  runtimeClassName: plain
-```""",
-                        run=r"""yq -i '.spec.runtimeClassName = "plain"' classy.yaml"""),
-                   cmd(r"""k -n plug apply -f classy.yaml
-k -n plug get po classy""")]),
         step("Storage drivers",
              r"""Write to `{WS}/lesson-4.7/csi.txt`:
 
@@ -923,16 +792,11 @@ n=$($K get csidrivers --no-headers 2>/dev/null | grep -c .)
 p=$($K get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.provisioner}{end}')
 eq "$(tr -d ' \r' < "$WS/lesson-4.7/csi.txt" 2>/dev/null | grep .)" "$(printf 'csidrivers=%s\nprovisioner=%s' "$n" "$p")" """,
              hint=r"""`kubectl get csidrivers`, `kubectl get csinodes`, `kubectl get sc`.""",
-             show=[cmd(r"""k get csidrivers
+             show=r"""kubectl get csidrivers
-k get csinodes
+kubectl get csinodes
-k get sc"""),
+kubectl get sc
-                   hand(r"""vim {WS}/lesson-4.7/csi.txt""",
+printf 'csidrivers=%s\nprovisioner=%s\n' "$(kubectl get csidrivers --no-headers 2>/dev/null | grep -c .)" "$(kubectl get sc standard -o jsonpath='{.provisioner}')" > {WS}/lesson-4.7/csi.txt
-                        r"""```
+cat {WS}/lesson-4.7/csi.txt"""),
-csidrivers=N
-provisioner=PROVISIONER
-```
-`N`: how many CSIDrivers `get csidrivers` listed (`0` for "No resources found"); `PROVISIONER`: the `(default)` class's.""",
-                        run=r"""printf 'csidrivers=%s\nprovisioner=%s\n' "$(kubectl get csidrivers --no-headers 2>/dev/null | grep -c .)" "$(kubectl get sc standard -o jsonpath='{.provisioner}')" > {WS}/lesson-4.7/csi.txt""")]),
     ],
     "outro": r"""You read the CNI chain on a node, the runtime the kubelet talks to, a container as the runtime sees it, ran a Pod with a RuntimeClass, and counted the storage drivers.
 
@@ -986,9 +850,7 @@ py crd $c 's = o["spec"]["versions"][0]; p = s["schema"]["openAPIV3Schema"]["pro
 s["served"] and s["storage"] and "status" in s.get("subresources", {}) and "schedule" in p.get("required", []) and p["properties"]["retention"].get("default") == 7 and p["properties"]["retention"].get("minimum") == 1' \
   && eq "$(j crd $c '{.status.conditions[?(@.type=="Established")].status}')" True""",
              hint=r"""The docs page "Extend the Kubernetes API with CustomResourceDefinitions" has the skeleton. `required:` sits beside `properties:`, at the level of `spec`'s schema; `default:` and `minimum:` inside `retention`. `subresources: {status: {}}` goes under the version.""",
-             show=[hand(r"""vim crd.yaml""",
+             show=r"""kubectl apply -f - <<'EOF'
-                        r"""Paste the example from the docs page *Extend the Kubernetes API with CustomResourceDefinitions*, and make it:
-```
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
@@ -1017,101 +879,54 @@ spec:
             type: object
             properties:
               phase: {type: string}
-```""",
+EOF
-                        run=r"""cat > crd.yaml <<'EOF'
+kubectl wait --for condition=established crd/backups.data.example.com
-apiVersion: apiextensions.k8s.io/v1
+kubectl api-resources --api-group=data.example.com"""),
-kind: CustomResourceDefinition
-metadata:
-  name: backups.data.example.com
-spec:
-  group: data.example.com
-  scope: Namespaced
-  names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]}
-  versions:
-  - name: v1
-    served: true
-    storage: true
-    subresources:
-      status: {}
-    schema:
-      openAPIV3Schema:
-        type: object
-        properties:
-          spec:
-            type: object
-            required: [schedule]
-            properties:
-              schedule: {type: string}
-              retention: {type: integer, minimum: 1, default: 7}
-          status:
-            type: object
-            properties:
-              phase: {type: string}
-EOF"""),
-                   cmd(r"""k apply -f crd.yaml
-k api-resources --api-group=data.example.com""")]),
         step("Defaults and validation",
              r"""In Namespace `data`, create Backup `nightly` with `schedule: "0 2 * * *"` and no retention: the API server fills in `7`. Then try a Backup `broken` with `retention: 0` - it is refused. Save the refusal (kubectl writes it to **stderr**) in `{WS}/lesson-4.8/refused.txt`.""",
              check=r"""
 eq "$(j -n data bk nightly '{.spec.schedule} {.spec.retention}')" "0 2 * * * 7" && [ -z "$(j -n data bk broken '{.metadata.name}')" ] \
   && grep -q 'retention' "$WS/lesson-4.8/refused.txt" 2>/dev/null && grep -qi 'greater than or equal to 1' "$WS/lesson-4.8/refused.txt" """,
              hint=r"""`apiVersion: data.example.com/v1`, `kind: Backup`. Redirect with `2>` - with `>` the file stays empty.""",
-             show=[hand(r"""vim nightly.yaml""",
+             show=r"""kubectl -n data apply -f - <<'EOF'
-                        r"""```
 apiVersion: data.example.com/v1
 kind: Backup
 metadata:
   name: nightly
 spec:
   schedule: "0 2 * * *"
-```""",
+EOF
-                        run=r"""cat > nightly.yaml <<'EOF'
+kubectl -n data get bk nightly -o jsonpath='{.spec}'; echo
-apiVersion: data.example.com/v1
+printf 'apiVersion: data.example.com/v1\nkind: Backup\nmetadata: {name: broken}\nspec: {schedule: "@hourly", retention: 0}\n' | kubectl -n data apply -f - 2> {WS}/lesson-4.8/refused.txt
-kind: Backup
+cat {WS}/lesson-4.8/refused.txt"""),
-metadata:
-  name: nightly
-spec:
-  schedule: "0 2 * * *"
-EOF"""),
-                   cmd(r"""k -n data apply -f nightly.yaml
-k -n data get bk nightly -o yaml | grep -A2 '^spec:' """, r"""`retention: 7`, filled in by the API server."""),
-                   hand(r"""cp nightly.yaml broken.yaml
-vim broken.yaml""",
-                        r"""Name it `broken`, and add `retention: 0` under `spec:`.""",
-                        run=r"""printf 'apiVersion: data.example.com/v1\nkind: Backup\nmetadata: {name: broken}\nspec: {schedule: "@hourly", retention: 0}\n' > broken.yaml"""),
-                   cmd(r"""k -n data apply -f broken.yaml 2> {WS}/lesson-4.8/refused.txt
-cat {WS}/lesson-4.8/refused.txt""")]),
         step("What the schema says",
              r"""`kubectl explain` reads the CRD's schema like any built-in kind's. Save the output of `kubectl explain backup.spec` in `{WS}/lesson-4.8/explain.txt`.""",
              check=r"""f="$WS/lesson-4.8/explain.txt"; grep -q 'retention' "$f" 2>/dev/null && grep -q 'schedule' "$f" && grep -qi 'required' "$f" """,
              hint=r"""`kubectl explain backup.spec` - or `--recursive` for the whole tree.""",
-             show=[cmd(r"""k explain backup.spec | tee {WS}/lesson-4.8/explain.txt""")]),
+             show=r"""kubectl explain backup.spec | tee {WS}/lesson-4.8/explain.txt"""),
         step("Install the operator",
              r"""Install the operator from `{WS}/lesson-4.8/operator/` and wait until its controller runs. Read the manifests first: a Namespace, a ServiceAccount, a ClusterRole for `backups` and ConfigMaps, the binding, and the controller's Deployment.""",
              check=r"""eq "$(j -n backup-operator deploy controller '{.status.readyReplicas}')" 1 && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller""",
              hint=r"""`kubectl apply -f DIR` applies every file in it; `kubectl -n backup-operator rollout status deploy controller`.""",
-             show=[cmd(r"""grep -e ^kind: -e '^  name:' {WS}/lesson-4.8/operator/operator.yaml
+             show=r"""grep -E '^kind:|^  name:' {WS}/lesson-4.8/operator/operator.yaml
-k apply -f {WS}/lesson-4.8/operator/
+kubectl apply -f {WS}/lesson-4.8/operator/
-k -n backup-operator rollout status deploy controller""")]),
+kubectl -n backup-operator rollout status deploy controller --timeout=120s"""),
         step("Let it work",
              r"""Create Backup `weekly` in `data` with `schedule: "0 3 * * 0"`. Within a few seconds the controller has acted on it - and on `nightly`: each has `status.phase: Scheduled`, and a ConfigMap `backup-NAME` holds its schedule. `kubectl -n backup-operator logs deploy/controller` is where you would look if it did not.""",
              check=r"""
 eq "$(j -n data bk weekly '{.status.phase}')" Scheduled && eq "$(j -n data cm backup-weekly '{.data.schedule}')" "0 3 * * 0" && eq "$(j -n data bk nightly '{.status.phase}')" Scheduled""",
              hint=r"""A Backup like `nightly`. `kubectl -n data get bk` - add `additionalPrinterColumns` to the CRD and the phase would show there too.""",
-             show=[hand(r"""cp nightly.yaml weekly.yaml
+             show=r"""kubectl -n data apply -f - <<'EOF'
-vim weekly.yaml""",
-                        r"""Name it `weekly`, with `schedule: "0 3 * * 0"`.""",
-                        run=r"""cat > weekly.yaml <<'EOF'
 apiVersion: data.example.com/v1
 kind: Backup
 metadata:
   name: weekly
 spec:
   schedule: "0 3 * * 0"
-EOF"""),
+EOF
-                   cmd(r"""k -n data apply -f weekly.yaml
+sleep 10
-k -n data get bk -o yaml | grep phase
+kubectl -n data get bk -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PHASE:.status.phase
-k -n data get cm""", r"""Again in a few seconds: both Backups `phase: Scheduled`, and a ConfigMap `backup-NAME` for each.""")]),
+kubectl -n data get cm"""),
     ],
     "outro": r"""You defined a kind with validation and defaults, saw the schema refuse bad objects, installed an operator with its RBAC, and watched it act on your custom resources.
 

cka/lessons_net.py

@@ -1,6 +1,6 @@
 """CKA lessons of the Services and Networking domain (5.1-5.6). See cka/lessons.py for what a lesson is."""
 from cka.common import DNS
-from sim.bank import NET, NS, PODS, step, cmd, hand
+from sim.bank import NET, NS, PODS, step
 
 # ---- 5.1 connectivity between Pods ------------------------------------------------------------
 L51 = {
@@ -35,55 +35,35 @@ ready net || exit 1
 want=$($K get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' | sort)
 eq "$(tr -s ' \t' ' ' < "$WS/lesson-5.1/cidrs.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """,
              hint=r"""`.spec.podCIDR` of each node: `-o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}'`, or `custom-columns`. The cluster-wide range is the `podSubnet` in kubeadm's configuration.""",
-             show=[cmd(r"""k get no -o custom-columns=NODE:.metadata.name,CIDR:.spec.podCIDR --no-headers > {WS}/lesson-5.1/cidrs.txt
+             show=r"""kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' > {WS}/lesson-5.1/cidrs.txt
 cat {WS}/lesson-5.1/cidrs.txt
-k -n net get po -o wide""", r"""Every Pod's IP is from the range of the node it runs on.""")]),
+kubectl -n net get pods -o wide"""),
         step("Across nodes, Pod to Pod",
              r"""From Pod `a`, send a request straight to Pod `b`'s **IP address** on port 80 - no Service - and save the answer in `{WS}/lesson-5.1/answer.txt`. It crossed from `{W1}` to `{W2}`.""",
              check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/answer.txt" 2>/dev/null)" b""",
              hint=r"""`kubectl -n net get pod b -o jsonpath='{.status.podIP}'`, then `kubectl -n net exec a -- wget -qO- -T3 http://IP`.""",
-             show=[cmd(r"""k -n net get po -o wide"""),
+             show=r"""ip=$(kubectl -n net get pod b -o jsonpath='{.status.podIP}')
-                   hand(r"""k -n net exec a -- wget -qO- -T3 IP > {WS}/lesson-5.1/answer.txt""",
+kubectl -n net exec a -- wget -qO- -T3 http://$ip | tee {WS}/lesson-5.1/answer.txt"""),
-                        r"""`IP`: Pod `b`'s address, from the list.""",
-                        run=r"""ip=$(kubectl -n net get pod b -o jsonpath='{.status.podIP}')
-kubectl -n net exec a -- wget -qO- -T3 http://$ip > {WS}/lesson-5.1/answer.txt"""),
-                   cmd(r"""cat {WS}/lesson-5.1/answer.txt""")]),
         step("The route that carries it",
              r"""On `{W1}`, look at the routing table: there is a route for `{W2}`'s Pod CIDR, via `{W2}`'s node address. Write that route's destination and gateway to `{WS}/lesson-5.1/route.txt`, as `CIDR via ADDRESS`.""",
              check=r"""
 c=$(j node "$W2" '{.spec.podCIDR}'); gw=$(j node "$W2" '{.status.addresses[?(@.type=="InternalIP")].address}')
 nx "$W1" "ip route | grep -q '^$c via $gw '" </dev/null && eq "$(tr -s ' ' < "$WS/lesson-5.1/route.txt" 2>/dev/null | tr -d '\r\n' | sed 's/ *$//')" "$c via $gw" """,
              hint=r"""`ip route` on the node. The node's address is its `InternalIP` (`kubectl get nodes -o wide`).""",
-             show=[cmd(r"""k get no -o wide
+             show=r"""ssh -n {W1} ip route
-k get no -o custom-columns=NODE:.metadata.name,CIDR:.spec.podCIDR""", r"""`{W2}`'s address (`INTERNAL-IP`) and its Pod CIDR."""),
+c=$(kubectl get node {W2} -o jsonpath='{.spec.podCIDR}')
-                   hand(r"""ssh {W1} ip route""",
+ssh -n {W1} ip route | grep "^$c " | awk '{print $1, $2, $3}' > {WS}/lesson-5.1/route.txt
-                        r"""The line for `{W2}`'s Pod CIDR reads `CIDR via ADDRESS dev ...` - the route to the other node's Pods.""",
+cat {WS}/lesson-5.1/route.txt"""),
-                        run=r"""ssh -n {W1} ip route"""),
-                   hand(r"""echo 'CIDR via ADDRESS' > {WS}/lesson-5.1/route.txt""",
-                        r"""The first three words of that line.""",
-                        run=r"""c=$(kubectl get node {W2} -o jsonpath='{.spec.podCIDR}')
-ssh -n {W1} ip route | grep "^$c " | awk '{print $1, $2, $3}' > {WS}/lesson-5.1/route.txt""")]),
         step("A Pod on the host's network",
              r"""Run Pod `hostnet` in `net` (`busybox:1.36`, `sleep 86400`) with `hostNetwork: true`, on `{W1}`. Its IP is not from the Pod CIDR: it is `{W1}`'s own address, and it sees the node's interfaces. Write its IP to `{WS}/lesson-5.1/hostnet.txt`.""",
              check=r"""
 want=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}')
 eq "$(j -n net pod hostnet '{.spec.hostNetwork} {.status.podIP}')" "true $want" && eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/hostnet.txt" 2>/dev/null)" "$want" """,
              hint=r"""`spec.hostNetwork: true` and `spec.nodeName: {W1}` (or a nodeSelector). `kubectl run ... --overrides='{"spec":{...}}'` adds both.""",
-             show=[cmd(r"""k -n net run hostnet --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > hostnet.yaml"""),
+             show=r"""kubectl -n net run hostnet --image=busybox:1.36 --overrides='{"spec":{"hostNetwork":true,"nodeName":"{W1}"}}' -- sleep 86400
-                   hand(r"""vim hostnet.yaml""",
+kubectl -n net wait --for=condition=Ready pod/hostnet --timeout=120s
-                        r"""Add to the Pod's `spec:`, above `containers:`:
+kubectl -n net get pod hostnet -o jsonpath='{.status.podIP}' > {WS}/lesson-5.1/hostnet.txt
-```
+kubectl -n net exec hostnet -- ip -4 addr show eth0"""),
-  hostNetwork: true
-  nodeName: {W1}
-```""",
-                        run=r"""yq -i '.spec.hostNetwork = true | .spec.nodeName = "{W1}"' hostnet.yaml"""),
-                   cmd(r"""k -n net apply -f hostnet.yaml
-k -n net get po hostnet -o wide""", r"""Its IP is `{W1}`'s own address, not one from the Pod CIDR."""),
-                   hand(r"""echo IP > {WS}/lesson-5.1/hostnet.txt""",
-                        r"""`IP`: that address.""",
-                        run=r"""kubectl -n net wait --for=condition=Ready pod/hostnet --timeout=120s
-kubectl -n net get pod hostnet -o jsonpath='{.status.podIP}' > {WS}/lesson-5.1/hostnet.txt"""),
-                   cmd(r"""k -n net exec hostnet -- ip -4 addr show eth0""")]),
     ],
     "outro": r"""You read how the Pod network is divided among nodes, sent a request from one node's Pod to another's, found the route that carried it, and ran a Pod on the host network.
 
@@ -128,9 +108,7 @@ ready prod && ready dev || exit 1
 eq "$(j -n prod netpol default-deny '{.spec.policyTypes[0]}')" Ingress && [ -z "$(j -n prod netpol default-deny '{.spec.ingress}')" ] || exit 1
 blocked prod api http://db && blocked dev tester http://web.prod""",
              hint=r"""`podSelector: {}` and `policyTypes: [Ingress]`, with no `ingress:` list. Test with `kubectl -n prod exec api -- wget -qO- -T3 db`.""",
-             show=[hand(r"""vim deny.yaml""",
+             show=r"""kubectl -n prod apply -f - <<'EOF'
-                        r"""No `kubectl create` for a NetworkPolicy: paste *Default deny all ingress traffic* from the docs page *Network Policies*, and name it:
-```
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
@@ -138,25 +116,13 @@ metadata:
 spec:
   podSelector: {}
   policyTypes: [Ingress]
-```""",
+EOF
-                        run=r"""cat > deny.yaml <<'EOF'
+kubectl -n prod exec api -- wget -qO- -T3 db || echo blocked"""),
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: default-deny
-spec:
-  podSelector: {}
-  policyTypes: [Ingress]
-EOF"""),
-                   cmd(r"""k -n prod apply -f deny.yaml
-k -n prod exec api -- wget -qO- -T3 db""", r"""It times out: blocked.""")]),
         step("Let the API reach the database",
              r"""Create NetworkPolicy `db-from-api` in `prod`: Pods `app=db` accept TCP 80 from Pods `app=api` **of `prod`**. `web` still cannot reach `db`, and neither can `dev`'s `api` - a `podSelector` alone means "in this Namespace".""",
              check=NET + r"""okr prod api http://db && blocked prod web http://db && blocked dev api http://db.prod""",
              hint=r"""`podSelector: {matchLabels: {app: db}}`, `ingress: [{from: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}]`.""",
-             show=[hand(r"""vim db-from-api.yaml""",
+             show=r"""kubectl -n prod apply -f - <<'EOF'
-                        r"""Paste the example from the docs page *Network Policies* and cut it down to:
-```
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
@@ -171,36 +137,16 @@ spec:
         matchLabels: {app: api}
     ports:
     - {protocol: TCP, port: 80}
-```""",
+EOF
-                        run=r"""cat > db-from-api.yaml <<'EOF'
+kubectl -n prod exec api -- wget -qO- -T3 db
-apiVersion: networking.k8s.io/v1
+kubectl -n prod exec web -- wget -qO- -T3 db || echo blocked"""),
-kind: NetworkPolicy
-metadata:
-  name: db-from-api
-spec:
-  podSelector:
-    matchLabels: {app: db}
-  policyTypes: [Ingress]
-  ingress:
-  - from:
-    - podSelector:
-        matchLabels: {app: api}
-    ports:
-    - {protocol: TCP, port: 80}
-EOF"""),
-                   cmd(r"""k -n prod apply -f db-from-api.yaml
-k -n prod exec api -- wget -qO- -T3 db
-k -n prod exec web -- wget -qO- -T3 db""", r"""`api` gets the page; `web` times out.""")]),
         step("Those Pods in that Namespace",
              r"""Create NetworkPolicy `web-from-dev-api` in `prod`: Pods `app=web` accept TCP 80 from Pods `app=api` **in** Namespaces labelled `team=dev` - both conditions in **one** element of `from`.
 
 `dev`'s `api` gets in; `dev`'s `tester` does not, and neither does `prod`'s own `api`. Written as two elements, the rule would let every Pod of `dev` in, and every `api` Pod of `prod` too.""",
              check=NET + r"""okr dev api http://web.prod && blocked dev tester http://web.prod && blocked prod api http://web""",
              hint=r"""`from: [{namespaceSelector: {matchLabels: {team: dev}}, podSelector: {matchLabels: {app: api}}}]` - mind the single `-`.""",
-             show=[hand(r"""cp db-from-api.yaml web-from-dev-api.yaml
+             show=r"""kubectl -n prod apply -f - <<'EOF'
-vim web-from-dev-api.yaml""",
-                        r"""Make it - `namespaceSelector` and `podSelector` in **one** element of `from` (no `-` before `podSelector`):
-```
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
@@ -217,28 +163,9 @@ spec:
         matchLabels: {app: api}
     ports:
     - {protocol: TCP, port: 80}
-```""",
+EOF
-                        run=r"""cat > web-from-dev-api.yaml <<'EOF'
+kubectl -n dev exec api -- wget -qO- -T3 web.prod
-apiVersion: networking.k8s.io/v1
+kubectl -n dev exec tester -- wget -qO- -T3 web.prod || echo blocked"""),
-kind: NetworkPolicy
-metadata:
-  name: web-from-dev-api
-spec:
-  podSelector:
-    matchLabels: {app: web}
-  policyTypes: [Ingress]
-  ingress:
-  - from:
-    - namespaceSelector:
-        matchLabels: {team: dev}
-      podSelector:
-        matchLabels: {app: api}
-    ports:
-    - {protocol: TCP, port: 80}
-EOF"""),
-                   cmd(r"""k -n prod apply -f web-from-dev-api.yaml
-k -n dev exec api -- wget -qO- -T3 web.prod
-k -n dev exec tester -- wget -qO- -T3 web.prod""", r"""`dev`'s `api` gets in; `tester` times out.""")]),
         step("Egress, and DNS",
              r"""Now limit what `tester` in `dev` may **send**: NetworkPolicy `tester-egress` in `dev` selects `app=tester` and allows egress only to `dev`'s `app=api` Pods on TCP 80 - and DNS, to `kube-dns` in `kube-system` on UDP and TCP 53.
 
@@ -247,9 +174,7 @@ k -n dev exec tester -- wget -qO- -T3 web.prod""", r"""`dev`'s `api` gets in; `t
 eq "$(j -n dev netpol tester-egress '{.spec.policyTypes[0]}')" Egress || exit 1
 okr dev tester http://api && blocked dev tester http://echo""",
              hint=r"""Two egress rules: `{to: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}` and `{to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}], ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]}`. Every Namespace carries the label `kubernetes.io/metadata.name`.""",
-             show=[hand(r"""vim tester-egress.yaml""",
+             show=r"""kubectl -n dev apply -f - <<'EOF'
-                        r"""An egress policy - two elements of `egress`, one for `api`, one for DNS:
-```
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
@@ -272,34 +197,9 @@ spec:
     ports:
     - {protocol: UDP, port: 53}
     - {protocol: TCP, port: 53}
-```""",
+EOF
-                        run=r"""cat > tester-egress.yaml <<'EOF'
+kubectl -n dev exec tester -- wget -qO- -T3 api
-apiVersion: networking.k8s.io/v1
+kubectl -n dev exec tester -- wget -qO- -T3 echo || echo blocked"""),
-kind: NetworkPolicy
-metadata:
-  name: tester-egress
-spec:
-  podSelector:
-    matchLabels: {app: tester}
-  policyTypes: [Egress]
-  egress:
-  - to:
-    - podSelector:
-        matchLabels: {app: api}
-    ports:
-    - {protocol: TCP, port: 80}
-  - to:
-    - namespaceSelector:
-        matchLabels: {kubernetes.io/metadata.name: kube-system}
-      podSelector:
-        matchLabels: {k8s-app: kube-dns}
-    ports:
-    - {protocol: UDP, port: 53}
-    - {protocol: TCP, port: 53}
-EOF"""),
-                   cmd(r"""k -n dev apply -f tester-egress.yaml
-k -n dev exec tester -- wget -qO- -T3 api
-k -n dev exec tester -- wget -qO- -T3 echo""", r"""`api` answers; `echo` times out.""")]),
     ],
     "outro": r"""You closed a Namespace, opened single paths into it, used the AND form of a rule, and limited a Pod's egress without breaking its DNS.
 
@@ -343,28 +243,20 @@ $K -n svc rollout status deploy web --timeout=180s >/dev/null && ready svc
              r"""Expose Deployment `web` as Service `web`, port 80. From `client`, `wget -qO- web` returns nginx's page.""",
              check=NET + r"""eq "$(j -n svc svc web '{.spec.type} {.spec.ports[0].port}')" "ClusterIP 80" && okr svc client http://web""",
              hint=r"""`kubectl expose deploy web --port=80` - it copies the Deployment's selector.""",
-             show=[cmd(r"""k -n svc expose deploy web --port=80
+             show=r"""kubectl -n svc expose deploy web --port=80
-k -n svc get svc,endpointslices
+kubectl -n svc get svc,endpointslices
-k -n svc exec client -- wget -qO- -T3 web | head -4""")]),
+kubectl -n svc exec client -- wget -qO- -T3 web | head -4"""),
         step("NodePort",
              r"""Create Service `web-np` of type **NodePort** for the same Pods, with node port **30080**. Then reach it from the control-plane node: `curl http://{W1}-ADDRESS:30080` - every node answers on that port, whether or not it runs one of the Pods.""",
              check=r"""
 eq "$(j -n svc svc web-np '{.spec.type} {.spec.ports[0].nodePort}')" "NodePort 30080" || exit 1
 ip=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}'); nx "$CP" "curl -s -m 3 http://$ip:30080 | grep -q nginx" </dev/null""",
              hint=r"""`kubectl create service nodeport web-np --tcp=80:80 --node-port=30080` makes a selector `app=web-np` - change it to `app=web`; or write the YAML, or `kubectl expose deploy web --name=web-np --type=NodePort` and then set the `nodePort`.""",
-             show=[cmd(r"""k -n svc expose deploy web --name=web-np --type=NodePort --port=80 --dry-run=client -o yaml > web-np.yaml"""),
+             show=r"""kubectl -n svc expose deploy web --name=web-np --type=NodePort --port=80 --dry-run=client -o yaml > {WS}/lesson-5.3/web-np.yaml
-                   hand(r"""vim web-np.yaml""",
+yq -i '.spec.ports[0].nodePort = 30080' {WS}/lesson-5.3/web-np.yaml
-                        r"""Add to the port:
+kubectl -n svc apply -f {WS}/lesson-5.3/web-np.yaml
-```
+ip=$(kubectl get node {W1} -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
-    nodePort: 30080
+curl -s -m 3 http://$ip:30080 | head -4"""),
-```""",
-                        run=r"""yq -i '.spec.ports[0].nodePort = 30080' web-np.yaml"""),
-                   cmd(r"""k -n svc apply -f web-np.yaml
-k get no -o wide""", r"""`{W1}`'s `INTERNAL-IP`:"""),
-                   hand(r"""curl IP:30080""",
-                        r"""nginx's page, from the node port.""",
-                        run=r"""ip=$(kubectl get node {W1} -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
-curl -s -m 3 http://$ip:30080 | head -4""")]),
         step("Headless",
              r"""Create Service `web-headless` with `clusterIP: None` for the `web` Pods. Look it up from `client` with `nslookup web-headless`: you get the two Pod IPs, not a Service IP. Write them, one per line, to `{WS}/lesson-5.3/headless.txt`.""",
              check=DNS + r"""
@@ -372,19 +264,18 @@ eq "$(j -n svc svc web-headless '{.spec.clusterIP}')" None || exit 1
 want=$($K -n svc get pods -l app=web -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' | sort)
 eq "$(tr -d ' \r' < "$WS/lesson-5.3/headless.txt" 2>/dev/null | grep . | sort)" "$want" """,
              hint=r"""`kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80`. In busybox's `nslookup` output the addresses follow the `Name:` line.""",
-             show=[cmd(r"""k -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80
+             show=r"""kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80
-k -n svc exec client -- nslookup web-headless""", r"""Two addresses: the Pods' own. Again in a few seconds if it does not resolve yet."""),
+sleep 5
-                   hand(r"""vim {WS}/lesson-5.3/headless.txt""",
+kubectl -n svc exec client -- nslookup web-headless
-                        r"""The two addresses, one per line.""",
+kubectl -n svc exec client -- nslookup web-headless | awk '/^Name:/ {n=1} n && /^Address/ {print $NF}' > {WS}/lesson-5.3/headless.txt
-                        run=r"""sleep 5
+cat {WS}/lesson-5.3/headless.txt"""),
-kubectl -n svc exec client -- nslookup web-headless | awk '/^Name:/ {n=1} n && /^Address/ {print $NF}' > {WS}/lesson-5.3/headless.txt""")]),
         step("ExternalName",
              r"""Create Service `docs` of type **ExternalName** pointing at `kubernetes.io`. Inside the cluster, `docs.svc.svc.cluster.local` is now a CNAME for that name - DNS only: nothing is proxied, and the port does not matter.""",
              check=r"""eq "$(j -n svc svc docs '{.spec.type} {.spec.externalName}')" "ExternalName kubernetes.io" """,
              hint=r"""`kubectl -n svc create service externalname docs --external-name=kubernetes.io`.""",
-             show=[cmd(r"""k -n svc create svc externalname docs --external-name=kubernetes.io
+             show=r"""kubectl -n svc create service externalname docs --external-name=kubernetes.io
-k -n svc get svc docs
+kubectl -n svc get svc docs
-k -n svc exec client -- nslookup docs | head -6""")]),
+kubectl -n svc exec client -- nslookup docs | head -6"""),
         step("A Service without a selector",
              r"""Create Service `legacy` (port 80) with **no selector**, and an EndpointSlice for it by hand that holds `legacy-box`'s Pod IP on port 80. From `client`, `wget -qO- legacy` then answers `legacy`.
 
@@ -393,32 +284,8 @@ An EndpointSlice belongs to a Service by the label `kubernetes.io/service-name`;
 [ -z "$(j -n svc svc legacy '{.spec.selector}')" ] || exit 1
 for i in 1 2 3 4 5; do [ "$($K -n svc exec client -- wget -qO- -T3 legacy 2>/dev/null)" = legacy ] && exit 0; sleep 3; done; exit 1""",
              hint=r"""Service: `spec.ports: [{port: 80, targetPort: 80}]` and no `selector`. EndpointSlice (`discovery.k8s.io/v1`): `addressType: IPv4`, `endpoints: [{addresses: [IP]}]`, `ports: [{port: 80, protocol: TCP}]`, and the label.""",
-             show=[cmd(r"""k -n svc get po legacy-box -o wide""", r"""Its IP goes into the EndpointSlice."""),
+             show=r"""ip=$(kubectl -n svc get pod legacy-box -o jsonpath='{.status.podIP}')
-                   hand(r"""vim legacy.yaml""",
+kubectl -n svc apply -f - <<EOF
-                        r"""No `kubectl create` makes a Service without a selector, nor an EndpointSlice: paste them from the docs page *Service* (*Services without selectors*), and make them, with that IP:
-```
-apiVersion: v1
-kind: Service
-metadata:
-  name: legacy
-spec:
-  ports:
-  - {port: 80, targetPort: 80, protocol: TCP}
----
-apiVersion: discovery.k8s.io/v1
-kind: EndpointSlice
-metadata:
-  name: legacy-1
-  labels:
-    kubernetes.io/service-name: legacy
-addressType: IPv4
-endpoints:
-- addresses: ["IP"]
-ports:
-- {port: 80, protocol: TCP}
-```""",
-                        run=r"""ip=$(kubectl -n svc get pod legacy-box -o jsonpath='{.status.podIP}')
-cat > legacy.yaml <<EOF
 apiVersion: v1
 kind: Service
 metadata:
@@ -438,9 +305,8 @@ endpoints:
 - addresses: ["$ip"]
 ports:
 - {port: 80, protocol: TCP}
-EOF"""),
+EOF
-                   cmd(r"""k -n svc apply -f legacy.yaml
+kubectl -n svc exec client -- wget -qO- -T3 legacy"""),
-k -n svc exec client -- wget -qO- -T3 legacy""")]),
     ],
     "outro": r"""You exposed the same Pods four ways, saw what headless DNS returns, aliased an outside name, and backed a Service with endpoints you wrote yourself.
 
@@ -482,9 +348,7 @@ mkdir -p "$WS/lesson-5.4"
 eq "$(j gatewayclass example '{.spec.controllerName}')" example.com/gateway-controller || exit 1
 eq "$(j -n gw gateway shop '{.spec.gatewayClassName} {.spec.listeners[0].name} {.spec.listeners[0].protocol} {.spec.listeners[0].port} {.spec.listeners[0].hostname} {.spec.listeners[0].allowedRoutes.namespaces.from}')" "example http HTTP 80 shop.example.com Same" """,
              hint=r"""Both are `gateway.networking.k8s.io/v1`. `kubectl explain gateway.spec.listeners` once the CRDs are in; the docs at gateway-api.sigs.k8s.io have every example.""",
-             show=[hand(r"""vim gw.yaml""",
+             show=r"""kubectl apply -f - <<'EOF'
-                        r"""No `kubectl create` for these: paste a GatewayClass and a Gateway from gateway-api.sigs.k8s.io (*API overview*), and make them:
-```
 apiVersion: gateway.networking.k8s.io/v1
 kind: GatewayClass
 metadata:
@@ -507,33 +371,8 @@ spec:
     allowedRoutes:
       namespaces:
         from: Same
-```""",
+EOF
-                        run=r"""cat > gw.yaml <<'EOF'
+kubectl -n gw get gateway shop"""),
-apiVersion: gateway.networking.k8s.io/v1
-kind: GatewayClass
-metadata:
-  name: example
-spec:
-  controllerName: example.com/gateway-controller
----
-apiVersion: gateway.networking.k8s.io/v1
-kind: Gateway
-metadata:
-  name: shop
-  namespace: gw
-spec:
-  gatewayClassName: example
-  listeners:
-  - name: http
-    protocol: HTTP
-    port: 80
-    hostname: shop.example.com
-    allowedRoutes:
-      namespaces:
-        from: Same
-EOF"""),
-                   cmd(r"""k apply -f gw.yaml
-k -n gw get gateway shop""")]),
         step("Route by path",
              r"""Create HTTPRoute `shop` in `gw`, attached to Gateway `shop`: requests for paths under `/api` go to Service `api` port 8080, everything else to `web` port 80.""",
              check=r"""
@@ -547,9 +386,7 @@ def to(path):
                 return [(b["name"], b.get("port")) for b in x["backendRefs"]]
 sys.exit(0 if p == "shop" and to("/api") == [("api", 8080)] and to("/") == [("web", 80)] else 1)'""",
              hint=r"""`parentRefs: [{name: shop}]`, then two `rules`, each with `matches: [{path: {type: PathPrefix, value: ...}}]` and `backendRefs: [{name: ..., port: ...}]`.""",
-             show=[hand(r"""vim route.yaml""",
+             show=r"""kubectl -n gw apply -f - <<'EOF'
-                        r"""Paste an HTTPRoute from gateway-api.sigs.k8s.io (*HTTP routing*), and make it:
-```
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
 metadata:
@@ -567,28 +404,8 @@ spec:
     - path: {type: PathPrefix, value: /}
     backendRefs:
     - {name: web, port: 80}
-```""",
+EOF
-                        run=r"""cat > route.yaml <<'EOF'
+kubectl -n gw describe httproute shop | grep -A3 'Backend Refs' | head -8"""),
-apiVersion: gateway.networking.k8s.io/v1
-kind: HTTPRoute
-metadata:
-  name: shop
-spec:
-  parentRefs:
-  - name: shop
-  hostnames: [shop.example.com]
-  rules:
-  - matches:
-    - path: {type: PathPrefix, value: /api}
-    backendRefs:
-    - {name: api, port: 8080}
-  - matches:
-    - path: {type: PathPrefix, value: /}
-    backendRefs:
-    - {name: web, port: 80}
-EOF"""),
-                   cmd(r"""k -n gw apply -f route.yaml
-k -n gw describe httproute shop""")]),
         step("A canary by weight",
              r"""Change the `/` rule of route `shop` so that **90%** of its requests go to `web` and **10%** to `web-v2` (both port 80). The `/api` rule stays as it is.""",
              check=r"""
@@ -599,20 +416,10 @@ root, api = by("/"), by("/api")
 w = sorted((b["name"], b.get("weight", 1)) for b in root["backendRefs"]) if root else []
 sys.exit(0 if w == [("web", 90), ("web-v2", 10)] and api and [b["name"] for b in api["backendRefs"]] == ["api"] else 1)'""",
              hint=r"""Two `backendRefs` in the same rule, each with a `weight`. The weights are relative: 9 and 1 would do the same.""",
-             show=[hand(r"""k -n gw edit httproute shop""",
+             show=r"""kubectl -n gw get httproute shop -o yaml > {WS}/lesson-5.4/shop.yaml
-                        r"""In the rule whose path is `/`, give `web` a `weight: 90` and add `web-v2` beside it:
-```
-    - name: web
-      port: 80
-      weight: 90
-    - name: web-v2
-      port: 80
-      weight: 10
-```""",
-                        run=r"""kubectl -n gw get httproute shop -o yaml > {WS}/lesson-5.4/shop.yaml
 yq -i '(.spec.rules[] | select(.matches[0].path.value == "/")).backendRefs = [{"name": "web", "port": 80, "weight": 90}, {"name": "web-v2", "port": 80, "weight": 10}]' {WS}/lesson-5.4/shop.yaml
-kubectl apply -f {WS}/lesson-5.4/shop.yaml"""),
+kubectl apply -f {WS}/lesson-5.4/shop.yaml
-                   cmd(r"""k -n gw describe httproute shop""")]),
+kubectl -n gw get httproute shop -o jsonpath='{.spec.rules[1].backendRefs}'; echo"""),
         step("Beta testers by header",
              r"""Add HTTPRoute `shop-beta` in `gw`, attached to `shop`: requests carrying the header `x-env: beta` go **all** to `web-v2`. A more specific match wins, so these requests no longer take part in the 90/10 split.""",
              check=r"""
@@ -621,9 +428,7 @@ import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0]
 h = r["matches"][0].get("headers", [{}])[0]
 sys.exit(0 if s["parentRefs"][0]["name"] == "shop" and h.get("name", "").lower() == "x-env" and h.get("value") == "beta" and [b["name"] for b in r["backendRefs"]] == ["web-v2"] else 1)'""",
              hint=r"""`matches: [{headers: [{name: x-env, value: beta}]}]` (type `Exact` by default).""",
-             show=[hand(r"""vim beta.yaml""",
+             show=r"""kubectl -n gw apply -f - <<'EOF'
-                        r"""Paste an HTTPRoute with a header match from gateway-api.sigs.k8s.io (*HTTP header matching*), and make it:
-```
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
 metadata:
@@ -638,25 +443,8 @@ spec:
       - {name: x-env, value: beta}
     backendRefs:
     - {name: web-v2, port: 80}
-```""",
+EOF
-                        run=r"""cat > beta.yaml <<'EOF'
+kubectl -n gw get httproute"""),
-apiVersion: gateway.networking.k8s.io/v1
-kind: HTTPRoute
-metadata:
-  name: shop-beta
-spec:
-  parentRefs:
-  - name: shop
-  hostnames: [shop.example.com]
-  rules:
-  - matches:
-    - headers:
-      - {name: x-env, value: beta}
-    backendRefs:
-    - {name: web-v2, port: 80}
-EOF"""),
-                   cmd(r"""k -n gw apply -f beta.yaml
-k -n gw get httproute""")]),
         step("HTTPS on the Gateway",
              r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `gw`, and add a second listener to Gateway `shop`: `https`, protocol HTTPS, port 443, the same hostname, TLS mode `Terminate` with that Secret as its certificate.""",
              check=r"""
@@ -664,27 +452,12 @@ eq "$(j -n gw secret shop-tls '{.type}')" kubernetes.io/tls || exit 1
 eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="https")].protocol} {.spec.listeners[?(@.name=="https")].port} {.spec.listeners[?(@.name=="https")].tls.mode} {.spec.listeners[?(@.name=="https")].tls.certificateRefs[0].name}')" "HTTPS 443 Terminate shop-tls" \
   && eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="http")].port}')" 80""",
              hint=r"""`openssl req -x509 ...`, `kubectl create secret tls`, then a listener with `tls: {mode: Terminate, certificateRefs: [{name: shop-tls}]}`. Keep the `http` listener.""",
-             show=[cmd(r"""cd {WS}/lesson-5.4
+             show=r"""cd {WS}/lesson-5.4
-openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com
+openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
-k -n gw create secret tls shop-tls --cert=tls.crt --key=tls.key
+kubectl -n gw create secret tls shop-tls --cert=tls.crt --key=tls.key
+kubectl -n gw patch gateway shop --type json -p '[{"op":"add","path":"/spec/listeners/-","value":{"name":"https","protocol":"HTTPS","port":443,"hostname":"shop.example.com","tls":{"mode":"Terminate","certificateRefs":[{"name":"shop-tls"}]},"allowedRoutes":{"namespaces":{"from":"Same"}}}}]'
+kubectl -n gw get gateway shop -o jsonpath='{range .spec.listeners[*]}{.name} {.port}{"\n"}{end}'
 cd"""),
-                   hand(r"""k -n gw edit gateway shop""",
-                        r"""Add a second listener below `http`, in `listeners:`:
-```
-  - name: https
-    protocol: HTTPS
-    port: 443
-    hostname: shop.example.com
-    tls:
-      mode: Terminate
-      certificateRefs:
-      - name: shop-tls
-    allowedRoutes:
-      namespaces:
-        from: Same
-```""",
-                        run=r"""kubectl -n gw patch gateway shop --type json -p '[{"op":"add","path":"/spec/listeners/-","value":{"name":"https","protocol":"HTTPS","port":443,"hostname":"shop.example.com","tls":{"mode":"Terminate","certificateRefs":[{"name":"shop-tls"}]},"allowedRoutes":{"namespaces":{"from":"Same"}}}}]' """),
-                   cmd(r"""k -n gw describe gateway shop | grep -A3 -i listener""")]),
     ],
     "outro": r"""You built a Gateway with HTTP and HTTPS listeners, routed by path, split traffic by weight, and sent one header's requests elsewhere.
 
@@ -725,9 +498,7 @@ mkdir -p "$WS/lesson-5.5"
              r"""Create IngressClass `nginx` for controller `k8s.io/ingress-nginx`, and make it the cluster's **default** class: an Ingress without `ingressClassName` then belongs to it.""",
              check=r"""eq "$(j ingressclass nginx '{.spec.controller} {.metadata.annotations.ingressclass\.kubernetes\.io/is-default-class}')" "k8s.io/ingress-nginx true" """,
              hint=r"""`networking.k8s.io/v1`, kind `IngressClass`, `spec.controller`, and the annotation `ingressclass.kubernetes.io/is-default-class: "true"`. Installing ingress-nginx makes this object for you.""",
-             show=[hand(r"""vim ic.yaml""",
+             show=r"""kubectl apply -f - <<'EOF'
-                        r"""No `kubectl create` for an IngressClass: paste the example from the docs page *Ingress* (*Default IngressClass*), and make it:
-```
 apiVersion: networking.k8s.io/v1
 kind: IngressClass
 metadata:
@@ -736,19 +507,8 @@ metadata:
     ingressclass.kubernetes.io/is-default-class: "true"
 spec:
   controller: k8s.io/ingress-nginx
-```""",
+EOF
-                        run=r"""cat > ic.yaml <<'EOF'
+kubectl get ingressclass"""),
-apiVersion: networking.k8s.io/v1
-kind: IngressClass
-metadata:
-  name: nginx
-  annotations:
-    ingressclass.kubernetes.io/is-default-class: "true"
-spec:
-  controller: k8s.io/ingress-nginx
-EOF"""),
-                   cmd(r"""k apply -f ic.yaml
-k get ingressclass""")]),
         step("Host and paths, from the command line",
              r"""Create Ingress `shop` in `ing` with class `nginx` and host `shop.example.com`: paths under `/api` go to `api:8080`, everything else to `web:80`. `kubectl create ingress` can do it in one line.""",
              check=r"""
@@ -757,8 +517,8 @@ import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0]
 p = {x["path"]: (x["pathType"], x["backend"]["service"]["name"], x["backend"]["service"]["port"]["number"]) for x in r["http"]["paths"]}
 sys.exit(0 if s.get("ingressClassName") == "nginx" and r["host"] == "shop.example.com" and p.get("/api") == ("Prefix", "api", 8080) and p.get("/") == ("Prefix", "web", 80) else 1)'""",
              hint=r"""`kubectl create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'` - a trailing `*` means `Prefix`, without it `Exact`.""",
-             show=[cmd(r"""k -n ing create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'
+             show=r"""kubectl -n ing create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'
-k -n ing describe ingress shop""")]),
+kubectl -n ing describe ingress shop | sed -n '/Rules/,/Annotations/p'"""),
         step("An exact path",
              r"""Add a rule to Ingress `shop`: exactly `/healthz` (pathType `Exact`) goes to `health:8081`. `/healthz/deep` would not match it - it would fall through to the `/` prefix.""",
              check=r"""
@@ -767,53 +527,26 @@ import json, sys; ps = json.load(sys.stdin)["spec"]["rules"][0]["http"]["paths"]
 h = [x for x in ps if x["path"] == "/healthz"]
 sys.exit(0 if h and h[0]["pathType"] == "Exact" and h[0]["backend"]["service"]["name"] == "health" and h[0]["backend"]["service"]["port"]["number"] == 8081 and len(ps) == 3 else 1)'""",
              hint=r"""`kubectl edit ingress shop` and add a third entry under `paths:`, or a JSON patch that appends to `/spec/rules/0/http/paths`.""",
-             show=[hand(r"""k -n ing edit ingress shop""",
+             show=r"""kubectl -n ing patch ingress shop --type json -p '[{"op":"add","path":"/spec/rules/0/http/paths/-","value":{"path":"/healthz","pathType":"Exact","backend":{"service":{"name":"health","port":{"number":8081}}}}}]'
-                        r"""Add a third path to `paths:`, indented like the other two:
+kubectl -n ing get ingress shop -o jsonpath='{range .spec.rules[0].http.paths[*]}{.pathType} {.path} -> {.backend.service.name}{"\n"}{end}'"""),
-```
-      - path: /healthz
-        pathType: Exact
-        backend:
-          service:
-            name: health
-            port:
-              number: 8081
-```""",
-                        run=r"""kubectl -n ing patch ingress shop --type json -p '[{"op":"add","path":"/spec/rules/0/http/paths/-","value":{"path":"/healthz","pathType":"Exact","backend":{"service":{"name":"health","port":{"number":8081}}}}}]' """),
-                   cmd(r"""k -n ing describe ingress shop""")]),
         step("TLS",
              r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `ing`, and add a `tls` section to Ingress `shop` for that host with that Secret. The controller would then serve HTTPS for it and, by default, redirect HTTP there.""",
              check=r"""
 eq "$(j -n ing secret shop-tls '{.type}')" kubernetes.io/tls && eq "$(j -n ing ingress shop '{.spec.tls[0].hosts[0]} {.spec.tls[0].secretName}')" "shop.example.com shop-tls" \
   && j -n ing secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q shop.example.com""",
              hint=r"""`spec.tls: [{hosts: [shop.example.com], secretName: shop-tls}]`. The Secret must be in the Ingress's Namespace.""",
-             show=[cmd(r"""cd {WS}/lesson-5.5
+             show=r"""cd {WS}/lesson-5.5
-openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com
+openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
-k -n ing create secret tls shop-tls --cert=tls.crt --key=tls.key
+kubectl -n ing create secret tls shop-tls --cert=tls.crt --key=tls.key
+kubectl -n ing patch ingress shop --type merge -p '{"spec":{"tls":[{"hosts":["shop.example.com"],"secretName":"shop-tls"}]}}'
+kubectl -n ing get ingress shop
 cd"""),
-                   hand(r"""k -n ing edit ingress shop""",
-                        r"""Add to `spec:`, beside `rules:`:
-```
-  tls:
-  - hosts: [shop.example.com]
-    secretName: shop-tls
-```""",
-                        run=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"tls":[{"hosts":["shop.example.com"],"secretName":"shop-tls"}]}}' """),
-                   cmd(r"""k -n ing get ingress shop""")]),
         step("What matches nothing",
              r"""Give Ingress `shop` a **default backend**: `web:80`, for requests whose host or path no rule matches.""",
              check=r"""eq "$(j -n ing ingress shop '{.spec.defaultBackend.service.name} {.spec.defaultBackend.service.port.number}')" "web 80" """,
              hint=r"""`spec.defaultBackend: {service: {name: web, port: {number: 80}}}`; `kubectl create ingress --default-backend=web:80` does it at creation.""",
-             show=[hand(r"""k -n ing edit ingress shop""",
+             show=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"defaultBackend":{"service":{"name":"web","port":{"number":80}}}}}'
-                        r"""Add to `spec:`, beside `rules:`:
+kubectl -n ing describe ingress shop | head -12"""),
-```
-  defaultBackend:
-    service:
-      name: web
-      port:
-        number: 80
-```""",
-                        run=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"defaultBackend":{"service":{"name":"web","port":{"number":80}}}}}' """),
-                   cmd(r"""k -n ing describe ingress shop""")]),
     ],
     "outro": r"""You made a default IngressClass and an Ingress with host, prefix and exact paths, TLS and a default backend.
 
@@ -875,18 +608,16 @@ ready dns-lab && $K -n dns-lab rollout status sts db --timeout=180s >/dev/null
              r"""From `client`, look up `web` by its **fully qualified** name, and write the address you get to `{WS}/lesson-5.6/web.txt`. It is the Service's ClusterIP.""",
              check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/web.txt" 2>/dev/null)" "$(j -n dns-lab svc web '{.spec.clusterIP}')" """,
              hint=r"""`kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local`.""",
-             show=[cmd(r"""k -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local""", r"""The `Address` below `Name:` is the Service's ClusterIP."""),
+             show=r"""kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local
-                   hand(r"""echo IP > {WS}/lesson-5.6/web.txt""",
+kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local | awk '/^Name:/ {n=1} n && /^Address/ {print $NF; exit}' > {WS}/lesson-5.6/web.txt"""),
-                        r"""`IP`: that address.""",
-                        run=r"""kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local | awk '/^Name:/ {n=1} n && /^Address/ {print $NF; exit}' > {WS}/lesson-5.6/web.txt""")]),
         step("One Pod of a StatefulSet",
              r"""A StatefulSet's Pods get stable names through their headless Service. Write the fully qualified DNS name of Pod `db-0` to `{WS}/lesson-5.6/db0.txt`, and check from `client` that it resolves to `db-0`'s IP.""",
              check=DNS + r"""
 eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/db0.txt" 2>/dev/null | sed 's/\.$//')" db-0.db.dns-lab.svc.cluster.local && resolves dns-lab client db-0.db.dns-lab.svc.cluster.local "$(j -n dns-lab pod db-0 '{.status.podIP}')" """,
              hint=r"""`POD.SERVICE.NAMESPACE.svc.cluster.local` - the Service is the StatefulSet's `serviceName`.""",
-             show=[cmd(r"""echo db-0.db.dns-lab.svc.cluster.local > {WS}/lesson-5.6/db0.txt
+             show=r"""echo db-0.db.dns-lab.svc.cluster.local > {WS}/lesson-5.6/db0.txt
-k -n dns-lab exec client -- nslookup db-0.db.dns-lab.svc.cluster.local
+kubectl -n dns-lab exec client -- nslookup db-0.db.dns-lab.svc.cluster.local
-k -n dns-lab get po db-0 -o wide""")]),
+kubectl -n dns-lab get pod db-0 -o wide"""),
         step("Names of your own",
              r"""Add a `hosts` block to CoreDNS's Corefile so that every Pod resolves **`legacy.internal`** to `10.96.0.250`, and every other name still resolves as before (`fallthrough`). Put it inside the `.:53 { ... }` server block, before `forward`.
 
@@ -895,19 +626,12 @@ CoreDNS reloads the changed ConfigMap by itself within a minute; `client` then r
 $K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep -q 'kubernetes cluster.local' || exit 1
 resolves dns-lab client legacy.internal 10.96.0.250 && resolves dns-lab client web.dns-lab.svc.cluster.local "$(j -n dns-lab svc web '{.spec.clusterIP}')" """,
              hint=r"""`kubectl -n kube-system edit cm coredns`. The block: `hosts {` / `10.96.0.250 legacy.internal` / `fallthrough` / `}`. A syntax error leaves CoreDNS on its old configuration: `kubectl -n kube-system logs -l k8s-app=kube-dns` says so.""",
-             show=[hand(r"""k -n kube-system edit cm coredns""",
+             show=r"""kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > {WS}/lesson-5.6/Corefile
-                        r"""In `Corefile`, add above the `forward` line, indented like it:
-```
-        hosts {
-           10.96.0.250 legacy.internal
-           fallthrough
-        }
-```""",
-                        run=r"""kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > {WS}/lesson-5.6/Corefile
 sed -i '0,/^    forward/s//    hosts {\n       10.96.0.250 legacy.internal\n       fallthrough\n    }\n    forward/' {WS}/lesson-5.6/Corefile
+cat {WS}/lesson-5.6/Corefile
 kubectl -n kube-system create configmap coredns --from-file=Corefile={WS}/lesson-5.6/Corefile --dry-run=client -o yaml | kubectl apply -f -
-sleep 40"""),
+sleep 40
-                   cmd(r"""k -n dns-lab exec client -- nslookup legacy.internal""", r"""Again until it answers `10.96.0.250`: CoreDNS reloads the ConfigMap by itself within a minute or two.""")]),
+kubectl -n dns-lab exec client -- nslookup legacy.internal"""),
         step("A Pod's own resolver settings",
              r"""Run Pod `tuned` in `dns-lab` (`busybox:1.36`, `sleep 86400`) whose resolver you set yourself: `dnsPolicy: None`, with the `kube-dns` Service's IP as its only nameserver, the search domains `dns-lab.svc.cluster.local` and `svc.cluster.local`, and the option `ndots: 2`. Its `/etc/resolv.conf` shows exactly that.""",
              check=r"""
@@ -916,21 +640,26 @@ r=$($K -n dns-lab exec tuned -- cat /etc/resolv.conf 2>/dev/null)
 echo "$r" | grep -qx "nameserver $ip" && echo "$r" | grep -qx 'search dns-lab.svc.cluster.local svc.cluster.local' && echo "$r" | grep -q 'ndots:2' \
   && eq "$(j -n dns-lab pod tuned '{.spec.dnsPolicy}')" None""",
              hint=r"""`spec.dnsPolicy: None` and `spec.dnsConfig: {nameservers: [IP], searches: [...], options: [{name: ndots, value: "2"}]}`. The other policies: `ClusterFirst` (the default), `Default` (the node's resolver), `ClusterFirstWithHostNet`.""",
-             show=[cmd(r"""k -n kube-system get svc kube-dns""", r"""Its `CLUSTER-IP` is the nameserver."""),
+             show=r"""ip=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}')
-                   cmd(r"""k -n dns-lab run tuned --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > tuned.yaml"""),
+kubectl -n dns-lab apply -f - <<EOF
-                   hand(r"""vim tuned.yaml""",
+apiVersion: v1
-                        r"""Change `dnsPolicy: ClusterFirst` to `dnsPolicy: None`, and add beside it (the docs page *DNS for Services and Pods* has it, under *Pod's DNS Config*):
+kind: Pod
-```
+metadata:
+  name: tuned
+spec:
+  dnsPolicy: None
   dnsConfig:
-    nameservers: [IP]
+    nameservers: [$ip]
     searches: [dns-lab.svc.cluster.local, svc.cluster.local]
     options:
     - {name: ndots, value: "2"}
-```""",
+  containers:
-                        run=r"""export IP=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}')
+  - name: tuned
-yq -i '.spec.dnsPolicy = "None" | .spec.dnsConfig = {"nameservers": [strenv(IP)], "searches": ["dns-lab.svc.cluster.local", "svc.cluster.local"], "options": [{"name": "ndots", "value": "2"}]}' tuned.yaml"""),
+    image: busybox:1.36
-                   cmd(r"""k -n dns-lab apply -f tuned.yaml
+    command: [sleep, '86400']
-k -n dns-lab exec tuned -- cat /etc/resolv.conf""")]),
+EOF
+kubectl -n dns-lab wait --for=condition=Ready pod/tuned --timeout=120s
+kubectl -n dns-lab exec tuned -- cat /etc/resolv.conf"""),
     ],
     "outro": r"""You resolved a Service and a StatefulSet Pod by their full names, taught CoreDNS a name of your own, and set a Pod's resolver by hand.
 

cka/lessons_storage.py

@@ -1,5 +1,5 @@
 """CKA lessons of the Storage domain (1.1-1.3). See cka/lessons.py for what a lesson is."""
-from sim.bank import NS, step, cmd, hand
+from sim.bank import NS, step
 
 # ---- 1.1 StorageClasses and dynamic provisioning ---------------------------------------------
 L11 = {
@@ -29,10 +29,9 @@ mkdir -p "$WS/lesson-1.1"
 want="$(j sc standard '{.metadata.name} {.provisioner} {.volumeBindingMode}')"
 [ -n "$want" ] && eq "$(tr -s ' \t' ' ' < "$WS/lesson-1.1/default.txt" 2>/dev/null | tr -d '\r\n' | sed 's/ *$//')" "$want" """,
              hint=r"""`kubectl get sc` shows all three columns. For exactly the values: `-o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}'`.""",
-             show=[cmd(r"""k get sc"""),
+             show=r"""kubectl get sc
-                   hand(r"""echo 'NAME PROVISIONER MODE' > {WS}/lesson-1.1/default.txt""",
+kubectl get sc standard -o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}' > {WS}/lesson-1.1/default.txt
-                        r"""From the `(default)` line: its name, `PROVISIONER` and `VOLUMEBINDINGMODE`.""",
+cat {WS}/lesson-1.1/default.txt"""),
-                        run=r"""kubectl get sc standard -o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}' > {WS}/lesson-1.1/default.txt""")]),
         step("A class that keeps its volumes",
              r"""Create StorageClass `keep`: the same provisioner as `standard` (`rancher.io/local-path`), binding mode `WaitForFirstConsumer`, and reclaim policy **`Retain`**, so a volume outlives its claim.
 
@@ -41,9 +40,7 @@ There is no `kubectl create storageclass`: write the manifest. `kubectl get sc s
 eq "$(j sc keep '{.provisioner} {.reclaimPolicy} {.volumeBindingMode}')" "rancher.io/local-path Retain WaitForFirstConsumer" \
   && [ "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""",
              hint=r"""A StorageClass has no `spec`: `provisioner`, `reclaimPolicy` and `volumeBindingMode` sit at the top level, beside `metadata`. Its `apiVersion` is `storage.k8s.io/v1`.""",
-             show=[hand(r"""vim keep.yaml""",
+             show=r"""kubectl apply -f - <<'EOF'
-                        r"""No `kubectl create` for a StorageClass: paste the example from the docs page *Storage Classes*, and make it:
-```
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
@@ -51,18 +48,8 @@ metadata:
 provisioner: rancher.io/local-path
 reclaimPolicy: Retain
 volumeBindingMode: WaitForFirstConsumer
-```""",
+EOF
-                        run=r"""cat > keep.yaml <<'EOF'
+kubectl get sc"""),
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata:
-  name: keep
-provisioner: rancher.io/local-path
-reclaimPolicy: Retain
-volumeBindingMode: WaitForFirstConsumer
-EOF"""),
-                   cmd(r"""k apply -f keep.yaml
-k get sc""")]),
         step("A claim that waits",
              r"""In Namespace `store`, create PersistentVolumeClaim `ledger`: class `keep`, `ReadWriteOnce`, `100Mi`.
 
@@ -71,9 +58,7 @@ It stays `Pending`, and that is correct. `kubectl -n store describe pvc ledger`
 eq "$(j -n store pvc ledger '{.spec.storageClassName}')" keep && eq "$(j -n store pvc ledger '{.spec.resources.requests.storage}')" 100Mi \
   && grep -qx 'WaitForFirstConsumer' <(tr -d ' \r' < "$WS/lesson-1.1/why.txt" 2>/dev/null)""",
              hint=r"""The events at the bottom of `describe` have a Reason column. A claim is namespaced; the class is not.""",
-             show=[hand(r"""vim ledger.yaml""",
+             show=r"""kubectl -n store apply -f - <<'EOF'
-                        r"""No `kubectl create` for a claim: paste the example from the docs page *Persistent Volumes*, and make it:
-```
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -84,22 +69,9 @@ spec:
   resources:
     requests:
       storage: 100Mi
-```""",
+EOF
-                        run=r"""cat > ledger.yaml <<'EOF'
+kubectl -n store describe pvc ledger | tail -4
-apiVersion: v1
+echo WaitForFirstConsumer > {WS}/lesson-1.1/why.txt"""),
-kind: PersistentVolumeClaim
-metadata:
-  name: ledger
-spec:
-  storageClassName: keep
-  accessModes: [ReadWriteOnce]
-  resources:
-    requests:
-      storage: 100Mi
-EOF"""),
-                   cmd(r"""k -n store apply -f ledger.yaml
-k -n store describe pvc ledger | tail -4""", r"""The event's reason: `WaitForFirstConsumer`."""),
-                   cmd(r"""echo WaitForFirstConsumer > {WS}/lesson-1.1/why.txt""")]),
         step("Give it a consumer",
              r"""Run Pod `bookkeeper` in `store` (image `busybox:1.36`) that mounts claim `ledger` at `/data` and runs `sh -c 'echo first > /data/entry.txt && sleep 86400'`.
 
@@ -109,22 +81,27 @@ eq "$(j -n store pvc ledger '{.status.phase}')" Bound || exit 1
 pv=$(j -n store pvc ledger '{.spec.volumeName}'); eq "$(j pv "$pv" '{.spec.persistentVolumeReclaimPolicy}')" Retain || exit 1
 eq "$($K -n store exec bookkeeper -- cat /data/entry.txt 2>/dev/null)" first""",
              hint=r"""`kubectl run bookkeeper --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'` gives you the Pod; add `volumes:` (with `persistentVolumeClaim.claimName`) at Pod level and `volumeMounts:` in the container.""",
-             show=[cmd(r"""k -n store run bookkeeper --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo first > /data/entry.txt && sleep 86400' > bk.yaml"""),
+             show=r"""kubectl -n store apply -f - <<'EOF'
-                   hand(r"""vim bk.yaml""",
+apiVersion: v1
-                        r"""Add the mount to the container, below `resources: {}`:
+kind: Pod
-```
+metadata:
+  name: bookkeeper
+spec:
+  containers:
+  - name: bookkeeper
+    image: busybox:1.36
+    command: [sh, -c, 'echo first > /data/entry.txt && sleep 86400']
     volumeMounts:
-    - {name: data, mountPath: /data}
+    - name: data
-```
+      mountPath: /data
-and the volume to the Pod's `spec:`, beside `restartPolicy:`:
-```
   volumes:
-  - {name: data, persistentVolumeClaim: {claimName: ledger}}
+  - name: data
-```""",
+    persistentVolumeClaim:
-                        run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}] | .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "ledger"}}]' bk.yaml"""),
+      claimName: ledger
-                   cmd(r"""k -n store apply -f bk.yaml
+EOF
-k -n store get pvc ledger
+kubectl -n store wait --for=condition=Ready pod/bookkeeper --timeout=120s
-k get pv""", r"""Again once the Pod runs: the claim is `Bound`, and its volume has `RECLAIM POLICY` `Retain`.""")]),
+kubectl -n store get pvc ledger
+kubectl get pv"""),
         step("Delete the claim, keep the volume",
              r"""Delete Pod `bookkeeper` and claim `ledger`. Because of `Retain`, the volume stays, with status `Released`: its data is still on disk, and no new claim can bind it until an admin says so.
 
@@ -134,22 +111,19 @@ Write the PersistentVolume's name to `{WS}/lesson-1.1/released.txt`.""",
 pv=$(tr -d ' \r\n' < "$WS/lesson-1.1/released.txt" 2>/dev/null); [ -n "$pv" ] || exit 1
 eq "$(j pv "$pv" '{.status.phase} {.spec.claimRef.namespace}/{.spec.claimRef.name}')" "Released store/ledger" """,
              hint=r"""Read the volume name off the claim before you delete it (`kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}'`), or find it afterwards in `kubectl get pv`: the CLAIM column still names `store/ledger`.""",
-             show=[cmd(r"""k -n store get pvc ledger""", r"""`VOLUME` is the PersistentVolume's name."""),
+             show=r"""kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}' > {WS}/lesson-1.1/released.txt
-                   hand(r"""echo PV > {WS}/lesson-1.1/released.txt""",
+kubectl -n store delete pod bookkeeper --grace-period=1
-                        r"""`PV`: that name.""",
+kubectl -n store delete pvc ledger
-                        run=r"""kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}' > {WS}/lesson-1.1/released.txt"""),
+kubectl get pv"""),
-                   cmd(r"""k -n store delete po bookkeeper --now
-k -n store delete pvc ledger
-k get pv""", r"""The volume stays, `Released`.""")]),
         step("Change the default class",
              r"""Make `keep` the cluster's default class, and `standard` no longer the default. Two defaults would be ambiguous: set the annotation to `"true"` on one and `"false"` on the other.""",
              check=r"""
 eq "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" true \
   && [ "$(j sc standard '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""",
              hint=r"""`kubectl annotate sc NAME storageclass.kubernetes.io/is-default-class=true --overwrite`, or `kubectl patch` with a `metadata.annotations` map.""",
-             show=[cmd(r"""k annotate sc standard storageclass.kubernetes.io/is-default-class=false --overwrite
+             show=r"""kubectl annotate sc standard storageclass.kubernetes.io/is-default-class=false --overwrite
-k annotate sc keep storageclass.kubernetes.io/is-default-class=true --overwrite
+kubectl annotate sc keep storageclass.kubernetes.io/is-default-class=true --overwrite
-k get sc""")]),
+kubectl get sc"""),
     ],
     "outro": r"""You read a class, made one that keeps its volumes, watched a claim wait for its Pod, kept a volume past its claim, and moved the default.
 
@@ -198,29 +172,29 @@ The reader sees what the writer wrote: an emptyDir is shared by every container
              check=r"""eq "$($K -n vols exec pair -c reader -- cat /shared/msg 2>/dev/null)" hello \
   && [ -n "$(j -n vols pod pair '{.spec.volumes[?(@.name=="shared")].emptyDir}')" ]""",
              hint=r"""Both containers list the same `volumeMounts` entry (`name: shared`); the Pod lists the volume once, as `emptyDir: {}`.""",
-             show=[cmd(r"""k -n vols run pair --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo hello > /shared/msg && sleep 86400' > pair.yaml"""),
+             show=r"""kubectl -n vols apply -f - <<'EOF'
-                   hand(r"""vim pair.yaml""",
+apiVersion: v1
-                        r"""Name the container `writer` and give it the mount; add `reader` below it; add the volume:
+kind: Pod
-```
+metadata:
+  name: pair
+spec:
   containers:
   - name: writer
     image: busybox:1.36
-    args: [sh, -c, 'echo hello > /shared/msg && sleep 86400']
+    command: [sh, -c, 'echo hello > /shared/msg && sleep 86400']
     volumeMounts:
     - {name: shared, mountPath: /shared}
   - name: reader
     image: busybox:1.36
-    args: [sleep, '86400']
+    command: [sleep, '86400']
     volumeMounts:
     - {name: shared, mountPath: /shared}
   volumes:
-  - {name: shared, emptyDir: {}}
+  - name: shared
-```""",
+    emptyDir: {}
-                        run=r"""yq -i '.spec.containers[0].name = "writer" | .spec.containers[0].volumeMounts = [{"name": "shared", "mountPath": "/shared"}]
+EOF
-  | .spec.containers += [{"name": "reader", "image": "busybox:1.36", "args": ["sleep", "86400"], "volumeMounts": [{"name": "shared", "mountPath": "/shared"}]}]
+kubectl -n vols wait --for=condition=Ready pod/pair --timeout=120s
-  | .spec.volumes = [{"name": "shared", "emptyDir": {}}]' pair.yaml"""),
+kubectl -n vols exec pair -c reader -- cat /shared/msg"""),
-                   cmd(r"""k -n vols apply -f pair.yaml
-k -n vols exec pair -c reader -- cat /shared/msg""", r"""Once it runs: `hello`.""")]),
         step("Scratch space in memory",
              r"""Create Pod `scratch` in `vols` (`busybox:1.36`, `sleep 86400`) with an emptyDir at `/scratch` that is held in **memory**, at most `64Mi`.
 
@@ -229,21 +203,24 @@ Inside the Pod, `/proc/mounts` shows the difference: `/scratch` is a `tmpfs`. Wh
 eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.medium}')" Memory && eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.sizeLimit}')" 64Mi \
   && $K -n vols exec scratch -- grep -q ' /scratch tmpfs ' /proc/mounts 2>/dev/null""",
              hint=r"""`emptyDir: {medium: Memory, sizeLimit: 64Mi}`.""",
-             show=[cmd(r"""k -n vols run scratch --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > scratch.yaml"""),
+             show=r"""kubectl -n vols apply -f - <<'EOF'
-                   hand(r"""vim scratch.yaml""",
+apiVersion: v1
-                        r"""Add the mount to the container, below `resources: {}`:
+kind: Pod
-```
+metadata:
+  name: scratch
+spec:
+  containers:
+  - name: scratch
+    image: busybox:1.36
+    command: [sleep, '86400']
     volumeMounts:
     - {name: tmp, mountPath: /scratch}
-```
-and the volume to the Pod's `spec:`, beside `restartPolicy:`:
-```
   volumes:
-  - {name: tmp, emptyDir: {medium: Memory, sizeLimit: 64Mi}}
+  - name: tmp
-```""",
+    emptyDir: {medium: Memory, sizeLimit: 64Mi}
-                        run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "tmp", "mountPath": "/scratch"}] | .spec.volumes = [{"name": "tmp", "emptyDir": {"medium": "Memory", "sizeLimit": "64Mi"}}]' scratch.yaml"""),
+EOF
-                   cmd(r"""k -n vols apply -f scratch.yaml
+kubectl -n vols wait --for=condition=Ready pod/scratch --timeout=120s
-k -n vols exec scratch -- grep /scratch /proc/mounts""", r"""Once it runs: a `tmpfs`.""")]),
+kubectl -n vols exec scratch -- grep /scratch /proc/mounts"""),
         step("One directory from three sources",
              r"""Create Pod `bundle` in `vols` (`busybox:1.36`, `sleep 86400`) with **one** `projected` volume mounted at `/etc/bundle`, made of:
 
@@ -256,15 +233,18 @@ Separate volumes would need three mount points; a projected volume puts them sid
 x() { $K -n vols exec bundle -- cat "/etc/bundle/$1" 2>/dev/null; }
 [ -n "$(j -n vols pod bundle '{.spec.volumes[0].projected}')" ] && eq "$(x mode)" prod && eq "$(x password)" s3cr3t && eq "$(x podname)" bundle""",
              hint=r"""`projected: {sources: [...]}` - each source is `configMap: {name: ...}`, `secret: {name: ...}` or `downwardAPI: {items: [{path: podname, fieldRef: {fieldPath: metadata.name}}]}`. `kubectl explain pod.spec.volumes.projected.sources` lists them.""",
-             show=[cmd(r"""k -n vols run bundle --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > bundle.yaml"""),
+             show=r"""kubectl -n vols apply -f - <<'EOF'
-                   hand(r"""vim bundle.yaml""",
+apiVersion: v1
-                        r"""Add the mount to the container, below `resources: {}`:
+kind: Pod
-```
+metadata:
+  name: bundle
+spec:
+  containers:
+  - name: app
+    image: busybox:1.36
+    command: [sleep, '86400']
     volumeMounts:
     - {name: bundle, mountPath: /etc/bundle, readOnly: true}
-```
-and the volume to the Pod's `spec:`, beside `restartPolicy:` (the docs page *Projected Volumes* has an example):
-```
   volumes:
   - name: bundle
     projected:
@@ -275,11 +255,9 @@ and the volume to the Pod's `spec:`, beside `restartPolicy:` (the docs page *Pro
           items:
           - path: podname
             fieldRef: {fieldPath: metadata.name}
-```""",
+EOF
-                        run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "bundle", "mountPath": "/etc/bundle", "readOnly": true}]
+kubectl -n vols wait --for=condition=Ready pod/bundle --timeout=120s
-  | .spec.volumes = [{"name": "bundle", "projected": {"sources": [{"configMap": {"name": "app-config"}}, {"secret": {"name": "app-secret"}}, {"downwardAPI": {"items": [{"path": "podname", "fieldRef": {"fieldPath": "metadata.name"}}]}}]}}]' bundle.yaml"""),
+kubectl -n vols exec bundle -- ls /etc/bundle"""),
-                   cmd(r"""k -n vols apply -f bundle.yaml
-k -n vols exec bundle -- ls /etc/bundle""", r"""Once it runs: `mode`, `password` and `podname`.""")]),
         step("A volume for exactly one Pod",
              r"""Create PersistentVolume `archive`: `hostPath` `/srv/archive` (type `DirectoryOrCreate`), `200Mi`, access mode **`ReadWriteOncePod`**, reclaim policy `Retain`, and `storageClassName: ""`.
 
@@ -290,9 +268,7 @@ The empty class matters on both sides. A claim with no class at all gets the def
 eq "$(j pv archive '{.spec.accessModes[0]} {.spec.persistentVolumeReclaimPolicy} {.spec.hostPath.path}')" "ReadWriteOncePod Retain /srv/archive" \
   && eq "$(j -n vols pvc archive-claim '{.status.phase} {.spec.volumeName}')" "Bound archive" """,
              hint=r"""`storageClassName: ""` is spelled out, with the quotes, under `spec` of both objects. A PersistentVolume is cluster-wide: no namespace.""",
-             show=[hand(r"""vim archive.yaml""",
+             show=r"""kubectl apply -f - <<'EOF'
-                        r"""No `kubectl create` for these: paste the PersistentVolume and the claim from the docs page *Configure a Pod to Use a PersistentVolume for Storage*, and make them:
-```
 apiVersion: v1
 kind: PersistentVolume
 metadata:
@@ -314,33 +290,9 @@ spec:
   accessModes: [ReadWriteOncePod]
   resources:
     requests: {storage: 200Mi}
-```""",
+EOF
-                        run=r"""cat > archive.yaml <<'EOF'
+kubectl get pv archive
-apiVersion: v1
+kubectl -n vols get pvc archive-claim"""),
-kind: PersistentVolume
-metadata:
-  name: archive
-spec:
-  capacity: {storage: 200Mi}
-  accessModes: [ReadWriteOncePod]
-  persistentVolumeReclaimPolicy: Retain
-  storageClassName: ""
-  hostPath: {path: /srv/archive, type: DirectoryOrCreate}
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: archive-claim
-  namespace: vols
-spec:
-  storageClassName: ""
-  accessModes: [ReadWriteOncePod]
-  resources:
-    requests: {storage: 200Mi}
-EOF"""),
-                   cmd(r"""k apply -f archive.yaml
-k get pv archive
-k -n vols get pvc archive-claim""", r"""Both `Bound`, to each other.""")]),
         step("Hand a Released volume to a new claim",
              r"""Delete claim `archive-claim`. The volume becomes `Released`: it still remembers its old claim in `spec.claimRef`, and nothing binds it while it does.
 
@@ -348,14 +300,10 @@ Make it `Available` again by removing `spec.claimRef` from the PV, then create c
              check=r"""
 [ -z "$(j -n vols pvc archive-claim '{.metadata.name}')" ] && eq "$(j -n vols pvc archive-claim-2 '{.status.phase} {.spec.volumeName}')" "Bound archive" """,
              hint=r"""`kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'` - or `kubectl edit pv archive` and delete the `claimRef:` block.""",
-             show=[cmd(r"""k -n vols delete pvc archive-claim
+             show=r"""kubectl -n vols delete pvc archive-claim
-k get pv archive""", r"""`Released`."""),
+kubectl get pv archive
-                   hand(r"""k edit pv archive""",
+kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'
-                        r"""Delete the whole `claimRef:` block, then `:wq`. The volume is `Available` again.""",
+kubectl apply -f - <<'EOF'
-                        run=r"""kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' """),
-                   hand(r"""vim claim2.yaml""",
-                        r"""The claim of `archive.yaml`, named `archive-claim-2`:
-```
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -366,21 +314,8 @@ spec:
   accessModes: [ReadWriteOncePod]
   resources:
     requests: {storage: 200Mi}
-```""",
+EOF
-                        run=r"""cat > claim2.yaml <<'EOF'
+kubectl get pv archive"""),
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: archive-claim-2
-  namespace: vols
-spec:
-  storageClassName: ""
-  accessModes: [ReadWriteOncePod]
-  resources:
-    requests: {storage: 200Mi}
-EOF"""),
-                   cmd(r"""k apply -f claim2.yaml
-k get pv archive""")]),
     ],
     "outro": r"""You shared an emptyDir between containers, put one in memory, combined three sources in one projected volume, and took a `Retain` volume through `Bound`, `Released` and back.
 
@@ -466,9 +401,7 @@ $K -n claims wait --for=condition=Ready pod/grower --timeout=180s >/dev/null
 Without the selector it would bind `pv-small`, the smallest volume that fits. With it, it binds `pv-large`.""",
              check=r"""eq "$(j -n claims pvc gold-claim '{.status.phase} {.spec.volumeName}')" "Bound pv-large" """,
              hint=r"""`spec.selector.matchLabels: {tier: gold}` on the claim.""",
-             show=[hand(r"""vim gold.yaml""",
+             show=r"""kubectl -n claims apply -f - <<'EOF'
-                        r"""A claim with a `selector` (the docs page *Persistent Volumes* has one):
-```
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -480,30 +413,14 @@ spec:
     requests: {storage: 100Mi}
   selector:
     matchLabels: {tier: gold}
-```""",
+EOF
-                        run=r"""cat > gold.yaml <<'EOF'
+kubectl get pv"""),
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: gold-claim
-spec:
-  storageClassName: manual
-  accessModes: [ReadWriteOnce]
-  resources:
-    requests: {storage: 100Mi}
-  selector:
-    matchLabels: {tier: gold}
-EOF"""),
-                   cmd(r"""k -n claims apply -f gold.yaml
-k get pv""", r"""`gold-claim` bound to `pv-large`.""")]),
         step("Bind by name, and use it",
              r"""Create claim `pinned` in `claims` that binds `pv-small` **by name** (class `manual`, `ReadWriteOnce`, `100Mi`), and Pod `user` (`busybox:1.36`, `sleep 86400`) that mounts it at `/data`.""",
              check=r"""eq "$(j -n claims pvc pinned '{.status.phase} {.spec.volumeName}')" "Bound pv-small" && eq "$(j -n claims pod user '{.status.phase}')" Running \
   && eq "$(j -n claims pod user '{.spec.volumes[0].persistentVolumeClaim.claimName}')" pinned""",
              hint=r"""`spec.volumeName: pv-small` on the claim. The volume's class and modes must still match the claim's.""",
-             show=[hand(r"""vim pinned.yaml""",
+             show=r"""kubectl -n claims apply -f - <<'EOF'
-                        r"""A claim that names its volume:
-```
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -514,35 +431,24 @@ spec:
   accessModes: [ReadWriteOnce]
   resources:
     requests: {storage: 100Mi}
-```""",
+---
-                        run=r"""cat > pinned.yaml <<'EOF'
 apiVersion: v1
-kind: PersistentVolumeClaim
+kind: Pod
 metadata:
-  name: pinned
+  name: user
 spec:
-  storageClassName: manual
+  containers:
-  volumeName: pv-small
+  - name: user
-  accessModes: [ReadWriteOnce]
+    image: busybox:1.36
-  resources:
+    command: [sleep, '86400']
-    requests: {storage: 100Mi}
-EOF"""),
-                   cmd(r"""k -n claims apply -f pinned.yaml
-k -n claims run user --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > user.yaml"""),
-                   hand(r"""vim user.yaml""",
-                        r"""Add the mount to the container, below `resources: {}`:
-```
     volumeMounts:
     - {name: data, mountPath: /data}
-```
-and the volume to the Pod's `spec:`, beside `restartPolicy:`:
-```
   volumes:
-  - {name: data, persistentVolumeClaim: {claimName: pinned}}
+  - name: data
-```""",
+    persistentVolumeClaim: {claimName: pinned}
-                        run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}] | .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "pinned"}}]' user.yaml"""),
+EOF
-                   cmd(r"""k -n claims apply -f user.yaml
+kubectl -n claims wait --for=condition=Ready pod/user --timeout=120s
-k -n claims get pvc,po""")]),
+kubectl -n claims get pvc"""),
         step("A claim in use does not go",
              r"""Delete claim `pinned` without waiting (`--wait=false`). It does not disappear: `kubectl -n claims get pvc` shows it `Terminating`, held by a **finalizer**.
 
@@ -551,19 +457,17 @@ Write that finalizer's name to `{WS}/lesson-1.3/finalizer.txt`.""",
 [ -n "$(j -n claims pvc pinned '{.metadata.deletionTimestamp}')" ] \
   && eq "$(tr -d ' \r\n' < "$WS/lesson-1.3/finalizer.txt" 2>/dev/null)" kubernetes.io/pvc-protection""",
              hint=r"""`kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers}'`. Without `--wait=false`, `kubectl delete` would sit there until the claim is really gone.""",
-             show=[cmd(r"""k -n claims delete pvc pinned --wait=false
+             show=r"""kubectl -n claims delete pvc pinned --wait=false
-k -n claims get pvc pinned
+kubectl -n claims get pvc pinned
-k -n claims get pvc pinned -o yaml | grep -A1 finalizers"""),
+kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers[0]}' > {WS}/lesson-1.3/finalizer.txt
-                   hand(r"""echo FINALIZER > {WS}/lesson-1.3/finalizer.txt""",
+cat {WS}/lesson-1.3/finalizer.txt"""),
-                        r"""`FINALIZER`: the name under `finalizers:`.""",
-                        run=r"""kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers[0]}' > {WS}/lesson-1.3/finalizer.txt""")]),
         step("Let it go",
              r"""Delete Pod `user`. With no Pod using it any more, the protection lifts and claim `pinned` is removed. `pv-small` (reclaim policy `Retain`) is now `Released`.""",
              check=r"""[ -z "$(j -n claims pod user '{.metadata.name}')" ] && [ -z "$(j -n claims pvc pinned '{.metadata.name}')" ] && eq "$(j pv pv-small '{.status.phase}')" Released""",
              hint=r"""`kubectl -n claims delete pod user`, then `kubectl -n claims get pvc` and `kubectl get pv`.""",
-             show=[cmd(r"""k -n claims delete po user --now
+             show=r"""kubectl -n claims delete pod user --grace-period=1
-k -n claims get pvc
+kubectl -n claims get pvc
-k get pv pv-small""", r"""The claim is gone; `pv-small` is `Released`.""")]),
+kubectl get pv pv-small"""),
         step("Claims made by a StatefulSet",
              r"""Create a StatefulSet `db` in `claims` with 2 replicas (`busybox:1.36`, `sleep 86400`) and its headless Service `db`. Give it a `volumeClaimTemplates` entry `data` (`ReadWriteOnce`, `50Mi`, class `standard`) mounted at `/var/lib/db`, and a `persistentVolumeClaimRetentionPolicy` that **deletes** a replica's claim when the StatefulSet is scaled down and **retains** the claims when the StatefulSet is deleted.
 
@@ -572,9 +476,7 @@ Each replica gets a claim of its own: `data-db-0`, `data-db-1`.""",
 eq "$(j -n claims sts db '{.spec.persistentVolumeClaimRetentionPolicy.whenScaled} {.spec.persistentVolumeClaimRetentionPolicy.whenDeleted}')" "Delete Retain" || exit 1
 eq "$(j -n claims sts db '{.status.readyReplicas}')" 2 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound && eq "$(j -n claims pvc data-db-1 '{.status.phase}')" Bound""",
              hint=r"""`kubectl create deploy db --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400` is a start: change the kind, add `serviceName: db`, drop `strategy`, add `volumeClaimTemplates` beside `template`. The Service: `clusterIP: None`. `kubectl explain sts.spec.persistentVolumeClaimRetentionPolicy`.""",
-             show=[hand(r"""vim db.yaml""",
+             show=r"""kubectl -n claims apply -f - <<'EOF'
-                        r"""No `kubectl create` for a StatefulSet: paste the example from the docs page *StatefulSets* (it comes with its headless Service), and make it:
-```
 apiVersion: v1
 kind: Service
 metadata:
@@ -615,52 +517,9 @@ spec:
       accessModes: [ReadWriteOnce]
       resources:
         requests: {storage: 50Mi}
-```""",
+EOF
-                        run=r"""cat > db.yaml <<'EOF'
+kubectl -n claims rollout status sts db --timeout=180s
-apiVersion: v1
+kubectl -n claims get pvc"""),
-kind: Service
-metadata:
-  name: db
-spec:
-  clusterIP: None
-  selector: {app: db}
-  ports:
-  - port: 5432
----
-apiVersion: apps/v1
-kind: StatefulSet
-metadata:
-  name: db
-spec:
-  serviceName: db
-  replicas: 2
-  selector:
-    matchLabels: {app: db}
-  persistentVolumeClaimRetentionPolicy:
-    whenScaled: Delete
-    whenDeleted: Retain
-  template:
-    metadata:
-      labels: {app: db}
-    spec:
-      containers:
-      - name: db
-        image: busybox:1.36
-        command: [sleep, '86400']
-        volumeMounts:
-        - {name: data, mountPath: /var/lib/db}
-  volumeClaimTemplates:
-  - metadata:
-      name: data
-    spec:
-      storageClassName: standard
-      accessModes: [ReadWriteOnce]
-      resources:
-        requests: {storage: 50Mi}
-EOF"""),
-                   cmd(r"""k -n claims apply -f db.yaml
-k -n claims rollout status sts db
-k -n claims get pvc""", r"""`data-db-0` and `data-db-1`.""")]),
         step("Scale down, and the claim goes with the replica",
              r"""Scale `db` to 1 replica. Pod `db-1` goes, and because `whenScaled` is `Delete`, so does `data-db-1`. `data-db-0` stays.
 
@@ -669,26 +528,23 @@ The default for both fields is `Retain`: without the policy, `data-db-1` would w
 eq "$(j -n claims sts db '{.spec.replicas}')" 1 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound || exit 1
 for i in $(seq 1 10); do [ -z "$(j -n claims pvc data-db-1 '{.metadata.name}')" ] && exit 0; sleep 3; done; exit 1""",
              hint=r"""`kubectl -n claims scale sts db --replicas=1`, then `kubectl -n claims get pvc -w`.""",
-             show=[cmd(r"""k -n claims scale sts db --replicas=1
+             show=r"""kubectl -n claims scale sts db --replicas=1
-k -n claims rollout status sts db
+kubectl -n claims rollout status sts db --timeout=120s
-k -n claims get pvc""", r"""Again in a moment: `data-db-1` is gone, `data-db-0` stays.""")]),
+kubectl -n claims get pvc"""),
         step("Grow a claim in use",
              r"""Claim `growing` is full. Its class allows expansion (`kubectl get sc csi-hostpath-sc`: `ALLOWVOLUMEEXPANSION true`), so make it **50Mi** - while Pod `grower` keeps running. Raise the request; the driver does the rest, and `status.capacity` follows.""",
              check=r"""eq "$(j -n claims pvc growing '{.spec.resources.requests.storage} {.status.capacity.storage}')" "50Mi 50Mi" && eq "$(j -n claims pod grower '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """,
              hint=r"""`kubectl -n claims edit pvc growing`, or `kubectl patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}'`. A claim never shrinks; with `standard` (local-path) the same patch is refused.""",
-             show=[cmd(r"""k get sc csi-hostpath-sc"""),
+             show=r"""kubectl get sc csi-hostpath-sc
-                   hand(r"""k -n claims edit pvc growing""",
+kubectl -n claims patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}'
-                        r"""Make `storage:` under `spec.resources.requests` read `50Mi`, then `:wq`.""",
+sleep 10
-                        run=r"""kubectl -n claims patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}' """),
+kubectl -n claims get pvc growing"""),
-                   cmd(r"""k -n claims get pvc growing""", r"""Again until `CAPACITY` shows `50Mi`.""")]),
         step("Snapshot it, and restore the snapshot",
              r"""Take VolumeSnapshot `growing-snap` of claim `growing` (snapshot class `csi-hostpath-snapclass`), wait until it is ready to use, then create claim `restored` (`csi-hostpath-sc`, 50Mi) **from** the snapshot: its `dataSource` names the VolumeSnapshot. The new volume holds what `growing` held when the snapshot was taken.""",
              check=r"""
 eq "$(j -n claims volumesnapshot growing-snap '{.spec.source.persistentVolumeClaimName} {.status.readyToUse}')" "growing true"   && eq "$(j -n claims pvc restored '{.spec.dataSource.kind}/{.spec.dataSource.name} {.status.phase}')" "VolumeSnapshot/growing-snap Bound" """,
              hint=r"""`apiVersion: snapshot.storage.k8s.io/v1`, `kind: VolumeSnapshot`, `spec: {volumeSnapshotClassName: ..., source: {persistentVolumeClaimName: growing}}`. The claim: `dataSource: {apiGroup: snapshot.storage.k8s.io, kind: VolumeSnapshot, name: growing-snap}`. A clone is the same with `dataSource: {kind: PersistentVolumeClaim, name: growing}`.""",
-             show=[hand(r"""vim snap.yaml""",
+             show=r"""kubectl -n claims apply -f - <<'EOF'
-                        r"""Paste a VolumeSnapshot from the docs page *Volume Snapshots*, and make it:
-```
 apiVersion: snapshot.storage.k8s.io/v1
 kind: VolumeSnapshot
 metadata:
@@ -697,24 +553,9 @@ spec:
   volumeSnapshotClassName: csi-hostpath-snapclass
   source:
     persistentVolumeClaimName: growing
-```""",
+EOF
-                        run=r"""cat > snap.yaml <<'EOF'
+kubectl -n claims wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/growing-snap --timeout=120s
-apiVersion: snapshot.storage.k8s.io/v1
+kubectl -n claims apply -f - <<'EOF'
-kind: VolumeSnapshot
-metadata:
-  name: growing-snap
-spec:
-  volumeSnapshotClassName: csi-hostpath-snapclass
-  source:
-    persistentVolumeClaimName: growing
-EOF"""),
-                   cmd(r"""k -n claims apply -f snap.yaml"""),
-                   hand(r"""k -n claims get volumesnapshot""",
-                        r"""Again until `READYTOUSE` is `true`.""",
-                        run=r"""kubectl -n claims wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/growing-snap --timeout=120s"""),
-                   hand(r"""vim restored.yaml""",
-                        r"""A claim whose `dataSource` is the snapshot (the same docs page, *Provisioning Volumes from Snapshots*):
-```
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -728,24 +569,8 @@ spec:
     apiGroup: snapshot.storage.k8s.io
     kind: VolumeSnapshot
     name: growing-snap
-```""",
+EOF
-                        run=r"""cat > restored.yaml <<'EOF'
+kubectl -n claims get volumesnapshot,pvc"""),
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: restored
-spec:
-  storageClassName: csi-hostpath-sc
-  accessModes: [ReadWriteOnce]
-  resources:
-    requests: {storage: 50Mi}
-  dataSource:
-    apiGroup: snapshot.storage.k8s.io
-    kind: VolumeSnapshot
-    name: growing-snap
-EOF"""),
-                   cmd(r"""k -n claims apply -f restored.yaml
-k -n claims get volumesnapshot,pvc""")]),
     ],
     "outro": r"""You steered binding with a selector and with a volume's name, watched claim protection hold a claim until its Pod went, let a StatefulSet's claims follow its replicas, and grew, snapshotted and restored a claim on a CSI class.
 

cka/lessons_tshoot.py

@@ -1,5 +1,5 @@
 """CKA lessons of the Troubleshooting domain (2.1-2.5). See cka/lessons.py for what a lesson is."""
-from sim.bank import NS, step, cmd, hand
+from sim.bank import NS, step
 
 # ---- 2.1 clusters and nodes -------------------------------------------------------------------
 L21 = {
@@ -35,11 +35,9 @@ echo "$W1 is still Ready a minute after its kubelet was stopped" >&2; exit 1
 Then look at `kubectl describe node` for it: under **Conditions** every row says `Kubelet stopped posting node status` - the control plane has not heard from the kubelet, so it knows nothing more. The answer is on the node.""",
              check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/node.txt" 2>/dev/null)" "$W1" """,
              hint=r"""`kubectl get nodes`. `--no-headers` and `awk '$2 != "Ready" {print $1}'` give just the name.""",
-             show=[cmd(r"""k get no
+             show=r"""kubectl get nodes
-k describe no {W1} | grep -A8 Conditions"""),
+kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt
-                   hand(r"""echo NODE > {WS}/lesson-2.1/node.txt""",
+kubectl describe node {W1} | grep -A8 Conditions"""),
-                        r"""`NODE`: the one that is not `Ready`.""",
-                        run=r"""kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt""")]),
         step("Bring the kubelet back - for good",
              r"""On `{W1}` (`ssh {W1}`, then `sudo -i`), look at the kubelet: `systemctl status kubelet` says `inactive (dead)` and `disabled`. Start it, and enable it so that it survives a reboot - a fix that is gone after the next reboot is not a fix.
 
@@ -48,15 +46,12 @@ Within a few seconds the node is `Ready` again.""",
 nx "$W1" 'systemctl is-active -q kubelet && systemctl is-enabled -q kubelet' </dev/null || exit 1
 eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
              hint=r"""`systemctl enable --now kubelet` does both at once. `journalctl -u kubelet -n 20` tells you when it is running happily.""",
-             show=[hand(r"""ssh {W1}
+             show=r"""ssh {W1} sudo bash <<'EOF'
-sudo -i
+systemctl status kubelet --no-pager | head -4
-systemctl status kubelet
-systemctl enable --now kubelet""",
-                        r"""Then `exit` twice.""",
-                        run=r"""ssh {W1} sudo bash <<'EOF'
 systemctl enable --now kubelet
-EOF"""),
+systemctl is-active kubelet; systemctl is-enabled kubelet
-                   cmd(r"""k get no""", r"""Again in a few seconds: `Ready`.""")]),
+EOF
+kubectl get nodes"""),
         step("Where the kubelet's configuration is",
              r"""A kubelet reads its settings from the file named by its `--config` flag, and the kubeadm flags from `/var/lib/kubelet/kubeadm-flags.env`. On `{W1}`, find the file (`ps -o args= -C kubelet`, or the drop-in under `/etc/systemd/system/kubelet.service.d/`).
 
@@ -65,16 +60,9 @@ The kubelet also tells the API server what it actually runs with: `kubectl get -
 want=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["cgroupDriver"])')
 [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/cgroup.txt" 2>/dev/null)" "$want" """,
              hint=r"""`configz` returns JSON: `| jq -r .kubeletconfig.cgroupDriver`. It must match containerd's `SystemdCgroup` setting, or Pods fail to start.""",
-             show=[hand(r"""ssh {W1}
+             show=r"""ssh -n {W1} 'ps -o args= -C kubelet | tr " " "\n" | grep -- --config; sudo grep -n cgroupDriver /var/lib/kubelet/config.yaml'
-sudo -i
+kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt
-ps -o args= -C kubelet
+cat {WS}/lesson-2.1/cgroup.txt"""),
-grep cgroupDriver /var/lib/kubelet/config.yaml""",
-                        r"""`--config=` names the kubelet's file. Then `exit` twice.""",
-                        run=r"""ssh -n {W1} 'ps -o args= -C kubelet; sudo grep cgroupDriver /var/lib/kubelet/config.yaml' """),
-                   cmd(r"""k get --raw /api/v1/nodes/{W1}/proxy/configz | tr , '\n' | grep cgroupDriver""", r"""The live value."""),
-                   hand(r"""echo VALUE > {WS}/lesson-2.1/cgroup.txt""",
-                        r"""`VALUE`: `systemd` or `cgroupfs`.""",
-                        run=r"""kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt""")]),
         step("Set the eviction thresholds",
              r"""Look at `evictionHard` in `/var/lib/kubelet/config.yaml` on `{W1}`: kind sets every threshold to `0%`, which means "never evict" - fine for a test cluster, not for a real one.
 
@@ -86,26 +74,15 @@ e = json.load(sys.stdin)["kubeletconfig"].get("evictionHard") or {}
 sys.exit(0 if e.get("memory.available") == "100Mi" and e.get("nodefs.available") == "10%" else 1)' || exit 1
 eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
              hint=r"""Edit the file with `vim`, or `sed`: change the `nodefs.available` line and add a `memory.available` line inside the `evictionHard:` block (two spaces of indent). A kubelet that does not come back after the restart: `journalctl -u kubelet -n 20` shows the line it could not parse.""",
-             show=[hand(r"""ssh {W1}
+             show=r"""ssh {W1} sudo bash <<'EOF'
-sudo -i
-vim /var/lib/kubelet/config.yaml""",
-                        r"""Under `evictionHard:`, set `nodefs.available` to `10%` and add `memory.available`, indented like the others:
-```
-evictionHard:
-  memory.available: 100Mi
-  nodefs.available: 10%
-```
-then restart the kubelet, and `exit` twice:
-```
-systemctl restart kubelet
-```""",
-                        run=r"""ssh {W1} sudo bash <<'EOF'
 f=/var/lib/kubelet/config.yaml
+grep -n -A4 '^evictionHard:' $f
 sed -i 's/^  nodefs.available: .*/  nodefs.available: 10%/' $f
 sed -i '/^evictionHard:/a\  memory.available: 100Mi' $f
+grep -n -A5 '^evictionHard:' $f
 systemctl restart kubelet
-EOF"""),
+EOF
-                   cmd(r"""k get --raw /api/v1/nodes/{W1}/proxy/configz | tr , '\n' | grep -A4 evictionHard""", r"""Again if the kubelet is not back yet.""")]),
+kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq .kubeletconfig.evictionHard"""),
         step("Read the pressure conditions",
              r"""The kubelet reports the result of those thresholds as node conditions. Write the status of `{W1}`'s three pressure conditions to `{WS}/lesson-2.1/pressure.txt`, one per line, as `TYPE STATUS` - for example `MemoryPressure False`.
 
@@ -114,15 +91,8 @@ EOF"""),
 want=$(for c in MemoryPressure DiskPressure PIDPressure; do echo "$c $(j node "$W1" "{.status.conditions[?(@.type==\"$c\")].status}")"; done | sort)
 eq "$(tr -s ' \t' ' ' < "$WS/lesson-2.1/pressure.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """,
              hint=r"""`kubectl describe node {W1}` lists them under Conditions. As a list: `-o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}'`, then keep the three `*Pressure` lines.""",
-             show=[cmd(r"""k describe no {W1} | grep Pressure"""),
+             show=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt
-                   hand(r"""vim {WS}/lesson-2.1/pressure.txt""",
+cat {WS}/lesson-2.1/pressure.txt"""),
-                        r"""Three lines, `TYPE STATUS`, with the statuses you read:
-```
-MemoryPressure False
-DiskPressure False
-PIDPressure False
-```""",
-                        run=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt""")]),
     ],
     "outro": r"""You found a `NotReady` node, brought its kubelet back for good, read its configuration from the file and from the running kubelet, set eviction thresholds and read what the kubelet reports about them.
 
@@ -174,12 +144,10 @@ Two things in Namespace `ops` do not work: Deployment `queue` has no Pods, and P
 `kubectl -n kube-system get pods` shows them; on the control-plane node, `sudo crictl ps -a` shows their containers, and `ls /etc/kubernetes/manifests` what the kubelet is told to run.""",
              check=r"""eq "$(tr -d ' \r' < "$WS/lesson-2.2/down.txt" 2>/dev/null | grep . | sort | tr '\n' ' ')" "kube-controller-manager kube-scheduler " """,
              hint=r"""A Pod that is not `Running` with all its containers ready. The Pod names end in the node's name; the component is the part before it.""",
-             show=[cmd(r"""k -n kube-system get po
+             show=r"""kubectl -n kube-system get pods
 ls /etc/kubernetes/manifests
-sudo crictl ps -a | grep kube-"""),
+sudo crictl ps -a | grep kube-
-                   hand(r"""vim {WS}/lesson-2.2/down.txt""",
+printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt"""),
-                        r"""The components that do not run, one per line.""",
-                        run=r"""printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt""")]),
         step("Repair the scheduler",
              r"""Find out why `kube-scheduler` does not start, and fix its manifest. `kubectl -n kube-system describe pod kube-scheduler-{CP}` shows the error in its events, and so do `sudo crictl ps -a --name kube-scheduler` and `sudo journalctl -u kubelet`.
 
@@ -188,33 +156,30 @@ The kubelet notices the saved file on its own: no restart, no `kubectl apply`. O
 nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml && [ -n "$(crictl ps --name kube-scheduler -q)" ]' </dev/null || exit 1
 [ -n "$(j -n ops pod waiting '{.spec.nodeName}')" ]""",
              hint=r"""Look at the first entry of the container's `command:` - the program the container runs. A static Pod's manifest is edited in place (`sudo vim`), never with `kubectl edit`.""",
-             show=[cmd(r"""k -n kube-system describe po kube-scheduler-{CP} | tail -5""", r"""The command it cannot find: a typo."""),
+             show=r"""kubectl -n kube-system describe pod kube-scheduler-{CP} | tail -5
-                   hand(r"""sudo vim /etc/kubernetes/manifests/kube-scheduler.yaml""",
+sudo grep -n -A2 'command:' /etc/kubernetes/manifests/kube-scheduler.yaml
-                        r"""Under `command:`, make `- kube-schedulerr` read `- kube-scheduler`, then `:wq`.""",
+sudo sed -i 's/- kube-schedulerr$/- kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
-                        run=r"""sudo sed -i 's/- kube-schedulerr$/- kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml"""),
+sudo crictl ps --name kube-scheduler
-                   cmd(r"""sudo crictl ps --name kube-scheduler
+kubectl -n ops get pod waiting -o wide"""),
-k -n ops get po waiting -o wide""", r"""Again in a few seconds: the scheduler runs, and `waiting` has a node.""")]),
         step("Repair the controller manager",
              r"""Now `kube-controller-manager`. Its Pod is not crashing - its container never starts. `describe` on it tells you which step fails. Fix the manifest; once it runs, Deployment `queue` gets its ReplicaSet and both Pods.""",
              check=r"""
 nx "$CP" '! grep -q -- "-broken" /etc/kubernetes/manifests/kube-controller-manager.yaml && [ -n "$(crictl ps --name kube-controller-manager -q)" ]' </dev/null || exit 1
 eq "$(j -n ops deploy queue '{.status.availableReplicas}')" 2""",
              hint=r"""`ErrImagePull` / `ImagePullBackOff`: the image tag does not exist. Every control-plane image runs the cluster's version - compare with `kube-apiserver.yaml`.""",
-             show=[cmd(r"""k -n kube-system describe po kube-controller-manager-{CP} | tail -5
+             show=r"""kubectl -n kube-system describe pod kube-controller-manager-{CP} | tail -5
-sudo grep image: /etc/kubernetes/manifests/*.yaml""", r"""Its image tag ends in `-broken`; the other components show the right tag."""),
+sudo grep -n 'image:' /etc/kubernetes/manifests/*.yaml
-                   hand(r"""sudo vim /etc/kubernetes/manifests/kube-controller-manager.yaml""",
+sudo sed -i 's/-broken$//' /etc/kubernetes/manifests/kube-controller-manager.yaml
-                        r"""Remove `-broken` from the end of the `image:` line, then `:wq`.""",
+sudo crictl ps --name kube-controller-manager
-                        run=r"""sudo sed -i 's/-broken$//' /etc/kubernetes/manifests/kube-controller-manager.yaml"""),
+kubectl -n ops get deploy queue"""),
-                   cmd(r"""sudo crictl ps --name kube-controller-manager
-k -n ops get deploy queue""", r"""Again until `queue` shows `2/2`.""")]),
         step("Ask the API server how it feels",
              r"""The API server has health endpoints of its own: `/livez` (restart me if this fails) and `/readyz` (send me traffic). With `?verbose` they list every check.
 
 Save the output of `kubectl get --raw '/readyz?verbose'` in `{WS}/lesson-2.2/readyz.txt`.""",
              check=r"""grep -q '^\[+\]etcd ok' "$WS/lesson-2.2/readyz.txt" 2>/dev/null && grep -q 'readyz check passed' "$WS/lesson-2.2/readyz.txt" """,
              hint=r"""`kubectl get --raw` sends a GET to any API path, with your credentials. Quote the path: `?` means something to the shell.""",
-             show=[cmd(r"""k get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt
+             show=r"""kubectl get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt | tail -3
-k get --raw /livez""")]),
+kubectl get --raw '/livez'; echo"""),
     ],
     "outro": r"""You read the control plane from both sides - `kubectl` and the node's container runtime - and repaired two static Pods by editing their manifests in place.
 
@@ -296,46 +261,38 @@ echo "kubectl top shows no metrics for the lesson's Pods" >&2; exit 1
              r"""Write the name of the Pod in `metrics` that uses the most memory to `{WS}/lesson-2.3/memory.txt`. `kubectl top` can sort for you.""",
              check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/memory.txt" 2>/dev/null)" hoarder""",
              hint=r"""`kubectl top pod -n metrics --sort-by=memory`. `--no-headers | head -1 | awk '{print $1}'` keeps just the name.""",
-             show=[cmd(r"""k top po -n metrics --sort-by=memory"""),
+             show=r"""kubectl top pod -n metrics --sort-by=memory
-                   hand(r"""echo POD > {WS}/lesson-2.3/memory.txt""",
+kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt"""),
-                        r"""`POD`: the first one.""",
-                        run=r"""kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt""")]),
         step("The busiest Pod in the cluster",
              r"""Across **all** Namespaces, which Pod uses the most CPU? Write it as `NAMESPACE/NAME` to `{WS}/lesson-2.3/cpu.txt`.""",
              check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/cpu.txt" 2>/dev/null)" metrics/burner""",
              hint=r"""`kubectl top pod -A --sort-by=cpu`; with `-A` the first column is the Namespace.""",
-             show=[cmd(r"""k top po -A --sort-by=cpu | head -5"""),
+             show=r"""kubectl top pod -A --sort-by=cpu | head -5
-                   hand(r"""echo NAMESPACE/POD > {WS}/lesson-2.3/cpu.txt""",
+kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt"""),
-                        r"""From the first line.""",
-                        run=r"""kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt""")]),
         step("Inside one Pod",
              r"""`kubectl top pod` adds up a Pod's containers. Which container of Pod `duo` uses the memory? Write its name to `{WS}/lesson-2.3/container.txt`.""",
              check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/container.txt" 2>/dev/null)" cache""",
              hint=r"""`--containers` splits the Pod into its containers.""",
-             show=[cmd(r"""k top po duo -n metrics --containers"""),
+             show=r"""kubectl top pod duo -n metrics --containers
-                   hand(r"""echo CONTAINER > {WS}/lesson-2.3/container.txt""",
+kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt"""),
-                        r"""The container with the most `MEMORY(bytes)`.""",
-                        run=r"""kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt""")]),
         step("Usage against what is reserved",
              r"""`kubectl top node` shows use; `kubectl describe node {W1}`, under **Allocated resources**, shows what Pods have **requested** - which is what decides whether another Pod fits.
 
 Write the memory that `{W1}` can give to Pods - its **allocatable** memory, as the API reports it (for example `8012345Ki`) - to `{WS}/lesson-2.3/allocatable.txt`.""",
              check=r"""want=$(j node "$W1" '{.status.allocatable.memory}'); [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/allocatable.txt" 2>/dev/null)" "$want" """,
              hint=r"""Allocatable is capacity minus what the node keeps for itself and for the eviction threshold: `.status.allocatable.memory`.""",
-             show=[cmd(r"""k top no
+             show=r"""kubectl top node
-k describe no {W1} | grep -A6 -e Allocatable: -e 'Allocated resources' """, r"""`memory:` under `Allocatable:` is what `{W1}` can give to Pods, for example `8012345Ki`."""),
+kubectl describe node {W1} | grep -A9 'Allocated resources'
-                   hand(r"""echo MEMORY > {WS}/lesson-2.3/allocatable.txt""",
+kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt"""),
-                        r"""`MEMORY`: that value, as it stands.""",
-                        run=r"""kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt""")]),
         step("The metrics API itself",
              r"""`kubectl top` only reads an API. Read it yourself: save what `/apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder` returns in `{WS}/lesson-2.3/raw.json`.
 
 If this path answers `NotFound` or `ServiceUnavailable` on some cluster, `kubectl top` fails there too: look at the `metrics-server` Deployment in `kube-system`, and at `kubectl get apiservice v1beta1.metrics.k8s.io`.""",
              check=r"""python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); sys.exit(0 if d.get("kind")=="PodMetrics" and d["metadata"]["name"]=="hoarder" and d.get("containers") else 1)' "$WS/lesson-2.3/raw.json" 2>/dev/null""",
              hint=r"""`kubectl get --raw PATH`.""",
-             show=[cmd(r"""k get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json
+             show=r"""kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json
-cat {WS}/lesson-2.3/raw.json
+jq '.containers' {WS}/lesson-2.3/raw.json
-k get apiservice v1beta1.metrics.k8s.io""")]),
+kubectl get apiservice v1beta1.metrics.k8s.io"""),
     ],
     "outro": r"""You ranked Pods by memory and CPU, looked inside a Pod, compared use with what a node can give, and read the metrics API that `kubectl top` uses.
 
@@ -404,17 +361,16 @@ echo "flaky did not restart" >&2; exit 1
 Save those in `{WS}/lesson-2.4/previous.log`.""",
              check=r"""grep -q 'FATAL config /etc/flaky.conf not found' "$WS/lesson-2.4/previous.log" 2>/dev/null && ! grep -q 'second attempt' "$WS/lesson-2.4/previous.log" """,
              hint=r"""`kubectl logs --previous` (or `-p`).""",
-             show=[cmd(r"""k -n logs logs flaky
+             show=r"""kubectl -n logs logs flaky
-k -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log
+kubectl -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log
-cat {WS}/lesson-2.4/previous.log""")]),
+cat {WS}/lesson-2.4/previous.log"""),
         step("The one replica that complains",
              r"""Pods `api-1`, `api-2` and `api-3` all carry `app=api`. One of them logs errors. Read all three at once, find it, and write its name to `{WS}/lesson-2.4/erroring.txt`.""",
              check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/erroring.txt" 2>/dev/null)" api-2""",
              hint=r"""`kubectl logs -l app=api --prefix` puts `[pod/NAME/CONTAINER]` before each line. With a selector, kubectl shows only the last 10 lines of each Pod unless you say `--tail`.""",
-             show=[cmd(r"""k -n logs logs -l app=api --prefix --tail=20 | grep ERROR""", r"""`[pod/NAME/...]` at the start of the line names the Pod."""),
+             show=r"""kubectl -n logs logs -l app=api --prefix --tail=2
-                   hand(r"""echo POD > {WS}/lesson-2.4/erroring.txt""",
+kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt
-                        r"""`POD`: that name.""",
+cat {WS}/lesson-2.4/erroring.txt"""),
-                        run=r"""kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt""")]),
         step("The log file on the node",
              r"""The node keeps one file per container instance. Write the full path of the file that holds the output of `flaky`'s **current** instance to `{WS}/lesson-2.4/logfile.txt`.
 
@@ -424,15 +380,11 @@ p=$(tr -d ' \r\n' < "$WS/lesson-2.4/logfile.txt" 2>/dev/null)
 uid=$(j -n logs pod flaky '{.metadata.uid}'); n=$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}'); node=$(j -n logs pod flaky '{.spec.nodeName}')
 eq "$p" "/var/log/pods/logs_flaky_$uid/flaky/$n.log" && nx "$node" "grep -q 'second attempt' '$p'" </dev/null""",
              hint=r"""`/var/log/pods/NAMESPACE_POD_UID/CONTAINER/RESTARTS.log`. `sudo ls -R /var/log/pods/ | grep flaky` on the node gets you close.""",
-             show=[cmd(r"""k -n logs get po flaky -o wide
+             show=r"""kubectl -n logs get pod flaky -o wide
-k -n logs get po flaky -o yaml | grep uid:""", r"""Its node, and its UID."""),
+uid=$(kubectl -n logs get pod flaky -o jsonpath='{.metadata.uid}')
-                   hand(r"""ssh {W1} sudo ls /var/log/pods/""",
+ssh -n {W1} "sudo ls /var/log/pods/logs_flaky_$uid/flaky/"
-                        r"""`logs_flaky_UID` holds `flaky/`, one file per instance: `0.log` the first, `1.log` the current one.""",
+echo "/var/log/pods/logs_flaky_$uid/flaky/1.log" > {WS}/lesson-2.4/logfile.txt
-                        run=r"""ssh -n {W1} sudo ls /var/log/pods/"""),
+ssh -n {W1} "sudo cat $(cat {WS}/lesson-2.4/logfile.txt)"""),
-                   hand(r"""echo /var/log/pods/logs_flaky_UID/flaky/1.log > {WS}/lesson-2.4/logfile.txt""",
-                        r"""With the Pod's UID.""",
-                        run=r"""uid=$(kubectl -n logs get pod flaky -o jsonpath='{.metadata.uid}')
-echo "/var/log/pods/logs_flaky_$uid/flaky/1.log" > {WS}/lesson-2.4/logfile.txt""")]),
         step("Debug a container that has no shell",
              r"""`kubectl -n logs exec quiet -- sh` fails: the `pause` image has nothing but its one program. Add an ephemeral container to the running Pod instead: name `dbg`, image `busybox:1.36`, **targeting** container `quiet` so that it sees its processes, running `sleep 3600`.
 
@@ -441,13 +393,10 @@ Then run `ps` in `dbg` and write the command of the target's process - the first
 eq "$(j -n logs pod quiet '{.spec.ephemeralContainers[0].name} {.spec.ephemeralContainers[0].targetContainerName}')" "dbg quiet" || exit 1
 grep -qx '/pause' <(tr -d ' \r' < "$WS/lesson-2.4/target.txt" 2>/dev/null)""",
              hint=r"""`kubectl debug POD --image=IMAGE --target=CONTAINER -c NAME -- COMMAND`. Interactively you would add `-it` and use the shell straight away; without it, `kubectl exec POD -c dbg -- ps` reaches it once it runs. Ephemeral containers cannot be removed - the Pod keeps them until it goes.""",
-             show=[cmd(r"""k -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600"""),
+             show=r"""kubectl -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600
-                   hand(r"""k -n logs exec quiet -c dbg -- ps""",
+for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done
-                        r"""Again if `dbg` is not up yet. PID 1 is the target's process.""",
+kubectl -n logs exec quiet -c dbg -- ps
-                        run=r"""for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done"""),
+kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt"""),
-                   hand(r"""echo COMMAND > {WS}/lesson-2.4/target.txt""",
-                        r"""`COMMAND`: PID 1's command, as `ps` shows it.""",
-                        run=r"""kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt""")]),
         step("Debug the node itself",
              r"""`kubectl debug node/{W1} --image=busybox:1.36` starts a Pod on that node with the node's whole filesystem mounted at `/host` - useful when you cannot `ssh` to a node, but can reach the API.
 
@@ -457,16 +406,11 @@ want=$(nx "$W1" '. /etc/os-release; echo "$PRETTY_NAME"' </dev/null); [ -n "$wan
 eq "$(tr -d '\r"' < "$WS/lesson-2.4/os.txt" 2>/dev/null | sed 's/^PRETTY_NAME=//; s/ *$//')" "$want" \
   && [ -n "$($K get pods -A -o name 2>/dev/null | grep node-debugger)" ]""",
              hint=r"""Without `-it`, give it a command and read its output with `kubectl logs`: the Pod is called `node-debugger-NODE-xxxxx`, in your current Namespace. Delete it afterwards.""",
-             show=[cmd(r"""k debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release
+             show=r"""kubectl debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release
-k get po""", r"""The `node-debugger-...` Pod: once it is `Completed`, its log is the file."""),
+pod=$(kubectl get pods -o name | grep node-debugger | head -1)
-                   hand(r"""k logs node-debugger-NAME""",
+kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s
-                        r"""`node-debugger-NAME`: its full name, from `get po`.""",
+kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt
-                        run=r"""pod=$(kubectl get pods -o name | grep node-debugger | head -1)
+cat {WS}/lesson-2.4/os.txt"""),
-kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s"""),
-                   hand(r"""echo 'VALUE' > {WS}/lesson-2.4/os.txt""",
-                        r"""`VALUE`: what `PRETTY_NAME=` says, without the double quotes.""",
-                        run=r"""pod=$(kubectl get pods -o name | grep node-debugger | head -1)
-kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt""")]),
     ],
     "outro": r"""You read a crashed instance's logs, searched several Pods at once, found the log file on the node, and debugged both a Pod without a shell and a node.
 
@@ -533,20 +477,19 @@ eq "$(j -n shop svc web '{.spec.selector.app}')" web || exit 1
 n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' | grep -c .)
 [ "$n" -eq 2 ]""",
              hint=r"""Compare `kubectl -n shop get svc web -o wide` (SELECTOR) with `kubectl -n shop get pods --show-labels`. `kubectl set selector` or `kubectl edit svc`.""",
-             show=[cmd(r"""k -n shop get endpointslices -l kubernetes.io/service-name=web
+             show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
-k -n shop get svc web -o wide
+kubectl -n shop get svc web -o wide
-k -n shop get po --show-labels
+kubectl -n shop get pods --show-labels
-k -n shop set selector svc web app=web
+kubectl -n shop set selector svc web app=web
-k -n shop get endpointslices -l kubernetes.io/service-name=web""")]),
+kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""),
         step("The wrong port",
              r"""Now the Service has endpoints, but they point at port `8080` - and nginx listens on `80`. Make the Service send traffic to the port the container serves.""",
              check=r"""
 eq "$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{.items[0].ports[0].port}')" 80 && eq "$(j -n shop svc web '{.spec.ports[0].port}')" 80""",
              hint=r"""`targetPort` is the container's port; `port` is the Service's own. `kubectl -n shop edit svc web`, or a JSON patch on `/spec/ports/0/targetPort`.""",
-             show=[hand(r"""k -n shop edit svc web""",
+             show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
-                        r"""Make `targetPort: 8080` read `targetPort: 80`, then `:wq`.""",
+kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]'
-                        run=r"""kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]' """),
+kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""),
-                   cmd(r"""k -n shop get endpointslices -l kubernetes.io/service-name=web""")]),
         step("Let the right client in",
              r"""Still nothing: NetworkPolicy `lockdown` selects the `web` Pods and allows no ingress at all.
 
@@ -557,17 +500,7 @@ ok() { $K -n shop exec "$1" -- wget -qO- -T 3 web >/dev/null 2>&1; }
 for i in 1 2 3 4 5; do ok client && break; sleep 3; done; ok client || exit 1
 ! ok stranger""",
              hint=r"""Add an `ingress` rule to `lockdown`: `from: [{podSelector: {matchLabels: {app: client}}}]` with `ports: [{protocol: TCP, port: 80}]`. A NetworkPolicy's ports are the Pod's, not the Service's.""",
-             show=[hand(r"""k -n shop edit netpol lockdown""",
+             show=r"""kubectl -n shop apply -f - <<'EOF'
-                        r"""Add the rule to its `spec:`, beside `policyTypes:`:
-```
-  ingress:
-  - from:
-    - podSelector:
-        matchLabels: {app: client}
-    ports:
-    - {protocol: TCP, port: 80}
-```""",
-                        run=r"""kubectl -n shop apply -f - <<'EOF'
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
@@ -582,9 +515,9 @@ spec:
         matchLabels: {app: client}
     ports:
     - {protocol: TCP, port: 80}
-EOF"""),
+EOF
-                   cmd(r"""k -n shop exec client -- wget -qO- -T3 web | head -4
+kubectl -n shop exec client -- wget -qO- -T3 web | head -4
-k -n shop exec stranger -- wget -qO- -T3 web""", r"""`client` gets the page; `stranger` times out.""")]),
+kubectl -n shop exec stranger -- wget -qO- -T3 web || echo refused"""),
         step("How kube-proxy routes it",
              r"""kube-proxy turned Service `web` into rules on every node. Find out how: its mode is in ConfigMap `kube-proxy` in `kube-system`. Write the mode to `{WS}/lesson-2.5/mode.txt`.
 
@@ -596,18 +529,11 @@ ip=$(j -n shop svc web '{.spec.clusterIP}')
 want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" </dev/null)
 [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.5/chain.txt" 2>/dev/null)" "$want" """,
              hint=r"""`kubectl -n kube-system get cm kube-proxy -o yaml | grep mode`. On the node: `sudo iptables-save -t nat | grep CLUSTER-IP` - the line for the Service jumps (`-j`) to its chain. In `nftables` mode the rules are in `sudo nft list ruleset` instead.""",
-             show=[cmd(r"""k -n kube-system get cm kube-proxy -o yaml | grep mode:"""),
+             show=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt
-                   hand(r"""echo MODE > {WS}/lesson-2.5/mode.txt""",
+ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}')
-                        r"""`MODE`: its value.""",
+sudo iptables-save -t nat | grep -- "-d $ip/32"
-                        run=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt"""),
+sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt
-                   cmd(r"""k -n shop get svc web""", r"""Its `CLUSTER-IP`:"""),
+sudo iptables-save -t nat | grep "^-A $(cat {WS}/lesson-2.5/chain.txt)" """),
-                   hand(r"""sudo iptables-save -t nat | grep IP""",
-                        r"""`IP`: that address. The rule for it jumps (`-j`) to a `KUBE-SVC-...` chain.""",
-                        run='ip=$(kubectl -n shop get svc web -o jsonpath=\'{.spec.clusterIP}\')\nsudo iptables-save -t nat | grep -- "-d $ip/32"'),
-                   hand(r"""echo KUBE-SVC-NAME > {WS}/lesson-2.5/chain.txt""",
-                        r"""The chain's full name.""",
-                        run=r"""ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}')
-sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt""")]),
     ],
     "outro": r"""You took a Service from "times out" to answering by fixing its selector, its target port and a NetworkPolicy, and followed it into kube-proxy's rules.
 

cka/lessons_work.py

@@ -1,5 +1,5 @@
 """CKA lessons of the Workloads and Scheduling domain (3.1-3.5). See cka/lessons.py for what a lesson is."""
-from sim.bank import NS, step, cmd, hand
+from sim.bank import NS, step
 
 # ---- 3.1 rolling updates and rollbacks --------------------------------------------------------
 L31 = {
@@ -27,10 +27,8 @@ $K -n rollout rollout status deploy api --timeout=180s >/dev/null
              r"""Change the update strategy of `api` so that an update first adds **one** Pod and never takes one away before its replacement is ready: `maxSurge: 1`, `maxUnavailable: 0`.""",
              check=r"""eq "$(j -n rollout deploy api '{.spec.strategy.type} {.spec.strategy.rollingUpdate.maxSurge} {.spec.strategy.rollingUpdate.maxUnavailable}')" "RollingUpdate 1 0" """,
              hint=r"""`kubectl -n rollout edit deploy api`, or `kubectl patch` with `{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}`.""",
-             show=[hand(r"""k -n rollout edit deploy api""",
+             show=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
-                        r"""Under `strategy.rollingUpdate`, set `maxSurge: 1` and `maxUnavailable: 0`, then save and quit (`:wq`).""",
+kubectl -n rollout get deploy api -o jsonpath='{.spec.strategy}'; echo"""),
-                        run=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' """),
-                   cmd(r"""k -n rollout describe deploy api | grep -i strategy""")]),
         step("A release with a reason",
              r"""Update `api` to `nginx:1.27`, and record why: the annotation `kubernetes.io/change-cause` on the Deployment, set to `upgrade to 1.27`. `kubectl rollout history` shows it beside the revision.""",
              check=r"""
@@ -38,10 +36,10 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin
 eq "$(j -n rollout deploy api '{.metadata.annotations.kubernetes\.io/change-cause}')" "upgrade to 1.27" || exit 1
 eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas}')" "4 4" """,
              hint=r"""`kubectl set image deploy/api nginx=nginx:1.27` (the container is called `nginx`), then `kubectl annotate deploy api kubernetes.io/change-cause='...'`. The `--record` flag is gone.""",
-             show=[cmd(r"""k -n rollout set image deploy/api nginx=nginx:1.27
+             show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27
-k -n rollout annotate deploy api kubernetes.io/change-cause='upgrade to 1.27' --overwrite
+kubectl -n rollout annotate deploy api kubernetes.io/change-cause='upgrade to 1.27' --overwrite
-k -n rollout rollout status deploy api
+kubectl -n rollout rollout status deploy api --timeout=180s
-k -n rollout rollout history deploy api""")]),
+kubectl -n rollout rollout history deploy api"""),
         step("A release that cannot start",
              r"""Now a mistake: set the image of `api` to `nginx:1.27-broken`, a tag that does not exist.
 
@@ -51,27 +49,29 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin
 [ "$(j -n rollout deploy api '{.status.availableReplicas}')" = 4 ] || exit 1
 $K -n rollout get pods -l app=api --no-headers 2>/dev/null | grep -Eq 'ErrImagePull|ImagePullBackOff' """,
              hint=r"""`kubectl set image` again. `kubectl -n rollout get pods` shows the Pod that cannot pull; `kubectl -n rollout get rs` shows the two ReplicaSets side by side.""",
-             show=[cmd(r"""k -n rollout set image deploy/api nginx=nginx:1.27-broken
+             show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27-broken
-k -n rollout get po,rs""", r"""Run `get` again after a few seconds: one new Pod is stuck in `ErrImagePull`, and the four old ones still run.""")]),
+sleep 10
+kubectl -n rollout get pods
+kubectl -n rollout get rs"""),
         step("Undo",
              r"""Roll `api` back to the revision before - `nginx:1.27` - and let the rollout finish.""",
              check=r"""
 eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1
 eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
              hint=r"""`kubectl rollout undo deploy/api`. It does not create a ReplicaSet: it scales the old one back up, and that revision gets a new number.""",
-             show=[cmd(r"""k -n rollout rollout undo deploy/api
+             show=r"""kubectl -n rollout rollout undo deploy/api
-k -n rollout rollout status deploy api
+kubectl -n rollout rollout status deploy api --timeout=180s
-k -n rollout rollout history deploy api""")]),
+kubectl -n rollout rollout history deploy api"""),
         step("Back to a revision by number",
              r"""Look at the history, then go back to the revision that ran **`nginx:1.26`**, by its number. `kubectl rollout history deploy api --revision=N` shows what a revision's template was.""",
              check=r"""
 eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.26 || exit 1
 eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
              hint=r"""Revision 1 is where the Deployment started. `kubectl rollout undo deploy/api --to-revision=1`.""",
-             show=[cmd(r"""k -n rollout rollout history deploy api
+             show=r"""kubectl -n rollout rollout history deploy api
-k -n rollout rollout history deploy api --revision=1""", r"""Find the revision whose `Image:` is `nginx:1.26` - revision 1 here."""),
+kubectl -n rollout rollout history deploy api --revision=1 | grep Image
-                   cmd(r"""k -n rollout rollout undo deploy/api --to-revision=1
+kubectl -n rollout rollout undo deploy/api --to-revision=1
-k -n rollout rollout status deploy api""")]),
+kubectl -n rollout rollout status deploy api --timeout=180s"""),
         step("Two changes, one rollout",
              r"""Each `set` command starts a rollout of its own. To make two changes at once, **pause** the Deployment, make them - image `nginx:1.27`, and an environment variable `MODE=prod` in the container - then **resume**: one new revision, one rollout.""",
              check=r"""
@@ -80,12 +80,12 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin
 eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].env[?(@.name=="MODE")].value}')" prod || exit 1
 eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
              hint=r"""`kubectl rollout pause deploy/api`, `kubectl set image ...`, `kubectl set env deploy/api MODE=prod`, `kubectl rollout resume deploy/api`.""",
-             show=[cmd(r"""k -n rollout rollout pause deploy/api
+             show=r"""kubectl -n rollout rollout pause deploy/api
-k -n rollout set image deploy/api nginx=nginx:1.27
+kubectl -n rollout set image deploy/api nginx=nginx:1.27
-k -n rollout set env deploy/api MODE=prod
+kubectl -n rollout set env deploy/api MODE=prod
-k -n rollout rollout resume deploy/api
+kubectl -n rollout rollout resume deploy/api
-k -n rollout rollout status deploy api
+kubectl -n rollout rollout status deploy api --timeout=180s
-k -n rollout rollout history deploy api""")]),
+kubectl -n rollout rollout history deploy api"""),
     ],
     "outro": r"""You tuned the strategy, released with a recorded cause, watched a broken release stall safely, undid it, went back to a numbered revision, and batched two changes into one rollout.
 
@@ -132,15 +132,15 @@ printf 'DB_USER=ledger\nDB_PASS=t0ps3cret\n' > "$WS/lesson-3.2/db.env"
              check=r"""
 eq "$(j -n conf cm app-config '{.data.MODE}')" prod && $K -n conf get cm app-config -o jsonpath='{.data.app\.properties}' | grep -q '^cache.size=128$'""",
              hint=r"""`kubectl create configmap NAME --from-file=PATH --from-literal=KEY=VALUE`. With `--from-file`, the key is the file's name.""",
-             show=[cmd(r"""k -n conf create cm app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod
+             show=r"""kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod
-k -n conf describe cm app-config""")]),
+kubectl -n conf describe cm app-config"""),
         step("A Secret from an env file",
              r"""Create Secret `db` (type `Opaque`) in `conf` from `{WS}/lesson-3.2/db.env`, one key per line of the file: `DB_USER` and `DB_PASS`. Then read `DB_PASS` back, decoded.""",
              check=r"""
 eq "$(j -n conf secret db '{.type}')" Opaque && eq "$(j -n conf secret db '{.data.DB_USER}' | base64 -d)" ledger && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""",
              hint=r"""`kubectl create secret generic NAME --from-env-file=PATH`. Reading one key: `-o jsonpath='{.data.DB_PASS}' | base64 -d`.""",
-             show=[cmd(r"""k -n conf create secret generic db --from-env-file={WS}/lesson-3.2/db.env
+             show=r"""kubectl -n conf create secret generic db --from-env-file={WS}/lesson-3.2/db.env
-k -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d""")]),
+kubectl -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d; echo"""),
         step("Consume both",
              r"""Run Pod `app` in `conf` (`busybox:1.36`, `sleep 86400`) that gets:
 
@@ -151,10 +151,16 @@ k -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d""")]),
 x() { $K -n conf exec app -- "$@" 2>/dev/null; }
 eq "$(x printenv DB_PASS)" t0ps3cret && eq "$(x printenv APP_MODE)" prod && x cat /etc/app/app.properties | grep -q '^log.level=info$'""",
              hint=r"""`envFrom: [{secretRef: {name: db}}]`, `env: [{name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: MODE}}}]`, and a `configMap` volume.""",
-             show=[cmd(r"""k -n conf run app --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > app.yaml"""),
+             show=r"""kubectl -n conf apply -f - <<'EOF'
-                   hand(r"""vim app.yaml""",
+apiVersion: v1
-                        r"""Add to the container, below `resources: {}` (the docs page *Configure a Pod to Use a ConfigMap* has each of these):
+kind: Pod
-```
+metadata:
+  name: app
+spec:
+  containers:
+  - name: app
+    image: busybox:1.36
+    command: [sleep, '86400']
     envFrom:
     - secretRef: {name: db}
     env:
@@ -163,38 +169,33 @@ eq "$(x printenv DB_PASS)" t0ps3cret && eq "$(x printenv APP_MODE)" prod && x ca
         configMapKeyRef: {name: app-config, key: MODE}
     volumeMounts:
     - {name: config, mountPath: /etc/app}
-```
-and to the Pod's `spec:`, beside `restartPolicy: Always` and indented like it:
-```
   volumes:
-  - {name: config, configMap: {name: app-config}}
+  - name: config
-```""",
+    configMap: {name: app-config}
-                        run=r"""yq -i '.spec.containers[0].envFrom = [{"secretRef": {"name": "db"}}]
+EOF
-  | .spec.containers[0].env = [{"name": "APP_MODE", "valueFrom": {"configMapKeyRef": {"name": "app-config", "key": "MODE"}}}]
+kubectl -n conf wait --for=condition=Ready pod/app --timeout=120s
-  | .spec.containers[0].volumeMounts = [{"name": "config", "mountPath": "/etc/app"}]
+kubectl -n conf exec app -- sh -c 'printenv DB_USER APP_MODE; cat /etc/app/app.properties'"""),
-  | .spec.volumes = [{"name": "config", "configMap": {"name": "app-config"}}]' app.yaml"""),
-                   cmd(r"""k -n conf apply -f app.yaml
-k -n conf exec app -- env | grep -e DB_ -e APP_""")]),
         step("Change the configuration, live",
              r"""Change `cache.size` to `256` in ConfigMap `app-config` - edit the file and re-create the ConfigMap's key from it, or `kubectl edit`.
 
 Then watch `/etc/app/app.properties` inside Pod `app`: within a minute the kubelet replaces the file. `APP_MODE` would not change, even if you edited `MODE`: the environment is fixed at start.""",
              check=r"""$K -n conf exec app -- cat /etc/app/app.properties 2>/dev/null | grep -q '^cache.size=256$'""",
              hint=r"""Regenerate and apply: `kubectl create configmap app-config --from-file=... --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f -` - the usual way to update a ConfigMap from files.""",
-             show=[hand(r"""k -n conf edit cm app-config""",
+             show=r"""sed -i 's/^cache.size=.*/cache.size=256/' {WS}/lesson-3.2/app.properties
-                        r"""In `app.properties`, change the line to `cache.size=256`, then `:wq`.""",
+kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f -
-                        run=r"""kubectl -n conf get cm app-config -o yaml | sed 's/cache.size=.*/cache.size=256/' | kubectl replace -f -"""),
+sleep 30
-                   cmd(r"""k -n conf exec app -- cat /etc/app/app.properties""", r"""Again until it shows `cache.size=256`: the kubelet rewrites the file within a minute.""")]),
+kubectl -n conf exec app -- cat /etc/app/app.properties"""),
         step("A TLS Secret",
              r"""Make a self-signed certificate for `shop.example.com` (`openssl` is on this host), and store it as Secret `shop-tls` in `conf`, of type `kubernetes.io/tls` - the type an Ingress or Gateway expects. Its keys are `tls.crt` and `tls.key`.""",
              check=r"""
 eq "$(j -n conf secret shop-tls '{.type}')" kubernetes.io/tls || exit 1
 j -n conf secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'shop.example.com'""",
              hint=r"""`openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com`, then `kubectl create secret tls NAME --cert=tls.crt --key=tls.key`.""",
-             show=[cmd(r"""cd {WS}/lesson-3.2
+             show=r"""cd {WS}/lesson-3.2
-openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com
+openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
-k -n conf create secret tls shop-tls --cert=tls.crt --key=tls.key
+kubectl -n conf create secret tls shop-tls --cert=tls.crt --key=tls.key
-cd""")]),
+kubectl -n conf get secret shop-tls
+cd"""),
         step("Encrypt Secrets at rest",
              r"""Read Secret `db` straight out of etcd on `{CP}`, and you see `t0ps3cret` in plain text. Make the API server encrypt Secrets before it stores them:
 
@@ -205,23 +206,7 @@ cd""")]),
 Edit a copy of the manifest outside `/etc/kubernetes/manifests` and copy it back: the kubelet runs every file in that directory, backups too.""",
              check=_ENC_CHECK + r""" && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""",
              hint=r"""The docs page "Encrypting Confidential Data at Rest" has the file. Stored values of encrypted Secrets start with `k8s:enc:aescbc:v1:key1:`: `sudo ETCDCTL_API=3 etcdctl ... get /registry/secrets/conf/db | od -c | head` (or `hexdump -C`). Rewriting: `kubectl get secret db -o json | kubectl replace -f -`. If the API server does not come back, `sudo crictl ps -a` and the logs under `/var/log/pods/kube-system_kube-apiserver-*` say why.""",
-             show=[cmd(r"""head -c 32 /dev/urandom | base64""", r"""Copy the output: that is the key."""),
+             show=r"""sudo mkdir -p /etc/kubernetes/enc
-                   hand(r"""sudo mkdir -p /etc/kubernetes/enc
-sudo vim /etc/kubernetes/enc/enc.yaml""",
-                        r"""Paste the example from the docs page *Encrypting Confidential Data at Rest* (search for `aescbc`) and cut it down to:
-```
-apiVersion: apiserver.config.k8s.io/v1
-kind: EncryptionConfiguration
-resources:
-- resources: [secrets]
-  providers:
-  - aescbc:
-      keys:
-      - name: key1
-        secret: <your key>
-  - identity: {}
-```""",
-                        run=r"""sudo mkdir -p /etc/kubernetes/enc
 sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
 apiVersion: apiserver.config.k8s.io/v1
 kind: EncryptionConfiguration
@@ -233,30 +218,19 @@ resources:
       - name: key1
         secret: $(head -c 32 /dev/urandom | base64)
   - identity: {}
-EOF"""),
+EOF
-                   cmd(r"""sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/"""),
+sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak
-                   hand(r"""sudo vim ~/kube-apiserver.yaml""",
+sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
-                        r"""Add three lines, each beside its kind (`yy` and `p` on an existing `k8s-certs` mount or volume, then change it):
+sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
-```
-    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
-```
-in the `command:` list,
-```
-    - {name: enc, mountPath: /etc/kubernetes/enc, readOnly: true}
-```
-under `volumeMounts:`, and
-```
-  - {name: enc, hostPath: {path: /etc/kubernetes/enc, type: DirectoryOrCreate}}
-```
-under `volumes:`.""",
-                        run=r"""sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
   | .spec.containers[0].volumeMounts += [{"name": "enc", "mountPath": "/etc/kubernetes/enc", "readOnly": true}]
-  | .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' ~/kube-apiserver.yaml"""),
+  | .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' /tmp/kube-apiserver.yaml
-                   cmd(r"""sudo cp ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
+sudo cp /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
-                   hand(r"""k get po -n kube-system""",
+sleep 20
-                        r"""Again until it answers: the API server is back in a minute or so.""",
+for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
-                        run=r"""sleep 20; for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
+kubectl -n conf get secret db -o json | kubectl replace -f -
-                   cmd(r"""k -n conf get secret db -o json | k replace -f -""", r"""Written again, the Secret is stored encrypted. Reading it from etcd would now show `k8s:enc:aescbc:v1:key1` and no plain text.""")]),
+sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
+  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
+  get /registry/secrets/conf/db | od -c | head -4"""),
     ],
     "outro": r"""You built a ConfigMap and Secrets from files, consumed them three ways, updated configuration in a running Pod, and turned on encryption at rest for Secrets.
 
@@ -329,8 +303,9 @@ $K -n scale rollout status deploy api --timeout=180s >/dev/null && $K -n scale r
 eq "$(j -n scale hpa api '{.spec.minReplicas} {.spec.maxReplicas} {.spec.metrics[0].resource.name} {.spec.metrics[0].resource.target.averageUtilization}')" "2 6 cpu 50" || exit 1
 [ -n "$(j -n scale hpa api '{.status.currentMetrics[0].resource.current.averageUtilization}')" ] && [ "$(j -n scale deploy api '{.status.readyReplicas}')" -ge 2 ] 2>/dev/null""",
              hint=r"""`kubectl autoscale deploy api --min=2 --max=6 --cpu=50%` (kubectl before 1.34: `--cpu-percent=50`). `<unknown>` in the TARGETS column means no metrics yet - or a Pod without a CPU request.""",
-             show=[cmd(r"""k -n scale autoscale deploy api --min=2 --max=6 --cpu=50%
+             show=r"""kubectl -n scale autoscale deploy api --min=2 --max=6 --cpu=50%
-k -n scale get hpa api""", r"""Again in a minute: `TARGETS` shows a value once metrics arrive.""")]),
+sleep 45
+kubectl -n scale get hpa api"""),
         step("Scale down calmly",
              r"""Give the HPA of `api` a `behavior`: when scaling **down**, wait **120** seconds of stable recommendations (`stabilizationWindowSeconds`), and remove at most **1** Pod per **60** seconds.
 
@@ -339,27 +314,36 @@ k -n scale get hpa api""", r"""Again in a minute: `TARGETS` shows a value once m
 eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.stabilizationWindowSeconds}')" 120 || exit 1
 eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.policies[0].type} {.spec.behavior.scaleDown.policies[0].value} {.spec.behavior.scaleDown.policies[0].periodSeconds}')" "Pods 1 60" """,
              hint=r"""Under `spec`: `behavior: {scaleDown: {stabilizationWindowSeconds: 120, policies: [{type: Pods, value: 1, periodSeconds: 60}]}}`. `kubectl explain hpa.spec.behavior.scaleDown`.""",
-             show=[hand(r"""k -n scale edit hpa api""",
+             show=r"""kubectl -n scale patch hpa api --type merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":120,"policies":[{"type":"Pods","value":1,"periodSeconds":60}]}}}}'
-                        r"""Add under `spec:` (the docs page *Horizontal Pod Autoscaling* has it under *Configurable scaling behavior*):
+kubectl -n scale describe hpa api | grep -A4 Behavior"""),
-```
-  behavior:
-    scaleDown:
-      stabilizationWindowSeconds: 120
-      policies:
-      - {type: Pods, value: 1, periodSeconds: 60}
-```""",
-                        run=r"""kubectl -n scale patch hpa api --type merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":120,"policies":[{"type":"Pods","value":1,"periodSeconds":60}]}}}}' """),
-                   cmd(r"""k -n scale describe hpa api""")]),
         step("Watch it scale up",
              r"""Write an HPA for `cruncher` as a manifest (`autoscaling/v2`): 1 to **4** replicas at **60%** average CPU. `cruncher` uses about three times its request, so the HPA soon adds Pods: wait until it wants more than one.""",
              check=r"""
 eq "$(j -n scale hpa cruncher '{.spec.maxReplicas} {.spec.metrics[0].resource.target.averageUtilization}')" "4 60" || exit 1
 [ "$(j -n scale hpa cruncher '{.status.desiredReplicas}')" -gt 1 ] 2>/dev/null""",
              hint=r"""`kubectl autoscale ... --dry-run=client -o yaml` writes the manifest for you. `kubectl -n scale get hpa cruncher -w` shows it react; `describe hpa` lists its decisions as events.""",
-             show=[cmd(r"""k -n scale autoscale deploy cruncher --min=1 --max=4 --cpu=60% --dry-run=client -o yaml > hpa.yaml
+             show=r"""kubectl -n scale apply -f - <<'EOF'
-cat hpa.yaml""", r"""That is the manifest the step asks for: `autoscaling/v2`, `minReplicas`, `maxReplicas` and a `Resource` metric for CPU."""),
+apiVersion: autoscaling/v2
-                   cmd(r"""k -n scale apply -f hpa.yaml
+kind: HorizontalPodAutoscaler
-k -n scale get hpa cruncher""", r"""Again in a minute or two: `REPLICAS` goes above 1.""")]),
+metadata:
+  name: cruncher
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: cruncher
+  minReplicas: 1
+  maxReplicas: 4
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      target:
+        type: Utilization
+        averageUtilization: 60
+EOF
+sleep 60
+kubectl -n scale get hpa cruncher"""),
         step("Resize a running Pod",
              r"""Run Pod `sizer` in `scale` (`busybox:1.36`, `sleep 86400`) with CPU request `100m` and limit `200m`, memory request and limit `64Mi`.
 
@@ -368,23 +352,11 @@ Then give it more CPU **without replacing it**: request `150m`, limit `300m`, th
 eq "$(j -n scale pod sizer '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.limits.cpu}')" "150m 300m" || exit 1
 eq "$(j -n scale pod sizer '{.status.containerStatuses[0].resources.limits.cpu} {.status.containerStatuses[0].restartCount}')" "300m 0" """,
              hint=r"""`kubectl patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{...}}]}}'`. A plain `kubectl edit` of a Pod's resources is refused; the subresource is the way in. A container's `resizePolicy` decides whether a change needs a restart.""",
-             show=[cmd(r"""k -n scale run sizer --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > sizer.yaml"""),
+             show=r"""kubectl -n scale run sizer --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"sizer","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"200m","memory":"64Mi"}}}]}}'
-                   hand(r"""vim sizer.yaml""",
+kubectl -n scale wait --for=condition=Ready pod/sizer --timeout=120s
-                        r"""Make `resources: {}`:
+kubectl -n scale patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{"requests":{"cpu":"150m"},"limits":{"cpu":"300m"}}}]}}'
-```
+sleep 5
-    resources:
+kubectl -n scale get pod sizer -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}{.status.containerStatuses[0].restartCount}{"\n"}'"""),
-      requests: {cpu: 100m, memory: 64Mi}
-      limits: {cpu: 200m, memory: 64Mi}
-```""",
-                        run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "100m", "memory": "64Mi"}, "limits": {"cpu": "200m", "memory": "64Mi"}}' sizer.yaml"""),
-                   cmd(r"""k -n scale apply -f sizer.yaml"""),
-                   hand(r"""k -n scale get po sizer""",
-                        r"""Once it is `Running`:""",
-                        run=r"""kubectl -n scale wait --for=condition=Ready pod/sizer --timeout=120s"""),
-                   hand(r"""k -n scale edit po sizer --subresource resize""",
-                        r"""Set the CPU request to `150m` and the CPU limit to `300m`, then `:wq`.""",
-                        run=r"""kubectl -n scale patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{"requests":{"cpu":"150m"},"limits":{"cpu":"300m"}}}]}}' """),
-                   cmd(r"""k -n scale describe po sizer | grep -A6 Limits""", r"""The new values, and `Restart Count: 0`.""")]),
     ],
     "outro": r"""You made HPAs both ways, slowed down their scale-down, watched one scale up, and resized a running Pod without replacing it.
 
@@ -449,24 +421,17 @@ rs=$(tr -d ' \r\n' < "$WS/lesson-3.4/rs.txt" 2>/dev/null); [ -n "$rs" ] || exit
 eq "$(j -n heal rs "$rs" '{.metadata.ownerReferences[0].name} {.status.readyReplicas}')" "web 3" || exit 1
 now=$(j -n heal pods -l app=web '{.items[*].metadata.uid}'); for u in $now; do grep -q "$u" "$PRIV/lesson-3.4.uids" || exit 0; done; exit 1""",
              hint=r"""`kubectl -n heal delete pod $(kubectl -n heal get pod -l app=web -o name | head -1)`. Pods of a Deployment are named DEPLOYMENT-RSHASH-RANDOM; the middle part is the ReplicaSet's suffix.""",
-             show=[cmd(r"""k -n heal get po -l app=web"""),
+             show=r"""kubectl -n heal delete $(kubectl -n heal get pod -l app=web -o name | head -1)
-                   hand(r"""k -n heal delete po POD""",
+kubectl -n heal get pods -l app=web
-                        r"""`POD`: any one of the three names.""",
+kubectl -n heal get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].name}' > {WS}/lesson-3.4/rs.txt
-                        run=r"""kubectl -n heal delete $(kubectl -n heal get pod -l app=web -o name | head -1)"""),
+kubectl -n heal get rs"""),
-                   cmd(r"""k -n heal get po -l app=web
-k -n heal get rs""", r"""A new Pod already, and one ReplicaSet behind them all: `web-` and a hash."""),
-                   hand(r"""echo RS > {WS}/lesson-3.4/rs.txt""",
-                        r"""`RS`: that ReplicaSet's name.""",
-                        run=r"""kubectl -n heal get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].name}' > {WS}/lesson-3.4/rs.txt""")]),
         step("A readiness probe that asks the wrong thing",
              r"""The Pods of `api` run, but none is `Ready`: `kubectl -n heal describe pod -l app=api` shows the readiness probe failing with `404`. nginx has no `/healthz`. Make the probe ask for `/` instead, and both Pods become ready.""",
              check=r"""eq "$(j -n heal deploy api '{.spec.template.spec.containers[0].readinessProbe.httpGet.path}')" / && eq "$(j -n heal deploy api '{.status.readyReplicas}')" 2""",
              hint=r"""A JSON patch on `/spec/template/spec/containers/0/readinessProbe/httpGet/path`, or `kubectl edit deploy api`. The fix is a change of template: a rollout.""",
-             show=[cmd(r"""k -n heal describe po -l app=api | grep -i readiness"""),
+             show=r"""kubectl -n heal describe pod -l app=api | grep -m2 Readiness
-                   hand(r"""k -n heal edit deploy api""",
+kubectl -n heal patch deploy api --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]'
-                        r"""In the `readinessProbe`, change `path: /healthz` to `path: /`, then `:wq`.""",
+kubectl -n heal rollout status deploy api --timeout=120s"""),
-                        run=r"""kubectl -n heal patch deploy api --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]' """),
-                   cmd(r"""k -n heal rollout status deploy api""")]),
         step("A liveness probe that restarts",
              r"""Run Pod `fragile` in `heal` (`busybox:1.36`) with the command `sh -c 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400'` and a **liveness** probe that runs `cat /tmp/healthy` every 5 seconds, starting after 5.
 
@@ -475,19 +440,24 @@ After 20 seconds the file is gone, the probe fails three times, and the kubelet
 eq "$(j -n heal pod fragile '{.spec.containers[0].livenessProbe.exec.command[1]}')" /tmp/healthy || exit 1
 [ "$(j -n heal pod fragile '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null""",
              hint=r"""`livenessProbe: {exec: {command: [cat, /tmp/healthy]}, initialDelaySeconds: 5, periodSeconds: 5}`. `kubectl -n heal get pod fragile -w` shows RESTARTS go up; `describe` shows `Liveness probe failed`.""",
-             show=[cmd(r"""k -n heal run fragile --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400' > fragile.yaml"""),
+             show=r"""kubectl -n heal apply -f - <<'EOF'
-                   hand(r"""vim fragile.yaml""",
+apiVersion: v1
-                        r"""Add to the container, below `resources: {}` (the docs page *Configure Liveness, Readiness and Startup Probes* has it):
+kind: Pod
-```
+metadata:
+  name: fragile
+spec:
+  containers:
+  - name: fragile
+    image: busybox:1.36
+    command: [sh, -c, 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400']
     livenessProbe:
       exec:
         command: [cat, /tmp/healthy]
       initialDelaySeconds: 5
       periodSeconds: 5
-```""",
+EOF
-                        run=r"""yq -i '.spec.containers[0].livenessProbe = {"exec": {"command": ["cat", "/tmp/healthy"]}, "initialDelaySeconds": 5, "periodSeconds": 5}' fragile.yaml"""),
+sleep 50
-                   cmd(r"""k -n heal apply -f fragile.yaml
+kubectl -n heal get pod fragile"""),
-k -n heal get po fragile""", r"""Again after about 40 seconds: `RESTARTS` is 1.""")]),
         step("A budget for disruptions",
              r"""Create PodDisruptionBudget `web-pdb` in `heal`: at least **2** Pods of `app=web` must stay available.
 
@@ -498,25 +468,20 @@ Then try to drain `{W1}`, where all three `web` Pods run: `kubectl drain {W1} --
 eq "$(j -n heal pdb web-pdb '{.spec.minAvailable} {.spec.selector.matchLabels.app}')" "2 web" || exit 1
 grep -qi 'disruption budget' "$WS/lesson-3.4/drain.txt" 2>/dev/null && [ "$(j node "$W1" '{.spec.unschedulable}')" != true ]""",
              hint=r"""`kubectl create pdb web-pdb --selector=app=web --min-available=2`. Drain prints the refusals on stderr: `2>&1 | tee FILE`. `kubectl uncordon {W1}` afterwards - drain cordons first.""",
-             show=[cmd(r"""k -n heal create pdb web-pdb --selector=app=web --min-available=2
+             show=r"""kubectl -n heal create pdb web-pdb --selector=app=web --min-available=2
-k -n heal delete po fragile --now
+kubectl -n heal get pdb web-pdb
-k drain {W1} --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 | tee {WS}/lesson-3.4/drain.txt
+kubectl -n heal delete pod fragile --grace-period=1
-k uncordon {W1}""", r"""The drain evicts one `web` Pod, then refuses the others: "Cannot evict pod as it would violate the pod's disruption budget".""")]),
+kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 | tee {WS}/lesson-3.4/drain.txt | tail -3
+kubectl uncordon {W1}"""),
         step("A Job that runs to completion",
              r"""Create Job `batch` in `heal`: `busybox:1.36` running `sh -c 'echo processed; sleep 2'`, **3** completions, **2** at a time, giving up after **2** retries (`backoffLimit`). Wait until it is complete.""",
              check=r"""eq "$(j -n heal job batch '{.spec.completions} {.spec.parallelism} {.spec.backoffLimit} {.status.succeeded}')" "3 2 2 3" """,
              hint=r"""`kubectl create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'`, then add `completions`, `parallelism` and `backoffLimit` under `spec`. `kubectl wait --for=condition=complete job/batch`.""",
-             show=[cmd(r"""k -n heal create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo processed; sleep 2' > batch.yaml"""),
+             show=r"""kubectl -n heal create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo processed; sleep 2' > {WS}/lesson-3.4/batch.yaml
-                   hand(r"""vim batch.yaml""",
+yq -i '.spec.completions = 3 | .spec.parallelism = 2 | .spec.backoffLimit = 2' {WS}/lesson-3.4/batch.yaml
-                        r"""Add to the Job's own `spec:`, above `template:`:
+kubectl -n heal apply -f {WS}/lesson-3.4/batch.yaml
-```
+kubectl -n heal wait --for=condition=complete job/batch --timeout=120s
-  completions: 3
+kubectl -n heal get pods -l job-name=batch"""),
-  parallelism: 2
-  backoffLimit: 2
-```""",
-                        run=r"""yq -i '.spec.completions = 3 | .spec.parallelism = 2 | .spec.backoffLimit = 2' batch.yaml"""),
-                   cmd(r"""k apply -f batch.yaml
-k -n heal get job,po""", r"""Again until `COMPLETIONS` is `3/3`.""")]),
         step("A CronJob, and a run on demand",
              r"""Create CronJob `report` in `heal`: every 30 minutes (`*/30 * * * *`), `busybox:1.36` running `date`, never two runs at once (`concurrencyPolicy: Forbid`), keeping the last **2** successful Jobs.
 
@@ -525,17 +490,11 @@ Do not wait half an hour: start a Job `report-now` from the CronJob yourself, an
 eq "$(j -n heal cronjob report '{.spec.schedule} {.spec.concurrencyPolicy} {.spec.successfulJobsHistoryLimit}')" "*/30 * * * * Forbid 2" || exit 1
 eq "$(j -n heal job report-now '{.metadata.annotations.cronjob\.kubernetes\.io/instantiate} {.status.succeeded}')" "manual 1" """,
              hint=r"""`kubectl create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date`, then the two fields. `kubectl create job report-now --from=cronjob/report`.""",
-             show=[cmd(r"""k -n heal create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date > report.yaml"""),
+             show=r"""kubectl -n heal create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date > {WS}/lesson-3.4/report.yaml
-                   hand(r"""vim report.yaml""",
+yq -i '.spec.concurrencyPolicy = "Forbid" | .spec.successfulJobsHistoryLimit = 2' {WS}/lesson-3.4/report.yaml
-                        r"""Add to the CronJob's own `spec:`, beside `schedule:`:
+kubectl -n heal apply -f {WS}/lesson-3.4/report.yaml
-```
+kubectl -n heal create job report-now --from=cronjob/report
-  concurrencyPolicy: Forbid
+kubectl -n heal wait --for=condition=complete job/report-now --timeout=120s"""),
-  successfulJobsHistoryLimit: 2
-```""",
-                        run=r"""yq -i '.spec.concurrencyPolicy = "Forbid" | .spec.successfulJobsHistoryLimit = 2' report.yaml"""),
-                   cmd(r"""k apply -f report.yaml
-k -n heal create job report-now --from=cronjob/report
-k -n heal get job,po""", r"""Again until `report-now` shows `1/1`.""")]),
     ],
     "outro": r"""You watched a ReplicaSet replace a Pod, fixed a readiness probe, saw a liveness probe restart a container, stopped a drain with a disruption budget, and ran a Job and a CronJob.
 
@@ -578,48 +537,35 @@ mkns sched
              check=r"""
 for p in gold:Guaranteed silver:Burstable bronze:BestEffort; do eq "$(j -n sched pod "${p%%:*}" '{.status.qosClass}')" "${p##*:}" || exit 1; done""",
              hint=r"""`kubectl run NAME --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400`, then `resources:` in the container. `.status.qosClass` shows the result.""",
-             show=[cmd(r"""k -n sched run bronze --image=busybox:1.36 -- sleep 86400
+             show=r"""kubectl -n sched run bronze --image=busybox:1.36 -- sleep 86400
-k -n sched run silver --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > silver.yaml"""),
+kubectl -n sched run silver --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"silver","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"memory":"32Mi"}}}]}}'
-                   hand(r"""vim silver.yaml""",
+kubectl -n sched run gold --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"gold","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"64Mi"}}}]}}'
-                        r"""Make `resources: {}`:
+kubectl -n sched get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass"""),
-```
-    resources:
-      requests: {memory: 32Mi}
-```""",
-                        run=r"""yq -i '.spec.containers[0].resources = {"requests": {"memory": "32Mi"}}' silver.yaml"""),
-                   cmd(r"""k -n sched apply -f silver.yaml
-k -n sched run gold --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > gold.yaml"""),
-                   hand(r"""vim gold.yaml""",
-                        r"""Make `resources: {}`:
-```
-    resources:
-      requests: {cpu: 100m, memory: 64Mi}
-      limits: {cpu: 100m, memory: 64Mi}
-```""",
-                        run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "100m", "memory": "64Mi"}, "limits": {"cpu": "100m", "memory": "64Mi"}}' gold.yaml"""),
-                   cmd(r"""k -n sched apply -f gold.yaml
-k -n sched describe po | grep -e ^Name: -e QoS""")]),
         step("Only on SSD nodes",
              r"""Label `{W2}` with `disk=ssd`. Then run Pod `fast` in `sched` (`busybox:1.36`, `sleep 86400`) with a **required** node affinity for `disk In (ssd)`. It lands on `{W2}` - and if no node had the label, it would stay `Pending`.""",
              check=r"""
 eq "$(j node "$W2" '{.metadata.labels.disk}')" ssd && eq "$(j -n sched pod fast '{.spec.nodeName}')" "$W2" \
   && eq "$(j -n sched pod fast '{.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key}')" disk""",
              hint=r"""`kubectl label node {W2} disk=ssd`. The affinity block: `affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[].matchExpressions[]` with `key`, `operator: In`, `values`. The docs page "Assign Pods to Nodes using Node Affinity" has it to copy.""",
-             show=[cmd(r"""k label node {W2} disk=ssd
+             show=r"""kubectl label node {W2} disk=ssd
-k -n sched run fast --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > fast.yaml"""),
+kubectl -n sched apply -f - <<'EOF'
-                   hand(r"""vim fast.yaml""",
+apiVersion: v1
-                        r"""Add to the Pod's `spec:`, above `containers:` (the docs page *Assign Pods to Nodes using Node Affinity* has it):
+kind: Pod
-```
+metadata:
+  name: fast
+spec:
   affinity:
     nodeAffinity:
       requiredDuringSchedulingIgnoredDuringExecution:
         nodeSelectorTerms:
         - matchExpressions:
           - {key: disk, operator: In, values: [ssd]}
-```""",
+  containers:
-                        run=r"""yq -i '.spec.affinity = {"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "disk", "operator": "In", "values": ["ssd"]}]}]}}}' fast.yaml"""),
+  - name: fast
-                   cmd(r"""k -n sched apply -f fast.yaml
+    image: busybox:1.36
-k -n sched get po fast -o wide""")]),
+    command: [sleep, '86400']
+EOF
+kubectl -n sched get pod fast -o wide"""),
         step("One replica per node",
              r"""Create Deployment `spread` in `sched` (2 replicas, `busybox:1.36`, `sleep 86400`, label `app=spread`) whose Pods **refuse** to share a node with each other: a required pod anti-affinity on `app=spread`, with `topologyKey: kubernetes.io/hostname`. One Pod lands on each worker; a third replica would stay `Pending`.""",
              check=r"""
@@ -627,20 +573,32 @@ eq "$(j -n sched deploy spread '{.status.readyReplicas}')" 2 || exit 1
 eq "$(j -n sched deploy spread '{.spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey}')" kubernetes.io/hostname || exit 1
 [ "$($K -n sched get pods -l app=spread -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 2 ]""",
              hint=r"""`podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution: [{labelSelector: {matchLabels: {app: spread}}, topologyKey: kubernetes.io/hostname}]` in the Pod template. `topologySpreadConstraints` is the softer, counting way to spread.""",
-             show=[cmd(r"""k -n sched create deploy spread --image=busybox:1.36 --replicas=2 --dry-run=client -o yaml -- sleep 86400 > spread.yaml"""),
+             show=r"""kubectl -n sched apply -f - <<'EOF'
-                   hand(r"""vim spread.yaml""",
+apiVersion: apps/v1
-                        r"""Add to the Pod template's `spec:`, above `containers:` (the docs page *Assigning Pods to Nodes* has an example):
+kind: Deployment
-```
+metadata:
+  name: spread
+spec:
+  replicas: 2
+  selector:
+    matchLabels: {app: spread}
+  template:
+    metadata:
+      labels: {app: spread}
+    spec:
       affinity:
         podAntiAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
           - labelSelector:
               matchLabels: {app: spread}
             topologyKey: kubernetes.io/hostname
-```""",
+      containers:
-                        run=r"""yq -i '.spec.template.spec.affinity = {"podAntiAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": [{"labelSelector": {"matchLabels": {"app": "spread"}}, "topologyKey": "kubernetes.io/hostname"}]}}' spread.yaml"""),
+      - name: spread
-                   cmd(r"""k apply -f spread.yaml
+        image: busybox:1.36
-k -n sched get po -l app=spread -o wide""")]),
+        command: [sleep, '86400']
+EOF
+kubectl -n sched rollout status deploy spread --timeout=120s
+kubectl -n sched get pods -l app=spread -o wide"""),
         step("A node for batch work only",
              r"""Taint `{W1}` with `dedicated=batch:NoSchedule`: from now on only Pods that tolerate it are scheduled there (Pods already running stay - `NoExecute` would evict them).
 
@@ -649,19 +607,23 @@ Then run Pod `nightly` in `sched` (`busybox:1.36`, `sleep 86400`) that tolerates
 [ -n "$($K get node "$W1" -o jsonpath='{range .spec.taints[*]}{.key}={.value}:{.effect}{"\n"}{end}' | grep -x 'dedicated=batch:NoSchedule')" ] || exit 1
 eq "$(j -n sched pod nightly '{.spec.nodeName} {.status.phase}')" "$W1 Running" && eq "$(j -n sched pod nightly '{.spec.tolerations[?(@.key=="dedicated")].value}')" batch""",
              hint=r"""`kubectl taint node {W1} dedicated=batch:NoSchedule`. The toleration: `{key: dedicated, operator: Equal, value: batch, effect: NoSchedule}`.""",
-             show=[cmd(r"""k taint node {W1} dedicated=batch:NoSchedule
+             show=r"""kubectl taint node {W1} dedicated=batch:NoSchedule
-k -n sched run nightly --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > nightly.yaml"""),
+kubectl -n sched apply -f - <<'EOF'
-                   hand(r"""vim nightly.yaml""",
+apiVersion: v1
-                        r"""Add to the Pod's `spec:`, above `containers:`:
+kind: Pod
-```
+metadata:
+  name: nightly
+spec:
   nodeSelector:
     kubernetes.io/hostname: {W1}
   tolerations:
   - {key: dedicated, operator: Equal, value: batch, effect: NoSchedule}
-```""",
+  containers:
-                        run=r"""yq -i '.spec.nodeSelector = {"kubernetes.io/hostname": "{W1}"} | .spec.tolerations = [{"key": "dedicated", "operator": "Equal", "value": "batch", "effect": "NoSchedule"}]' nightly.yaml"""),
+  - name: nightly
-                   cmd(r"""k -n sched apply -f nightly.yaml
+    image: busybox:1.36
-k -n sched get po nightly -o wide""")]),
+    command: [sleep, '86400']
+EOF
+kubectl -n sched get pod nightly -o wide"""),
         step("Defaults for Pods that bring none",
              r"""Create LimitRange `defaults` in `sched`: containers that say nothing get a **request** of `50m` CPU and `32Mi` memory and a **limit** of `200m` and `128Mi`.
 
@@ -670,9 +632,7 @@ Then run Pod `plain` (`busybox:1.36`, `sleep 86400`, no resources) and look at w
 eq "$(j -n sched limitrange defaults '{.spec.limits[0].type}')" Container || exit 1
 eq "$(j -n sched pod plain '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.requests.memory} {.spec.containers[0].resources.limits.cpu} {.spec.containers[0].resources.limits.memory}')" "50m 32Mi 200m 128Mi" """,
              hint=r"""`spec.limits: [{type: Container, defaultRequest: {cpu: 50m, memory: 32Mi}, default: {cpu: 200m, memory: 128Mi}}]` - `default` is the limit.""",
-             show=[hand(r"""vim lr.yaml""",
+             show=r"""kubectl -n sched apply -f - <<'EOF'
-                        r"""No `kubectl create` for a LimitRange: paste the example from the docs page *Configure Default CPU Requests and Limits for a Namespace* and make it:
-```
 apiVersion: v1
 kind: LimitRange
 metadata:
@@ -682,37 +642,19 @@ spec:
   - type: Container
     defaultRequest: {cpu: 50m, memory: 32Mi}
     default: {cpu: 200m, memory: 128Mi}
-```""",
+EOF
-                        run=r"""cat > lr.yaml <<'EOF'
+kubectl -n sched run plain --image=busybox:1.36 -- sleep 86400
-apiVersion: v1
+kubectl -n sched get pod plain -o jsonpath='{.spec.containers[0].resources}'; echo"""),
-kind: LimitRange
-metadata:
-  name: defaults
-spec:
-  limits:
-  - type: Container
-    defaultRequest: {cpu: 50m, memory: 32Mi}
-    default: {cpu: 200m, memory: 128Mi}
-EOF"""),
-                   cmd(r"""k -n sched apply -f lr.yaml
-k -n sched run plain --image=busybox:1.36 -- sleep 86400
-k -n sched describe po plain | grep -A6 Limits""")]),
         step("Past the scheduler",
              r"""Cordon `{W2}`. Then create Pod `manual` in `sched` (`busybox:1.36`, `sleep 86400`) with **`nodeName: {W2}`**. It runs there anyway: `nodeName` hands the Pod straight to that node's kubelet, and cordoning only tells the *scheduler* to stay away.
 
 Leave `{W2}` cordoned.""",
              check=r"""eq "$(j node "$W2" '{.spec.unschedulable}')" true && eq "$(j -n sched pod manual '{.spec.nodeName} {.status.phase}')" "$W2 Running" """,
              hint=r"""`kubectl cordon {W2}`, then a Pod manifest with `spec.nodeName`. Without a running scheduler, this is also how you place a Pod by hand.""",
-             show=[cmd(r"""k cordon {W2}
+             show=r"""kubectl cordon {W2}
-k -n sched run manual --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > manual.yaml"""),
+kubectl -n sched run manual --image=busybox:1.36 --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 86400
-                   hand(r"""vim manual.yaml""",
+kubectl -n sched get pod manual -o wide
-                        r"""Add to the Pod's `spec:`, above `containers:`:
+kubectl get nodes"""),
-```
-  nodeName: {W2}
-```""",
-                        run=r"""yq -i '.spec.nodeName = "{W2}"' manual.yaml"""),
-                   cmd(r"""k -n sched apply -f manual.yaml
-k -n sched get po manual -o wide""")]),
     ],
     "outro": r"""You produced all three QoS classes, steered Pods with node affinity, kept replicas apart, reserved a node with a taint, set defaults with a LimitRange, and bypassed the scheduler.
 

cka/questions.py

@@ -667,8 +667,7 @@ _ids = [q["id"] for q in QUESTIONS]
 assert len(_ids) == len(set(_ids)), "duplicate question ids"
 from cka.solutions import S as _SOL
 for _q in QUESTIONS:
-    _s = _q.get("solution") or _SOL.get(_q["id"], "")
+    _q["solution"] = (_q.get("solution") or _SOL.get(_q["id"], "")).strip()
-    _q["solution"] = _s if isinstance(_s, list) else _s.strip()   # a list: parts (sim/bank.py cmd, hand)
     _q.setdefault("set", None)
 _missing = [q["id"] for q in QUESTIONS if not q["solution"]]
 assert not _missing, f"tasks without a solution: {_missing}"

cka/solve.py

@@ -8,10 +8,6 @@ substituted as in the task. A task without a script is only audited for a clean
 A command that reads stdin (ssh, kubectl run -i) would swallow the rest of the script: ssh -n,
 </dev/null.
 
-A task whose reference solution is in parts (sim/bank.py: cmd, hand) has no script here: the audit
-runs the parts - what is typed as it stands, and for each edit by hand its script - so the solution
-the candidate reads is the one that is proven.
-
 SOLVE: task id -> bash
 """
 
@@ -29,6 +25,142 @@ cpwait() {
 
 SOLVE = {
 # ---- 114-124: bank_storage.py
+114: r"""
+cat <<EOF | kubectl apply -f -
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata: {name: %{sc}}
+provisioner: rancher.io/local-path
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: %{pvc}, namespace: %{ns}}
+spec:
+  storageClassName: %{sc}
+  accessModes: [ReadWriteOnce]
+  resources: {requests: {storage: 300Mi}}
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  terminationGracePeriodSeconds: 2
+  volumes: [{name: data, persistentVolumeClaim: {claimName: %{pvc}}}]
+  containers:
+    - name: writer
+      image: busybox:1.36
+      command: [sh, -c, "echo %{text} > /data/keep.txt; sleep 86400"]
+      volumeMounts: [{name: data, mountPath: /data}]
+EOF
+kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
+kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt
+kubectl -n %{ns} delete pod %{pod}
+kubectl -n %{ns} delete pvc %{pvc}
+sleep 4
+kubectl get pv $(cat {WS}/q114/pv.txt) -o jsonpath='{.status.phase}' > {WS}/q114/phase.txt
+""",
+115: r"""
+cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  volumes:
+    - name: scratch
+      ephemeral:
+        volumeClaimTemplate:
+          spec:
+            storageClassName: standard
+            accessModes: [ReadWriteOnce]
+            resources: {requests: {storage: 100Mi}}
+    - name: cache
+      emptyDir: {medium: Memory, sizeLimit: 32Mi}
+  containers:
+    - name: encoder
+      image: busybox:1.36
+      command: [sleep, "86400"]
+      volumeMounts:
+        - {name: scratch, mountPath: /scratch}
+        - {name: cache, mountPath: /cache}
+EOF
+kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
+kubectl -n %{ns} get pvc -o jsonpath='{.items[0].metadata.name}' > {WS}/q115/claim.txt
+kubectl -n %{ns} get pvc %{pod}-scratch -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}' > {WS}/q115/owner.txt
+""",
+116: r"""
+kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'
+kubectl -n %{ns} scale sts %{sts} --replicas=1
+for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc --no-headers | wc -l)" = 1 ] && break; sleep 3; done
+kubectl -n %{ns} get pvc -o name | cut -d/ -f2 > {WS}/q116/claims.txt
+""",
+117: r"""
+cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: PersistentVolume
+metadata: {name: %{pv}}
+spec:
+  capacity: {storage: 1Gi}
+  accessModes: [ReadWriteOnce]
+  storageClassName: manual
+  persistentVolumeReclaimPolicy: Retain
+  hostPath: {path: /mnt/%{pv}}
+  claimRef: {namespace: %{ns}, name: %{pvc}}
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: %{pvc}, namespace: %{ns}}
+spec:
+  storageClassName: manual
+  accessModes: [ReadWriteOnce]
+  resources: {requests: {storage: 1Gi}}
+EOF
+sleep 5
+""",
+118: r"""
+kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt
+kubectl -n %{ns} scale deploy %{dep} --replicas=0
+kubectl -n %{ns} wait --for=delete pod -l app=%{dep} --timeout=120s
+kubectl -n %{ns} delete pvc %{pvc}
+cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: %{pvc}, namespace: %{ns}}
+spec:
+  storageClassName: ""
+  volumeName: %{pv}
+  accessModes: [ReadWriteOnce]
+  resources: {requests: {storage: 500Mi}}
+EOF
+kubectl -n %{ns} scale deploy %{dep} --replicas=1
+kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
+""",
+119: r"""
+printf 'reports capacity\narchive accessmode\nmedia class\n' > {WS}/q119/reasons.txt
+kubectl -n %{ns} delete pvc reports archive media
+for c in "reports 1Gi" "archive 2Gi" "media 1Gi"; do set -- $c; cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: $1, namespace: %{ns}}
+spec:
+  storageClassName: manual
+  accessModes: [ReadWriteOnce]
+  selector: {matchLabels: {use: $1}}
+  resources: {requests: {storage: $2}}
+EOF
+done
+sleep 5
+""",
+120: r"""
+for i in $(seq 1 40); do
+  M=$(kubectl -n %{ns} get events --field-selector reason=FailedScheduling -o jsonpath='{.items[-1:].message}')
+  [ -n "$M" ] && break; sleep 3
+done
+echo "$M" > {WS}/q120/event.txt
+kubectl -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
+kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
+""",
 121: r"""
 kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.metadata.finalizers[0]}' > {WS}/q121/finalizer.txt
 echo deployment/%{dep} > {WS}/q121/used-by.txt
@@ -44,6 +176,55 @@ for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.sta
 kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
 echo rancher.io/local-path > {WS}/q122/provisioner.txt
 """,
+123: r"""
+kubectl annotate sc %{sc} storageclass.kubernetes.io/is-default-class=true
+for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.status.phase}')" = Bound ] && break; sleep 3; done
+kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
+kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.storageClassName}' > {WS}/q123/class.txt
+""",
+124: r"""
+cat <<EOF | kubectl apply -f -
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata: {name: %{sc}}
+provisioner: kubernetes.io/no-provisioner
+volumeBindingMode: WaitForFirstConsumer
+---
+apiVersion: v1
+kind: PersistentVolume
+metadata: {name: %{pv}}
+spec:
+  capacity: {storage: 1Gi}
+  accessModes: [ReadWriteOnce]
+  storageClassName: %{sc}
+  persistentVolumeReclaimPolicy: Retain
+  local: {path: /mnt/disks/%{pv}}
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+        - matchExpressions: [{key: kubernetes.io/hostname, operator: In, values: [{W2}]}]
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: %{pvc}, namespace: %{ns}}
+spec:
+  storageClassName: %{sc}
+  accessModes: [ReadWriteOnce]
+  resources: {requests: {storage: 1Gi}}
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  volumes: [{name: disk, persistentVolumeClaim: {claimName: %{pvc}}}]
+  containers:
+    - name: prom
+      image: busybox:1.36
+      command: [sleep, "86400"]
+      volumeMounts: [{name: disk, mountPath: /disk}]
+EOF
+kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
+""",
 # ---- 125-128: bank_net.py
 125: r"""
 kubectl -n kube-system get cm coredns -o yaml > {WS}/q125/coredns-backup.yaml
@@ -52,6 +233,36 @@ kubectl -n kube-system get cm coredns -o json | jq --arg add "$ADD" '.data.Coref
 kubectl -n kube-system rollout restart deploy coredns
 kubectl -n kube-system rollout status deploy coredns --timeout=240s
 """,
+126: r"""
+cat <<EOF | kubectl apply -f -
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata: {name: %{gw}, namespace: %{ns}}
+spec:
+  gatewayClassName: %{gc}
+  listeners:
+    - name: https
+      protocol: HTTPS
+      port: 443
+      hostname: %{host}.example.com
+      tls:
+        mode: Terminate
+        certificateRefs: [{kind: Secret, name: %{host}-tls}]
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata: {name: %{host}, namespace: %{ns}}
+spec:
+  parentRefs: [{name: %{gw}}]
+  hostnames: [%{host}.example.com]
+  rules:
+    - matches: [{path: {type: PathPrefix, value: /api}}]
+      backendRefs: [{name: api, port: 8080}]
+    - matches: [{path: {type: PathPrefix, value: /}}]
+      backendRefs: [{name: web, port: 80}]
+EOF
+kubectl -n %{ns} delete ingress %{host}
+""",
 127: r"""
 kubectl annotate ingressclass %{old} ingressclass.kubernetes.io/is-default-class-
 kubectl annotate ingressclass %{cls} ingressclass.kubernetes.io/is-default-class=true
@@ -61,6 +272,16 @@ kubectl -n %{ns} patch ingress %{ing} --type=json -p '[
   {"op": "replace", "path": "/spec/rules/0/http/paths/1/pathType", "value": "Prefix"}]'
 kubectl apply -f {WS}/q127/blog-ingress.yaml
 """,
+128: r"""
+kubectl -n %{ns} patch ds %{ds} --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/ports/0/hostPort", "value": %{hp}}]'
+kubectl -n %{ns} rollout status ds %{ds} --timeout=240s
+yq -i '.spec.dnsPolicy = "ClusterFirstWithHostNet"' {WS}/q128/collector.yaml
+kubectl -n %{ns} delete pod %{pod} --grace-period=1
+kubectl apply -f {WS}/q128/collector.yaml
+kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=180s
+kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt
+sleep 8
+""",
 # ---- 129-131: bank_tshoot.py
 129: r"""
 ssh -n {W1} 'sudo systemctl enable --now containerd; echo "runtime-endpoint: unix:///run/containerd/containerd.sock" | sudo tee /etc/crictl.yaml >/dev/null'
@@ -100,10 +321,25 @@ sudo mkdir -p /opt/cka
 sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
   --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/cka/etcd-snapshot.db
 """,
+5: r"""
+ssh -n {W3} 'sudo tee /etc/kubernetes/manifests/static-web.yaml >/dev/null <<EOF
+apiVersion: v1
+kind: Pod
+metadata: {name: static-web, namespace: default}
+spec:
+  containers: [{name: web, image: "nginx:1.27"}]
+EOF'
+for i in $(seq 1 60); do [ "$(kubectl get pod static-web-{W3} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
+""",
 17: r"""
 ssh -n {W2} 'sudo systemctl enable --now kubelet'
 for i in $(seq 1 40); do [ "$(kubectl get node {W2} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
 """,
+18: r"""
+kubectl -n troubleshoot set image deploy payments payments=nginx:1.27
+kubectl -n troubleshoot patch svc payments -p '{"spec":{"selector":{"app":"payments"},"ports":[{"port":80,"targetPort":80}]}}'
+kubectl -n troubleshoot rollout status deploy payments --timeout=240s
+""",
 20: CPWAIT + r"""
 sudo sed -i 's/kube-schedulerr/kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
 cpwait kube-scheduler
@@ -123,6 +359,44 @@ kubectl -n kube-system rollout status deploy coredns --timeout=240s
 sudo sed -i '/--leader-elect-lease-duration=1s/d' /etc/kubernetes/manifests/kube-controller-manager.yaml
 cpwait kube-controller-manager
 """,
+39: r"""
+ssh -n {W1} "sudo sed -i 's|ca-bundle.crt|ca.crt|' /var/lib/kubelet/config.yaml; sudo systemctl restart kubelet; sudo systemctl enable kubelet"
+for i in $(seq 1 40); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
+""",
+45: CPWAIT + r"""
+sudo etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore
+sudo sed -i 's|path: /var/lib/etcd$|path: /var/lib/etcd-restore|' /etc/kubernetes/manifests/etcd.yaml
+sleep 30
+for i in $(seq 1 80); do kubectl -n default get cm %{cm} >/dev/null 2>&1 && break; sleep 3; done
+for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
+""",
+46: r"""
+N=$(kubectl get nodes -l tier=edge -o jsonpath='{.items[0].metadata.name}')
+D=$(ssh -n $N "sudo awk '/^staticPodPath:/ {print \$2}' /var/lib/kubelet/config.yaml")
+kubectl create ns %{ns} 2>/dev/null
+ssh -n $N "sudo tee $D/%{pod}.yaml >/dev/null <<EOF
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}, labels: {role: edge}}
+spec:
+  containers:
+    - name: web
+      image: nginx:1.27
+      resources: {requests: {cpu: 50m, memory: 32Mi}}
+EOF"
+for i in $(seq 1 60); do [ "$(kubectl -n %{ns} get pod %{pod}-$N -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
+""",
+51: r"""
+for r in $(kubectl -n %{ns} rollout history deploy %{dep} | awk '/^[0-9]/{print $1}'); do
+  kubectl -n %{ns} rollout history deploy %{dep} --revision=$r | grep -q 'nginx:1.25' && N=$r
+done
+kubectl -n %{ns} rollout undo deploy %{dep} --to-revision=$N
+kubectl -n %{ns} scale deploy %{dep} --replicas=3
+kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
+kubectl -n %{ns} rollout pause deploy %{dep}
+kubectl -n %{ns} set image deploy/%{dep} $(kubectl -n %{ns} get deploy %{dep} -o jsonpath='{.spec.template.spec.containers[0].name}')=nginx:1.27
+kubectl -n %{ns} set env deploy/%{dep} RELEASE=next
+""",
 57: r"""
 kubectl -n kube-system delete netpol maintenance-lockdown
 kubectl -n kube-system scale deploy coredns --replicas=2
@@ -130,6 +404,11 @@ kubectl -n kube-system get cm coredns -o json | jq '.data.Corefile |= sub("forwa
 kubectl -n kube-system rollout restart deploy coredns
 kubectl -n kube-system rollout status deploy coredns --timeout=240s
 """,
+60: r"""
+ssh -n {W2} "sudo sed -i 's|ca-wrong.crt|ca.crt|' /var/lib/kubelet/config.yaml; sudo systemctl enable --now kubelet"
+kubectl uncordon {W2}
+for i in $(seq 1 40); do [ "$(kubectl get node {W2} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
+""",
 61: CPWAIT + r"""
 sudo sed -i 's/kube-schedulerr/kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
 sudo sed -i 's|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)-broken|\1|' /etc/kubernetes/manifests/kube-controller-manager.yaml
@@ -142,6 +421,17 @@ for i in $(seq 1 80); do kubectl get nodes >/dev/null 2>&1 && break; sleep 3; do
 for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
 sudo kubeadm certs check-expiration 2>/dev/null | grep '^apiserver ' > {WS}/q65/expiry.txt
 """,
+66: CPWAIT + r"""
+sudo sed -i 's/kube-apiserverr/kube-apiserver/' /etc/kubernetes/manifests/kube-apiserver.yaml
+for i in $(seq 1 80); do kubectl get nodes >/dev/null 2>&1 && break; sleep 3; done
+for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
+""",
+67: r"""
+T=$(sudo kubeadm token create --ttl 2h)
+H=$(sudo cat /etc/kubernetes/pki/ca.crt | openssl x509 -pubkey | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')
+E=$(kubectl get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}')
+printf 'JOIN_TOKEN=%s\nCA_HASH=sha256:%s\nAPI_ENDPOINT=%s\n' "$T" "$H" "$E" > {WS}/q67/node-join.env
+""",
 70: r"""
 kubectl -n driver-onboard set image deploy/onboard-app onboard=nginx:1.25
 kubectl -n driver-onboard patch deploy onboard-app --type=json -p '[
@@ -149,6 +439,15 @@ kubectl -n driver-onboard patch deploy onboard-app --type=json -p '[
   {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/"}]'
 kubectl -n driver-onboard rollout status deploy onboard-app --timeout=240s
 """,
+71: r"""
+kubectl -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
+kubectl -n dispatch-hub rollout status deploy dispatch-api --timeout=240s
+for i in 1 2 3 4 5; do
+  kubectl -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt </dev/null 2>/dev/null
+  grep -qi nginx {WS}/q71/test-result.txt && break; kubectl -n dispatch-hub delete pod probe --ignore-not-found >/dev/null 2>&1; sleep 3
+done
+kubectl -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt
+""",
 76: r"""
 kubectl -n trip-tracker get deploy trip-logger -o json | jq '
   .spec.template.spec.volumes += [{"name": "config-volume", "emptyDir": {}}]
@@ -168,23 +467,79 @@ kubectl -n surge-pricing get deploy surge-engine -o json | jq '
   | .spec.template.spec.volumes += [{"name": "config-vol", "configMap": {"name": "surge-config"}}]' | kubectl apply -f -
 kubectl -n surge-pricing rollout status deploy surge-engine --timeout=240s
 """,
+81: r"""
+kubectl -n fare-engine delete pvc fare-data-pvc
+cat <<EOF | kubectl apply -f -
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: fare-data-pvc, namespace: fare-engine}
+spec:
+  accessModes: [ReadWriteOnce]
+  resources: {requests: {storage: 100Mi}}
+  storageClassName: standard
+EOF
+kubectl label node {W1} disk=nvme
+kubectl -n fare-engine rollout status deploy fare-calculator --timeout=240s
+""",
 84: r"""
 kubectl -n booking-tls get cm booking-nginx-config -o json | jq '.data |= with_entries(.value |= sub("ssl_protocols TLSv1.2 TLSv1.3;"; "ssl_protocols TLSv1.3;"))' | kubectl apply -f -
 kubectl -n booking-tls rollout restart deploy booking-nginx
 kubectl -n booking-tls rollout status deploy booking-nginx --timeout=240s
 echo 'ssl_protocols TLSv1.3;' > {WS}/q84/ssl-line.txt
 """,
+86: r"""
+cat > {WS}/q86/controlplane.txt <<'EOF'
+kubelet: process
+kube-apiserver: static-pod
+kube-scheduler: static-pod
+kube-controller-manager: static-pod
+etcd: static-pod
+dns: deployment coredns
+EOF
+""",
+87: r"""
+C=$(kubectl get nodes --no-headers -l node-role.kubernetes.io/control-plane | wc -l)
+W=$(kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' | wc -l)
+CIDR=$(kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -o 'service-cluster-ip-range=[^ ]*' | head -1 | cut -d= -f2)
+F=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
+printf '1: %s\n2: %s\n3: %s\n4: kindnet, %s\n5: -{W3}\n' "$C" "$W" "$CIDR" "$F" > {WS}/q87/cluster-info.txt
+""",
 88: r"""
 kubectl -n oslo-web run gatekeeper --image=busybox:1.36 --dry-run=client -o json -- sleep 86400 | jq '.spec.containers[0].readinessProbe = {"exec": {"command": ["sh", "-c", "wget -T2 -qO- http://ledger-api:8080"]}, "initialDelaySeconds": 5, "periodSeconds": 5}' | kubectl apply -f -
 sleep 12
 kubectl -n oslo-web expose deploy ledger --name=ledger-api --port=8080 --target-port=80
 kubectl -n oslo-web wait --for=condition=Ready pod/gatekeeper --timeout=180s
 """,
+90: r"""
+cat > {WS}/q90/cluster-events.sh <<'EOF'
+kubectl get events -A --sort-by=.metadata.creationTimestamp
+EOF
+chmod +x {WS}/q90/cluster-events.sh
+P=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].metadata.name}')
+kubectl -n kube-system delete pod $P
+for i in $(seq 1 40); do
+  [ "$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].status.phase}' 2>/dev/null)" = Running ] && break; sleep 3
+done
+sleep 5
+kubectl -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
+ssh -n {W2} 'sudo crictl stop $(sudo crictl ps --name kube-proxy -q)'
+sleep 25
+kubectl -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
+echo container > {WS}/q90/restarts.txt
+""",
 92: r"""
 ssh -n {W1} "sudo sed -i 's|^ExecStart=/usr/local/bin/kubelet |ExecStart=/usr/bin/kubelet |' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; sudo systemctl daemon-reload; sudo systemctl restart kubelet"
 echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q92/fixed-file.txt
 for i in $(seq 1 40); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
 """,
+94: r"""
+kubectl run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon --dry-run=client -o yaml \
+  | yq '.spec.containers[0].resources = {"requests": {"cpu": "10m", "memory": "20Mi"}}' | sudo tee /etc/kubernetes/manifests/ops-beacon.yaml >/dev/null
+for i in $(seq 1 60); do [ "$(kubectl -n default get pod ops-beacon-{CP} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
+kubectl -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
+kubectl -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt
+sleep 5
+""",
 95: r"""
 CI=$(ssh -n {W2} 'sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -issuer' | sed 's/^issuer= *//')
 SI=$(ssh -n {W2} 'sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -issuer' | sed 's/^issuer= *//')
@@ -232,11 +587,47 @@ kubectl -n %{ns} patch deploy %{dep} --type=json -p '[
   {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/%{ready}"}]'
 kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
 """,
+107: r"""
+ssh -n {W3} "printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/%{name}.conf >/dev/null
+printf 'net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\n' | sudo tee /etc/sysctl.d/%{name}.conf >/dev/null"
+IPF=$(ssh -n {W3} 'sudo sysctl -n net.ipv4.ip_forward')
+SW=off; [ "$(ssh -n {W3} 'sed 1d /proc/swaps | grep -c .')" -gt 0 ] && SW=on
+FS=$(ssh -n {W3} "sudo awk '/^failSwapOn:/ {print \$2}' /var/lib/kubelet/config.yaml"); [ -n "$FS" ] || FS=true
+CG=$(ssh -n {W3} "sudo awk '/^cgroupDriver:/ {print \$2}' /var/lib/kubelet/config.yaml")
+printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s\n' "$IPF" "$SW" "$FS" "$CG" > {WS}/q107/node.txt
+""",
+108: r"""
+V=$(ssh -n {W2} 'sudo crictl version 2>/dev/null')
+RN=$(echo "$V" | awk '/^RuntimeName:/ {print $2}'); RV=$(echo "$V" | awk '/^RuntimeVersion:/ {print $2}')
+SOCK=$(ssh -n {W2} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
+CID=$(kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#^[a-z-]*://##')
+PID=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.info.pid}}' $CID 2>/dev/null")
+LOG=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.status.logPath}}' $CID 2>/dev/null")
+printf 'runtime=%s %s\nsocket=%s\ncontainer-id=%s\npid=%s\nlog=%s\n' "$RN" "$RV" "$SOCK" "$CID" "$PID" "$LOG" > {WS}/q108/runtime.txt
+""",
 110: r"""
 kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/^controlPlaneEndpoint:/{print $2}' | tr -d '"' > {WS}/q110/endpoint.txt
 KEY=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
 sudo kubeadm token create --print-join-command --certificate-key $KEY 2>/dev/null | grep 'kubeadm join' > {WS}/q110/join-control-plane.sh
 """,
+111: r"""
+E='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
+N=$(kubectl get nodes -l node-role.kubernetes.io/control-plane --no-headers | wc -l)
+M=$($E member list | wc -l)
+L=$($E member list | awk -F', ' 'NR==1 {print $3}')
+S=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
+C=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
+printf 'control-plane-nodes=%s\netcd-topology=stacked\netcd-members=%s\netcd-leader=%s\nscheduler-leader=%s\ncontroller-manager-leader=%s\n' "$N" "$M" "$L" "$S" "$C" > {WS}/q111/ha.txt
+""",
+112: r"""
+CONF=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
+PL=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[].type] | join(",")')
+IPAM=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[] | select(.ipam) | .ipam.type][0]')
+SOCK=$(ssh -n {W1} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
+N=$(kubectl get csidrivers --no-headers 2>/dev/null | wc -l)
+PROV=$(kubectl get sc -o json | jq -r '.items[] | select(.metadata.annotations["storageclass.kubernetes.io/is-default-class"] == "true") | .provisioner')
+printf 'cni-config=%s\ncni-plugins=%s\ncni-ipam=%s\ncri-socket=%s\ncsi-drivers=%s\ndefault-provisioner=%s\n' "$CONF" "$PL" "$IPAM" "$SOCK" "$N" "$PROV" > {WS}/q112/interfaces.txt
+""",
 19: r"""
 for i in $(seq 1 40); do kubectl -n troubleshoot logs crasher --previous 2>/dev/null | grep -q DB_HOST && break; sleep 3; done
 kubectl -n troubleshoot logs crasher --previous > {WS}/q19/crasher.log
@@ -253,6 +644,71 @@ KV=$(kubectl get node %{node} -o jsonpath='{.status.nodeInfo.kubeletVersion}')
 R=$(kubectl get node %{node} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}')
 printf 'apiserver=%s\netcd=%s\nkubelet=%s\nruntime=%s\n' "$A" "$E" "$KV" "$R" > {WS}/q41/versions.txt
 """,
+59: r"""
+kubectl -n %{ns} apply -f - <<EOF
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: new-data, namespace: %{ns}}
+spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 1Gi}}}
+---
+apiVersion: batch/v1
+kind: Job
+metadata: {name: migrate, namespace: %{ns}}
+spec:
+  template:
+    spec:
+      restartPolicy: Never
+      volumes:
+        - {name: src, persistentVolumeClaim: {claimName: old-data}}
+        - {name: dst, persistentVolumeClaim: {claimName: new-data}}
+      containers:
+        - name: cp
+          image: busybox:1.36
+          command: [sh, -c, "cp -a /src/. /dst/ && ls /dst"]
+          volumeMounts: [{name: src, mountPath: /src}, {name: dst, mountPath: /dst}]
+EOF
+kubectl -n %{ns} wait --for=condition=complete job/migrate --timeout=240s
+kubectl -n %{ns} get pod legacy -o json | jq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .status)
+  | (.spec.volumes[] | select(.persistentVolumeClaim.claimName == "old-data") | .persistentVolumeClaim.claimName) = "new-data"' > /tmp/legacy.json
+kubectl -n %{ns} delete pod legacy --now
+kubectl apply -f /tmp/legacy.json
+kubectl -n %{ns} delete pod -l job-name=migrate
+kubectl -n %{ns} delete pvc old-data
+kubectl -n %{ns} wait --for=condition=Ready pod/legacy --timeout=180s
+""",
+77: r"""
+kubectl -n kube-system get ds -o name | sed 's|.*/||' | grep -i kindnet > {WS}/q77/cni.txt
+kubectl -n zero-trust run srv --image=nginx:1.27 --labels=app=srv
+kubectl -n zero-trust expose pod srv --port 80
+kubectl -n zero-trust run cli --image=busybox:1.36 -- sleep 3600
+kubectl -n zero-trust wait --for=condition=Ready pod/srv pod/cli --timeout=180s
+kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null && echo reachable
+cat <<EOF | kubectl apply -f -
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: default-deny-all, namespace: zero-trust}
+spec:
+  podSelector: {}
+  policyTypes: [Ingress, Egress]
+EOF
+sleep 6
+if kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null 2>&1; then echo no; else echo yes; fi > {WS}/q77/enforced.txt
+kubectl -n zero-trust delete pod srv cli --now
+kubectl -n zero-trust delete svc srv
+""",
+91: r"""
+kubectl api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q91/namespaced.txt
+for ns in census-north census-south census-east census-west; do echo "$ns $(kubectl -n $ns get role --no-headers | wc -l)"; done | sort -k2 -n | tail -1 > {WS}/q91/most-roles.txt
+""",
+102: r"""
+kubectl -n oslo-files get pvc shared-files -o json | jq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .metadata.annotations, .metadata.finalizers, .status)
+  | .spec.accessModes = ["ReadWriteOnce"]' > /tmp/pvc.json
+kubectl -n oslo-files delete pvc shared-files
+kubectl apply -f /tmp/pvc.json
+kubectl -n oslo-files rollout restart deploy uploader
+kubectl -n oslo-files rollout status deploy uploader --timeout=240s
+echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
+""",
 93: r"""
 kubectl -n oslo-secure create secret generic bundle --from-file={WS}/q93/tls-bundle.pem
 kubectl -n oslo-secure create secret generic db-login --from-literal=user=svc-ledger --from-literal=pass='%{pw}'
@@ -263,14 +719,48 @@ kubectl -n oslo-secure run vault-reader --image=busybox:1.36 --dry-run=client -o
                                {"name": "APP_PASS", "valueFrom": {"secretKeyRef": {"name": "db-login", "key": "pass"}}}]' | kubectl apply -f -
 kubectl -n oslo-secure wait --for=condition=Ready pod/vault-reader --timeout=180s
 """,
+# ---- 132-145: bank_course.py (the others are solved by their solution as written, below)
+132: r"""
+apt_get() { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq -y -o Dpkg::Options::=--force-confold "$@" </dev/null >/dev/null; }
+sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
+apt_get update
+sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null
+sudo kubeadm upgrade apply v1.37.1 -y </dev/null | tail -2
+kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
+sudo apt-mark unhold kubelet kubectl >/dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null
+sudo systemctl daemon-reload && sudo systemctl restart kubelet
+for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done
+kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data
+ssh {W1} sudo bash <<'EOF'
+export DEBIAN_FRONTEND=noninteractive
+sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
+apt-get update -qq </dev/null
+apt-mark unhold kubeadm kubelet kubectl >/dev/null
+apt-get install -y -qq kubeadm=1.37.1-1.1 </dev/null >/dev/null
+kubeadm upgrade node </dev/null | tail -1
+apt-get install -y -qq kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 </dev/null >/dev/null
+apt-mark hold kubeadm kubelet kubectl >/dev/null
+systemctl daemon-reload && systemctl restart kubelet
+EOF
+kubectl uncordon {W1}
+""",
+139: r"""
+ssh {W1} sudo bash <<'EOF'
+yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
+systemctl restart kubelet
+EOF
+for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
+""",
 }
 
 # ---- 1-113: the banks carried over from cka-sim.
 # For these tasks the reference solution is a script as it stands, so the audit runs exactly the
 # text the candidate is shown - and a solution that stops working is found the next time it runs.
-AS_WRITTEN = [1, 21, 40, 55, 69, 75, 85, 98, 104, 109,
+AS_WRITTEN = [1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
+              40, 42, 43, 44, 47, 48, 49, 50, 52, 53, 54, 55, 56, 58, 62, 63, 64, 68, 69, 72, 73, 74, 75, 78, 80, 82, 83,
+              85, 89, 98, 99, 100, 101, 103, 104, 109, 113,
               # bank_course.py: written to run as they stand
-              133]
+              133, 134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 145, 146, 147]
 
 
 def _as_written():

ksim

@@ -22,7 +22,7 @@ from collections.abc import Callable
 from datetime import datetime
 from typing import Any
 
-VERSION = "2026.10.27.2"
+VERSION = "2026.10.26.3"
 ROOT = os.path.dirname(os.path.abspath(__file__))
 sys.path.insert(0, ROOT)
 from sim import desktop, envs  # noqa: E402
@@ -96,11 +96,7 @@ def load_track(track_id):
         q.setdefault("set", None)
         if q["id"] in workers:
             q["workers"] = workers[q["id"]]
-        if isinstance(q.get("solution"), list):   # a solution in parts: the audit runs what it shows
+        if q["id"] in solve:
-            if q["id"] in solve:
-                sys.exit(f"{track_id} task {q['id']}: its solution is in parts, which the audit runs - drop its script from {track_id}/solve.py")
-            q["solve"] = show_script({"show": q["solution"]})
-        elif q["id"] in solve:
             q["solve"] = solve[q["id"]]
         # Every task has a cluster to itself, so nothing it breaks can reach another task: the flags
         # that kept such tasks apart on a shared cluster ("breaks", "solo", "disruptive") are void.
@@ -128,29 +124,7 @@ def lesson_task(l, objectives):
     o = next((o for o in objectives if o["id"] == l["objective"]), None)
     return dict(l, lesson=True, domain=o["domain"] if o else "", difficulty="normal", set=None,
                 checks=[{"name": s["title"], "pts": 1, "script": s["check"]} for s in l["steps"]], weight=len(l["steps"]),
-                task=l["intro"], solution=[p for s in l["steps"] for p in show_parts(s)])
+                task=l["intro"], solution="\n\n".join(s["show"] for s in l["steps"] if s["show"]))
-
-
-def show_parts(s):
-    """How to do a step, as parts (sim/bank.py: cmd, hand): commands typed as they stand, and what
-    is done by hand - an edit, a value read off the screen - with the script the audit runs for it."""
-    sh = s.get("show") or ""
-    return sh if isinstance(sh, list) else ([{"type": sh, "how": "", "run": sh}] if sh.strip() else [])
-
-def show_md(s):
-    """How to do a step, as you read it: each thing to type in a block of its own (a click copies
-    it), and beneath it what to do there by hand."""
-    return "\n\n".join(f"```\n{p['type'].strip(chr(10))}\n```" + (f"\n\n{p['how'].strip()}" if p["how"] else "")
-                       for p in show_parts(s))
-
-def show_script(s):
-    """What the audit runs for a step: what you type, and for each part done by hand its script."""
-    return "\n".join(p["run"] for p in show_parts(s))
-
-def solution_parts(q):
-    """A task's reference solution as parts, like a lesson step's: a solution in parts (cmd, hand)
-    as it stands, one written as a single script as one part."""
-    return show_parts({"show": q.get("solution") or ""})
 
 
 def set_questions(name):
@@ -464,9 +438,7 @@ def task_text(q, params=None):
     return prose(q["task"], params if params is not None else session_params(load_state(), q), q)
 
 def solution_text(q, params=None):
-    """A task's reference solution as you read it (Markdown): each thing to type in a block of its
+    return prose(str(q.get("solution", "")), params if params is not None else session_params(load_state(), q), q)
-    own, and beneath it what to do there by hand."""
-    return prose(show_md({"show": solution_parts(q)}), params if params is not None else session_params(load_state(), q), q)
 
 def make_params(q, rng):
     fn = q.get("params")
@@ -776,15 +748,12 @@ def solution_allowed(st: dict[str, Any] | None, qid: int | str) -> bool:
 def cmd_solution(args):
     if not args:
         die("usage: ./ksim solution N   (drills: after grading the task; exams: after submit; --force to peek anyway)")
-    if args[0] == "lint":   # no cluster needed, and no solution shown: only where one has you type a script
-        names = [a.upper() for a in args[1:]]
-        return print_solution_lint(solution_lint([q for q in QUESTIONS if not names or str(q["id"]) in names or q["domain"] in names]))
     force = "--force" in args; args = [a for a in args if a != "--force"]
     q = q_by_id(int(args[0])); st = load_state()
     if not solution_allowed(st, int(q["id"])) and not force:
         die("Not yet: in a drill, grade the task first (./ksim grade N); in an exam, submit first. --force overrides.")
     print(bold(f"Solution - Q{q['id']} {q['title']}")); print("-" * 64)
-    print(cli_md(solution_text(q, session_params(st, q)))); print("-" * 64)
+    print(solution_text(q, session_params(st, q))); print("-" * 64)
     print(dim(f"One correct way, not the only one. $do = --dry-run=client -o yaml, k = kubectl. Run it on the task's host: ssh {host_of(q)}"))
 
 def cmd_progress(args):
@@ -1597,9 +1566,7 @@ def begin_session(qs, mode, duration, force, log: Callable[..., Any] = _log_msg,
     st = {"mode": mode, "track": TRACK_ID, "hard": hard, "ids": [q["id"] for q in qs],
           "params": params_by_id, "duration": duration, "started_at": time.time(), "ended_at": None,
           "k8s": k8s_version(), "results": {}, "flags": [], "notes": "",
-          "objective": objective,   # the curriculum objective a drill or a lesson was started for, if any
+          "objective": objective}   # the curriculum objective a drill or a lesson was started for, if any
-          # the tasks or lessons whose host failed its setup: the page and ./ksim lesson say so, and how to rebuild it
-          "broken": [q["id"] for q in qs if any(p.startswith(host_of(q) + ":") for p in problems)]}
     lesson = mode == "lesson"
     if lesson:
         st["steps"] = {}   # lesson id -> step index -> its latest check (a pass sticks)
@@ -1775,9 +1742,6 @@ def cli_md(text):
 def print_step(l, params, i, st=None):
     s = l["steps"][i]
     done = bool(st and steps_of(st, l).get(str(i), {}).get("pass"))
-    if st and l["id"] in st.get("broken", []):
-        print(red(f"This lesson's host failed its setup when the session started: it is not as the steps describe.\n"
-                  f"  Rebuild it (about a minute):  ./ksim reset {l['id']}"))
     print(bold(f"Step {i + 1} of {len(l['steps'])}: {render(s['title'], params, l)}") + (green("   ✓ checked") if done else ""))
     print(cli_md(render(s["text"], params, l)))
     print(dim("  ./ksim lesson check   when it is done        ./ksim lesson hint / show   if you are stuck"))
@@ -1806,9 +1770,6 @@ def cmd_lesson(args):
     sub = args[0] if args else ""
     if sub == "audit":
         return lesson_audit(args[1:])
-    if sub == "lint":   # no cluster needed: what each step has you type, against TYPING_TRAPS
-        names = [a for a in args[1:] if not a.startswith("--")]
-        return print_lint(lesson_lint(LESSONS if not names else [x for x in LESSONS if x["objective"] in names or x["domain"] in names]))
     st = load_state()
     live = lesson_of(st) if st and not st.get("ended_at") else None
     if sub == "next":
@@ -1833,9 +1794,9 @@ def cmd_lesson(args):
         if sub == "step":
             return print_step(live, params, i, st)
         if sub in ("hint", "show"):
-            print(bold(f"{'Hint' if sub == 'hint' else 'How to do it'} - step {i + 1}: {render(s['title'], params, live)}"))
+            print(bold(f"{'Hint' if sub == 'hint' else 'The commands'} - step {i + 1}: {render(s['title'], params, live)}"))
-            text = render(s["hint"] if sub == "hint" else show_md(s), params, live)
+            text = render(s[sub], params, live)
-            print(cli_md(text) or dim("Nothing for this step."))
+            print((cli_md(text) if sub == "hint" else text) or dim("Nothing for this step."))
             return
         r, why = lesson_check(st, i)
         if why:
@@ -1879,51 +1840,6 @@ def cmd_lesson(args):
     print(dim(f"\n  ./ksim lesson {LESSONS[0]['objective']}   starts that lesson      ./ksim lesson audit   proves them on real clusters"))
     print(dim(f"  ./ksim lesson {LESSONS[0]['domain']}   a domain's lessons not done yet, back to back ({runs} done)"))
 
-# What a step must not ask you to type: in the exam you have a keyboard, vim and the docs, and no
-# time to compose scripts. Such things belong in the script of a part done by hand (sim/bank.py).
-TYPING_TRAPS = [(re.compile(p, re.M), why) for p, why in (
-    (r"\b(sed|awk|yq|jq|perl|python3?)\b", "a text tool - edit in vim, or read the value and echo it"),
-    (r"<<", "a heredoc - generate the YAML, or paste the docs' example into vim"),
-    (r"^\s*(sleep|for|while|until)\b", "a pause or a loop - look again yourself"),
-    (r"\b(k|kubectl)\b[^|\n]* wait\b", "kubectl wait - look with get"),
-    (r"--overrides", "--overrides - generate the YAML and edit it"),
-    (r"\bpatch\b[^\n]*(-p|--patch)[ =]'[\[{]", "a JSON patch - kubectl edit"),
-    (r"jsonpath[^\n|]*>", "jsonpath into a file - read the value, then echo it"),
-    (r"\$\(", "command substitution - read the value, then type it"),
-    (r"(^|[|;&(]\s*)(sudo\s+)?kubectl\b", "kubectl in full - the exam has k"))]   # the command, not the package
-
-def lesson_lint(ls):
-    """What each step has you type, against TYPING_TRAPS. Returns [(lesson, step number, why, line)]."""
-    out = []
-    for l in ls:
-        for i, s in enumerate(l["steps"]):
-            for p in show_parts(s):
-                for line in p["type"].splitlines():
-                    for rx, why in TYPING_TRAPS:
-                        if rx.search(line):
-                            out.append((l, i + 1, why, line.strip()))
-    return out
-
-def print_lint(problems):
-    for l, n, why, line in problems:
-        print(f"  {yellow(l['objective'] + ' step ' + str(n))}  {why}\n      {dim(line[:150])}")
-    steps = len({(l["id"], n) for l, n, _, _ in problems})
-    print(bold(f"{steps} step(s) ask you to type what you would not type in the exam.") if steps
-          else green("Nothing a step asks you to type is a script: every step is done as in the exam."))
-
-def solution_lint(qs):
-    """What each task's reference solution has you type, its comments aside, against TYPING_TRAPS.
-    Returns [(task, why, line)]."""
-    return [(q, why, line.strip()) for q in qs for p in solution_parts(q) for line in p["type"].splitlines()
-            if not line.lstrip().startswith("#") for rx, why in TYPING_TRAPS if rx.search(line)]
-
-def print_solution_lint(problems):
-    for q, why, line in problems:
-        print(f"  {yellow('Q' + str(q['id']))}  {why}\n      {dim(line[:150])}")
-    tasks = len({q["id"] for q, _, _ in problems})
-    print(bold(f"{tasks} solution(s) ask you to type what you would not type in the exam.") if tasks
-          else green("Nothing a solution asks you to type is a script: every one is done as in the exam."))
-
 def lesson_audit(args):
     """Prove lessons on real clusters: each is seeded on a host of its own; then every step must
     fail its check, have its commands run as `candidate`, and pass its check - so a lesson can be
@@ -1946,8 +1862,6 @@ def lesson_audit(args):
     open(lock, "w").close()
     import atexit
     atexit.register(lambda: os.path.exists(lock) and os.remove(lock))
-    print(bold(f"What the {len(ls)} lesson(s) have you type:"))
-    print_lint(lesson_lint(ls))
     print(bold(f"Lesson audit of {len(ls)} {TRACK_ID.upper()} lesson(s), {PARALLEL} at a time - each step must fail, be done by its own commands, then pass."))
     def one(l):
         params = make_params(l, random.Random(l["id"])); host = host_of(l); t0 = time.time()
@@ -1969,7 +1883,7 @@ def lesson_audit(args):
                 if check_step(l, params, i)["pass"]:
                     res.update(verdict=f"step {i + 1} passes before its commands")
                     break
-                _, sout = solve_on_host(l, params, script=show_script(s))
+                _, sout = solve_on_host(l, params, script=s["show"])
                 # what the commands created may still be starting: a learner would watch it come up
                 r, waited = check_step(l, params, i), 0
                 while not r["pass"] and waited < 90:
@@ -2241,24 +2155,13 @@ def cmd_history(args):
                 cells.append(f"{round(100 * de / dt):>3}%" if dt else "   -")
             print(f"  {d:<7} " + " → ".join(cells))
 
-def rebuild(st, q, log: Callable[..., Any] = _log_msg):
-    """A task's or a lesson's host again from scratch, scenario included. A lesson's steps start over
-    on it; a host that comes up whole is no longer listed as broken. Returns what went wrong, or None."""
-    problem = provision(q, session_params(st, q), log=log)
-    st = load_state() or st
-    if q.get("lesson"):
-        st.get("steps", {}).pop(str(q["id"]), None)
-    st["broken"] = [i for i in st.get("broken", []) if i != q["id"]] + ([q["id"]] if problem else [])
-    save_state(st)
-    return problem
-
 def cmd_reset(args):
     """Throw a task's cluster away and build it again, scenario included."""
     if not args:
         die("usage: ./ksim reset N   (rebuild one task's host from scratch: new cluster, scenario seeded again)")
     st = require_started(); q = q_by_id(int(args[0]))
     print(dim(f"  rebuilding {host_of(q)} - about a minute..."))
-    problem = rebuild(st, q)
+    problem = provision(q, session_params(st, q))
     print(("rebuilt " if not problem else "problem rebuilding ") + f"Q{q['id']:02d}: {q['title']}")
 
 # --------------------------------------------------------------- web ----
@@ -2427,9 +2330,7 @@ class Api(http.server.BaseHTTPRequestHandler):
                         return self._json({"error": down_message(down)}, 503)
                     finish_session(st, log=lambda *_: None)
             elif p.path == "/api/reset":
-                problem = rebuild(st, q_by_id(int(body["id"])), log=lambda *_: None)
+                q = q_by_id(int(body["id"])); provision(q, session_params(st, q), log=lambda *_: None)
-                if problem:
-                    return self._json({"error": f"Rebuilding failed again ({problem}) - try ./ksim reset {body['id']} in a terminal to see why."}, 500)
             else:
                 return self._json({"error": "not found"}, 404)
             return self._json(self.session_payload())
@@ -2467,7 +2368,6 @@ class Api(http.server.BaseHTTPRequestHandler):
                                   "paused": bool(st.get("paused_at")), "paused_total": paused_total(st),
                                   "notes": st.get("notes", ""),
                                   "objective": st.get("objective"), "milestones": st.get("milestones", []), "aborted": bool(st.get("aborted")),
-                                  "broken": st.get("broken", []),
                                   "results": st["results"], "score": {"earned": earned, "total": total, "pct": pct, "pass_pct": PASS_PCT},
                                   "domains": self.domain_scores(st), "server_time": time.time()}
             l = lesson_of(st)
@@ -2489,7 +2389,7 @@ class Api(http.server.BaseHTTPRequestHandler):
         r = lambda t: render(t, params, l).strip("\n")
         return {"id": l["id"], "objective": l["objective"], "title": l["title"], "host": host_of(l), "nodes": 1 + workers_of(l),
                 "intro": r(l["intro"]), "outro": r(l["outro"]),
-                "steps": [{"title": r(s["title"]), "text": r(s["text"]), "hint": r(s["hint"]), "show": r(show_md(s))} for s in l["steps"]]}
+                "steps": [{"title": r(s["title"]), "text": r(s["text"]), "hint": r(s["hint"]), "show": r(s["show"])} for s in l["steps"]]}
 
     def path_payload(self) -> dict[str, Any]:
         """The curriculum as the UI shows it: every objective with its notes, tasks and state, how
@@ -2799,8 +2699,7 @@ def cmd_help(args):
         "  lesson DOMAIN [--all]    teach a domain: its lessons not done yet (--all: every one) back to back, hosts made up front\n"
         "  lesson next | OBJ        in a domain's run: on to its next lesson, or to any lesson of it\n"
         "  lesson step|check|hint|show [N]   the step you are on, check it, a hint, the commands;  ./ksim end finishes\n"
-        "  lesson audit [OBJ..]     prove lessons: every step fails before its commands and passes after them\n"
+        "  lesson audit [OBJ..]     prove lessons: every step fails before its commands and passes after them\n")
-        "  lesson lint [OBJ|DOMAIN..]   what the steps have you type: no scripts, only what you would type in the exam\n")
     print(f"""usage: ./ksim [--track cka|ckad|cks] <command>        (current track: {TRACK_ID}, {len(QUESTIONS)} tasks)
        Every task runs on its own cluster at the exam's Kubernetes version ({TRACK_ID}: {k8s_version()}),
        reached with `ssh <host>` from the remote desktop.
@@ -2815,7 +2714,6 @@ def cmd_help(args):
                            whatever else you named: ./ksim next OBS:3
   progress                attempted/passed per domain, normal and hard
   solution N               reference solution (drills: after grading N; exams: after submit)
-  solution lint [N|DOMAIN..]   what the solutions have you type: no scripts, only what you would type in the exam
   sim {'|'.join(SETS) or '<set>'}{' ' * max(1, 20 - len('|'.join(SETS)))}fixed simulator set, exam rules, {TRACK['minutes']} min
   start [--hard]           full exam: {sum(QUOTA.values()) or '?'} tasks sampled by domain ({sum(QUOTA_HARD.values()) or '?'} in hard mode), {TRACK['minutes']} min.
                            Requires every domain drilled once first (--force skips). Hard mode adds traps,

sim/bank.py

@@ -21,22 +21,10 @@ def C(name, pts, script):
 
 def step(title, text, check, hint="", show=""):
     """One step of a lesson (<track>/lessons.py): what to do (Markdown), the check that proves it
-    (bash, exit 0 = done, run like a task's checks), a hint, and how to do it as you would in the
+    (bash, exit 0 = done, run like a task's checks), a hint, and the commands that do it - shown on
-    exam - shown on request, and done by ./ksim lesson audit to prove the step works as shown.
+    request, and run by ./ksim lesson audit to prove the step can be done as shown."""
-    `show` is commands you type as they stand, or a list of cmd() and hand() parts."""
     return {"title": title, "text": text, "check": check, "hint": hint, "show": show}
 
-def cmd(text, how=""):
-    """Commands you type as they stand, and what to look for (Markdown); the audit runs them as
-    they stand."""
-    return {"type": text, "how": how, "run": text}
-
-def hand(type_, how, run):
-    """What you do by hand: what you type to begin (`vim f`, `k edit ...`, `echo NAME > f`), what
-    to do there (Markdown), and a script that does the same, which the audit runs instead - an
-    editor needs a person, and so does a value read off the screen."""
-    return {"type": type_, "how": how, "run": run}
-
 def ns_pick(r, base):
     """Namespace name with a random suffix so the same task never lands on the same namespace twice."""
     return f"{base}-{r.choice(['alpha', 'bravo', 'delta', 'echo', 'kilo', 'nova', 'omega', 'zulu'])}"

web/index.html

@@ -607,10 +607,8 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
   // minimal markdown: paragraphs, **bold**, `code` (click to copy), ```fenced blocks``` (kept as
   // written, click to copy), and a paragraph made only of "- " lines as a list. Other lists stay as
   // literal lines.
-  // `code` first, so that the stars of a cron schedule in it are not read as **bold** or *italics*
+  const inlineMd = s => esc(s).replace(/\*\*(.+?)\*\*/g, '<b>$1</b>')
-  const inlineMd = s => s.split(/(`[^`]+`)/).map((p, i) => i % 2
+    .replace(/`([^`]+)`/g, (m, c) => `<code class="ic" title="click to copy">${c}</code>`);
-    ? `<code class="ic" title="click to copy">${esc(p.slice(1, -1))}</code>`
-    : esc(p).replace(/\*\*(.+?)\*\*/g, '<b>$1</b>').replace(/(^|[^\w*])\*(?!\s)([^*]+?)\*(?![\w*])/g, '$1<i>$2</i>')).join('');
   const render = md => md.split(/^```[^\n]*\n([\s\S]*?)\n?^```[ \t]*$/m).map((part, i) => i % 2
     ? `<pre class="code" title="click to copy">${esc(part)}</pre>`
     : part.split(/\n\s*\n/).filter(p => p.trim()).map(p => /^(- .*(\n|$))+$/.test(p.trim() + '\n')
@@ -927,7 +925,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
       $('#startBody').innerHTML = `<div class="alert alert-info"><b>No clock.</b> Take your time: the lessons teach, the drills test.</div>
         <ul>
           <li>every lesson has a host of its own, and all ${n} are up already: the top of each lesson says which to <code class="ic">ssh</code> to;</li>
-          <li>a lesson is an introduction, then steps - each with a <b>Check step</b> button, a hint, and <b>Show me how</b>: the exam way, if you are stuck; steps count in order;</li>
+          <li>a lesson is an introduction, then steps - each with a <b>Check step</b> button, a hint, and the commands if you are stuck; steps count in order;</li>
           <li>a lesson's wrap-up leads to the next one, and the list at the top goes to any lesson of the run;</li>
           <li>a lesson counts as done the moment its last step is checked; nothing goes into your history or scores;</li>
           <li>finish the run from <b>Controls</b>, then drill ${esc(C.domain)} against the clock.</li>
@@ -941,7 +939,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
       $('#startBody').innerHTML = `<div class="alert alert-info"><b>No clock.</b> Take your time: a lesson teaches, the drills test.</div>
         <ul>
           <li>it runs on a host of its own: <code class="ic">${esc('ssh ' + L.host)}</code> in the desktop's terminal;</li>
-          <li>read the introduction, then do the ${L.steps.length} steps - each has a <b>Check step</b> button, a hint, and <b>Show me how</b>: the exam way, if you are stuck;</li>
+          <li>read the introduction, then do the ${L.steps.length} steps - each has a <b>Check step</b> button, a hint, and the commands if you are stuck;</li>
           <li>steps count in order, and nothing goes into your history or scores;</li>
           <li>the wrap-up sends you to the objective's timed drill.</li>
         </ul>`;
@@ -1017,7 +1015,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
     const r = sess.results[q.id], show = r && (sess.mode === 'drill' || sess.ended_at);
     const oids = q.objectives || [], oid = oids.includes(sess.objective) ? sess.objective : oids[0];
     const obj = oid && (S.objectives || []).find(o => o.id === oid);   // which objective a task belongs to is a hint: it follows the solution's rule
-    $('#qContent').innerHTML = brokenHost(q.id) + `
+    $('#qContent').innerHTML = `
       <div class="qhead">
         <div class="line"><b>Solve this question on instance:</b> <code class="ic" title="click to copy">ssh ${esc(q.host)}</code></div>
         <div class="line sub">${q.nodes}-node cluster · task files in ${esc(S.workspace)} on that host</div>
@@ -1030,10 +1028,9 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
       <div class="qbody">${render(q.task)}</div>
       ${on && sess.mode === 'drill' ? '<div class="checkrow"><button class="btn btn-outline-primary btn-sm" id="btnCheck">Check this task</button><span class="muted">grades it per check; its reference solution opens once it is graded</span></div>' : ''}
       ${show ? `<div class="checks"><b>${r.earned}/${r.total} points</b>${r.checks.map(c => `<div class="${c.pass ? 'ok' : 'no'}">${esc(c.name)} <span class="pts">(${c.pts})</span></div>`).join('')}</div>` : ''}
-      ${q.solution ? `<details class="fold"><summary>Reference solution</summary><div class="qbody" style="margin-top:10px">${render(q.solution)}</div></details>` : ''}
+      ${q.solution ? `<details class="fold"><summary>Reference solution</summary><pre>${esc(q.solution)}</pre></details>` : ''}
       ${obj ? `<details class="fold"><summary>Objective ${obj.id} · ${esc(obj.text)}</summary><div class="notes">${notesHtml(obj)}</div></details>` : ''}`;
     copyable($('#qContent'));
-    wireRebuild();
     const bc = $('#btnCheck'); if (bc) bc.onclick = checkTask;
     const fl = sess.flags.includes(q.id);
     $('#btnFlag').className = 'btn ' + (fl ? 'btn-warning' : 'btn-outline-warning') + (on ? '' : ' hidden');
@@ -1041,24 +1038,6 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
     $('#btnSatisfied').disabled = i >= n - 1;
     paintDesk();
   }
-  // a host whose setup failed when the session started: say so where it is used, and offer to rebuild it
-  const brokenHost = id => live() && (S.session.broken || []).includes(id)
-    ? `<div class="alert alert-danger"><b>This host's setup failed</b> when the session started, so it is not as the text describes.
-       <div style="margin-top:.5rem"><button class="btn btn-danger btn-sm" id="btnRebuild" data-id="${id}">Rebuild this host</button>
-       <span class="muted"> about a minute - or <code class="ic">./ksim reset ${id}</code> in a terminal</span></div></div>` : '';
-  function wireRebuild() {
-    const b = $('#btnRebuild'); if (!b) return;
-    b.onclick = async () => {
-      $('#busyTitle').textContent = 'Rebuilding the host';
-      $('#busyBody').innerHTML = '<p class="mb0">A new cluster, and its setup run again - about a minute.</p>';
-      openModal('#mBusy');
-      try { S = await api('/api/reset', {id: +b.dataset.id}); }
-      catch (e) { alert(e.message); }
-      finally { closeModal('#mBusy'); }
-      if (isLesson()) lpage = null;
-      paintExam();
-    };
-  }
   // what the exam and a lesson share: the notepad, the remote desktop, the clock
   function paintDesk() {
     if (document.activeElement !== $('#notes')) $('#notes').value = S.session.notes || '';
@@ -1122,7 +1101,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
       + C.lessons.map(x => { const c = x.steps.filter(s => s.pass).length;
           return `<option value="${x.id}"${x.id === C.at ? ' selected' : ''}>${c === x.steps.length ? '✓' : '·'} ${esc(x.objective)} ${esc(x.title)} (${c}/${x.steps.length})</option>`; }).join('')
       + '</select></div>';
-    let body = brokenHost(L.id) + `<div class="qhead">${pick}<div class="line"><b>This lesson runs on:</b> <code class="ic" title="click to copy">${esc('ssh ' + L.host)}</code></div>
+    let body = `<div class="qhead">${pick}<div class="line"><b>This lesson runs on:</b> <code class="ic" title="click to copy">${esc('ssh ' + L.host)}</code></div>
       <div class="line sub">${L.nodes}-node cluster · files in ${esc(S.workspace)} on that host · no clock</div></div><hr>`;
     if (lpage === 0) {
       body += `<div class="stepsum">Lesson ${esc(L.objective)} · ${n} steps</div><h2 class="qtitle">${esc(L.title)}</h2><div class="qbody">${render(L.intro)}</div>`;
@@ -1141,11 +1120,10 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
           + (r && !r.pass && r.out ? `<details class="fold"><summary>What the check saw</summary><pre>${esc(r.out)}</pre></details>` : '');
       } else if (r && r.pass) body += '<div class="lstate"><span class="ok">✓ Done</span></div>';
       if (s.hint) body += `<details class="fold"><summary>Hint</summary><div class="qbody" style="margin-top:10px">${render(s.hint)}</div></details>`;
-      if (s.show) body += `<details class="fold"><summary>Show me how</summary><div class="qbody" style="margin-top:10px">${render(s.show)}</div></details>`;
+      if (s.show) body += `<details class="fold"><summary>Show me the commands</summary><pre class="code" title="click to copy" style="margin:10px 0 0">${esc(s.show)}</pre></details>`;
     }
     $('#qContent').innerHTML = body;
     copyable($('#qContent'));
-    wireRebuild();
     const bs = $('#btnStep'); if (bs) bs.onclick = () => checkStep(lpage - 1);
     const bf = $('#btnFinish'); if (bf) bf.onclick = finishLesson;
     const bn = $('#btnNextLesson'); if (bn) bn.onclick = () => goLesson(nx.id);
@@ -1316,9 +1294,8 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
       const cls = r.pass ? 't-pass' : (r.earned ? 't-partial' : 't-fail');
       return `<div class="qcard ${cls}"><div class="qcard-h"><h3>Question ${i + 1} · Q${q.id} ${esc(q.title)}</h3><span class="qs">${r.earned} of ${r.total}</span></div>
         <ul class="vitems">${r.checks.map(c => `<li class="vitem"><span>${esc(c.name)} <span class="muted">(${c.pts})</span></span><span class="vstat ${c.pass ? 'ok' : 'no'}">${c.pass ? '✓' : '✗'}</span></li>`).join('')}</ul>
-        ${q.solution ? `<details class="fold"><summary>Reference solution</summary><div class="qbody" style="margin-top:10px">${render(q.solution)}</div></details>` : '<div class="pad"></div>'}</div>`;
+        ${q.solution ? `<details class="fold"><summary>Reference solution</summary><pre>${esc(q.solution)}</pre></details>` : '<div class="pad"></div>'}</div>`;
     }).join('');
-    copyable($('#resQuestions'));
     const missed = S.questions.filter(q => s.results[q.id] && !s.results[q.id].pass).map(q => q.id);
     $('#btnDrillMissed').disabled = !missed.length;
     $('#btnDrillMissed').onclick = () => start('drill', missed);
@@ -1326,9 +1303,8 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
   function showAnswers() {
     const s = S.session;
     $('#answersBody').innerHTML = S.questions.filter(q => q.solution).map((q, i) =>
-      `<h4>Question ${S.questions.indexOf(q) + 1} · Q${q.id} ${esc(q.title)}${s && s.results[q.id] ? ` <span class="muted">(${s.results[q.id].earned}/${s.results[q.id].total})</span>` : ''}</h4><div class="qbody">${render(q.solution)}</div>`).join('')
+      `<h4>Question ${S.questions.indexOf(q) + 1} · Q${q.id} ${esc(q.title)}${s && s.results[q.id] ? ` <span class="muted">(${s.results[q.id].earned}/${s.results[q.id].total})</span>` : ''}</h4><pre>${esc(q.solution)}</pre>`).join('')
       || '<p class="muted">No reference solutions are open yet.</p>';
-    copyable($('#answersBody'));
     openModal('#mAnswers');
   }