chore: release 2026.10.27.1

Bump VERSION to 2026.10.27.1 and date its CHANGELOG.md entry: every
CKA reference solution shown the exam way and audited by its own parts,
./ksim solution lint, from e761d53.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNSZLLsGXTMBsw17kDunkQ

CHANGELOG.md

@@ -4,6 +4,26 @@ Notable changes to kubestronaut-sim, newest first. A version is the `VERSION` st
 (year.month.day.n); `./ksim help` and the top bar of the web page show the one you are running.
 Every entry names the commit it came in.
 
+## 2026.10.27.1 — 2026-10-27
+
+Everything below came in `e761d53`.
+
+### Changed
+
+- **Every CKA reference solution shows the exam way**, as the lessons do: a generator and the few
+  lines to add in `vim`, a manifest pasted from the docs page it names, `kubectl edit`, `ssh NODE`,
+  or a value you read and `echo` - no heredocs, `sed`, JSON patches, `kubectl wait` or loops. 105
+  of the 147 were rewritten; the other 42 were commands you type already. The page and `./ksim
+  solution N` show each thing to type as a block of its own, which a click copies.
+- A solution in parts (`cmd`, `hand`) is its own audit script: `./ksim audit` runs what it shows,
+  and for each edit by hand the script beside it, so the solution you read is the one proven. Their
+  separate scripts in `cka/solve.py` are gone.
+
+### Added
+
+- `./ksim solution lint [N|DOMAIN..]`: what the reference solutions have you type, against the
+  list `./ksim lesson lint` uses.
+
 ## 2026.10.26.4 — 2026-10-26
 
 Everything below came in `226ef19`.

README.md

@@ -200,7 +200,7 @@ Each track is a directory: `cka/`, `ckad/`, `cks/`.
 | `bank_*.py` | more tasks; `bank_sim_*` are the fixed simulator sets |
 | `curriculum.py` | the exam's objectives in order, each with study notes and the tasks that practise it |
 | `solutions.py` | reference solutions (CKA keeps them apart; the others inline them) |
-| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved |
+| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved - for the tasks whose reference solution is not in parts |
 | `lessons.py` | guided labs, one per objective (CKA: all, one `lessons_*.py` per domain · CKAD: 2.1–2.4): introduction, steps with their check, hint and commands, wrap-up |
 
 | Track | Kubernetes | Bank today | Target (17–20 pool tasks per domain) |
@@ -242,7 +242,8 @@ v1.36), CKAD Q1 and CKS Q60, Q61 and Q64 passed again on the rebuilt images.
  "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
  "setup": "bash that seeds the scenario on the task's own cluster",
  "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
- "solution": "...",
+ "solution": [cmd("k ... --dry-run=client -o yaml > f.yaml"),   # or one string, a script as it stands
+              hand("vim f.yaml", "what to add, Markdown", run="the same, scripted: yq, a heredoc")],
  "workers": 2,                                              # optional: worker nodes of its cluster
  "k8s": "v1.36"}                                            # optional: a minor of its own (an upgrade task)
 ```
@@ -287,9 +288,16 @@ and the untouched grade, which catches a check that only passes once the seeded
 until a stopped kubelet shows as NotReady; `--keep` leaves the host up to look at
 (`KUBECONFIG=.state/envs/<host>/kubeconfig`, `./ksim down` removes it).
 
-In `cka/solve.py`, `AS_WRITTEN` lists the tasks whose reference solution is a script as it stands:
+A reference solution in parts - `cmd()` and `hand()`, as a lesson step's `show` - is its own
-for those the audit runs exactly the text the candidate is shown, so a solution that stops working
+script: the audit runs what is typed as it stands, and for each part done by hand its `run`, so the
-with a new Kubernetes release fails the next audit instead of misleading you after a drill.
+solution the candidate reads after a drill is the one that is proven, and one that stops working
+with a new Kubernetes release fails the next audit. 105 of the 147 CKA solutions are written so
+(the page and `./ksim solution N` show each thing to type as a block of its own, which a click
+copies, with the edit to make in `vim` beneath it); the other 42 are commands typed as they stand,
+each with its script in `solve.py`. `./ksim solution lint` reads what they have you type, against
+the same list of things you would not type in the exam as `./ksim lesson lint`. In `cka/solve.py`,
+`AS_WRITTEN` lists the few tasks whose reference solution is still one script, which the audit
+runs as it stands.
 
 **Writing a lesson.** A lesson in `<track>/lessons.py` is a dict with `id` (9000 plus the objective,
 `9201` for 2.1 — its host is `ckad9201`, which no task's can be), `objective`, `title`, `workers`,

cka/bank_course.py

@@ -12,6 +12,8 @@ operator with broken RBAC do its job, and 146-147 grow, snapshot, restore and cl
 the CSI hostpath driver's class `csi-hostpath-sc`, as the real exam's clusters have it
 (ensure_csi_hostpath).
 """
+
+from sim.bank import cmd, hand
 from cka.common import C, CAN, NET, NS, ONE, backup_operator, ns_pick, name_pick
 
 _ETCD = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
@@ -59,38 +61,65 @@ for n in "$CP" "$W1"; do
 done
 eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" v1.37.1 && eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" 2"""),
  ],
- "solution": """
+ "solution": [hand(r"""sudo vim /etc/apt/sources.list.d/kubernetes.list""",
-# the control plane - this host
+                   r"""The docs page *Upgrading kubeadm clusters* starts here: pkgs.k8s.io has one repository per minor, so 1.37 packages exist for apt only once the line says `/v1.37/` instead of `/v1.36/`.""",
-sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list   # one repository per minor
+                   run=r"""apt_get() { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq -y -o Dpkg::Options::=--force-confold "$@" </dev/null >/dev/null; }
-sudo apt-get update
+sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list"""),
+              hand(r"""sudo apt-get update
 apt-cache madison kubeadm | head -3
-sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm
+sudo apt-mark unhold kubeadm
-sudo kubeadm upgrade plan
+sudo apt-get install -y kubeadm=1.37.1-1.1
-sudo kubeadm upgrade apply v1.37.1 -y                                       # a few minutes
+sudo apt-mark hold kubeadm""",
-k drain {CP} --ignore-daemonsets --delete-emptydir-data
+                   r"""The control plane - this host - first, and kubeadm first: it drives the upgrade.""",
-sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl
+                   run=r"""apt_get update
-sudo systemctl daemon-reload && sudo systemctl restart kubelet
+sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null"""),
-k uncordon {CP}
+              hand(r"""sudo kubeadm upgrade plan
-# the worker
+sudo kubeadm upgrade apply v1.37.1""",
-k drain {W1} --ignore-daemonsets --delete-emptydir-data
+                   r"""Answer `y`; it takes a few minutes, and the API server restarts on the new version.""",
-{SHELL_W1}
+                   run=r"""sudo kubeadm upgrade apply v1.37.1 -y </dev/null | tail -2"""),
+              cmd(r"""k drain {CP} --ignore-daemonsets --delete-emptydir-data"""),
+              hand(r"""sudo apt-mark unhold kubelet kubectl
+sudo apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
+sudo apt-mark hold kubelet kubectl
+sudo systemctl daemon-reload
+sudo systemctl restart kubelet""",
+                   r"""Then the node's kubelet and kubectl, held again.""",
+                   run=r"""sudo apt-mark unhold kubelet kubectl >/dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null
+sudo systemctl daemon-reload && sudo systemctl restart kubelet"""),
+              hand(r"""k uncordon {CP}
+k get nodes""",
+                   r"""Again until `k` answers - the kubelet is restarting. `{CP}` shows `v1.37.1`.""",
+                   run=r"""for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done"""),
+              cmd(r"""k drain {W1} --ignore-daemonsets --delete-emptydir-data"""),
+              hand(r"""{SHELL_W1}
 sudo -i
-sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
+vim /etc/apt/sources.list.d/kubernetes.list""",
-apt-get update
+                   r"""`/v1.36/` to `/v1.37/` here too.""",
+                   run=r""""""),
+              hand(r"""apt-get update
 apt-mark unhold kubeadm kubelet kubectl
 apt-get install -y kubeadm=1.37.1-1.1
-kubeadm upgrade node                                                        # a worker: node, not apply
+kubeadm upgrade node
 apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
 apt-mark hold kubeadm kubelet kubectl
+systemctl daemon-reload
+systemctl restart kubelet
+exit
+exit""",
+                   r"""A worker runs `kubeadm upgrade node`, not `apply`: it hosts no control plane, only its kubelet's configuration to update.""",
+                   run=r"""ssh {W1} sudo bash <<'EOF'
+export DEBIAN_FRONTEND=noninteractive
+sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
+apt-get update -qq </dev/null
+apt-mark unhold kubeadm kubelet kubectl >/dev/null
+apt-get install -y -qq kubeadm=1.37.1-1.1 </dev/null >/dev/null
+kubeadm upgrade node </dev/null | tail -1
+apt-get install -y -qq kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 </dev/null >/dev/null
+apt-mark hold kubeadm kubelet kubectl >/dev/null
 systemctl daemon-reload && systemctl restart kubelet
-exit; exit
+EOF"""),
-k uncordon {W1}
+              cmd(r"""k uncordon {W1}
-k get nodes
+k get nodes""", r"""Both `Ready`, both `v1.37.1` - `VERSION` is each kubelet's.""")],
-# docs: "Upgrading kubeadm clusters". pkgs.k8s.io has one repository per minor, so 1.37 packages
-# appear only once the line names v1.37. kubeadm first - it drives the upgrade and rewrites the
-# kubelet's configuration; kubelet and kubectl after `upgrade apply`, then the restart: `k get
-# nodes` shows each kubelet's running version. Workers run `kubeadm upgrade node`.
-""",
 },
 # ------------------------------------------------------------------ 133 ----
 {
@@ -196,19 +225,16 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.ephemeralContainers[?(@.name=="debugger")].t
   && eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q134.uid") 0" """),
   C("mode.txt holds the value the process runs with", 3, ONE + r"""eq "$(one "$WS/q134/mode.txt")" %{mode}"""),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} exec %{pod} -- sh
-k -n %{ns} exec %{pod} -- sh                        # fails: no shell in the image
+k -n %{ns} get cm %{pod}-env -o yaml""", r"""No shell in the image. And the ConfigMap says `MODE=open` - now; the environment was read once, when the container started."""),
-k -n %{ns} get cm %{pod}-env -o yaml                # MODE=open - but that is the ConfigMap NOW
+              cmd(r"""k -n %{ns} debug %{pod} --image=busybox:1.36 --target=app -c debugger --profile=general -- sleep 3600""", r"""`--target` puts the debug container into `app`'s process namespace; the `general` profile gives it `SYS_PTRACE`, which reading another process's environment needs. Interactively: `k debug -it ... -- sh`."""),
-k -n %{ns} debug %{pod} --image=busybox:1.36 --target=app -c debugger --profile=general -- sleep 3600
+              hand(r"""k -n %{ns} exec %{pod} -c debugger -- ps""",
-for i in $(seq 1 30); do k -n %{ns} exec %{pod} -c debugger -- true 2>/dev/null && break; sleep 2; done
+                   r"""Again until it answers: PID 1 is `/pause`, `app`'s process.""",
-k -n %{ns} exec %{pod} -c debugger -- ps                   # PID 1 is /pause, the app's process
+                   run=r"""for i in $(seq 1 30); do kubectl -n %{ns} exec %{pod} -c debugger -- true 2>/dev/null && break; sleep 2; done"""),
-k -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\\0" "\\n" < /proc/1/environ' | grep '^MODE='
+              cmd(r"""k -n %{ns} exec %{pod} -c debugger -- cat /proc/1/environ | tr '\0' '\n'""", r"""`MODE=%{mode}`: what the process runs with."""),
-k -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\\0" "\\n" < /proc/1/environ' | sed -n 's/^MODE=//p' > {WS}/q134/mode.txt
+              hand(r"""echo %{mode} > {WS}/q134/mode.txt""",
-# Environment variables from a ConfigMap are read once, when the container starts: editing the
+                   r"""With the value after `MODE=`.""",
-# ConfigMap changes nothing for a running process. --target puts the debug container into app's
+                   run=r"""kubectl -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\0" "\n" < /proc/1/environ' | sed -n 's/^MODE=//p' > {WS}/q134/mode.txt""")],
-# process namespace, so /proc/1 is app's process; the `general` profile gives it SYS_PTRACE,
-# which reading another user's /proc/PID/environ needs. Interactively: `k debug -it ...  -- sh`.
-""",
 },
 # ------------------------------------------------------------------ 135 ----
 {
@@ -253,19 +279,19 @@ tok = [v for v in s.get("volumes", []) if any("serviceAccountToken" in src for s
 sys.exit(1 if tok else 0)' || exit 1
 done"""),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} create serviceaccount pipeline
-k -n %{ns} create serviceaccount pipeline
 k -n %{ns} create role pipeline --verb=get,list,create,update,patch --resource=deployments.apps
 k -n %{ns} create rolebinding pipeline --role=pipeline --serviceaccount=%{ns}:pipeline
-k auth can-i delete deployments -n %{ns} --as=system:serviceaccount:%{ns}:pipeline     # no
+k auth can-i delete deployments -n %{ns} --as=system:serviceaccount:%{ns}:pipeline""", r"""`no`."""),
-k -n %{ns} create token pipeline --duration=%{h}h > {WS}/q135/pipeline.token
+              cmd(r"""k -n %{ns} create token pipeline --duration=%{h}h > {WS}/q135/pipeline.token""", r"""The TokenRequest API issues a short-lived, bound token; nothing is stored in a Secret."""),
-k -n %{ns} patch deploy frontend -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":false}}}}'
+              hand(r"""k -n %{ns} edit deploy frontend""",
-k -n %{ns} rollout status deploy frontend
+                   r"""In the Pod template's `spec:`, beside `containers:` and indented like it, then `:wq`:
-# `create token` asks the TokenRequest API for a short-lived, bound token - nothing is stored in a
+```
-# Secret. automountServiceAccountToken: false on the Pod template (or on the ServiceAccount; the
+      automountServiceAccountToken: false
-# Pod's setting wins) leaves out the projected volume kube-api-access-*. To try a token alone, use
+```
-# no kubeconfig: a client certificate in yours would be checked first.
+It leaves out the projected volume `kube-api-access-*`. (On the ServiceAccount it works too; the Pod's setting wins.)""",
-""",
+                   run=r"""kubectl -n %{ns} patch deploy frontend -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":false}}}}'"""),
+              cmd(r"""k -n %{ns} rollout status deploy frontend""")],
 },
 # ------------------------------------------------------------------ 136 ----
 {
@@ -298,22 +324,20 @@ eq "$(x theme)" dark && eq "$(x locale)" en_GB && eq "$(x db/password)" pw-%{mem
 x() { $K -n %{ns} exec %{pod} -- cat "/etc/portal/$1" 2>/dev/null; }
 x labels | grep -qx 'app="%{pod}"' && eq "$(x mem-limit)" %{mem}"""),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} run %{pod} --image=busybox:1.36 --labels=app=%{pod} --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
-k -n %{ns} apply -f - <<'EOF'
+              hand(r"""vim pod.yaml""",
-apiVersion: v1
+                   r"""Replace the container's `resources: {}` with the limit and the mount:
-kind: Pod
+```
-metadata:
-  name: %{pod}
-  labels: {app: %{pod}}
-spec:
-  containers:
-  - name: portal
-    image: busybox:1.36
-    command: [sleep, '86400']
     resources:
-      limits: {memory: %{mem}Mi}
+      limits:
+        memory: %{mem}Mi
     volumeMounts:
-    - {name: portal, mountPath: /etc/portal, readOnly: true}
+    - name: portal
+      mountPath: /etc/portal
+      readOnly: true
+```
+and add to the Pod's `spec:`, right below `spec:` (the docs page *Projected Volumes* has an example with all three kinds):
+```
   volumes:
   - name: portal
     projected:
@@ -323,20 +347,33 @@ spec:
       - secret:
           name: portal-db
           items:
-          - {key: password, path: db/password}
+          - key: password
+            path: db/password
       - downwardAPI:
           items:
           - path: labels
-            fieldRef: {fieldPath: metadata.labels}
+            fieldRef:
+              fieldPath: metadata.labels
           - path: mem-limit
-            resourceFieldRef: {containerName: portal, resource: limits.memory, divisor: 1Mi}
+            resourceFieldRef:
-EOF
+              containerName: %{pod}
-k -n %{ns} wait --for=condition=Ready pod/%{pod}
+              resource: limits.memory
-k -n %{ns} exec %{pod} -- sh -c 'ls -R /etc/portal; cat /etc/portal/labels /etc/portal/mem-limit'
+              divisor: 1Mi
-# `items` picks keys and renames them: only `password` from the Secret, as db/password. A
+```
-# resourceFieldRef needs the container's name in a volume (in an env var it is implied), and the
+`items` picks keys and renames them; in a volume a `resourceFieldRef` needs the container's name, and the divisor turns bytes into MiB.""",
-# divisor turns bytes into MiB.
+                   run=r"""yq -i '.spec.containers[0].resources = {"limits": {"memory": "%{mem}Mi"}}
-""",
+  | .spec.containers[0].volumeMounts = [{"name": "portal", "mountPath": "/etc/portal", "readOnly": true}]
+  | .spec.volumes = [{"name": "portal", "projected": {"sources": [
+      {"configMap": {"name": "portal-config"}},
+      {"secret": {"name": "portal-db", "items": [{"key": "password", "path": "db/password"}]}},
+      {"downwardAPI": {"items": [{"path": "labels", "fieldRef": {"fieldPath": "metadata.labels"}},
+                                 {"path": "mem-limit", "resourceFieldRef": {"containerName": "%{pod}", "resource": "limits.memory", "divisor": "1Mi"}}]}}]}}]' pod.yaml"""),
+              cmd(r"""k apply -f pod.yaml"""),
+              hand(r"""k -n %{ns} get pod %{pod}""",
+                   r"""Again until it is `Running`.""",
+                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=120s"""),
+              cmd(r"""k -n %{ns} exec %{pod} -- ls -R /etc/portal
+k -n %{ns} exec %{pod} -- cat /etc/portal/labels /etc/portal/mem-limit""")],
 },
 # ------------------------------------------------------------------ 137 ----
 {
@@ -404,18 +441,19 @@ eq "$(j -n %{ns} deploy critical '{.status.availableReplicas} {.status.updatedRe
 eq "$(j -n %{ns} deploy critical '{.spec.template.spec.containers[0].resources.requests.cpu} {.spec.template.spec.containers[0].resources.limits.cpu} {.spec.template.spec.containers[0].resources.requests.memory} {.spec.template.spec.containers[0].resources.limits.memory}')" "200m 200m 128Mi 128Mi" || exit 1
 for q in $($K -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'); do [ "$q" = Guaranteed ] || exit 1; done"""),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass""", r"""Guaranteed: every container has requests equal to limits, CPU and memory. BestEffort: no container has any. Everything else is Burstable - `delta` too, as one of its containers sets nothing."""),
-k -n %{ns} get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
+              hand(r"""vim {WS}/q137/qos.txt""",
-k -n %{ns} get pods alpha beta delta gamma -o jsonpath='{range .items[*]}{.metadata.name} {.status.qosClass}{"\\n"}{end}' > {WS}/q137/qos.txt
+                   r"""```
-echo gamma > {WS}/q137/first.txt
+alpha Guaranteed
-k -n %{ns} set resources deploy critical --requests=cpu=200m,memory=128Mi --limits=cpu=200m,memory=128Mi
+beta Burstable
+gamma BestEffort
+delta Burstable
+```""",
+                   run=r"""printf 'alpha Guaranteed\nbeta Burstable\ngamma BestEffort\ndelta Burstable\n' > {WS}/q137/qos.txt"""),
+              cmd(r"""echo gamma > {WS}/q137/first.txt""", r"""Under memory pressure BestEffort goes first, then Burstable Pods using more than they requested, Guaranteed last."""),
+              cmd(r"""k -n %{ns} set resources deploy critical --requests=cpu=200m,memory=128Mi --limits=cpu=200m,memory=128Mi
 k -n %{ns} rollout status deploy critical
-k -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'; echo
+k -n %{ns} get pods -l app=critical -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass""")],
-# Guaranteed: every container has requests equal to limits for both CPU and memory. BestEffort: no
-# container has any. Everything else is Burstable - delta too, because one container sets nothing.
-# Under memory pressure the kubelet evicts BestEffort first, then Burstable Pods using more than
-# they requested; Guaranteed Pods last.
-""",
 },
 # ------------------------------------------------------------------ 138 ----
 {
@@ -452,15 +490,21 @@ eq "$(j -n %{ns} pod %{pod} '{.status.containerStatuses[0].resources.requests.cp
 eq "$(j -n %{ns} pod %{pod} '{.spec.containers[0].resources.limits.cpu}')" %{lim} \
   && eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q138.uid") 0" """),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} get pod %{pod} -o yaml | grep -A6 resources:"""),
-k -n %{ns} get pod %{pod} -o jsonpath='{.spec.containers[0].resources}'; echo
+              hand(r"""k -n %{ns} edit pod %{pod} --subresource resize""",
-k -n %{ns} patch pod %{pod} --subresource resize --patch '{"spec":{"containers":[{"name":"encoder","resources":{"requests":{"cpu":"%{req}"},"limits":{"cpu":"%{lim}"}}}]}}'
+                   r"""Through the `resize` subresource - a plain `k edit` of a Pod's resources is refused. Set the CPU, leave the memory, then `:wq`:
-k -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].resources}{"\\n"}{.status.containerStatuses[0].restartCount}{"\\n"}'
+```
-# A running Pod's CPU and memory can be changed in place, through the `resize` subresource - a
+    resources:
-# plain `k edit` or `k apply` of a Pod's resources is refused. The spec
+      limits:
-# says what you asked for; status.containerStatuses[].resources what the kubelet has applied.
+        cpu: %{lim}
-# A container's resizePolicy decides whether a resource's change needs a restart (CPU: no).
+        memory: 48Mi
-""",
+      requests:
+        cpu: %{req}
+        memory: 48Mi
+```""",
+                   run=r"""kubectl -n %{ns} patch pod %{pod} --subresource resize --patch '{"spec":{"containers":[{"name":"encoder","resources":{"requests":{"cpu":"%{req}"},"limits":{"cpu":"%{lim}"}}}]}}'"""),
+              cmd(r"""k -n %{ns} get pod %{pod} -o yaml | grep -A6 resources:
+k -n %{ns} get pod %{pod}""", r"""The spec says what you asked for, `status.containerStatuses[].resources` what the kubelet applied; `RESTARTS 0` - a container's `resizePolicy` decides, and CPU needs no restart.""")],
 },
 # ------------------------------------------------------------------ 139 ----
 {
@@ -495,24 +539,33 @@ sys.exit(0 if s == "%{soft}Mi" and secs == 90 else 1)'"""),
 nx "$W1" 'grep -q "%{soft}Mi" /var/lib/kubelet/config.yaml && grep -q "%{mem}Mi" /var/lib/kubelet/config.yaml && grep -q "^staticPodPath:" /var/lib/kubelet/config.yaml && systemctl is-active -q kubelet' </dev/null || exit 1
 eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True"""),
  ],
- "solution": """
+ "solution": [hand(r"""{SHELL_W1}
-{SHELL_W1}
 sudo -i
-grep -n -A4 '^eviction' /var/lib/kubelet/config.yaml       # kind sets the hard thresholds to 0%
+grep -n -A4 ^eviction /var/lib/kubelet/config.yaml
-yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%"
+vim /var/lib/kubelet/config.yaml""",
-  | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
+                   r"""Under `evictionHard:` set the two hard thresholds, and add the soft one with its grace period at the top level - a soft threshold without a grace period for the same signal, and the kubelet refuses to start:
+```
+evictionHard:
+  memory.available: "%{mem}Mi"
+  nodefs.available: "%{fs}%"
+evictionSoft:
+  memory.available: "%{soft}Mi"
+evictionSoftGracePeriod:
+  memory.available: "1m30s"
+```
+Keep the other `evictionHard` entries as they are.""",
+                   run=r"""ssh {W1} sudo bash <<'EOF'
+yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
 systemctl restart kubelet
+EOF"""),
+              hand(r"""systemctl restart kubelet
 journalctl -u kubelet -n 5 --no-pager
-exit; exit
+exit
-k get --raw /api/v1/nodes/{W1}/proxy/configz | jq '.kubeletconfig | {evictionHard, evictionSoft, evictionSoftGracePeriod}'
+exit""",
-# A soft threshold needs a grace period for the same signal, or the kubelet refuses to start. By
+                   r"""No complaint about the configuration; then back on the host.""",
-# hand: under evictionHard add `memory.available: "%{mem}Mi"` and change nodefs.available; then
+                   run=r"""for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done"""),
-#   evictionSoft:
+              cmd(r"""k get node {W1}
-#     memory.available: "%{soft}Mi"
+k get --raw /api/v1/nodes/{W1}/proxy/configz | tr ',' '\n' | grep -i eviction""", r"""`Ready`; `configz` shows the configuration the running kubelet uses.""")],
-#   evictionSoftGracePeriod:
-#     memory.available: "1m30s"
-# configz shows what the running kubelet uses.
-""",
 },
 # ------------------------------------------------------------------ 140 ----
 {
@@ -548,19 +601,24 @@ ch=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A
 n=$(nx "$CP" "iptables-save -t nat | grep -- '^-A $ch ' | grep -c -- '-j KUBE-SEP-'" </dev/null)
 eq "$n" %{n} && eq "$(one "$WS/q140/endpoints.txt")" "$n" """),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n kube-system get cm kube-proxy -o yaml | grep mode""", r"""`mode: iptables` - an empty mode means the default, iptables too."""),
-k -n kube-system get cm kube-proxy -o jsonpath='{.data.config\\.conf}' | grep '^mode'       # iptables
+              hand(r"""echo iptables > {WS}/q140/mode.txt""",
-echo iptables > {WS}/q140/mode.txt
+                   r"""The mode it printed.""",
-ip=$(k -n %{ns} get svc %{svc} -o jsonpath='{.spec.clusterIP}')
+                   run=r"""m=$(kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"')
-sudo iptables-save -t nat | grep -- "-d $ip/32"                 # KUBE-SERVICES ... -j KUBE-SVC-XXXX
+echo ${m:-iptables} > {WS}/q140/mode.txt"""),
-chain=$(sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1)
+              cmd(r"""k -n %{ns} get svc %{svc}""", r"""Note its `CLUSTER-IP`."""),
-echo $chain > {WS}/q140/chain.txt
+              hand(r"""sudo iptables-save -t nat | grep 10.96.12.34""",
-sudo iptables-save -t nat | grep -- "^-A $chain "               # one -j KUBE-SEP-... per endpoint
+                   r"""With that IP: `KUBE-SERVICES` matches the ClusterIP and port and jumps (`-j`) to the Service's own `KUBE-SVC-...` chain.""",
-sudo iptables-save -t nat | grep -- "^-A $chain " | grep -c -- '-j KUBE-SEP-' > {WS}/q140/endpoints.txt
+                   run=r""""""),
-# KUBE-SERVICES matches ClusterIP:port and jumps to the Service's KUBE-SVC chain; that picks an
+              hand(r"""echo KUBE-SVC-ABCDEFGHIJKLMNOP > {WS}/q140/chain.txt
-# endpoint chain at random (statistic mode random), and each KUBE-SEP chain DNATs to one Pod.
+sudo iptables-save -t nat | grep -- '-A KUBE-SVC-ABCDEFGHIJKLMNOP '""",
-# One KUBE-SEP per ready endpoint. In nftables mode: `sudo nft list ruleset | grep -A5 %{svc}`.
+                   r"""With the chain's name. Its rules pick one endpoint chain at random (`statistic mode random`); each `KUBE-SEP-...` DNATs to one Pod - count them. In nftables mode: `sudo nft list ruleset | grep -A5 %{svc}`.""",
-""",
+                   run=r"""ip=$(kubectl -n %{ns} get svc %{svc} -o jsonpath='{.spec.clusterIP}')
+sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/q140/chain.txt"""),
+              hand(r"""echo 3 > {WS}/q140/endpoints.txt""",
+                   r"""With the number of `-j KUBE-SEP-` rules: one per ready endpoint.""",
+                   run=r"""chain=$(cat {WS}/q140/chain.txt)
+sudo iptables-save -t nat | grep -- "^-A $chain " | grep -c -- '-j KUBE-SEP-' > {WS}/q140/endpoints.txt""")],
 },
 # ------------------------------------------------------------------ 141 ----
 {
@@ -593,36 +651,65 @@ eq "$(j -n %{ns} endpointslice billing-1 '{.metadata.labels.kubernetes\.io/servi
   C("wget billing from client returns the API's answer", 3, r"""
 for i in 1 2 3 4 5; do [ "$($K -n %{ns} exec client -- wget -qO- -T 3 billing 2>/dev/null)" = billing-api-%{port} ] && exit 0; sleep 3; done; exit 1"""),
  ],
- "solution": """
+ "solution": [cmd(r"""cat {WS}/q141/billing.txt""", r"""`ADDRESS:PORT`."""),
-cat {WS}/q141/billing.txt                                       # ADDRESS:PORT
+              hand(r"""vim billing.yaml""",
-addr=$(cut -d: -f1 {WS}/q141/billing.txt); port=$(cut -d: -f2 {WS}/q141/billing.txt)
+                   r"""A Service with no selector - no controller writes its endpoints - and the EndpointSlice you maintain, tied to it by the label `kubernetes.io/service-name` (the docs page *Service* has both under *Services without selectors*). With the address and port from the file:
-k -n %{ns} apply -f - <<EOF
+```
 apiVersion: v1
 kind: Service
 metadata:
   name: billing
+  namespace: %{ns}
 spec:
   ports:
-  - {port: 80, protocol: TCP}
+  - port: 80
+    protocol: TCP
 ---
 apiVersion: discovery.k8s.io/v1
 kind: EndpointSlice
 metadata:
   name: billing-1
+  namespace: %{ns}
   labels:
     kubernetes.io/service-name: billing
 addressType: IPv4
 endpoints:
-- addresses: ["$addr"]
+- addresses:
+  - 172.18.0.3
 ports:
-- {port: $port, protocol: TCP}
+- port: %{port}
-EOF
+  protocol: TCP
-k -n %{ns} exec client -- wget -qO- -T3 billing
+```
-# Without a selector, no controller writes endpoints for the Service: the label
+The slice's port is where traffic goes - the Service has no `targetPort` to speak of - and its name must match the Service port's: here both unnamed.""",
-# kubernetes.io/service-name ties a slice to it. The slice's port is where traffic goes (the
+                   run=r"""addr=$(cut -d: -f1 {WS}/q141/billing.txt); port=$(cut -d: -f2 {WS}/q141/billing.txt)
-# Service's targetPort plays no part), and its name must match the Service port's - here both
+cat > billing.yaml <<EOF
-# unnamed. ExternalName would need a DNS name; an address needs this.
+apiVersion: v1
-""",
+kind: Service
+metadata:
+  name: billing
+  namespace: %{ns}
+spec:
+  ports:
+  - port: 80
+    protocol: TCP
+---
+apiVersion: discovery.k8s.io/v1
+kind: EndpointSlice
+metadata:
+  name: billing-1
+  namespace: %{ns}
+  labels:
+    kubernetes.io/service-name: billing
+addressType: IPv4
+endpoints:
+- addresses:
+  - "$addr"
+ports:
+- port: $port
+  protocol: TCP
+EOF"""),
+              cmd(r"""k apply -f billing.yaml
+k -n %{ns} exec client -- wget -qO- -T3 billing""", r"""`billing-api-%{port}`. (`ExternalName` would need a DNS name; an address needs this.)""")],
 },
 # ------------------------------------------------------------------ 142 ----
 {
@@ -660,8 +747,24 @@ $K -n %{ns} create secret generic ksim-probe --from-literal=x=y >/dev/null 2>&1
 nx "$CP" '""" + _ETCD + r""" get /registry/secrets/%{ns}/ksim-probe --print-value-only 2>/dev/null | head -c 64' </dev/null | grep -aq '^k8s:enc:aescbc:v1:'; r=$?
 $K -n %{ns} delete secret ksim-probe --ignore-not-found >/dev/null 2>&1; exit $r"""),
  ],
- "solution": """
+ "solution": [cmd(r"""head -c 32 /dev/urandom | base64""", r"""The host is the control-plane node. Copy the output: that is the key."""),
-sudo mkdir -p /etc/kubernetes/enc
+              hand(r"""sudo mkdir -p /etc/kubernetes/enc
+sudo vim /etc/kubernetes/enc/enc.yaml""",
+                   r"""Paste the example from the docs page *Encrypting Confidential Data at Rest* (search for `aescbc`) and cut it down to:
+```
+apiVersion: apiserver.config.k8s.io/v1
+kind: EncryptionConfiguration
+resources:
+- resources: [secrets]
+  providers:
+  - aescbc:
+      keys:
+      - name: key1
+        secret: <your key>
+  - identity: {}
+```
+The first provider encrypts; `identity` after it still reads what was stored before.""",
+                   run=r"""sudo mkdir -p /etc/kubernetes/enc
 sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
 apiVersion: apiserver.config.k8s.io/v1
 kind: EncryptionConfiguration
@@ -673,23 +776,31 @@ resources:
       - name: key1
         secret: $(head -c 32 /dev/urandom | base64)
   - identity: {}
-EOF
+EOF"""),
-sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak     # a backup OUTSIDE manifests/
+              cmd(r"""sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/""", r"""Edit a copy: the kubelet acts on every save in `manifests/`, half-done edits too."""),
-sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
+              hand(r"""sudo vim ~/kube-apiserver.yaml""",
-sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
+                   r"""Add three lines, each beside its kind (`yy` and `p` on an existing `k8s-certs` mount or volume, then change it):
+```
+    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
+```
+in the `command:` list,
+```
+    - {name: enc, mountPath: /etc/kubernetes/enc, readOnly: true}
+```
+under `volumeMounts:`, and
+```
+  - {name: enc, hostPath: {path: /etc/kubernetes/enc, type: DirectoryOrCreate}}
+```
+under `volumes:` - the API server runs in a container, and sees only what is mounted.""",
+                   run=r"""sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
   | .spec.containers[0].volumeMounts += [{"name": "enc", "mountPath": "/etc/kubernetes/enc", "readOnly": true}]
-  | .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' /tmp/kube-apiserver.yaml
+  | .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' ~/kube-apiserver.yaml"""),
-sudo cp /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
+              cmd(r"""sudo cp ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
-sleep 20; for i in $(seq 1 40); do k get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
+              hand(r"""k -n kube-system get pods""",
-k -n %{ns} get secret %{sec} -o json | k replace -f -                 # stored again: encrypted now
+                   r"""Again until it answers: the API server is back in a minute or so.""",
-sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \\
+                   run=r"""sleep 20; for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
-  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \\
+              cmd(r"""k -n %{ns} get secret %{sec} -o yaml | k replace -f -""", r"""Written again, the Secret is stored encrypted; every Secret at once: `k get secrets -A -o json | k replace -f -`."""),
-  get /registry/secrets/%{ns}/%{sec} | od -c | head -3                 # k 8 s : e n c : a e s c b c : v 1 : k e y 1
+              cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/%{ns}/%{sec} | head -c 80""", r"""`k8s:enc:aescbc:v1:key1:` and no plain text.""")],
-# By hand: in kube-apiserver.yaml add the flag to the command, a volumeMount for /etc/kubernetes/enc
-# and a hostPath volume of the same path - the API server runs in a container and sees only what is
-# mounted. The FIRST provider encrypts; `identity` after it reads what was stored before. Every
-# Secret: `k get secrets -A -o json | k replace -f -`.
-""",
 },
 # ------------------------------------------------------------------ 143 ----
 {
@@ -730,20 +841,39 @@ sys.exit(0 if ok else 1)'"""),
 eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas} {.status.updatedReplicas} {.status.replicas}')" "3 3 3" || exit 1
 [ "$($K -n %{ns} get pods -l app=%{dep} --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 3 ]"""),
  ],
- "solution": """
+ "solution": [cmd(r"""k label node {W2} {W3} tier=edge"""),
-k label node {W2} {W3} tier=edge
+              hand(r"""k -n %{ns} edit deploy %{dep}""",
-k -n %{ns} patch deploy %{dep} --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}},"template":{"spec":{"affinity":{
+                   r"""In the Pod template's `spec:`, above `containers:` (the docs page *Assigning Pods to Nodes* has both kinds):
+```
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchLabels:
+                app: %{dep}
+            topologyKey: kubernetes.io/hostname
+        nodeAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+          - weight: 100
+            preference:
+              matchExpressions:
+              - key: tier
+                operator: In
+                values:
+                - edge
+```
+and in the same edit, under `strategy:`, let the rollout replace one Pod at a time:
+```
+    rollingUpdate:
+      maxSurge: 0
+      maxUnavailable: 1
+```
+With 3 workers and one Pod each, a surge Pod has nowhere to go, and 25% of 3 unavailable rounds down to 0: the default rollout would wait for ever. Then `:wq`.""",
+                   run=r"""kubectl -n %{ns} patch deploy %{dep} --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}},"template":{"spec":{"affinity":{
   "podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app":"%{dep}"}},"topologyKey":"kubernetes.io/hostname"}]},
-  "nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"tier","operator":"In","values":["edge"]}]}}]}}}}}}'
+  "nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"tier","operator":"In","values":["edge"]}]}}]}}}}}}'"""),
-k -n %{ns} rollout status deploy %{dep}
+              cmd(r"""k -n %{ns} rollout status deploy %{dep}
-k -n %{ns} get pods -o wide
+k -n %{ns} get pods -o wide""", r"""Three nodes, one Pod each. The anti-affinity's selector names the Pods to keep away from - its own - and `topologyKey` per what: one per hostname, one per node.""")],
-# The anti-affinity's labelSelector picks the Pods to stay away from - its own, by its template's
-# labels - and topologyKey says per what: one per hostname = one per node. With 3 workers a 4th
-# replica would stay Pending - and so does the rolling update's surge Pod, since every node already
-# has one of the old Pods (same labels). The default maxUnavailable of 25% is 0 Pods for 3
-# replicas, so the rollout would wait for ever: let it replace one Pod at a time instead
-# (maxSurge 0, maxUnavailable 1), in the same change.
-""",
 },
 # ------------------------------------------------------------------ 144 ----
 {
@@ -787,15 +917,18 @@ eq "$(j -n %{ns} pod %{sts}-0 '{.spec.containers[0].image}') $(j -n %{ns} pod %{
 eq "$(for i in 0 1; do j -n %{ns} pod %{sts}-$i '{.metadata.uid}'; echo; done)" "$(cat "$PRIV/q144.uids")" \
   && eq "$(j -n %{ns} sts %{sts} '{.spec.template.spec.containers[0].image}')" nginx:1.27"""),
  ],
- "solution": """
+ "solution": [hand(r"""k -n %{ns} edit sts %{sts}""",
-k -n %{ns} patch sts %{sts} -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'
+                   r"""The partition first - a StatefulSet updates from the highest ordinal down, and without one the new image would reach all three within moments. Under `updateStrategy:`, then `:wq`:
-k -n %{ns} set image sts/%{sts} nginx=nginx:1.27
+```
+  updateStrategy:
+    rollingUpdate:
+      partition: 2
+    type: RollingUpdate
+```""",
+                   run=r"""kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'"""),
+              cmd(r"""k -n %{ns} set image sts %{sts} nginx=nginx:1.27
 k -n %{ns} rollout status sts %{sts}
-k -n %{ns} get pods -l app=%{sts} -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image
+k -n %{ns} get pods -l app=%{sts} -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image""", r"""Ordinals at or above the partition get the new template: `%{sts}-2` alone. The next steps of the canary: partition 1, then 0.""")],
-# The partition FIRST: a StatefulSet updates from the highest ordinal down, and with no partition
-# the new image would reach -1 and -0 within moments. Pods with an ordinal >= partition get the new
-# template; the others keep theirs. Next step of the canary: partition 1, then 0.
-""",
 },
 # ------------------------------------------------------------------ 145 ----
 {
@@ -825,32 +958,53 @@ eq "$(j crd backups.data.example.com '{.status.conditions[?(@.type=="Established
   C("Backup %{name} is Scheduled: the controller may update its status", 3, r"""
 eq "$(j -n %{ns} backup %{name} '{.status.phase}')" Scheduled && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller"""),
  ],
- "solution": """
+ "solution": [cmd(r"""grep -e ^kind: -e '^  name:' {WS}/q145/operator.yaml
-grep -E '^kind:|^  name:' {WS}/q145/operator.yaml
+k apply -f {WS}/q145/operator.yaml"""),
-k apply -f {WS}/q145/operator.yaml
+              hand(r"""k get crd backups.data.example.com
-k wait --for condition=established crd/backups.data.example.com
+k -n backup-operator get pods""",
-k -n backup-operator rollout status deploy controller
+                   r"""Again until the controller is `Running`.""",
-k create ns %{ns}
+                   run=r"""kubectl wait --for condition=established crd/backups.data.example.com --timeout=60s
-k -n %{ns} apply -f - <<'EOF'
+kubectl -n backup-operator rollout status deploy controller --timeout=180s"""),
+              cmd(r"""k create ns %{ns}"""),
+              hand(r"""vim backup.yaml""",
+                   r"""```
 apiVersion: data.example.com/v1
 kind: Backup
 metadata:
   name: %{name}
+  namespace: %{ns}
 spec:
   schedule: "30 1 * * *"
-EOF
+```""",
-sleep 10
+                   run=r"""cat > backup.yaml <<'EOF'
-k -n %{ns} get cm backup-%{name}                                  # made - but:
+apiVersion: data.example.com/v1
-k -n %{ns} get backup %{name} -o jsonpath='{.status}'; echo       # no phase
+kind: Backup
-k -n backup-operator logs deploy/controller | tail -3             # ... cannot patch resource "backups/status" ...
+metadata:
-k auth can-i patch backups.data.example.com --subresource=status --as=system:serviceaccount:backup-operator:controller -A
+  name: %{name}
-k patch clusterrole backup-operator --type json -p '[{"op":"add","path":"/rules/0/resources/-","value":"backups/status"}]'
+  namespace: %{ns}
-sleep 10
+spec:
-k -n %{ns} get backup %{name} -o jsonpath='{.status.phase}'; echo
+  schedule: "30 1 * * *"
-# The CRD has a status subresource, and a subresource is a resource of its own for RBAC:
+EOF"""),
-# `backups/status`. The controller could read Backups and write ConfigMaps, but not set a phase.
+              cmd(r"""k apply -f backup.yaml"""),
-# The log of an operator's controller is where such a gap shows - here as `forbidden`.
+              hand(r"""k -n %{ns} get cm backup-%{name}
-""",
+k -n %{ns} get backup %{name} -o yaml | grep -A1 status
+k -n backup-operator logs deploy/controller | tail -3""",
+                   r"""The ConfigMap is made, but no `phase`: the log says `cannot patch resource "backups/status"`. The CRD has a status subresource, and for RBAC a subresource is a resource of its own.""",
+                   run=r"""sleep 10"""),
+              cmd(r"""k auth can-i patch backups.data.example.com --subresource=status --as=system:serviceaccount:backup-operator:controller -A""", r"""`no`."""),
+              hand(r"""k edit clusterrole backup-operator""",
+                   r"""Add `backups/status` to the first rule's resources, then `:wq` - the one thing wrong:
+```
+- apiGroups:
+  - data.example.com
+  resources:
+  - backups
+  - backups/status
+```""",
+                   run=r"""kubectl patch clusterrole backup-operator --type json -p '[{"op":"add","path":"/rules/0/resources/-","value":"backups/status"}]'"""),
+              hand(r"""k -n %{ns} get backup %{name} -o yaml | grep -A1 status""",
+                   r"""Again until `phase: Scheduled` - the controller tries every 5 seconds.""",
+                   run=r"""for i in $(seq 1 20); do [ "$(kubectl -n %{ns} get backup %{name} -o jsonpath='{.status.phase}')" = Scheduled ] && break; sleep 3; done""")],
 },
 # ------------------------------------------------------------------ 146 ----
 {
@@ -881,41 +1035,66 @@ eq "$(j -n %{ns} pod web-server "{.spec.volumes[?(@.name==\"$v\")].persistentVol
 eq "$(j -n %{ns} pvc pv-volume '{.spec.resources.requests.storage} {.status.capacity.storage}')" "%{size} %{size}" || exit 1
 eq "$(j -n %{ns} pod web-server '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """),
  ],
- "solution": """
+ "solution": [cmd(r"""k get sc csi-hostpath-sc""", r"""`ALLOWVOLUMEEXPANSION true`: a claim of this class may grow."""),
-k get sc csi-hostpath-sc                         # ALLOWVOLUMEEXPANSION: true - a claim of this class may grow
+              hand(r"""vim pvc.yaml""",
-k -n %{ns} apply -f - <<'EOF'
+                   r"""```
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: pv-volume
+  namespace: %{ns}
 spec:
   storageClassName: csi-hostpath-sc
-  accessModes: [ReadWriteOnce]
+  accessModes:
+  - ReadWriteOnce
   resources:
-    requests: {storage: 10Mi}
+    requests:
----
+      storage: 10Mi
+```""",
+                   run=r"""cat > pvc.yaml <<'EOF'
 apiVersion: v1
-kind: Pod
+kind: PersistentVolumeClaim
 metadata:
-  name: web-server
+  name: pv-volume
+  namespace: %{ns}
 spec:
-  containers:
+  storageClassName: csi-hostpath-sc
-  - name: web-server
+  accessModes:
-    image: nginx:1.27
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Mi
+EOF"""),
+              cmd(r"""k apply -f pvc.yaml
+k -n %{ns} run web-server --image=nginx:1.27 --dry-run=client -o yaml > pod.yaml"""),
+              hand(r"""vim pod.yaml""",
+                   r"""Add to the container, below `name: web-server`:
+```
     volumeMounts:
-    - {name: html, mountPath: /usr/share/nginx/html}
+    - name: html
+      mountPath: /usr/share/nginx/html
+```
+and to the Pod's `spec:`, right below `spec:`:
+```
   volumes:
   - name: html
-    persistentVolumeClaim: {claimName: pv-volume}
+    persistentVolumeClaim:
-EOF
+      claimName: pv-volume
-k -n %{ns} wait --for=condition=Ready pod/web-server
+```""",
-k -n %{ns} patch pvc pv-volume -p '{"spec":{"resources":{"requests":{"storage":"%{size}"}}}}'
+                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "html", "mountPath": "/usr/share/nginx/html"}]
-k -n %{ns} get pvc pv-volume                     # CAPACITY follows once the driver has resized it
+  | .spec.volumes = [{"name": "html", "persistentVolumeClaim": {"claimName": "pv-volume"}}]' pod.yaml"""),
-# `k edit pvc pv-volume` works just as well. A claim only grows, and only when its class has
+              cmd(r"""k apply -f pod.yaml"""),
-# allowVolumeExpansion: true and its driver can expand; spec.resources.requests is what you ask
+              hand(r"""k -n %{ns} get pod web-server""",
-# for, status.capacity what you have. Filesystem volumes of this driver grow online - the Pod
+                   r"""Again until it is `Running`.""",
-# keeps running.
+                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/web-server --timeout=180s"""),
-""",
+              hand(r"""k -n %{ns} edit pvc pv-volume""",
+                   r"""Under `spec.resources.requests`, then `:wq` - a claim only grows:
+```
+      storage: %{size}
+```""",
+                   run=r"""kubectl -n %{ns} patch pvc pv-volume -p '{"spec":{"resources":{"requests":{"storage":"%{size}"}}}}'"""),
+              cmd(r"""k -n %{ns} get pvc pv-volume
+k -n %{ns} get pod web-server""", r"""Again until `CAPACITY %{size}`: `spec.resources.requests` is what you asked for, `status.capacity` what the driver has done. This driver grows filesystems online - the Pod keeps running.""")],
 },
 # ------------------------------------------------------------------ 147 ----
 {
@@ -963,71 +1142,132 @@ x() { $K -n %{ns} exec auditor -- cat "$1/ledger.txt" 2>/dev/null; }
 eq "$(x /restore)" entries-%{n} && eq "$(x /copy)" entries-%{n} \
   && eq "$(j -n %{ns} pod auditor '{.spec.volumes[*].persistentVolumeClaim.claimName}' | tr ' ' '\n' | sort | tr '\n' ' ')" "ledger-copy ledger-restore " """),
  ],
- "solution": """
+ "solution": [cmd(r"""k get volumesnapshotclass""", r"""`csi-hostpath-snapclass`, driver `hostpath.csi.k8s.io`."""),
-k get volumesnapshotclass                        # csi-hostpath-snapclass, driver hostpath.csi.k8s.io
+              hand(r"""vim snap.yaml""",
-k -n %{ns} apply -f - <<'EOF'
+                   r"""The docs page *Volume Snapshots* has the VolumeSnapshot:
+```
 apiVersion: snapshot.storage.k8s.io/v1
 kind: VolumeSnapshot
 metadata:
   name: ledger-snap
+  namespace: %{ns}
 spec:
   volumeSnapshotClassName: csi-hostpath-snapclass
   source:
     persistentVolumeClaimName: ledger-data
-EOF
+```""",
-k -n %{ns} wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/ledger-snap --timeout=120s
+                   run=r"""cat > snap.yaml <<'EOF'
-k -n %{ns} apply -f - <<'EOF'
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshot
+metadata:
+  name: ledger-snap
+  namespace: %{ns}
+spec:
+  volumeSnapshotClassName: csi-hostpath-snapclass
+  source:
+    persistentVolumeClaimName: ledger-data
+EOF"""),
+              cmd(r"""k apply -f snap.yaml"""),
+              hand(r"""k -n %{ns} get volumesnapshot ledger-snap""",
+                   r"""Again until `READYTOUSE true`.""",
+                   run=r"""kubectl -n %{ns} wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/ledger-snap --timeout=120s"""),
+              hand(r"""vim claims.yaml""",
+                   r"""`dataSource` makes a claim from a snapshot (*Volume Snapshots*) or from another claim of the same class and namespace - a clone (*CSI Volume Cloning*; no `apiGroup`, claims are core). Both at least as big as the source:
+```
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: ledger-restore
+  namespace: %{ns}
 spec:
   storageClassName: csi-hostpath-sc
-  accessModes: [ReadWriteOnce]
-  resources:
-    requests: {storage: 50Mi}
   dataSource:
     apiGroup: snapshot.storage.k8s.io
     kind: VolumeSnapshot
     name: ledger-snap
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 50Mi
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: ledger-copy
+  namespace: %{ns}
 spec:
   storageClassName: csi-hostpath-sc
-  accessModes: [ReadWriteOnce]
-  resources:
-    requests: {storage: 50Mi}
   dataSource:
     kind: PersistentVolumeClaim
     name: ledger-data
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 50Mi
+```""",
+                   run=r"""cat > claims.yaml <<'EOF'
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: ledger-restore
+  namespace: %{ns}
+spec:
+  storageClassName: csi-hostpath-sc
+  dataSource:
+    apiGroup: snapshot.storage.k8s.io
+    kind: VolumeSnapshot
+    name: ledger-snap
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 50Mi
 ---
 apiVersion: v1
-kind: Pod
+kind: PersistentVolumeClaim
 metadata:
-  name: auditor
+  name: ledger-copy
+  namespace: %{ns}
 spec:
-  containers:
+  storageClassName: csi-hostpath-sc
-  - name: auditor
+  dataSource:
-    image: busybox:1.36
+    kind: PersistentVolumeClaim
-    command: [sleep, '86400']
+    name: ledger-data
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 50Mi
+EOF"""),
+              cmd(r"""k apply -f claims.yaml
+k -n %{ns} run auditor --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
+              hand(r"""vim pod.yaml""",
+                   r"""Add to the container, below `name: auditor`:
+```
     volumeMounts:
-    - {name: restore, mountPath: /restore}
+    - name: restore
-    - {name: copy, mountPath: /copy}
+      mountPath: /restore
+    - name: copy
+      mountPath: /copy
+```
+and to the Pod's `spec:`, right below `spec:`:
+```
   volumes:
   - name: restore
-    persistentVolumeClaim: {claimName: ledger-restore}
+    persistentVolumeClaim:
+      claimName: ledger-restore
   - name: copy
-    persistentVolumeClaim: {claimName: ledger-copy}
+    persistentVolumeClaim:
-EOF
+      claimName: ledger-copy
-k -n %{ns} wait --for=condition=Ready pod/auditor --timeout=180s
+```""",
-k -n %{ns} exec auditor -- cat /restore/ledger.txt /copy/ledger.txt
+                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "restore", "mountPath": "/restore"}, {"name": "copy", "mountPath": "/copy"}]
-# A snapshot is taken by the CSI driver (VolumeSnapshotClass -> driver); `dataSource` makes a new
+  | .spec.volumes = [{"name": "restore", "persistentVolumeClaim": {"claimName": "ledger-restore"}}, {"name": "copy", "persistentVolumeClaim": {"claimName": "ledger-copy"}}]' pod.yaml"""),
-# claim from it - apiGroup snapshot.storage.k8s.io - or from another claim of the same class and
+              cmd(r"""k apply -f pod.yaml"""),
-# Namespace, a clone (no apiGroup: claims are core). Both need a CSI driver that can do it, and
+              hand(r"""k -n %{ns} get pvc,pod""",
-# the new claim must be at least as big as the source.
+                   r"""Again until both claims are `Bound` and `auditor` `Running`.""",
-""",
+                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/auditor --timeout=180s"""),
+              cmd(r"""k -n %{ns} exec auditor -- cat /restore/ledger.txt /copy/ledger.txt""", r"""`entries-%{n}`, twice.""")],
 },
 ]

cka/bank_hard.py

@@ -186,7 +186,9 @@ $K delete ns %{ns} --ignore-not-found --wait=false >/dev/null
 },
 {
  "id": 49, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults",
- "params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])},
+ # low, short: the plural's stem and the shortName, which the solution spells out
+ "params": lambda r: (lambda p: dict(p, low=p["kind"].lower(), short=p["kind"][:2].lower()))(
+     {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])}),
  "task": CTX + """
 Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the kind in lowercase plus
 `s`, shortName = the first two letters of the kind in lowercase) with schema:

cka/bank_net.py

@@ -5,6 +5,8 @@ Written for the per-task hosts: a task may rewrite the cluster's CoreDNS configu
 resolution and hostPorts are tested for real; Ingress and Gateway objects are graded as objects,
 as no controller for them runs on the hosts.
 """
+
+from sim.bank import cmd, hand
 from cka.common import C, DNS, NS, ONE, PODS, PY, ns_pick, name_pick
 
 QUESTIONS = [
@@ -153,12 +155,15 @@ parents == [("%{gw}", "%{ns}")] and s.get("hostnames") == ["%{host}.example.com"
 out=$($K -n %{ns} get ingress %{host} 2>&1); case "$out" in *NotFound*) ;; *) exit 1 ;; esac
 eq "$(j -n %{ns} secret %{host}-tls '{.metadata.uid}')" "$(cat "$PRIV/q126.uid")" """),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} get ingress %{host} -o yaml""", r"""The host, the TLS Secret `%{host}-tls`, `/api` to `api:8080` and `/` to `web:80`."""),
-k -n %{ns} get ingress %{host} -o yaml     # host, the TLS Secret (%{host}-tls), /api -> api:8080, / -> web:80
+              hand(r"""vim gw.yaml""",
-cat <<EOF | k apply -f -
+                   r"""What the Ingress keeps in one object, the Gateway API splits: the listener - port, hostname, certificate - goes in the Gateway, the routing in the HTTPRoute. The docs page *Gateway API* has both; `k explain gateway.spec.listeners.tls` the TLS fields:
+```
 apiVersion: gateway.networking.k8s.io/v1
 kind: Gateway
-metadata: {name: %{gw}, namespace: %{ns}}
+metadata:
+  name: %{gw}
+  namespace: %{ns}
 spec:
   gatewayClassName: %{gc}
   listeners:
@@ -168,28 +173,84 @@ spec:
     hostname: %{host}.example.com
     tls:
       mode: Terminate
-        certificateRefs: [{kind: Secret, name: %{host}-tls}]
+      certificateRefs:
+      - kind: Secret
+        name: %{host}-tls
 ---
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
-metadata: {name: %{host}, namespace: %{ns}}
+metadata:
+  name: %{host}
+  namespace: %{ns}
 spec:
-  parentRefs: [{name: %{gw}}]
+  parentRefs:
-  hostnames: [%{host}.example.com]
+  - name: %{gw}
+  hostnames:
+  - %{host}.example.com
   rules:
-    - matches: [{path: {type: PathPrefix, value: /api}}]
+  - matches:
-      backendRefs: [{name: api, port: 8080}]
+    - path:
-    - matches: [{path: {type: PathPrefix, value: /}}]
+        type: PathPrefix
-      backendRefs: [{name: web, port: 80}]
+        value: /api
-EOF
+    backendRefs:
+    - name: api
+      port: 8080
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /
+    backendRefs:
+    - name: web
+      port: 80
+```""",
+                   run=r"""cat > gw.yaml <<'EOF'
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  name: %{gw}
+  namespace: %{ns}
+spec:
+  gatewayClassName: %{gc}
+  listeners:
+  - name: https
+    protocol: HTTPS
+    port: 443
+    hostname: %{host}.example.com
+    tls:
+      mode: Terminate
+      certificateRefs:
+      - kind: Secret
+        name: %{host}-tls
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: %{host}
+  namespace: %{ns}
+spec:
+  parentRefs:
+  - name: %{gw}
+  hostnames:
+  - %{host}.example.com
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /api
+    backendRefs:
+    - name: api
+      port: 8080
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /
+    backendRefs:
+    - name: web
+      port: 80
+EOF"""),
+              cmd(r"""k apply -f gw.yaml
 k -n %{ns} get gateway,httproute
-k -n %{ns} delete ingress %{host}
+k -n %{ns} delete ingress %{host}""")],
-# What an Ingress keeps in one object, the Gateway API splits by role: the Gateway is the
-# listener - port, hostname, certificate - and belongs to whoever runs the edge; the HTTPRoute is
-# the routing and belongs to the application. spec.tls of the Ingress becomes tls.certificateRefs
-# on the listener, each path becomes a rule with a PathPrefix match, and a Service backend becomes
-# a backendRef. `k explain gateway.spec.listeners.tls` and the Gateway API docs are allowed.
-""",
 },
 # ------------------------------------------------------------------ 127 ----
 {
@@ -399,23 +460,37 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.hostNetwork}')" true || exit 1
 pip=$(j -n %{ns} pod %{pod} '{.status.podIP}'); hip=$(j -n %{ns} pod %{pod} '{.status.hostIP}')
 [ -n "$pip" ] && eq "$pip" "$hip" && eq "$(tr -s ' \t\r\n' ' ' < "$WS/q128/ips.txt" 2>/dev/null | sed 's/^ //; s/ $//')" "$pip $hip" """),
  ],
- "solution": """
+ "solution": [hand(r"""k -n %{ns} edit ds %{ds}""",
-k -n %{ns} edit ds %{ds}                   # under the container:
+                   r"""Give the container's port a `hostPort`, then `:wq`:
-#   ports: [{containerPort: 8080, hostPort: %{hp}}]
+```
-k -n %{ns} rollout status ds %{ds}
+        ports:
-k get nodes -o wide                        # INTERNAL-IP
+        - containerPort: 8080
-curl -s http://<a worker's INTERNAL-IP>:%{hp}/            # agent on <that node>
+          hostPort: %{hp}
-
+          protocol: TCP
-k -n %{ns} logs %{pod} --tail=2            # bad address %{svc}
+```
-k -n %{ns} get pod %{pod} -o jsonpath='{.spec.dnsPolicy}'; echo       # ClusterFirst - which a hostNetwork Pod does not get
+`hostPort` maps one port of the node to the Pod on it (the CNI's `portmap` plugin writes the rule); `hostNetwork` would put the whole Pod in the node's network.""",
-vi {WS}/q128/collector.yaml                # spec.dnsPolicy: ClusterFirstWithHostNet
+                   run=r"""kubectl -n %{ns} patch ds %{ds} --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/ports/0/hostPort", "value": %{hp}}]'"""),
-k -n %{ns} delete pod %{pod}; k apply -f {WS}/q128/collector.yaml
+              cmd(r"""k -n %{ns} rollout status ds %{ds}
-k -n %{ns} logs %{pod} --tail=2            # resolved %{svc}
+k get nodes -o wide""", r"""Note a worker's `INTERNAL-IP`."""),
-k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt
+              hand(r"""curl -s http://172.18.0.3:%{hp}/""",
-# hostPort maps one port of the node to one Pod on it (the CNI's portmap plugin writes the rule);
+                   r"""With that IP: `agent on <that node>`.""",
-# hostNetwork puts the whole Pod into the node's network namespace - its IP IS the node's IP, and
+                   run=r""""""),
-# it uses the node's /etc/resolv.conf, which knows nothing of cluster.local, unless dnsPolicy is
+              cmd(r"""k -n %{ns} logs %{pod} --tail=2
-# ClusterFirstWithHostNet. dnsPolicy cannot be changed on a running Pod: recreate it.
+k -n %{ns} get pod %{pod} -o yaml | grep -e hostNetwork -e dnsPolicy""", r"""`bad address`: a `hostNetwork` Pod uses the node's `/etc/resolv.conf`, which knows nothing of `cluster.local` - unless its `dnsPolicy` is `ClusterFirstWithHostNet`."""),
-""",
+              hand(r"""vim {WS}/q128/collector.yaml""",
+                   r"""In the Pod's `spec:`, below `hostNetwork: true`:
+```
+  dnsPolicy: ClusterFirstWithHostNet
+```""",
+                   run=r"""yq -i '.spec.dnsPolicy = "ClusterFirstWithHostNet"' {WS}/q128/collector.yaml"""),
+              cmd(r"""k replace --force -f {WS}/q128/collector.yaml""", r"""A running Pod's `dnsPolicy` cannot change: `--force` deletes it and creates it again."""),
+              hand(r"""k -n %{ns} logs %{pod} --tail=2
+k -n %{ns} get pod %{pod} -o wide""",
+                   r"""Again until `resolved %{svc}`. Its IP is its node's: note it.""",
+                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=180s
+sleep 8"""),
+              hand(r"""echo "172.18.0.3 172.18.0.3" > {WS}/q128/ips.txt""",
+                   r"""The Pod's IP and its node's - the same address.""",
+                   run=r"""kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt""")],
 },
 ]

cka/bank_objectives.py

@@ -20,7 +20,7 @@ kubeadm, etcd, the runtime. All normal difficulty, all in the random pool; nothi
 node or a control-plane component (109 cordons the control-plane node while you work on it and has
 you uncordon it; the session baseline uncordons it if you do not).
 """
-from sim.bank import C, NS, ns_pick, name_pick
+from sim.bank import C, NS, ns_pick, name_pick, cmd, hand
 from ckad import bank_extra as _ckad_extra, bank_more as _ckad_more
 
 
@@ -30,18 +30,34 @@ def _port(bank, src_id, new_id, domain, **over):
     old, new = f"q{src_id}", f"q{new_id}"
     q = dict(src, id=new_id, domain=domain, **over)
     for k in ("task", "setup", "solution"):
+        if k not in over:   # a solution of its own (in parts) is the CKA one already
             q[k] = src[k].replace(old, new)
     q["checks"] = [dict(c, script=c["script"].replace(old, new), name=c["name"].replace(old, new)) for c in src["checks"]]
     return q
 
 
+# 103's own solution: the CKAD task's, done as in the exam
+_SOL103 = [hand(r"""k -n %{ns} top pod""",
+                r"""Again until it shows all three: metrics-server needs about 30 seconds after the Pods start. Without it, `Metrics API not available`.""",
+                run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} top pod --no-headers 2>/dev/null | wc -l)" -ge 3 ] && break; sleep 5; done"""),
+           cmd(r"""k -n %{ns} top pod --sort-by=cpu
+k -n %{ns} top pod --sort-by=memory""", r"""The first line of each is the heaviest."""),
+           hand(r"""echo %{cpu} > {WS}/q103/cpu.txt
+echo %{mem} > {WS}/q103/memory.txt""",
+                r"""With the names on those first lines.""",
+                run=r"""kubectl -n %{ns} top pod --sort-by=cpu --no-headers | head -1 | awk '{print $1}' > {WS}/q103/cpu.txt
+kubectl -n %{ns} top pod --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/q103/memory.txt"""),
+           cmd(r"""echo 'kubectl top node' > {WS}/q103/nodes.sh
+echo 'kubectl top pod -A --sort-by=memory' > {WS}/q103/pods.sh""", r"""`k top` reads the metrics API (`metrics.k8s.io`) that metrics-server serves; `--containers` breaks a Pod down per container.""")]
+
+
 ETCDCTL = ("etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
            "--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key")
 
 QUESTIONS = [
 # ---------------------------------------------------------------- TSHOOT ----
 # needs a working metrics API, which Q23 breaks on purpose: never in one session with it
-_port(_ckad_extra, 23, 103, "TSHOOT", breaks="metrics-server"),
+_port(_ckad_extra, 23, 103, "TSHOOT", breaks="metrics-server", solution=_SOL103),
 _port(_ckad_more, 64, 104, "TSHOOT"),
 # ------------------------------------------------------------------ WORK ----
 {
@@ -183,32 +199,44 @@ cg=$(nx "$W3" "awk '/^cgroupDriver:/{print \$2}' /var/lib/kubelet/config.yaml" <
 want=$(printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s' "$fwd" "$swap" "$fso" "$cg")
 eq "$(tr -d ' \t\r' < "$WS/q107/node.txt" 2>/dev/null | grep .)" "$want" """),
  ],
- "solution": """
+ "solution": [hand(r"""{SHELL_W3}
-{SHELL_W3}
 sudo -i
-printf 'overlay\\nbr_netfilter\\n' > /etc/modules-load.d/%{name}.conf
+vim /etc/modules-load.d/%{name}.conf""",
-cat > /etc/sysctl.d/%{name}.conf <<EOF
+                   r"""The docs page *Container Runtimes* has both files under *Install and configure prerequisites*. One module per line:
+```
+overlay
+br_netfilter
+```""",
+                   run=r"""ssh -n {W3} 'printf "overlay\nbr_netfilter\n" | sudo tee /etc/modules-load.d/%{name}.conf >/dev/null'"""),
+              hand(r"""vim /etc/sysctl.d/%{name}.conf""",
+                   r"""```
 net.bridge.bridge-nf-call-iptables  = 1
 net.bridge.bridge-nf-call-ip6tables = 1
 net.ipv4.ip_forward                 = 1
-EOF
+```
-# on a real machine you now load and apply them without a reboot:
+On a real machine they would now be loaded and applied without a reboot: `modprobe overlay`, `modprobe br_netfilter`, `sysctl --system`. `sysctl -w` and `modprobe` alone are gone after a reboot; the files make them stick.""",
-#   modprobe overlay && modprobe br_netfilter && sysctl --system
+                   run=r"""ssh -n {W3} 'printf "net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\n" | sudo tee /etc/sysctl.d/%{name}.conf >/dev/null'"""),
-sysctl -n net.ipv4.ip_forward                              # 1
+              hand(r"""sysctl -n net.ipv4.ip_forward
-cat /proc/swaps                                            # any line under the header = swap is on  (swapon --show)
+cat /proc/swaps
-grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml
+grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml
-exit; exit
+exit
-cat > {WS}/q107/node.txt <<EOF
+exit""",
-ip_forward=1
+                   r"""Any line below the header of `/proc/swaps` means swap is on. No `failSwapOn` line means the default, `true`.""",
-swap=<on|off>
+                   run=r""""""),
-fail-swap-on=<true|false>
+              hand(r"""vim {WS}/q107/node.txt""",
-cgroup-driver=<systemd|cgroupfs>
+                   r"""What the node printed:
-EOF
+```
-# `sysctl -w` and `modprobe` alone are gone after a reboot - the files are what makes them stick.
+ip_forward=<0 or 1>
-# kubeadm's preflight refuses a node with swap on unless the kubelet is told to tolerate it
+swap=<on or off>
-# (failSwapOn: false), and the kubelet's cgroup driver has to match the container runtime's
+fail-swap-on=<true or false>
-# (SystemdCgroup = true in /etc/containerd/config.toml goes with cgroupDriver: systemd).
+cgroup-driver=<systemd or cgroupfs>
-""",
+```
+kubeadm's preflight refuses a node with swap on unless the kubelet tolerates it (`failSwapOn: false`); and the kubelet's cgroup driver has to match the runtime's (`SystemdCgroup = true` in containerd goes with `cgroupDriver: systemd`).""",
+                   run=r"""IPF=$(ssh -n {W3} 'sudo sysctl -n net.ipv4.ip_forward')
+SW=off; [ "$(ssh -n {W3} 'sed 1d /proc/swaps | grep -c .')" -gt 0 ] && SW=on
+FS=$(ssh -n {W3} "sudo awk '/^failSwapOn:/ {print \$2}' /var/lib/kubelet/config.yaml"); [ -n "$FS" ] || FS=true
+CG=$(ssh -n {W3} "sudo awk '/^cgroupDriver:/ {print \$2}' /var/lib/kubelet/config.yaml")
+printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s\n' "$IPF" "$SW" "$FS" "$CG" > {WS}/q107/node.txt""")],
 },
 {
  "id": 108, "domain": "ARCH", "title": "The container runtime behind a Pod",
@@ -263,29 +291,35 @@ cid=$($K -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].cont
 log=$(nx "$W2" "crictl inspect -o go-template --template '{{.status.logPath}}' $cid 2>/dev/null" </dev/null | tr -d '[:space:]')
 [ -n "$cid" ] && [ -n "$log" ] && tr -d ' \t\r' < "$WS/q108/runtime.txt" 2>/dev/null | grep -qx "log=$log" """),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} get pod %{pod} -o wide""", r"""On `{W2}`."""),
-k -n %{ns} get pod %{pod} -o wide                       # on {W2}
+              hand(r"""{SHELL_W2}
-{SHELL_W2}
 sudo -i
-crictl version                                          # RuntimeName: containerd   RuntimeVersion: v2.x.y
+crictl version
-grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml       # unix:///run/containerd/containerd.sock
+grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
-crictl pods --name %{pod}                               # the Pod sandbox
+crictl ps --no-trunc --name app""",
-crictl ps --no-trunc --name app --pod $(crictl pods --name %{pod} -q)     # full container ID
+                   r"""`RuntimeName` and `RuntimeVersion`; the CRI socket the kubelet uses - crictl speaks the same protocol, which is why it works even when the API server does not; and the container's full ID (`--no-trunc`), the line whose `POD` is `%{pod}`.""",
-crictl inspect <id> | grep -E '"pid"|logPath'           # .info.pid and .status.logPath
+                   run=r""""""),
-#   or:  crictl inspect -o go-template --template '{{.info.pid}} {{.status.logPath}}' <id>
+              hand(r"""crictl inspect 0123abcd... | grep -e '"pid"' -e logPath
-exit; exit
+exit
-cat > {WS}/q108/runtime.txt <<EOF
+exit""",
+                   r"""With that ID: `.info.pid` and `.status.logPath`. The log is the file `kubectl logs` reads, under `/var/log/pods`.""",
+                   run=r""""""),
+              hand(r"""vim {WS}/q108/runtime.txt""",
+                   r"""The values you read:
+```
 runtime=containerd <version>
 socket=unix:///run/containerd/containerd.sock
-container-id=<64 hex characters>
+container-id=<the 64 hex characters>
-pid=<number>
+pid=<the number>
 log=/var/log/pods/%{ns}_%{pod}_<pod uid>/app/0.log
-EOF
+```""",
-# The kubelet drives the runtime over the CRI socket, and crictl speaks the same protocol - which
+                   run=r"""V=$(ssh -n {W2} 'sudo crictl version 2>/dev/null')
-# is why it still works when the API server does not. `kubectl get pod -o
+RN=$(echo "$V" | awk '/^RuntimeName:/ {print $2}'); RV=$(echo "$V" | awk '/^RuntimeVersion:/ {print $2}')
-# jsonpath='{.status.containerStatuses[0].containerID}'` shows the same ID with a containerd://
+SOCK=$(ssh -n {W2} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
-# prefix; /var/log/pods/... is the file `kubectl logs` reads.
+CID=$(kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#^[a-z-]*://##')
-""",
+PID=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.info.pid}}' $CID 2>/dev/null")
+LOG=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.status.logPath}}' $CID 2>/dev/null")
+printf 'runtime=%s %s\nsocket=%s\ncontainer-id=%s\npid=%s\nlog=%s\n' "$RN" "$RV" "$SOCK" "$CID" "$PID" "$LOG" > {WS}/q108/runtime.txt""")],
 },
 {
  # drains the control-plane node: CoreDNS moves off it, so never beside the tasks that break DNS
@@ -468,29 +502,33 @@ c=$(j -n kube-system lease kube-controller-manager '{.spec.holderIdentity}' | cu
 f=$(tr -d ' \t\r' < "$WS/q111/ha.txt" 2>/dev/null)
 [ -n "$s" ] && [ -n "$c" ] && echo "$f" | grep -qx "scheduler-leader=$s" && echo "$f" | grep -qx "controller-manager-leader=$c" """),
  ],
- "solution": """
+ "solution": [cmd(r"""k get nodes -l node-role.kubernetes.io/control-plane
-k get nodes -l node-role.kubernetes.io/control-plane              # how many
+k -n kube-system get pods -l component=etcd -o wide
-k -n kube-system get pods -l component=etcd -o wide               # an etcd static Pod on the control-plane node = stacked
+k -n kube-system get lease kube-scheduler kube-controller-manager""", r"""An etcd static Pod on the control-plane node: stacked. A Lease's `HOLDER` is `<node>_<uuid>`."""),
-k -n kube-system get lease kube-scheduler kube-controller-manager # HOLDER is <node>_<uuid>
+              hand(r"""sudo -i
-sudo -i                # the task's host is the control-plane node: no hop needed
+export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
-export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \\
+etcdctl member list -w table
-       ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
+etcdctl endpoint status -w table
-etcdctl member list -w table                                      # one line per member: ID, NAME, peer and client URLs
+exit""",
-etcdctl endpoint status -w table                                  # IS LEADER (add --cluster to ask every member)
+                   r"""The host is the control-plane node. One line per member; `IS LEADER` in the status (`--cluster` asks every member).""",
-exit
+                   run=r""""""),
-cat > {WS}/q111/ha.txt <<EOF
+              hand(r"""vim {WS}/q111/ha.txt""",
+                   r"""```
 control-plane-nodes=1
 etcd-topology=stacked
 etcd-members=1
 etcd-leader={CP}
 scheduler-leader={CP}
 controller-manager-leader={CP}
-EOF
+```
-# In an HA control plane every kube-apiserver serves at once, but only one kube-scheduler and one
+In an HA control plane every API server serves at once, but one scheduler and one controller manager act, through the leader Lease. etcd elects its own leader and needs a majority to stay writable: three members survive the loss of one, five of two.""",
-# kube-controller-manager act: they elect a leader through a Lease in kube-system, and the others
+                   run=r"""E='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
-# wait. etcd elects its own leader and needs a majority of members to stay writable - three
+N=$(kubectl get nodes -l node-role.kubernetes.io/control-plane --no-headers | wc -l)
-# members survive the loss of one, five of two, and an even number adds nothing.
+M=$($E member list | wc -l)
-""",
+L=$($E member list | awk -F', ' 'NR==1 {print $3}')
+S=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
+C=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
+printf 'control-plane-nodes=%s\netcd-topology=stacked\netcd-members=%s\netcd-leader=%s\nscheduler-leader=%s\ncontroller-manager-leader=%s\n' "$N" "$M" "$L" "$S" "$C" > {WS}/q111/ha.txt""")],
 },
 {
  "id": 112, "domain": "ARCH", "title": "CNI, CSI and CRI: what this cluster plugs in",
@@ -532,30 +570,35 @@ prov=$($K get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclas
 f=$(tr -d ' \t\r' < "$WS/q112/interfaces.txt" 2>/dev/null)
 echo "$f" | grep -qx "csi-drivers=$n" && echo "$f" | grep -qx "default-provisioner=$prov" """),
  ],
- "solution": """
+ "solution": [hand(r"""{SHELL_W1}
-{SHELL_W1}
 sudo -i
-ls /etc/cni/net.d/                                    # 10-kindnet.conflist - the kubelet's runtime takes the first file
+ls /etc/cni/net.d
-cat /etc/cni/net.d/10-kindnet.conflist                # "plugins": [{"type": "ptp", "ipam": {"type": "host-local"}}, {"type": "portmap"}]
+cat /etc/cni/net.d/10-kindnet.conflist
-ls /opt/cni/bin                                       # the plugin binaries those types name
 grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
-exit; exit
+exit
-k get csidrivers                                      # No resources found = 0
+exit""",
-k get sc                                              # the (default) class and its PROVISIONER
+                   r"""The runtime takes the first file of `/etc/cni/net.d`. Its `plugins` chain, in order - each `type` names a binary in `/opt/cni/bin` - and the `ipam` type inside one of them.""",
-cat > {WS}/q112/interfaces.txt <<EOF
+                   run=r""""""),
+              cmd(r"""k get csidrivers
+k get sc""", r"""`No resources found`: no CSI driver - the local-path provisioner is not one. The `(default)` class's `PROVISIONER`."""),
+              hand(r"""vim {WS}/q112/interfaces.txt""",
+                   r"""The values you read - here:
+```
 cni-config=/etc/cni/net.d/10-kindnet.conflist
 cni-plugins=ptp,portmap
 cni-ipam=host-local
 cri-socket=unix:///run/containerd/containerd.sock
 csi-drivers=0
 default-provisioner=rancher.io/local-path
-EOF
+```
-# CRI: the kubelet talks to the runtime over that socket (crictl uses the same one). CNI: the
+`ptp` wires each Pod's veth pair, `host-local` hands out addresses from the node's Pod CIDR, `portmap` serves hostPorts.""",
-# runtime calls the plugins of the chain in order for every Pod sandbox - here ptp wires the veth
+                   run=r"""CONF=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
-# pair, host-local hands out an address from the node's Pod CIDR, portmap serves hostPorts.
+PL=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[].type] | join(",")')
-# CSI: a storage driver registers a CSIDriver object and a StorageClass names it as provisioner;
+IPAM=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[] | select(.ipam) | .ipam.type][0]')
-# this cluster's local-path provisioner is not a CSI driver, hence none.
+SOCK=$(ssh -n {W1} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
-""",
+N=$(kubectl get csidrivers --no-headers 2>/dev/null | wc -l)
+PROV=$(kubectl get sc -o json | jq -r '.items[] | select(.metadata.annotations["storageclass.kubernetes.io/is-default-class"] == "true") | .provisioner')
+printf 'cni-config=%s\ncni-plugins=%s\ncni-ipam=%s\ncri-socket=%s\ncsi-drivers=%s\ndefault-provisioner=%s\n' "$CONF" "$PL" "$IPAM" "$SOCK" "$N" "$PROV" > {WS}/q112/interfaces.txt""")],
 },
 # ------------------------------------------------------------------- NET ----
 {
@@ -623,19 +666,18 @@ eq "$(tr -s ' \t' ' ' < "$WS/q113/pod-cidrs.txt" 2>/dev/null | tr -d '\r' | sed
   C("outside.txt names %{edge}, why.txt names hostNetwork", 2, r"""
 eq "$(tr -d '[:space:]' < "$WS/q113/outside.txt" 2>/dev/null | sed 's#^pod/##')" %{edge} && grep -qi 'hostNetwork' "$WS/q113/why.txt" 2>/dev/null"""),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} get pods -o wide
-k -n %{ns} get pods -o wide                           # IP and NODE of every Pod
+k get nodes -o wide""", r"""The `IP` and `NODE` of every Pod. `%{edge}`'s IP is its node's own `INTERNAL-IP`, not one from a Pod CIDR."""),
-k -n %{ns} get pod %{web} -o jsonpath='{.status.podIP} {.spec.nodeName}' > {WS}/q113/web.txt
+              hand(r"""echo "10.244.1.5 {W1}" > {WS}/q113/web.txt""",
-k -n %{ns} exec %{client} -- wget -qO- -T3 http://$(cut -d' ' -f1 {WS}/q113/web.txt) > {WS}/q113/direct.html
+                   r"""With `%{web}`'s IP and node.""",
-k get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\\n"}{end}' > {WS}/q113/pod-cidrs.txt
+                   run=r"""kubectl -n %{ns} get pod %{web} -o jsonpath='{.status.podIP} {.spec.nodeName}' > {WS}/q113/web.txt"""),
-#   %{edge}'s IP is its node's own address (compare with `k get nodes -o wide`), not one from 10.244.x.0/24
+              hand(r"""k -n %{ns} exec %{client} -- wget -qO- -T3 http://10.244.1.5 > {WS}/q113/direct.html""",
-echo %{edge} > {WS}/q113/outside.txt
+                   r"""With that IP: Pod to Pod, across nodes, no Service and no NAT - the network model the CNI makes true.""",
+                   run=r"""kubectl -n %{ns} exec %{client} -- wget -qO- -T3 http://$(kubectl -n %{ns} get pod %{web} -o jsonpath='{.status.podIP}') > {WS}/q113/direct.html"""),
+              cmd(r"""k get nodes -o custom-columns=NAME:.metadata.name,CIDR:.spec.podCIDR --no-headers > {WS}/q113/pod-cidrs.txt
+cat {WS}/q113/pod-cidrs.txt
 k -n %{ns} get pod %{edge} -o yaml | grep hostNetwork
-echo hostNetwork > {WS}/q113/why.txt
+echo %{edge} > {WS}/q113/outside.txt
-# Every Pod gets its own address from its node's Pod CIDR, and any Pod reaches any other Pod on
+echo hostNetwork > {WS}/q113/why.txt""", r"""A `hostNetwork` Pod lives in its node's network namespace and shares the node's address and ports.""")],
-# any node at that address without NAT - that is the whole network model, and the CNI plugin is
-# what makes it true. A hostNetwork Pod opts out: it lives in the node's network namespace and
-# shares its address and ports. Pod addresses change with every new Pod, which is why Services exist.
-""",
 },
 ]

cka/bank_sim_c.py

@@ -12,7 +12,7 @@ Two scenarios are deliberately adapted so they cannot wreck the practice cluster
 
 Run with:  ./ksim sim c
 """
-from sim.bank import C, NS
+from sim.bank import C, NS, cmd, hand
 
 Q = []
 
@@ -62,42 +62,87 @@ eq "$(j -n geo-fence pod geofence-reader '{.status.phase}')" Running \
  && eq "$(j -n geo-fence pod geofence-reader '{.spec.volumes[0].persistentVolumeClaim.claimName}')" geofence-pvc \
  && j -n geo-fence pod geofence-reader '{.spec.containers[0].volumeMounts[*].mountPath}' | grep -q /geo-data"""),
     ],
-    solution="""
+    solution=[hand(r"""vim pv.yaml""",
-cat <<EOF | k apply -f -
+                   r"""The PersistentVolume and PersistentVolumeClaim from the docs page *Configure a Pod to Use a PersistentVolume for Storage*, `---` between them. `volumeName` pins the claim to this PV: without it the binder takes any PV whose size, access mode and class fit.
+```
 apiVersion: v1
 kind: PersistentVolume
-metadata: {name: geofence-pv}
+metadata:
+  name: geofence-pv
 spec:
-  capacity: {storage: 250Mi}
-  accessModes: [ReadWriteOnce]
-  hostPath: {path: /data/geofence}
   storageClassName: manual
+  capacity:
+    storage: 250Mi
+  accessModes:
+  - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
+  hostPath:
+    path: /data/geofence
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata: {name: geofence-pvc, namespace: geo-fence}
+metadata:
+  name: geofence-pvc
+  namespace: geo-fence
 spec:
-  accessModes: [ReadWriteOnce]
-  resources: {requests: {storage: 250Mi}}
   storageClassName: manual
   volumeName: geofence-pv
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 250Mi
+```""",
+                   run=r"""cat > pv.yaml <<'EOF'
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: geofence-pv
+spec:
+  storageClassName: manual
+  capacity:
+    storage: 250Mi
+  accessModes:
+  - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  hostPath:
+    path: /data/geofence
 ---
 apiVersion: v1
-kind: Pod
+kind: PersistentVolumeClaim
-metadata: {name: geofence-reader, namespace: geo-fence}
+metadata:
+  name: geofence-pvc
+  namespace: geo-fence
 spec:
-  volumes: [{name: geo, persistentVolumeClaim: {claimName: geofence-pvc}}]
+  storageClassName: manual
-  containers:
+  volumeName: geofence-pv
-    - name: reader
+  accessModes:
-      image: busybox:1.36
+  - ReadWriteOnce
-      command: [sh, -c, "ls /geo-data && sleep 3600"]
+  resources:
-      volumeMounts: [{name: geo, mountPath: /geo-data}]
+    requests:
-EOF
+      storage: 250Mi
+EOF"""),
+              cmd(r"""k apply -f pv.yaml
 k -n geo-fence get pvc
-# volumeName pins the claim to one PV by name. Without it the binder picks ANY PV whose capacity,
+k -n geo-fence run geofence-reader --image=busybox:1.36 --dry-run=client -o yaml --command -- sh -c "ls /geo-data && sleep 3600" > pod.yaml""", r"""`Bound` to `geofence-pv`."""),
-# accessModes and storageClassName match - which on a busy cluster is rarely the one you meant.
+              hand(r"""vim pod.yaml""",
-""",
+                   r"""Add to the container, below `name: geofence-reader`:
+```
+    volumeMounts:
+    - name: geo
+      mountPath: /geo-data
+```
+and to the Pod's `spec:`, right below `spec:`:
+```
+  volumes:
+  - name: geo
+    persistentVolumeClaim:
+      claimName: geofence-pvc
+```""",
+                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "geo", "mountPath": "/geo-data"}]
+  | .spec.volumes = [{"name": "geo", "persistentVolumeClaim": {"claimName": "geofence-pvc"}}]' pod.yaml"""),
+              cmd(r"""k apply -f pod.yaml
+k -n geo-fence get pod geofence-reader""")],
 )
 
 # ------------------------------------------------------------------ 69 ----
@@ -260,15 +305,19 @@ want=$(j -n dispatch-hub svc dispatch-service '{.spec.clusterIP}')
 got=$(tr -d '[:space:]' < "$WS/q71/service-ip.txt" 2>/dev/null)
 [ -n "$want" ] && eq "$got" "$want" """),
     ],
-    solution="""
+    solution=[cmd(r"""k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
-k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
+k -n dispatch-hub get svc,endpointslices""", r"""`expose` copies the Deployment's selector, which is what makes the endpoints appear. Note the Service's `CLUSTER-IP`."""),
-k -n dispatch-hub get endpointslices -l kubernetes.io/service-name=dispatch-service   # must not be empty
+              hand(r"""k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt
-k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- \\
+cat {WS}/q71/test-result.txt""",
-  wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt
+                   r"""nginx's welcome page, fetched from a Pod in the cluster.""",
-k -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt
+                   run=r"""kubectl -n dispatch-hub rollout status deploy dispatch-api --timeout=240s
-# `expose` copies the Deployment's selector for you, which is what makes the endpoints appear.
+for i in 1 2 3 4 5; do
-# Write the IP with jsonpath, not by eyeballing `get svc` - the grader compares it literally.
+  kubectl -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt </dev/null 2>/dev/null
-""",
+  grep -qi nginx {WS}/q71/test-result.txt && break; kubectl -n dispatch-hub delete pod probe --ignore-not-found >/dev/null 2>&1; sleep 3
+done"""),
+              hand(r"""echo 10.96.123.45 > {WS}/q71/service-ip.txt""",
+                   r"""With the `CLUSTER-IP` that `get svc` showed - the address alone.""",
+                   run=r"""kubectl -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt""")],
 )
 
 # ------------------------------------------------------------------ 72 ----
@@ -358,54 +407,102 @@ j -n ride-history deploy ride-archive '{.spec.template.spec.volumes[*].persisten
 j -n ride-history deploy ride-archive '{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' | grep -qw ride-archive-pvc-new || exit 1
 for i in $(seq 1 20); do [ "$(j -n ride-history deploy ride-archive '{.status.availableReplicas}')" -ge 1 ] 2>/dev/null && exit 0; sleep 5; done; exit 1"""),
     ],
-    solution="""
+    solution=[cmd(r"""k -n ride-history get pvc
-k -n ride-history get pvc,pv                    # note reclaimPolicy Delete on the old PV
+k get pv ride-archive-pv""", r"""`RECLAIM POLICY Delete` on the old PV: losing the claim loses the data."""),
-cat <<EOF | k apply -f -
+              hand(r"""vim new.yaml""",
+                   r"""The new PV and its claim, pinned to it by name (the docs page *Configure a Pod to Use a PersistentVolume for Storage* has both):
+```
 apiVersion: v1
 kind: PersistentVolume
-metadata: {name: ride-archive-pv-new}
+metadata:
+  name: ride-archive-pv-new
 spec:
-  capacity: {storage: 500Mi}
-  accessModes: [ReadWriteOnce]
-  hostPath: {path: /data/ride-archive-new}
   storageClassName: manual
+  capacity:
+    storage: 500Mi
+  accessModes:
+  - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
+  hostPath:
+    path: /data/ride-archive-new
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata: {name: ride-archive-pvc-new, namespace: ride-history}
+metadata:
+  name: ride-archive-pvc-new
+  namespace: ride-history
 spec:
-  accessModes: [ReadWriteOnce]
-  resources: {requests: {storage: 500Mi}}
   storageClassName: manual
   volumeName: ride-archive-pv-new
-EOF
+  accessModes:
-k -n ride-history scale deploy ride-archive --replicas=0     # release the RWO claim first
+  - ReadWriteOnce
-cat <<EOF | k apply -f -
+  resources:
+    requests:
+      storage: 500Mi
+```""",
+                   run=r"""cat > new.yaml <<'EOF'
 apiVersion: v1
-kind: Pod
+kind: PersistentVolume
-metadata: {name: data-migrator, namespace: ride-history}
+metadata:
+  name: ride-archive-pv-new
 spec:
-  restartPolicy: Never
+  storageClassName: manual
-  volumes:
+  capacity:
-    - {name: old, persistentVolumeClaim: {claimName: ride-archive-pvc}}
+    storage: 500Mi
-    - {name: new, persistentVolumeClaim: {claimName: ride-archive-pvc-new}}
+  accessModes:
-  containers:
+  - ReadWriteOnce
-    - name: migrator
+  persistentVolumeReclaimPolicy: Retain
-      image: busybox:1.36
+  hostPath:
-      command: [sh, -c, "cp -a /old-data/. /new-data/ && echo migration-complete && sleep 3600"]
+    path: /data/ride-archive-new
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: ride-archive-pvc-new
+  namespace: ride-history
+spec:
+  storageClassName: manual
+  volumeName: ride-archive-pv-new
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 500Mi
+EOF"""),
+              cmd(r"""k apply -f new.yaml
+k -n ride-history scale deploy ride-archive --replicas=0
+k -n ride-history run data-migrator --image=busybox:1.36 --restart=Never --dry-run=client -o yaml -- sh -c "cp -a /old-data/. /new-data/ && echo migration-complete && sleep 3600" > m.yaml""", r"""Scale to 0 first: the running Pod holds the old claim. `cp -a /old-data/. /new-data/` copies the contents, dotfiles too, rather than the directory itself."""),
+              hand(r"""vim m.yaml""",
+                   r"""Both claims: add to the container, below `name: data-migrator`,
+```
     volumeMounts:
-        - {name: old, mountPath: /old-data}
+    - name: old
-        - {name: new, mountPath: /new-data}
+      mountPath: /old-data
-EOF
+    - name: new
-k -n ride-history logs data-migrator             # migration-complete
+      mountPath: /new-data
-k -n ride-history patch deploy ride-archive --type=json \\
+```
-  -p '[{"op":"replace","path":"/spec/template/spec/volumes/0/persistentVolumeClaim/claimName","value":"ride-archive-pvc-new"}]'
+and to the Pod's `spec:`, right below `spec:`:
-k -n ride-history scale deploy ride-archive --replicas=1
+```
-# ReadWriteOnce is per-node, but a second pod on the same node still cannot mount a claim that a
+  volumes:
-# running pod holds - hence the scale to 0 first. `cp -a /old/. /new/` copies the CONTENTS
+  - name: old
-# (including dotfiles) rather than nesting the directory inside the target.
+    persistentVolumeClaim:
-""",
+      claimName: ride-archive-pvc
+  - name: new
+    persistentVolumeClaim:
+      claimName: ride-archive-pvc-new
+```""",
+                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "old", "mountPath": "/old-data"}, {"name": "new", "mountPath": "/new-data"}]
+  | .spec.volumes = [{"name": "old", "persistentVolumeClaim": {"claimName": "ride-archive-pvc"}}, {"name": "new", "persistentVolumeClaim": {"claimName": "ride-archive-pvc-new"}}]' m.yaml"""),
+              cmd(r"""k apply -f m.yaml"""),
+              hand(r"""k -n ride-history logs data-migrator""",
+                   r"""Again until it prints `migration-complete`.""",
+                   run=r"""for i in $(seq 1 60); do kubectl -n ride-history logs data-migrator 2>/dev/null | grep -q migration-complete && break; sleep 2; done"""),
+              cmd(r"""k -n ride-history delete pod data-migrator --now"""),
+              hand(r"""k -n ride-history edit deploy ride-archive""",
+                   r"""In the Pod template's `volumes:`, change `claimName: ride-archive-pvc` to `claimName: ride-archive-pvc-new`, then `:wq`.""",
+                   run=r"""kubectl -n ride-history patch deploy ride-archive --type=json -p '[{"op":"replace","path":"/spec/template/spec/volumes/0/persistentVolumeClaim/claimName","value":"ride-archive-pvc-new"}]'"""),
+              cmd(r"""k -n ride-history scale deploy ride-archive --replicas=1
+k -n ride-history get pods""")],
 )
 
 # ------------------------------------------------------------------ 73 ----
@@ -445,17 +542,21 @@ for i in $(seq 1 12); do
   sleep 3
 done; exit 1"""),
     ],
-    solution="""
+    solution=[cmd(r"""k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend
-k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend    # empty
+k -n eta-service get pods --show-labels
-k -n eta-service get deploy eta-calculator -o jsonpath='{.spec.template.metadata.labels}'  # app=eta-calculator
+k -n eta-service describe svc eta-backend""", r"""No endpoints. The Pods are `app=eta-calculator`, the Service selects `app=eta-service`; and it sends to `TargetPort 9090`, where nginx listens on 80. Empty endpoints is almost always the selector; endpoints that refuse connections, the target port."""),
-k -n eta-service get svc eta-backend -o jsonpath='{.spec.selector}'                        # app=eta-service  <- fault 1
+              hand(r"""k -n eta-service edit svc eta-backend""",
-k -n eta-service patch svc eta-backend -p '{"spec":{"selector":{"app":"eta-calculator"}}}'
+                   r"""Fix both, then `:wq`:
-k -n eta-service patch svc eta-backend -p '{"spec":{"ports":[{"port":80,"targetPort":80,"protocol":"TCP"}]}}'
+```
-k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend
+  ports:
-# Empty endpoints is almost always the selector; a populated endpoint list that still refuses
+  - port: 80
-# connections is almost always targetPort. Check the endpoints first - it splits the two cases
+    protocol: TCP
-# in one command.
+    targetPort: 80
-""",
+  selector:
+    app: eta-calculator
+```""",
+                   run=r"""kubectl -n eta-service patch svc eta-backend -p '{"spec":{"selector":{"app":"eta-calculator"},"ports":[{"port":80,"targetPort":80,"protocol":"TCP"}]}}'"""),
+              cmd(r"""k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend""")],
 )
 
 # ------------------------------------------------------------------ 74 ----
@@ -521,11 +622,14 @@ sys.exit(0 if ok else 1)'"""),
         C("the HTTPRoute is attached to fleet-gateway", 1, r"""
 j -n fleet-ops httproute fleet-route '{.spec.parentRefs[*].name}' | grep -qw fleet-gateway"""),
     ],
-    solution="""
+    solution=[hand(r"""vim gw.yaml""",
-cat <<EOF | k apply -f -
+                   r"""The docs page *Gateway API* has a Gateway and an HTTPRoute to start from:
+```
 apiVersion: gateway.networking.k8s.io/v1
 kind: Gateway
-metadata: {name: fleet-gateway, namespace: fleet-ops}
+metadata:
+  name: fleet-gateway
+  namespace: fleet-ops
 spec:
   gatewayClassName: fleet-gateway-class
   listeners:
@@ -536,21 +640,73 @@ spec:
 ---
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
-metadata: {name: fleet-route, namespace: fleet-ops}
+metadata:
+  name: fleet-route
+  namespace: fleet-ops
 spec:
-  parentRefs: [{name: fleet-gateway}]
+  parentRefs:
-  hostnames: [fleet.example.com]
+  - name: fleet-gateway
+  hostnames:
+  - fleet.example.com
   rules:
   - matches:
-        - headers: [{type: Exact, name: X-Fleet-Version, value: canary}]
+    - headers:
-      backendRefs: [{name: fleet-dashboard-v2, port: 80}]
+      - name: X-Fleet-Version
+        value: canary
+    backendRefs:
+    - name: fleet-dashboard-v2
+      port: 80
   - matches:
-        - path: {type: PathPrefix, value: /}
+    - path:
-      backendRefs: [{name: fleet-dashboard-v1, port: 80}]
+        type: PathPrefix
-EOF
+        value: /
-# Order matters: HTTPRoute picks the most specific match, and a bare PathPrefix "/" matches
+    backendRefs:
-# everything - so the header rule must be able to win. Listing it first also keeps it readable.
+    - name: fleet-dashboard-v1
-""",
+      port: 80
+```
+The more specific match wins, so the header rule takes canary requests though `/` matches everything.""",
+                   run=r"""cat > gw.yaml <<'EOF'
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  name: fleet-gateway
+  namespace: fleet-ops
+spec:
+  gatewayClassName: fleet-gateway-class
+  listeners:
+  - name: http
+    protocol: HTTP
+    port: 80
+    hostname: fleet.example.com
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: fleet-route
+  namespace: fleet-ops
+spec:
+  parentRefs:
+  - name: fleet-gateway
+  hostnames:
+  - fleet.example.com
+  rules:
+  - matches:
+    - headers:
+      - name: X-Fleet-Version
+        value: canary
+    backendRefs:
+    - name: fleet-dashboard-v2
+      port: 80
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /
+    backendRefs:
+    - name: fleet-dashboard-v1
+      port: 80
+EOF"""),
+              cmd(r"""k apply -f gw.yaml
+k -n fleet-ops get gateway,httproute""")],
 )
 
 # ------------------------------------------------------------------ 75 ----
@@ -714,29 +870,49 @@ eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \
  && $n -o jsonpath='{.spec.policyTypes}' | grep -q Egress \
  && [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ]"""),
     ],
-    solution="""
+    solution=[cmd(r"""k -n kube-system get ds
-k -n kube-system get ds                       # kindnet  -> {WS}/q77/cni.txt
+echo kindnet > {WS}/q77/cni.txt""", r"""`kindnet` runs on every node: it is the CNI. `kube-proxy` is not."""),
-# Don't answer from memory - prove it. Two pods, one deny policy, one wget:
+              cmd(r"""k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv
-k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv; k -n zero-trust expose pod srv --port 80
+k -n zero-trust expose pod srv --port 80
-k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600
+k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600""", r"""Prove it rather than remember it: two Pods, a policy that denies everything, and a request between them."""),
-k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null && echo reachable
+              hand(r"""k -n zero-trust get pods""",
-cat <<EOF | k apply -f -
+                   r"""Again until both are `Running`.""",
+                   run=r"""kubectl -n zero-trust wait --for=condition=Ready pod/srv pod/cli --timeout=180s"""),
+              cmd(r"""k -n zero-trust exec cli -- wget -qO- -T3 http://srv""", r"""nginx's page: no policy yet."""),
+              hand(r"""vim deny.yaml""",
+                   r"""The default deny for both directions from the docs page *Network Policies*: every Pod, both policy types, no rules.
+```
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
-metadata: {name: default-deny-all, namespace: zero-trust}
+metadata:
+  name: default-deny-all
+  namespace: zero-trust
 spec:
   podSelector: {}
-  policyTypes: [Ingress, Egress]
+  policyTypes:
-EOF
+  - Ingress
-k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null || echo BLOCKED   # -> BLOCKED here
+  - Egress
-echo yes > {WS}/q77/enforced.txt
+```""",
-k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv                # leave only the policy
+                   run=r"""cat > deny.yaml <<'EOF'
-# podSelector {} means every pod in the namespace; naming both policyTypes with no ingress/egress
+apiVersion: networking.k8s.io/v1
-# blocks listed means "deny everything". Recent kindnet builds (this cluster's included) embed a
+kind: NetworkPolicy
-# policy engine - look for `nft list table inet kindnet-network-policies` on a node. Flannel stores
+metadata:
-# NetworkPolicies and enforces nothing; Calico and Cilium enforce. A policy that "works" on a
+  name: default-deny-all
-# non-enforcing CNI is the classic hardening-review trap.
+  namespace: zero-trust
-""",
+spec:
+  podSelector: {}
+  policyTypes:
+  - Ingress
+  - Egress
+EOF"""),
+              cmd(r"""k apply -f deny.yaml
+k -n zero-trust exec cli -- wget -qO- -T3 http://srv""", r"""`download timed out`: the CNI enforces it. (Flannel would store the policy and enforce nothing; Calico and Cilium enforce. Recent kindnet has a policy engine of its own.)"""),
+              hand(r"""echo yes > {WS}/q77/enforced.txt""",
+                   r"""What the test showed.""",
+                   run=r"""sleep 6
+if kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null 2>&1; then echo no; else echo yes; fi > {WS}/q77/enforced.txt"""),
+              cmd(r"""k -n zero-trust delete pod srv cli --now
+k -n zero-trust delete svc srv""", r"""Leave only the policy.""")],
 )
 
 # ------------------------------------------------------------------ 78 ----
@@ -767,23 +943,16 @@ for i in $(seq 1 20); do [ "$(j -n rider-support deploy support-portal '{.status
 eq "$(j -n rider-support deploy support-portal '{.status.readyReplicas}')" 2 \
  && eq "$(j -n rider-support pod -l app=support-portal '{.items[0].spec.priority}')" 1000000"""),
     ],
-    solution="""
+    solution=[cmd(r"""k create priorityclass rider-support-priority --value=1000000 --description='Priority for rider support services'""", r"""Not the global default unless `--global-default`."""),
-cat <<EOF | k apply -f -
+              hand(r"""k -n rider-support edit deploy support-portal""",
-apiVersion: scheduling.k8s.io/v1
+                   r"""In the Pod template's `spec:`, beside `containers:` and indented like it, then `:wq`:
-kind: PriorityClass
+```
-metadata: {name: rider-support-priority}
+      priorityClassName: rider-support-priority
-value: 1000000
+```
-globalDefault: false
+The value is copied into each Pod when it is created, so only new Pods carry it - the edit rolls them.""",
-description: Priority for rider support services
+                   run=r"""kubectl -n rider-support patch deploy support-portal -p '{"spec":{"template":{"spec":{"priorityClassName":"rider-support-priority"}}}}'"""),
-EOF
+              cmd(r"""k -n rider-support rollout status deploy support-portal
-k -n rider-support patch deploy support-portal \\
+k -n rider-support get pods -o custom-columns=NAME:.metadata.name,PRIORITY:.spec.priority""")],
-  -p '{"spec":{"template":{"spec":{"priorityClassName":"rider-support-priority"}}}}'
-k -n rider-support rollout status deploy support-portal
-k -n rider-support get pod -l app=support-portal -o jsonpath='{.items[0].spec.priority}'
-# The admission controller copies the class's value into pod.spec.priority at creation time, so
-# existing pods keep their old priority until they are recreated - patching the template does
-# that for you. globalDefault: true would apply the class to every pod without one, cluster-wide.
-""",
 )
 
 # ------------------------------------------------------------------ 79 ----
@@ -892,31 +1061,35 @@ un=$(grep -n 'uncordon' "$f" | head -1 | cut -d: -f1)
 grep -Eq 'kubeadm[[:space:]]+upgrade[[:space:]]+node' "$WS/q80/worker.txt" 2>/dev/null \
  && ! grep -q 'upgrade apply' "$WS/q80/worker.txt" """),
     ],
-    solution="""
+    solution=[cmd(r"""k version
-k version -o json | jq -r .serverVersion.gitVersion > {WS}/q80/current.txt
 k get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion --no-headers > {WS}/q80/kubelets.txt
-cat > {WS}/q80/steps.sh <<'EOF'
+cat {WS}/q80/kubelets.txt"""),
-sed -i 's|/v1.37/|/v1.38/|' /etc/apt/sources.list.d/kubernetes.list
+              hand(r"""echo v1.37.0 > {WS}/q80/current.txt""",
+                   r"""With the `Server Version` that `k version` printed.""",
+                   run=r"""kubectl version -o json | jq -r .serverVersion.gitVersion > {WS}/q80/current.txt"""),
+              hand(r"""vim {WS}/q80/steps.sh""",
+                   r"""The order of the docs page *Upgrading kubeadm clusters*: kubeadm first, which upgrades the control plane; then the node's kubelet and kubectl, drained; uncordon last. The next minor is 1.38 - first point `/etc/apt/sources.list.d/kubernetes.list` at the v1.38 repository, and take its newest patch (`apt-cache madison kubeadm`):
+```
 apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.38.0-1.1 && apt-mark hold kubeadm
 kubeadm upgrade plan
-kubeadm upgrade apply v1.38.0 -y
+kubeadm upgrade apply v1.38.0
-kubectl drain <control-plane> --ignore-daemonsets
+kubectl drain {CP} --ignore-daemonsets
-apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
+apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
 systemctl daemon-reload
 systemctl restart kubelet
-kubectl uncordon <control-plane>
+kubectl uncordon {CP}
-EOF
+```""",
-echo 'kubeadm upgrade node' > {WS}/q80/worker.txt
+                   run=r"""cat > {WS}/q80/steps.sh <<'EOF'
-# The cluster runs v1.37, so the next minor is 1.38 - and pkgs.k8s.io serves one apt repo per
+apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.38.0-1.1 && apt-mark hold kubeadm
-# minor, so the repo line has to move to v1.38 before 1.38 packages exist for apt at all. (1.38
+kubeadm upgrade plan
-# is due in December 2026; until then that repo is empty, which is why this is only a plan. Take
+kubeadm upgrade apply v1.38.0
-# the newest 1.38 patch on offer then: `apt-cache madison kubeadm`.)
+kubectl drain {CP} --ignore-daemonsets
-# kubeadm goes first because it drives the whole thing; the control plane is upgraded by
+apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
-# `upgrade apply`, and only then do the node binaries (kubelet/kubectl) follow, ending with
+systemctl daemon-reload
-# uncordon. Workers run `kubeadm upgrade node` - they host no control-plane static pods, so
+systemctl restart kubelet
-# there is nothing for `upgrade apply` to do there. The kubelet may trail the API server by up
+kubectl uncordon {CP}
-# to three minor versions, but must never be ahead of it.
+EOF"""),
-""",
+              cmd(r"""echo 'kubeadm upgrade node' > {WS}/q80/worker.txt""", r"""A worker hosts no control plane, so there is nothing for `upgrade apply` to do there; `upgrade node` updates its kubelet configuration.""")],
 )
 
 # ------------------------------------------------------------------ 81 ----
@@ -977,35 +1150,44 @@ done; exit 1"""),
         C("fare-calculator pods are Running and Ready", 3, r"""
 for i in $(seq 1 24); do [ "$(j -n fare-engine deploy fare-calculator '{.status.readyReplicas}')" -ge 1 ] 2>/dev/null && exit 0; sleep 5; done; exit 1"""),
     ],
-    solution="""
+    solution=[cmd(r"""k -n fare-engine get pvc,pods
-k -n fare-engine get pvc                       # Pending
+k -n fare-engine describe pvc fare-data-pvc | tail -3
-k -n fare-engine describe pvc fare-data-pvc    # storageclass.storage.k8s.io "fast-storage" not found
+k get sc""", r"""`storageclass.storage.k8s.io "fast-storage" not found`: the class here is `standard`."""),
-k get sc                                       # on kind the class is called `standard`
+              cmd(r"""k -n fare-engine delete pvc fare-data-pvc""", r"""A claim's `storageClassName` cannot change: make it again."""),
-# A PVC's storageClassName is immutable, so recreate it with the name your cluster really has:
+              hand(r"""vim pvc.yaml""",
-k -n fare-engine delete pvc fare-data-pvc
+                   r"""```
-cat <<EOF | k apply -f -
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata: {name: fare-data-pvc, namespace: fare-engine}
+metadata:
+  name: fare-data-pvc
+  namespace: fare-engine
 spec:
-  accessModes: [ReadWriteOnce]
-  resources: {requests: {storage: 100Mi}}
   storageClassName: standard
-EOF
+  accessModes:
-k -n fare-engine describe pod -l app=fare-calculator   # 0/4 nodes match node affinity disk in [nvme]
+  - ReadWriteOnce
-k get nodes --show-labels | grep -c 'disk=nvme'        # 0
+  resources:
-# Either label a node to satisfy the rule, or drop the rule. Labelling is the smaller change:
+    requests:
-k label node <a-worker> disk=nvme
+      storage: 100Mi
-k -n fare-engine rollout status deploy fare-calculator
+```""",
-# Two independent blockers stack up silently: the PVC never binds, AND nothing would schedule
+                   run=r"""cat > pvc.yaml <<'EOF'
-# even if it did. `describe pod` reports only the scheduling failure, so check the PVC too
+apiVersion: v1
-# whenever a pod that mounts one is stuck Pending.
+kind: PersistentVolumeClaim
-# Never assume the class name - `standard` here, `local-path` on a stock local-path-provisioner
+metadata:
-# install, something else on a cloud cluster. `k get sc` first, every time; a PVC naming a class
+  name: fare-data-pvc
-# that does not exist sits Pending forever with no other symptom.
+  namespace: fare-engine
-# `standard` is WaitForFirstConsumer, so the PVC only binds once the pod can actually be placed -
+spec:
-# which is why fixing the affinity is what finally unblocks the storage too.
+  storageClassName: standard
-""",
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 100Mi
+EOF"""),
+              cmd(r"""k apply -f pvc.yaml
+k -n fare-engine describe pod -l app=fare-calculator | tail -3
+k get nodes -L disk""", r"""`didn't match Pod's node affinity/selector`: it wants `disk=nvme`, which no node has. `describe pod` names only this one; the claim's fault you had to find yourself."""),
+              cmd(r"""k label node {W1} disk=nvme
+k -n fare-engine rollout status deploy fare-calculator""", r"""Labelling a node is the smaller change than dropping the rule. `standard` binds only once the Pod is placed (`WaitForFirstConsumer`), so the claim goes `Bound` now too.""")],
 )
 
 # ------------------------------------------------------------------ 82 ----
@@ -1054,30 +1236,20 @@ ok=any(p.get("type")=="Pods" and int(p.get("value",0))==2 and int(p.get("periodS
        for p in b.get("policies",[]))
 sys.exit(0 if ok else 1)'"""),
     ],
-    solution="""
+    solution=[cmd(r"""k -n route-planner autoscale deploy route-planner --name=route-planner-hpa --min=1 --max=6 --memory=70% --dry-run=client -o yaml > hpa.yaml""", r"""`autoscale` writes an `autoscaling/v2` HPA with the memory target; it has no flag for `behavior`."""),
-cat <<EOF | k apply -f -
+              hand(r"""vim hpa.yaml""",
-apiVersion: autoscaling/v2
+                   r"""Add to its `spec:`, beside `maxReplicas:` (the docs page *Horizontal Pod Autoscaling* has the block under *Configurable scaling behavior*):
-kind: HorizontalPodAutoscaler
+```
-metadata: {name: route-planner-hpa, namespace: route-planner}
-spec:
-  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: route-planner}
-  minReplicas: 1
-  maxReplicas: 6
-  metrics:
-    - type: Resource
-      resource:
-        name: memory
-        target: {type: Utilization, averageUtilization: 70}
   behavior:
     scaleUp:
       policies:
-        - {type: Pods, value: 2, periodSeconds: 60}
+      - type: Pods
-EOF
+        value: 2
-k -n route-planner get hpa route-planner-hpa
+        periodSeconds: 60
-# `kubectl autoscale` can set a memory target (--memory=70%) but has no flag for a behavior block,
+```""",
-# so memory + behavior means writing the v2 YAML by hand (or autoscale, then kubectl edit). Utilisation is a percentage OF THE REQUEST, so the Deployment must
+                   run=r"""yq -i '.spec.behavior.scaleUp.policies = [{"type": "Pods", "value": 2, "periodSeconds": 60}]' hpa.yaml"""),
-# declare a memory request or the HPA reports <unknown> and never scales.
+              cmd(r"""k apply -f hpa.yaml
-""",
+k -n route-planner get hpa route-planner-hpa""", r"""Utilisation is a percentage of the request: the Deployment has a memory request, so the target reads a number, not `<unknown>`.""")],
 )
 
 # ------------------------------------------------------------------ 83 ----
@@ -1131,26 +1303,81 @@ ok=any(not r.get("to") and {p.get("protocol") for p in r.get("ports",[]) if str(
        for r in eg)
 sys.exit(0 if ok else 1)'"""),
     ],
-    solution="""
+    solution=[hand(r"""vim netpol.yaml""",
-cat <<EOF | k apply -f -
+                   r"""Paste the example from the docs page *Network Policies* and change it to:
+```
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
-metadata: {name: safety-api-policy, namespace: safety-monitor}
+metadata:
+  name: safety-api-policy
+  namespace: safety-monitor
 spec:
-  podSelector: {matchLabels: {app: safety-api}}
+  podSelector:
-  policyTypes: [Ingress, Egress]
+    matchLabels:
+      app: safety-api
+  policyTypes:
+  - Ingress
+  - Egress
   ingress:
-    - from: [{podSelector: {matchLabels: {app: safety-dashboard}}}]
+  - from:
-      ports: [{protocol: TCP, port: 8443}]
+    - podSelector:
+        matchLabels:
+          app: safety-dashboard
+    ports:
+    - protocol: TCP
+      port: 8443
   egress:
-    - to: [{podSelector: {matchLabels: {app: safety-db}}}]
+  - to:
-      ports: [{protocol: TCP, port: 5432}]
+    - podSelector:
-    - ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
+        matchLabels:
-EOF
+          app: safety-db
-# The DNS rule deliberately has no `to:` - CoreDNS lives in kube-system, and an egress rule with
+    ports:
-# only `ports` allows that port to anywhere. Naming Egress in policyTypes denies all outbound
+    - protocol: TCP
-# traffic by default, so forgetting DNS is what silently breaks every name lookup in the pod.
+      port: 5432
-""",
+  - ports:
+    - protocol: UDP
+      port: 53
+    - protocol: TCP
+      port: 53
+```
+The DNS rule has no `to:` - only ports, so port 53 anywhere. With `Egress` in `policyTypes` everything else outbound is denied, and forgetting DNS breaks every name lookup in the Pod.""",
+                   run=r"""cat > netpol.yaml <<'EOF'
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: safety-api-policy
+  namespace: safety-monitor
+spec:
+  podSelector:
+    matchLabels:
+      app: safety-api
+  policyTypes:
+  - Ingress
+  - Egress
+  ingress:
+  - from:
+    - podSelector:
+        matchLabels:
+          app: safety-dashboard
+    ports:
+    - protocol: TCP
+      port: 8443
+  egress:
+  - to:
+    - podSelector:
+        matchLabels:
+          app: safety-db
+    ports:
+    - protocol: TCP
+      port: 5432
+  - ports:
+    - protocol: UDP
+      port: 53
+    - protocol: TCP
+      port: 53
+EOF"""),
+              cmd(r"""k apply -f netpol.yaml
+k -n safety-monitor describe netpol safety-api-policy""")],
 )
 
 # ------------------------------------------------------------------ 84 ----

cka/bank_sim_d.py

@@ -16,7 +16,7 @@ task was seeded, solved and graded on an isolated kind cluster before it went in
 
 Run with:  ./ksim sim d
 """
-from sim.bank import C, NS
+from sim.bank import C, NS, cmd, hand
 
 Q = []
 # Tasks written for the pre-2025 exam opened by switching kubectl context. Every task now has a host
@@ -73,25 +73,27 @@ if [ "$own" = ReplicaSet ]; then want="deployment $(j -n kube-system rs "$name"
 else want="$(echo "$own" | tr A-Z a-z) $name"; fi
 eq "$(sed -n 's/^ *dns *: *//p' "$f" | head -1 | tr -s ' ' | tr A-Z a-z | sed 's/ *$//')" "$want" """),
     ],
-    solution="""
+    solution=[cmd(r"""systemctl status kubelet | head -3
-sudo -i                # the task's host is the control-plane node: no hop needed
+sudo ls /etc/kubernetes/manifests""", r"""The host is the control-plane node. The kubelet is a systemd service - a process; etcd and the three control-plane components are manifests in the static Pod path."""),
-systemctl status kubelet | head -3          # a systemd service: a plain process on the node
+              cmd(r"""k -n kube-system get pods -o wide
-ls /etc/kubernetes/manifests                # etcd, kube-apiserver, kube-controller-manager, kube-scheduler
+k -n kube-system get deploy,ds""", r"""Static Pods carry the node's name as a suffix. What answers DNS is the `coredns` Deployment, though its Service is called `kube-dns`; `kube-proxy` and `kindnet` are DaemonSets."""),
-exit
+              hand(r"""vim {WS}/q86/controlplane.txt""",
-k -n kube-system get pod -o wide            # static pods carry the node name as a suffix...
+                   r"""```
-k -n kube-system get pod kube-apiserver-{CP} -o jsonpath='{.metadata.ownerReferences[0].kind}{"\\n"}'   # ...and are owned by the Node
-k -n kube-system get deploy,ds              # coredns is a Deployment (kube-proxy, kindnet are DaemonSets)
-cat > {WS}/q86/controlplane.txt <<'EOF'
 kubelet: process
 kube-apiserver: static-pod
 kube-scheduler: static-pod
 kube-controller-manager: static-pod
 etcd: static-pod
 dns: deployment coredns
-EOF
+```""",
-# The Service is called kube-dns for historical reasons; what actually answers is the coredns
+                   run=r"""cat > {WS}/q86/controlplane.txt <<'EOF'
-# Deployment. Follow the chain svc selector -> pod -> ownerReferences when unsure.
+kubelet: process
-""",
+kube-apiserver: static-pod
+kube-scheduler: static-pod
+kube-controller-manager: static-pod
+etcd: static-pod
+dns: deployment coredns
+EOF""")],
 )
 
 # ------------------------------------------------------------------ 87 ----
@@ -125,24 +127,27 @@ cfg=$(nx "$W1" 'ls /etc/cni/net.d/*.conflist /etc/cni/net.d/*.conf 2>/dev/null |
         C("static Pod name suffix on {W3} (line 5)", 1, r"""
 v=$(sed -n 's/^ *5 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]'); [ "$v" = "-$W3" ] || [ "$v" = "$W3" ]"""),
     ],
-    solution="""
+    solution=[cmd(r"""k get nodes
-k get nodes                                   # ROLES shows control-plane; count the rest
+k -n kube-system describe pod -l component=kube-apiserver | grep service-cluster-ip-range
-k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
+k -n kube-system get ds""", r"""`ROLES control-plane` on one node, the rest workers; the API server's `--service-cluster-ip-range`; and the CNI's DaemonSet, `kindnet`."""),
-k -n kube-system get ds                       # kindnet
+              hand(r"""ssh {W1} ls /etc/cni/net.d""",
-{SHELL_W1}
+                   r"""The CNI's configuration on a node: `10-kindnet.conflist`.""",
-sudo -i
+                   run=r""""""),
-ls /etc/cni/net.d/                            # 10-kindnet.conflist
+              hand(r"""vim {WS}/q87/cluster-info.txt""",
-exit; exit
+                   r"""The values you read - here:
-cat > {WS}/q87/cluster-info.txt <<'EOF'
+```
 1: 1
 2: 3
 3: 10.96.0.0/16
 4: kindnet, /etc/cni/net.d/10-kindnet.conflist
 5: -{W3}
-EOF
+```
-# A static Pod's mirror is named <pod>-<node name>, e.g. kube-apiserver-{CP}. Values come from THIS
+A static Pod's mirror is named `<pod>-<node>`, e.g. `kube-apiserver-{CP}`.""",
-# cluster - check each one instead of trusting the ones shown here.
+                   run=r"""C=$(kubectl get nodes --no-headers -l node-role.kubernetes.io/control-plane | wc -l)
-""",
+W=$(kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' | wc -l)
+CIDR=$(kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -o 'service-cluster-ip-range=[^ ]*' | head -1 | cut -d= -f2)
+F=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
+printf '1: %s\n2: %s\n3: %s\n4: kindnet, %s\n5: -{W3}\n' "$C" "$W" "$CIDR" "$F" > {WS}/q87/cluster-info.txt""")],
 )
 
 # ------------------------------------------------------------------ 88 ----
@@ -218,18 +223,19 @@ grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-n|--namespa
 timeout 60 bash "$f" 2>/dev/null | grep -q etcd"""),
         C("no aliases in either script", 1, r"""[ -s "$WS/q89/pods-by-age.sh" ] && [ -s "$WS/q89/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q89/pods-by-age.sh" "$WS/q89/pods-by-uid.sh" """),
     ],
-    solution="""
+    solution=[hand(r"""vim {WS}/q89/pods-by-age.sh""",
-cat > {WS}/q89/pods-by-age.sh <<'EOF'
+                   r"""`kubectl` in full: a script runs in a fresh shell, where the `k` alias does not exist.
+```
 kubectl get pods -A --sort-by=.metadata.creationTimestamp
-EOF
+```""",
-cat > {WS}/q89/pods-by-uid.sh <<'EOF'
+                   run=r"""echo 'kubectl get pods -A --sort-by=.metadata.creationTimestamp' > {WS}/q89/pods-by-age.sh"""),
+              hand(r"""vim {WS}/q89/pods-by-uid.sh""",
+                   r"""```
 kubectl -n kube-system get pods --sort-by=.metadata.uid
-EOF
+```""",
-chmod +x {WS}/q89/*.sh
+                   run=r"""echo 'kubectl -n kube-system get pods --sort-by=.metadata.uid' > {WS}/q89/pods-by-uid.sh"""),
-bash {WS}/q89/pods-by-age.sh | head -3        # the control-plane pods come first - they are oldest
+              cmd(r"""chmod +x {WS}/q89/*.sh
-# --sort-by takes a JSONPath into each item; ascending only, so "newest first" is `| tac` or
+{WS}/q89/pods-by-age.sh | head -3""", r"""The control-plane Pods first: they are the oldest. `--sort-by` takes a JSONPath into each item, ascending only.""")],
-# sorting by a field that grows the other way.
-""",
 )
 
 # ------------------------------------------------------------------ 90 ----
@@ -273,31 +279,35 @@ for f in pod-deleted.log container-killed.log; do
 done"""),
         C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q90/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""),
     ],
-    solution="""
+    solution=[hand(r"""vim {WS}/q90/cluster-events.sh""",
-cat > {WS}/q90/cluster-events.sh <<'EOF'
+                   r"""```
 kubectl get events -A --sort-by=.metadata.creationTimestamp
-EOF
+```""",
-chmod +x {WS}/q90/cluster-events.sh
+                   run=r"""echo 'kubectl get events -A --sort-by=.metadata.creationTimestamp' > {WS}/q90/cluster-events.sh"""),
-
+              cmd(r"""chmod +x {WS}/q90/cluster-events.sh
-k -n kube-system get pod -l k8s-app=kube-proxy -o wide              # the one on {W2}
+k -n kube-system get pods -l k8s-app=kube-proxy -o wide""", r"""Note the name of the one on `{W2}`."""),
-k -n kube-system delete pod <kube-proxy-on-{W2}>
+              hand(r"""k -n kube-system delete pod kube-proxy-xxxxx""",
-k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
+                   r"""With that name. The DaemonSet replaces it with a new Pod at once.""",
-#   Killing (old pod), SuccessfulCreate (DaemonSet), Scheduled, Pulled, Created, Started (new pod)
+                   run=r"""P=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].metadata.name}')
-
+kubectl -n kube-system delete pod $P
-{SHELL_W2}
+for i in $(seq 1 40); do
+  [ "$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].status.phase}' 2>/dev/null)" = Running ] && break; sleep 3
+done
+sleep 5"""),
+              cmd(r"""k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
+cat {WS}/q90/pod-deleted.log""", r"""`Killing` the old Pod, `SuccessfulCreate` by the DaemonSet, then `Scheduled`, `Pulled`, `Created`, `Started` for the new one."""),
+              hand(r"""{SHELL_W2}
 sudo -i
-crictl ps --name kube-proxy                 # container id
+crictl ps --name kube-proxy
-crictl stop <id>
+crictl stop 1a2b3c4d5e6f
-exit; exit
+exit
-k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
+exit""",
-#   Pulled, Created, Started - same Pod, new container
+                   r"""With the container ID `crictl ps` printed. Only the container goes: the kubelet starts a new one in the same Pod.""",
-k -n kube-system get pod -l k8s-app=kube-proxy -o wide              # RESTARTS 1 on {W2}
+                   run=r"""ssh -n {W2} 'sudo crictl stop $(sudo crictl ps --name kube-proxy -q)'
-echo container > {WS}/q90/restarts.txt
+sleep 25"""),
-# Deleting the Pod makes the DaemonSet create a brand-new Pod (RESTARTS 0). Killing only the
+              cmd(r"""k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
-# container makes the kubelet start a new container inside the SAME Pod - that is a restart.
+k -n kube-system get pods -l k8s-app=kube-proxy -o wide
-# Do not `crictl rm` it straight away: on v1.35 the kubelet counts the restart from the exited
+echo container > {WS}/q90/restarts.txt""", r"""`RESTARTS 1` on `{W2}`: a new container in the same Pod is a restart; a new Pod starts again at 0.""")],
-# container it sees, and if rm wins the race it starts a fresh attempt 0 and RESTARTS stays 0.
-""",
 )
 
 # ------------------------------------------------------------------ 91 ----
@@ -333,15 +343,15 @@ got=$(awk '{print $1}' "$f" | sed 's/\..*//' | grep . | sort -u)
 best=$(for ns in census-north census-south census-east census-west; do echo "$ns $($K -n $ns get role --no-headers 2>/dev/null | wc -l)"; done | sort -k2 -n | tail -1)
 eq "$(tr -s ' ' < "$WS/q91/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """),
     ],
-    solution="""
+    solution=[cmd(r"""k api-resources --namespaced=true -o name > {WS}/q91/namespaced.txt
-k api-resources --namespaced=true -o name > /tmp/r.txt       # deployments.apps, ... (group suffix)
+head -3 {WS}/q91/namespaced.txt""", r"""`-o name` prints the plural names alone (`deployments.apps`: name and group); `--namespaced=false` would give the cluster-scoped half."""),
-k api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q91/namespaced.txt
+              cmd(r"""k -n census-north get role --no-headers | wc -l
-for ns in census-north census-south census-east census-west; do echo "$ns $(k -n $ns get role --no-headers | wc -l)"; done
+k -n census-south get role --no-headers | wc -l
-#   pick the largest, e.g.  echo "census-east 8" > {WS}/q91/most-roles.txt
+k -n census-east get role --no-headers | wc -l
-# --namespaced=false gives the cluster-scoped half (nodes, pv, storageclasses, clusterroles...).
+k -n census-west get role --no-headers | wc -l""", r"""Or all at once: `k get role -A | grep census`."""),
-# `k get role -A --no-headers | awk '{print $1}' | sort | uniq -c | sort -n` counts every
+              hand(r"""echo "census-east 9" > {WS}/q91/most-roles.txt""",
-# namespace at once.
+                   r"""With the namespace that has the most, and its count.""",
-""",
+                   run=r"""for ns in census-north census-south census-east census-west; do echo "$ns $(kubectl -n $ns get role --no-headers | wc -l)"; done | sort -k2 -n | tail -1 > {WS}/q91/most-roles.txt""")],
 )
 
 # ------------------------------------------------------------------ 92 ----
@@ -480,20 +490,27 @@ ip=$(j -n default pod "ops-beacon-$CP" '{.status.podIP}'); [ -n "$ip" ] || exit
 j -n default endpointslices -l kubernetes.io/service-name=ops-beacon-svc '{.items[*].endpoints[*].addresses[*]}' | tr ' ' '\n' | grep -qx "$ip" """),
         C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q94/nodeport.txt" 2>/dev/null)" "$np" """),
     ],
-    solution="""
+    solution=[cmd(r"""k -n default run ops-beacon --image=nginx:1.27 --labels=app=ops-beacon --dry-run=client -o yaml > beacon.yaml"""),
-k run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon $do > beacon.yaml
+              hand(r"""vim beacon.yaml""",
-# add to the container:  resources: {requests: {cpu: 10m, memory: 20Mi}}
+                   r"""Replace the container's `resources: {}` with:
-sudo -i                # the task's host is the control-plane node: no hop needed
+```
-grep staticPodPath /var/lib/kubelet/config.yaml          # /etc/kubernetes/manifests
+    resources:
-vi /etc/kubernetes/manifests/ops-beacon.yaml             # paste beacon.yaml
+      requests:
-exit
+        cpu: 10m
-k -n default get pod ops-beacon-{CP} -o wide             # mirror pod, named <pod>-<node>
+        memory: 20Mi
-k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
+```""",
-k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc   # the mirror pod's IP
+                   run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "10m", "memory": "20Mi"}}' beacon.yaml"""),
-k -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt
+              cmd(r"""sudo grep staticPodPath /var/lib/kubelet/config.yaml
-# The control-plane taint only keeps the SCHEDULER from placing pods there; the kubelet runs its
+sudo cp beacon.yaml /etc/kubernetes/manifests/ops-beacon.yaml""", r"""The host is the control-plane node. Its taint keeps the scheduler away, not the kubelet: static Pods run regardless."""),
-# static pods regardless. `expose pod` copies the pod's labels into the selector.
+              hand(r"""k -n default get pod ops-beacon-{CP} -o wide""",
-""",
+                   r"""Again until the mirror Pod is `Running`.""",
+                   run=r"""for i in $(seq 1 60); do [ "$(kubectl -n default get pod ops-beacon-{CP} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done"""),
+              cmd(r"""k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
+k -n default get svc ops-beacon-svc
+k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc""", r"""`expose pod` makes the selector from the Pod's labels: the mirror Pod's IP is the endpoint. Note the node port, after `80:`."""),
+              hand(r"""echo 31234 > {WS}/q94/nodeport.txt""",
+                   r"""With the node port `get svc` showed.""",
+                   run=r"""kubectl -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt""")],
 )
 
 # ------------------------------------------------------------------ 95 ----
@@ -757,40 +774,107 @@ for i in 0 1; do eq "$($K -n lima-data exec ledger-db-$i -- cat /data/id.txt 2>/
 want=$($K -n lima-data get pvc --no-headers -o custom-columns=N:.metadata.name 2>/dev/null | grep '^data-ledger-db-' | sort)
 [ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q99/pvcs.txt" 2>/dev/null)" = "$want" ]"""),
     ],
-    solution="""
+    solution=[hand(r"""vim sts.yaml""",
-cat <<EOF | k apply -f -
+                   r"""The docs page *StatefulSets* has a headless Service and a StatefulSet with `volumeClaimTemplates` together; change them to:
+```
 apiVersion: v1
 kind: Service
-metadata: {name: ledger-db, namespace: lima-data}
+metadata:
-spec: {clusterIP: None, selector: {app: ledger-db}, ports: [{port: 5432}]}
+  name: ledger-db
+  namespace: lima-data
+spec:
+  clusterIP: None
+  selector:
+    app: ledger-db
+  ports:
+  - port: 5432
 ---
 apiVersion: apps/v1
 kind: StatefulSet
-metadata: {name: ledger-db, namespace: lima-data}
+metadata:
+  name: ledger-db
+  namespace: lima-data
 spec:
   serviceName: ledger-db
   replicas: 3
-  selector: {matchLabels: {app: ledger-db}}
+  selector:
+    matchLabels:
+      app: ledger-db
   template:
-    metadata: {labels: {app: ledger-db}}
+    metadata:
+      labels:
+        app: ledger-db
     spec:
       containers:
       - name: db
         image: busybox:1.36
-          command: [sh, -c, "hostname > /data/id.txt && sleep 86400"]
+        command: ["sh", "-c", "hostname > /data/id.txt && sleep 86400"]
-          volumeMounts: [{name: data, mountPath: /data}]
+        volumeMounts:
+        - name: data
+          mountPath: /data
   volumeClaimTemplates:
-    - metadata: {name: data}
+  - metadata:
-      spec: {accessModes: [ReadWriteOnce], storageClassName: standard, resources: {requests: {storage: 100Mi}}}
+      name: data
-EOF
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      storageClassName: standard
+      resources:
+        requests:
+          storage: 100Mi
+```""",
+                   run=r"""cat > sts.yaml <<'EOF'
+apiVersion: v1
+kind: Service
+metadata:
+  name: ledger-db
+  namespace: lima-data
+spec:
+  clusterIP: None
+  selector:
+    app: ledger-db
+  ports:
+  - port: 5432
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: ledger-db
+  namespace: lima-data
+spec:
+  serviceName: ledger-db
+  replicas: 3
+  selector:
+    matchLabels:
+      app: ledger-db
+  template:
+    metadata:
+      labels:
+        app: ledger-db
+    spec:
+      containers:
+      - name: db
+        image: busybox:1.36
+        command: ["sh", "-c", "hostname > /data/id.txt && sleep 86400"]
+        volumeMounts:
+        - name: data
+          mountPath: /data
+  volumeClaimTemplates:
+  - metadata:
+      name: data
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      storageClassName: standard
+      resources:
+        requests:
+          storage: 100Mi
+EOF"""),
+              cmd(r"""k apply -f sts.yaml
 k -n lima-data rollout status sts ledger-db
-k -n lima-data scale sts ledger-db --replicas=2
+k -n lima-data scale sts ledger-db --replicas=2"""),
-k -n lima-data get pvc -o custom-columns=N:.metadata.name --no-headers | grep '^data-ledger-db-' | sort > {WS}/q99/pvcs.txt
+              cmd(r"""k -n lima-data get pvc -o custom-columns=NAME:.metadata.name --no-headers | sort > {WS}/q99/pvcs.txt
-cat {WS}/q99/pvcs.txt           # all THREE: data-ledger-db-0, -1, -2
+cat {WS}/q99/pvcs.txt""", r"""All three, `data-ledger-db-0` to `-2`: claims are named `<template>-<statefulset>-<ordinal>` and outlive their Pods, so scaling up again gives `ledger-db-2` its old data back.""")],
-# Claims are named <template>-<statefulset>-<ordinal> and outlive their pods: scaling down (or
-# deleting the StatefulSet) keeps them unless spec.persistentVolumeClaimRetentionPolicy says
-# otherwise, so scaling back up re-attaches ledger-db-2 to its old data.
-""",
 )
 
 # ------------------------------------------------------------------ 100 ----
@@ -849,28 +933,63 @@ $K -n oslo-db get netpol db-ingress >/dev/null 2>&1 || exit 1
 ! $K -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1 \
  && ! $K -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1"""),
     ],
-    solution="""
+    solution=[cmd(r"""k get ns --show-labels | grep -e oslo-api -e lima-api
-k get ns --show-labels | grep -E 'oslo-api|lima-api'    # env=prod / env=dev
+k -n oslo-api get pods --show-labels""", r"""`env=prod` and `env=dev`; `api` is `role=backend`, `debug` is not."""),
-cat <<EOF | k apply -f -
+              hand(r"""vim netpol.yaml""",
+                   r"""One `from` entry holding both selectors - both must match:
+```
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
-metadata: {name: db-ingress, namespace: oslo-db}
+metadata:
+  name: db-ingress
+  namespace: oslo-db
 spec:
-  podSelector: {matchLabels: {app: db}}
+  podSelector:
-  policyTypes: [Ingress]
+    matchLabels:
+      app: db
+  policyTypes:
+  - Ingress
   ingress:
   - from:
-        - namespaceSelector: {matchLabels: {env: prod}}
+    - namespaceSelector:
-          podSelector: {matchLabels: {role: backend}}      # SAME list item = AND
+        matchLabels:
-      ports: [{protocol: TCP, port: 80}]
+          env: prod
-EOF
+      podSelector:
-k -n oslo-api exec deploy/api   -- wget -qO- -T3 http://db.oslo-db | head -1     # <!DOCTYPE html>
+        matchLabels:
-k -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db               # download timed out
+          role: backend
-k -n lima-api exec deploy/api   -- wget -qO- -T3 http://db.oslo-db               # download timed out
+    ports:
-# The trap is one dash: "- namespaceSelector: ...\\n  - podSelector: ..." (two list items) means OR -
+    - protocol: TCP
-# any pod in a prod namespace, OR a backend pod in oslo-db's own namespace. One item with both
+      port: 80
-# selectors means AND.
+```
-""",
+The trap is one dash: `- podSelector:` would make it a second entry, and the rule an OR - any Pod in a prod namespace, or any backend Pod in `oslo-db`.""",
+                   run=r"""cat > netpol.yaml <<'EOF'
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: db-ingress
+  namespace: oslo-db
+spec:
+  podSelector:
+    matchLabels:
+      app: db
+  policyTypes:
+  - Ingress
+  ingress:
+  - from:
+    - namespaceSelector:
+        matchLabels:
+          env: prod
+      podSelector:
+        matchLabels:
+          role: backend
+    ports:
+    - protocol: TCP
+      port: 80
+EOF"""),
+              cmd(r"""k apply -f netpol.yaml
+k -n oslo-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db
+k -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db
+k -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db""", r"""nginx's page, then `download timed out` twice.""")],
 )
 
 # ------------------------------------------------------------------ 101 ----
@@ -919,18 +1038,18 @@ for i in $(seq 1 15); do
 done; exit 1"""),
         C("fix.txt names dnsPolicy and its new value", 1, r"""grep -Eqi '^ *dnsPolicy *= *ClusterFirst *$' "$WS/q101/fix.txt" 2>/dev/null"""),
     ],
-    solution="""
+    solution=[cmd(r"""k -n lima-client logs deploy/reporter --tail=3
-k -n lima-client logs deploy/reporter --tail=3            # wget: bad address 'ledger:8080'
+k -n lima-client get endpointslices
-k -n lima-client get endpointslices                       # the Service has endpoints - not the problem
+k -n lima-client get deploy reporter -o yaml | grep dnsPolicy""", r"""`bad address 'ledger:8080'`, though the Service has endpoints. `dnsPolicy: Default` uses the *node's* resolver, not cluster DNS - the default policy is `ClusterFirst`, despite the name."""),
-k -n lima-client get deploy reporter -o yaml | grep -n dns
+              hand(r"""k -n lima-client edit deploy reporter""",
-#   dnsPolicy: Default   <- "Default" means: use the NODE's resolver, NOT cluster DNS
+                   r"""In the Pod template's `spec:`, then `:wq`:
-k -n lima-client exec deploy/reporter -- cat /etc/resolv.conf   # the podman/host resolver, no cluster search domains
+```
-k -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'
+      dnsPolicy: ClusterFirst
-k -n lima-client logs deploy/reporter -f                  # "ledger ok"
+```""",
-echo dnsPolicy=ClusterFirst > {WS}/q101/fix.txt
+                   run=r"""kubectl -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'"""),
-# The trap is the name: the DEFAULT dnsPolicy is ClusterFirst, while the policy literally called
+              cmd(r"""echo dnsPolicy=ClusterFirst > {WS}/q101/fix.txt
-# "Default" inherits the node's /etc/resolv.conf. ClusterFirstWithHostNet is for hostNetwork pods.
+k -n lima-client rollout status deploy reporter
-""",
+k -n lima-client logs deploy/reporter --tail=3""", r"""`ledger ok`.""")],
 )
 
 # ------------------------------------------------------------------ 102 ----
@@ -977,22 +1096,16 @@ eq "$(j -n oslo-files deploy uploader '{.status.availableReplicas}')" 1 \
  && j -n oslo-files deploy uploader '{.spec.template.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /files"""),
         C("reason.txt names the unsupported access mode", 2, r"""grep -Eqi 'ReadWriteMany|RWX|access ?mode' "$WS/q102/reason.txt" 2>/dev/null"""),
     ],
-    solution="""
+    solution=[cmd(r"""k -n oslo-files get pod,pvc
-k -n oslo-files get pod                          # Pending
+k -n oslo-files describe pvc shared-files | tail -4""", r"""`ProvisioningFailed ... NodePath only supports ReadWriteOnce and ReadWriteOncePod access modes`: the claim asks for `ReadWriteMany`, which local-path cannot give."""),
-k -n oslo-files describe pod -l app=uploader | tail -4    # unbound immediate PersistentVolumeClaims / waiting
+              cmd(r"""echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
-k -n oslo-files describe pvc shared-files | tail -4
+k -n oslo-files get pvc shared-files -o yaml > pvc.yaml"""),
-#   ProvisioningFailed ... NodePath only supports ReadWriteOnce and ReadWriteOncePod (1.22+) access modes
+              hand(r"""vim pvc.yaml""",
-# accessModes is immutable on a PVC, so recreate it. The Pending pod was never scheduled, so it
+                   r"""Under `accessModes:`, change `ReadWriteMany` to `ReadWriteOnce` - name, size and class stay.""",
-# does not hold the claim (pvc-protection only counts scheduled pods) and the delete goes through.
+                   run=r"""yq -i '.spec.accessModes = ["ReadWriteOnce"]' pvc.yaml"""),
-k -n oslo-files get pvc shared-files -o yaml > pvc.yaml     # change ReadWriteMany -> ReadWriteOnce, strip status/uid
+              cmd(r"""k replace --force -f pvc.yaml
-k -n oslo-files delete pvc shared-files
+k -n oslo-files rollout restart deploy uploader
-k apply -f pvc.yaml
+k -n oslo-files get pvc,pod""", r"""Access modes cannot change on a claim: `--force` deletes it and creates it again. The Pending Pod was never scheduled, so it does not hold the claim. Again until it is `Bound` and the Pod `Running`.""")],
-k -n oslo-files rollout restart deploy uploader            # or just wait - the pending pod retries
-k -n oslo-files get pvc,pod
-echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
-# RWX needs a filesystem many nodes can mount at once (NFS, CephFS, a cloud file service).
-# ReadWriteOnce is per NODE, not per pod - ReadWriteOncePod is the per-pod one.
-""",
 )
 
 QUESTIONS = Q

cka/bank_storage.py

@@ -5,6 +5,8 @@ served by the local-path provisioner in Namespace local-path-storage), so a task
 provisioner or change the default class. Volumes are tested for real - a claim that binds, a file
 that is still on the node, a Pod that reads it.
 """
+
+from sim.bank import cmd, hand
 from cka.common import C, NS, PY, STOR, ns_pick, name_pick
 
 # what a setup records about the objects a task must leave alone
@@ -50,44 +52,85 @@ gone -n %{ns} pvc %{pvc} || exit 1
 node=$(pvnode "$pv"); path=$(pvpath "$pv"); [ -n "$node" ] && [ -n "$path" ] || exit 1
 eq "$(nx "$node" "cat $path/keep.txt" </dev/null 2>/dev/null | tr -d '\r')" "%{text}" """),
  ],
- "solution": """
+ "solution": [cmd(r"""k get sc standard -o yaml | grep provisioner""", r"""`rancher.io/local-path`."""),
-cat <<EOF | k apply -f -
+              hand(r"""vim sc.yaml""",
+                   r"""The class, from the docs page *Storage Classes* - no `is-default-class` annotation, so not the default - and the claim:
+```
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
-metadata: {name: %{sc}}
+metadata:
-provisioner: rancher.io/local-path          # k get sc standard -o yaml shows it
+  name: %{sc}
+provisioner: rancher.io/local-path
 reclaimPolicy: Retain
 volumeBindingMode: WaitForFirstConsumer
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata: {name: %{pvc}, namespace: %{ns}}
+metadata:
+  name: %{pvc}
+  namespace: %{ns}
 spec:
   storageClassName: %{sc}
-  accessModes: [ReadWriteOnce]
+  accessModes:
-  resources: {requests: {storage: 300Mi}}
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 300Mi
+```""",
+                   run=r"""cat > sc.yaml <<'EOF'
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: %{sc}
+provisioner: rancher.io/local-path
+reclaimPolicy: Retain
+volumeBindingMode: WaitForFirstConsumer
 ---
 apiVersion: v1
-kind: Pod
+kind: PersistentVolumeClaim
-metadata: {name: %{pod}, namespace: %{ns}}
+metadata:
+  name: %{pvc}
+  namespace: %{ns}
 spec:
-  volumes: [{name: data, persistentVolumeClaim: {claimName: %{pvc}}}]
+  storageClassName: %{sc}
-  containers:
+  accessModes:
-    - name: writer
+  - ReadWriteOnce
-      image: busybox:1.36
+  resources:
-      command: [sh, -c, "echo %{text} > /data/keep.txt; sleep 86400"]
+    requests:
-      volumeMounts: [{name: data, mountPath: /data}]
+      storage: 300Mi
-EOF
+EOF"""),
-k -n %{ns} get pvc %{pvc}                 # Pending until the Pod is scheduled: WaitForFirstConsumer
+              cmd(r"""k apply -f sc.yaml
-k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt
+k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sh -c "echo %{text} > /data/keep.txt; sleep 86400" > pod.yaml""", r"""The claim stays `Pending` until a Pod uses it: `WaitForFirstConsumer`."""),
-k -n %{ns} delete pod %{pod}; k -n %{ns} delete pvc %{pvc}
+              hand(r"""vim pod.yaml""",
-k get pv $(cat {WS}/q114/pv.txt)          # STATUS Released, RECLAIM POLICY Retain
+                   r"""Add to the container, below `name: %{pod}`:
-echo Released > {WS}/q114/phase.txt
+```
-# A class's reclaimPolicy is copied into every volume it provisions. Delete (the default) removes
+    volumeMounts:
-# the volume and its data with the claim; Retain leaves a Released volume that no new claim can
+    - name: data
-# bind until an administrator clears spec.claimRef. A StorageClass cannot be edited afterwards,
+      mountPath: /data
-# except for a few fields - to change the policy of one volume, patch the PV.
+```
-""",
+and to the Pod's `spec:`, right below `spec:`:
+```
+  volumes:
+  - name: data
+    persistentVolumeClaim:
+      claimName: %{pvc}
+```""",
+                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}]
+  | .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "%{pvc}"}}]' pod.yaml"""),
+              cmd(r"""k apply -f pod.yaml"""),
+              hand(r"""k -n %{ns} get pod,pvc""",
+                   r"""Again until the Pod is `Running` and the claim `Bound`; note its `VOLUME`.""",
+                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
+              hand(r"""echo pvc-0b1c2d3e-... > {WS}/q114/pv.txt""",
+                   r"""With the volume's name.""",
+                   run=r"""kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt"""),
+              cmd(r"""k -n %{ns} delete pod %{pod} --now
+k -n %{ns} delete pvc %{pvc}"""),
+              hand(r"""k get pv
+echo Released > {WS}/q114/phase.txt""",
+                   r"""`STATUS Released`, `RECLAIM POLICY Retain`: the class's policy is copied into every volume it provisions. A Released volume binds no new claim until its `claimRef` is cleared.""",
+                   run=r"""sleep 4
+kubectl get pv $(cat {WS}/q114/pv.txt) -o jsonpath='{.status.phase}' > {WS}/q114/phase.txt""")],
 },
 # ------------------------------------------------------------------ 115 ----
 {
@@ -133,39 +176,43 @@ eq "$(one "$WS/q115/claim.txt" | sed 's#.*/##')" %{pod}-scratch || exit 1
 eq "$(j -n %{ns} pvc %{pod}-scratch '{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}')" Pod/%{pod} || exit 1
 eq "$(one "$WS/q115/owner.txt" | tr A-Z a-z)" pod/%{pod}"""),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
-cat <<EOF | k apply -f -
+              hand(r"""vim pod.yaml""",
-apiVersion: v1
+                   r"""Add to the container, below `name: %{pod}`:
-kind: Pod
+```
-metadata: {name: %{pod}, namespace: %{ns}}
+    volumeMounts:
-spec:
+    - name: scratch
+      mountPath: /scratch
+    - name: cache
+      mountPath: /cache
+```
+and to the Pod's `spec:`, right below `spec:` (the docs page *Ephemeral Volumes* has the generic ephemeral volume):
+```
   volumes:
   - name: scratch
     ephemeral:
       volumeClaimTemplate:
         spec:
           storageClassName: standard
-            accessModes: [ReadWriteOnce]
+          accessModes:
-            resources: {requests: {storage: 100Mi}}
+          - ReadWriteOnce
+          resources:
+            requests:
+              storage: 100Mi
   - name: cache
-      emptyDir: {medium: Memory, sizeLimit: 32Mi}
+    emptyDir:
-  containers:
+      medium: Memory
-    - name: encoder
+      sizeLimit: 32Mi
-      image: busybox:1.36
+```""",
-      command: [sleep, "86400"]
+                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "scratch", "mountPath": "/scratch"}, {"name": "cache", "mountPath": "/cache"}]
-      volumeMounts:
+  | .spec.volumes = [{"name": "scratch", "ephemeral": {"volumeClaimTemplate": {"spec": {"storageClassName": "standard", "accessModes": ["ReadWriteOnce"], "resources": {"requests": {"storage": "100Mi"}}}}}}, {"name": "cache", "emptyDir": {"medium": "Memory", "sizeLimit": "32Mi"}}]' pod.yaml"""),
-        - {name: scratch, mountPath: /scratch}
+              cmd(r"""k apply -f pod.yaml"""),
-        - {name: cache, mountPath: /cache}
+              hand(r"""k -n %{ns} get pod,pvc""",
-EOF
+                   r"""Again until the Pod is `Running`. The claim is `<pod>-<volume>`: `%{pod}-scratch`.""",
-k -n %{ns} get pvc                                   # %{pod}-scratch: <pod name>-<volume name>
+                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
-echo %{pod}-scratch > {WS}/q115/claim.txt
+              cmd(r"""echo %{pod}-scratch > {WS}/q115/claim.txt
-k -n %{ns} get pvc %{pod}-scratch -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}'
+k -n %{ns} get pvc %{pod}-scratch -o yaml | grep -A5 ownerReferences
-echo Pod/%{pod} > {WS}/q115/owner.txt
+echo Pod/%{pod} > {WS}/q115/owner.txt""", r"""The Pod owns it: delete the Pod, and the claim and its volume go too. An emptyDir in `Memory` is a tmpfs, counted as the container's memory.""")],
-# A generic ephemeral volume is a claim written into the Pod: Kubernetes creates the PVC, the Pod
-# owns it, and deleting the Pod deletes claim and volume. Unlike emptyDir it can come from any
-# StorageClass - size, speed, snapshots - and unlike a normal claim it never outlives the Pod.
-# An emptyDir with medium Memory is a tmpfs: what is written there counts as the container's memory.
-""",
 },
 # ------------------------------------------------------------------ 116 ----
 {
@@ -234,18 +281,20 @@ gone -n %{ns} pvc data-%{sts}-1 && gone -n %{ns} pvc data-%{sts}-2 && eq "$(j -n
   C("claims.txt lists what is left: data-%{sts}-0", 1, r"""
 eq "$(tr -d ' \t\r' < "$WS/q116/claims.txt" 2>/dev/null | sed 's#.*/##' | grep . | sort | tr '\n' ' ')" "data-%{sts}-0 " """),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} get sts %{sts} -o yaml | grep -A2 persistentVolumeClaimRetentionPolicy""", r"""`Retain` for both: by default a StatefulSet never deletes a claim."""),
-k -n %{ns} get sts %{sts} -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}'      # whenDeleted Retain, whenScaled Retain
+              hand(r"""k -n %{ns} edit sts %{sts}""",
-k -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'
+                   r"""Set it before scaling - it applies to the Pods being removed - then `:wq`:
-k -n %{ns} scale sts %{sts} --replicas=1
+```
-k -n %{ns} get pvc -w                     # data-%{sts}-2 and data-%{sts}-1 disappear with their Pods
+  persistentVolumeClaimRetentionPolicy:
-k -n %{ns} get pvc -o name | cut -d/ -f2 > {WS}/q116/claims.txt
+    whenDeleted: Retain
-# By default a StatefulSet never deletes a claim: scale down and up again and each replica finds
+    whenScaled: Delete
-# its old data. persistentVolumeClaimRetentionPolicy changes that per event - whenScaled for the
+```""",
-# replicas a scale-down removes, whenDeleted for all of them when the StatefulSet goes. Set the
+                   run=r"""kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'"""),
-# policy BEFORE scaling: it is applied to the Pods being removed, not to claims orphaned earlier
+              cmd(r"""k -n %{ns} scale sts %{sts} --replicas=1"""),
-# (those you delete by hand). It is one of the few StatefulSet fields that can be changed in place.
+              hand(r"""k -n %{ns} get pvc""",
-""",
+                   r"""Again until `data-%{sts}-2` and `data-%{sts}-1` have gone with their Pods.""",
+                   run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc --no-headers | wc -l)" = 1 ] && break; sleep 3; done"""),
+              cmd(r"""echo data-%{sts}-0 > {WS}/q116/claims.txt""")],
 },
 # ------------------------------------------------------------------ 117 ----
 {
@@ -292,34 +341,76 @@ eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}/{.spec.storageClassName}/{.spec.
 [ -n "$(j pv %{pv} '{.metadata.name}')" ] || exit 1
 eq "$(j -n %{oth} pvc any-volume '{.metadata.uid}')" "$(cat "$PRIV/q117.uid")" && eq "$(j -n %{oth} pvc any-volume '{.status.phase}')" Pending"""),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{oth} get pvc any-volume""", r"""`Pending`, for any 1Gi `manual` volume: an Available one would go to it."""),
-cat <<EOF | k apply -f -
+              hand(r"""vim pv.yaml""",
+                   r"""A `claimRef` on the PV reserves it for one claim, so the waiting one cannot take it:
+```
 apiVersion: v1
 kind: PersistentVolume
-metadata: {name: %{pv}}
+metadata:
+  name: %{pv}
 spec:
-  capacity: {storage: 1Gi}
-  accessModes: [ReadWriteOnce]
   storageClassName: manual
+  capacity:
+    storage: 1Gi
+  accessModes:
+  - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
-  hostPath: {path: /mnt/%{pv}}
+  hostPath:
-  claimRef: {namespace: %{ns}, name: %{pvc}}      # the reservation: no other claim can bind this volume
+    path: /mnt/%{pv}
+  claimRef:
+    namespace: %{ns}
+    name: %{pvc}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata: {name: %{pvc}, namespace: %{ns}}
+metadata:
+  name: %{pvc}
+  namespace: %{ns}
 spec:
   storageClassName: manual
-  accessModes: [ReadWriteOnce]
+  accessModes:
-  resources: {requests: {storage: 1Gi}}
+  - ReadWriteOnce
-EOF
+  resources:
-k get pv %{pv}; k -n %{ns} get pvc %{pvc}; k -n %{oth} get pvc any-volume      # Bound, Bound, Pending
+    requests:
-# Without claimRef the volume would be Available, and the control plane hands an Available volume
+      storage: 1Gi
-# to whichever matching claim it looks at first - here the one that has been waiting. That cannot
+```
-# be undone cleanly: a claim whose volume is taken away ends up Lost. claimRef on the PV reserves
+`volumeName` on a claim is the other half: it asks for one volume, but keeps no other claim away from it.""",
-# it; volumeName on a claim is the other half (it asks for one volume but does not keep others
+                   run=r"""cat > pv.yaml <<'EOF'
-# away from it). Use both when it has to be this pair and no other.
+apiVersion: v1
-""",
+kind: PersistentVolume
+metadata:
+  name: %{pv}
+spec:
+  storageClassName: manual
+  capacity:
+    storage: 1Gi
+  accessModes:
+  - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  hostPath:
+    path: /mnt/%{pv}
+  claimRef:
+    namespace: %{ns}
+    name: %{pvc}
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: %{pvc}
+  namespace: %{ns}
+spec:
+  storageClassName: manual
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+EOF"""),
+              cmd(r"""k apply -f pv.yaml
+k get pv %{pv}
+k -n %{ns} get pvc %{pvc}
+k -n %{oth} get pvc any-volume""", r"""`Bound`, `Bound`, still `Pending`.""")],
 },
 # ------------------------------------------------------------------ 118 ----
 {
@@ -405,30 +496,53 @@ done; exit 1"""),
 eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}')" %{pv} || exit 1
 for i in $(seq 1 10); do gone pv "$(cat "$PRIV/q118.wrong")" && exit 0; sleep 3; done; exit 1"""),
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} get pvc %{pvc}
-k -n %{ns} get pvc %{pvc}                  # Bound to pvc-..., STORAGECLASS standard - not to %{pv}
+k get pv %{pv}""", r"""Bound to a `pvc-...` volume of class `standard`, while `%{pv}` is `Available` with no class: the manifest names no class, so the claim got the default one, and a new empty volume."""),
-k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt
+              hand(r"""echo pvc-0b1c2d3e-... > {WS}/q118/wrong-pv.txt""",
-k get pv %{pv}                             # Available, STORAGECLASS empty
+                   r"""With the claim's `VOLUME`.""",
-# The manifest names no class, so the claim was given the cluster's default one (standard) and a
+                   run=r"""kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt"""),
-# brand-new, empty volume was provisioned for it. A claim's class and volume cannot be edited:
+              cmd(r"""k -n %{ns} scale deploy %{dep} --replicas=0""", r"""A claim's class and volume cannot be edited, and a claim in use is not deleted (pvc-protection): stop the Pod first."""),
-k -n %{ns} scale deploy %{dep} --replicas=0          # a claim in use is not deleted (pvc-protection)
+              hand(r"""k -n %{ns} get pods""",
-k -n %{ns} delete pvc %{pvc}                         # its volume goes with it: reclaim policy Delete
+                   r"""Again until the Pod has gone.""",
-cat <<EOF | k apply -f -
+                   run=r"""kubectl -n %{ns} wait --for=delete pod -l app=%{dep} --timeout=120s"""),
+              cmd(r"""k -n %{ns} delete pvc %{pvc}""", r"""Its volume goes with it: reclaim policy `Delete`."""),
+              hand(r"""vim pvc.yaml""",
+                   r"""`storageClassName: ""` - no class: never the default, never provisioned - and the volume by name:
+```
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata: {name: %{pvc}, namespace: %{ns}}
+metadata:
+  name: %{pvc}
+  namespace: %{ns}
 spec:
-  storageClassName: ""            # "no class": never the default, never provisioned
+  storageClassName: ""
   volumeName: %{pv}
-  accessModes: [ReadWriteOnce]
+  accessModes:
-  resources: {requests: {storage: 500Mi}}
+  - ReadWriteOnce
-EOF
+  resources:
+    requests:
+      storage: 500Mi
+```
+`volumeName` alone would not do: the default class is still filled in, and it does not match the class-less volume.""",
+                   run=r"""cat > pvc.yaml <<'EOF'
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: %{pvc}
+  namespace: %{ns}
+spec:
+  storageClassName: ""
+  volumeName: %{pv}
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 500Mi
+EOF"""),
+              cmd(r"""k apply -f pvc.yaml
 k -n %{ns} scale deploy %{dep} --replicas=1
-k -n %{ns} logs deploy/%{dep}              # ledger %{tok} balanced
+k -n %{ns} rollout status deploy %{dep}
-# storageClassName has three states: a name; "" = only volumes without a class, no provisioning;
+k -n %{ns} logs deploy/%{dep}""", r"""`ledger %{tok} balanced`.""")],
-# and absent = "give me the default class". volumeName alone would not have helped: the default
-# class is still filled in, and it does not match the class-less volume.
-""",
 },
 # ------------------------------------------------------------------ 119 ----
 {
@@ -492,34 +606,120 @@ eq "$(j pv %{{p}}-{u} '{{.metadata.uid}}')" "$(sed -n {n}p "$PRIV/q119.uids")" |
 eq "$(j pv %{{p}}-{u} '{{.spec.capacity.storage}}/{{.spec.accessModes[*]}}/{{.spec.storageClassName}}/{{.metadata.labels.use}}')" {size}/ReadWriteOnce/manual/{u}""")
   for n, (u, size) in enumerate([("reports", "1Gi"), ("archive", "2Gi"), ("media", "1Gi")], 1)
  ],
- "solution": """
+ "solution": [cmd(r"""k -n %{ns} get pvc
-k -n %{ns} get pvc; k get pv --show-labels
+k -n %{ns} describe pvc reports archive media | grep -e ^Name: -e Warning -e Normal""", r"""`media`: `storageclass.storage.k8s.io "manuel" not found` - the class, a typo. The other two only say no volume fits: compare claim and volume yourself."""),
-k -n %{ns} describe pvc reports archive media | grep -E '^Name:|Warning|Normal'
+              cmd(r"""k -n %{ns} get pvc -o custom-columns=NAME:.metadata.name,SIZE:.spec.resources.requests.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
-#   media:   storageclass.storage.k8s.io "manuel" not found           -> class (a typo for manual)
+k get pv -o custom-columns=NAME:.metadata.name,SIZE:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName""", r"""`reports` asks for 5Gi of a 1Gi volume; `archive` for `ReadWriteMany`, its volume offers `ReadWriteOnce`. A claim binds only when the class matches, the volume is large enough, it has every access mode asked for, and the selector matches."""),
-#   reports, archive: no persistent volumes available for this claim   -> compare claim and volume:
+              hand(r"""vim {WS}/q119/reasons.txt""",
-k -n %{ns} get pvc -o custom-columns=NAME:.metadata.name,SIZE:.spec.resources.requests.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
+                   r"""```
-k get pv -o custom-columns=NAME:.metadata.name,SIZE:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
+reports capacity
-#   reports asks for 5Gi, its volume has 1Gi                          -> capacity
+archive accessmode
-#   archive asks for ReadWriteMany, its volume offers ReadWriteOnce   -> accessmode
+media class
-printf 'reports capacity\\narchive accessmode\\nmedia class\\n' > {WS}/q119/reasons.txt
+```""",
-# size (downwards), access modes and class of a claim cannot be edited: recreate the three
+                   run=r"""printf 'reports capacity\narchive accessmode\nmedia class\n' > {WS}/q119/reasons.txt"""),
-k -n %{ns} delete pvc reports archive media
+              cmd(r"""k -n %{ns} delete pvc reports archive media""", r"""None of the three fields can be edited on a claim: recreate them."""),
-for c in "reports 1Gi" "archive 2Gi" "media 1Gi"; do set -- $c; cat <<EOF | k apply -f -
+              hand(r"""vim pvcs.yaml""",
+                   r"""The three claims again, each keeping its selector:
+```
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata: {name: $1, namespace: %{ns}}
+metadata:
+  name: reports
+  namespace: %{ns}
 spec:
   storageClassName: manual
-  accessModes: [ReadWriteOnce]
+  accessModes:
-  selector: {matchLabels: {use: $1}}
+  - ReadWriteOnce
-  resources: {requests: {storage: $2}}
+  selector:
-EOF
+    matchLabels:
-done
+      use: reports
-k -n %{ns} get pvc                         # all Bound, each to its own volume
+  resources:
-# A claim binds a volume only if ALL of these hold: same class, the volume is at least as large,
+    requests:
-# it offers every access mode the claim asks for, and it matches the claim's selector. The event
+      storage: 1Gi
-# only names the class problem - for the rest you compare the two objects yourself.
+---
-""",
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: archive
+  namespace: %{ns}
+spec:
+  storageClassName: manual
+  accessModes:
+  - ReadWriteOnce
+  selector:
+    matchLabels:
+      use: archive
+  resources:
+    requests:
+      storage: 2Gi
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: media
+  namespace: %{ns}
+spec:
+  storageClassName: manual
+  accessModes:
+  - ReadWriteOnce
+  selector:
+    matchLabels:
+      use: media
+  resources:
+    requests:
+      storage: 1Gi
+```""",
+                   run=r"""cat > pvcs.yaml <<'EOF'
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: reports
+  namespace: %{ns}
+spec:
+  storageClassName: manual
+  accessModes:
+  - ReadWriteOnce
+  selector:
+    matchLabels:
+      use: reports
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: archive
+  namespace: %{ns}
+spec:
+  storageClassName: manual
+  accessModes:
+  - ReadWriteOnce
+  selector:
+    matchLabels:
+      use: archive
+  resources:
+    requests:
+      storage: 2Gi
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: media
+  namespace: %{ns}
+spec:
+  storageClassName: manual
+  accessModes:
+  - ReadWriteOnce
+  selector:
+    matchLabels:
+      use: media
+  resources:
+    requests:
+      storage: 1Gi
+EOF"""),
+              cmd(r"""k apply -f pvcs.yaml
+k -n %{ns} get pvc""", r"""All three `Bound`, each to its own volume.""")],
 },
 # ------------------------------------------------------------------ 120 ----
 {
@@ -605,24 +805,20 @@ done; exit 1"""),
 eq "$(j pv %{pv} '{.metadata.uid}')" "$(sed -n 1p "$PRIV/q120.uids")" && eq "$(j -n %{ns} pvc %{dep}-data '{.metadata.uid}')" "$(sed -n 2p "$PRIV/q120.uids")" \
  && eq "$(pvnode %{pv})/$(pvpath %{pv})" "$W1//mnt/%{pv}" """),
  ],
- "solution": """
+ "solution": [hand(r"""k -n %{ns} describe pod -l app=%{dep} | grep FailedScheduling > {WS}/q120/event.txt
-k -n %{ns} describe pod -l app=%{dep} | tail -5
+cat {WS}/q120/event.txt""",
-#   0/3 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s)
+                   r"""`1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) didn't match Pod's node affinity/selector`: the scheduler ANDs every constraint - taints, the Pod's selector, and the node affinity of each volume it mounts.""",
-#   didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s). ...
+                   run=r"""for i in $(seq 1 40); do
-k -n %{ns} get events --field-selector reason=FailedScheduling -o jsonpath='{.items[-1:].message}' > {WS}/q120/event.txt
+  kubectl -n %{ns} describe pod -l app=%{dep} | grep FailedScheduling > {WS}/q120/event.txt && break; sleep 3
-k get pv %{pv} -o jsonpath='{.spec.nodeAffinity}'; echo       # the volume is on {W1}
+done"""),
-k -n %{ns} get deploy %{dep} -o jsonpath='{.spec.template.spec.nodeSelector}'; echo    # disk=ssd
+              cmd(r"""k describe pv %{pv} | grep -A3 'Node Affinity'
-k get nodes -L disk                                           # only {W2} carries disk=ssd
+k -n %{ns} get deploy %{dep} -o yaml | grep -A1 nodeSelector
-# The volume cannot move, so the Pod has to: drop the nodeSelector (the volume's own node
+k get nodes -L disk""", r"""The volume is on `{W1}`; the Pod must go to `disk=ssd`, which only `{W2}` has. The volume cannot move, so the Pod has to."""),
-# affinity already sends the Pod to the right node) ...
+              hand(r"""k -n %{ns} edit deploy %{dep}""",
-k -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
+                   r"""Delete the two lines of the `nodeSelector` (`nodeSelector:` and `disk: ssd`), then `:wq` - the volume's node affinity already sends the Pod to the right node. (If the selector has a reason to exist, `k label node {W1} disk=ssd` instead.)""",
-# ... or, if the selector has a reason to exist, give {W1} the label:  k label node {W1} disk=ssd
+                   run=r"""kubectl -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'"""),
-k -n %{ns} rollout status deploy %{dep}; k -n %{ns} logs deploy/%{dep}
+              cmd(r"""k -n %{ns} rollout status deploy %{dep}
-# The scheduler ANDs every constraint: taints, the Pod's nodeSelector/affinity, and the node
+k -n %{ns} logs deploy/%{dep}""", r"""`orders %{tok} open`.""")],
-# affinity of each volume it mounts. "didn't match PersistentVolume's node affinity" (before v1.35:
-# "had volume node affinity conflict") always means: the Pod is not allowed onto the one node its
-# volume lives on.
-""",
 },
 # ------------------------------------------------------------------ 121 ----
 {
@@ -882,19 +1078,15 @@ running %{ns} %{pod} 60"""),
   C("class.txt holds %{sc}", 1, STOR + r"""
 eq "$(j -n %{ns} pvc %{pvc} '{.spec.storageClassName}')" %{sc} && eq "$(one "$WS/q123/class.txt")" %{sc}"""),
  ],
- "solution": """
+ "solution": [cmd(r"""k get sc
-k get sc                                   # no "(default)" behind any name
+k -n %{ns} describe pvc %{pvc} | tail -2""", r"""No `(default)` anywhere, and the claim waits: `no storage class is set`."""),
-k -n %{ns} describe pvc %{pvc} | tail -2   # no persistent volumes available for this claim and no storage class is set
+              cmd(r"""k annotate sc %{sc} storageclass.kubernetes.io/is-default-class=true
-k patch sc %{sc} -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
+k get sc""", r"""`%{sc} (default)`. A claim without a class gets the default when it is created - or, if there is none, as soon as one appears: no need to recreate it."""),
-k get sc                                   # %{sc} (default)
+              hand(r"""k -n %{ns} get pvc,pod""",
-k -n %{ns} get pvc %{pvc} -w               # within seconds: STORAGECLASS %{sc}, then Bound once the Pod is scheduled
+                   r"""Again until the claim shows `STORAGECLASS %{sc}` and `Bound`, and the Pod `Running`.""",
-k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.storageClassName}' > {WS}/q123/class.txt
+                   run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.status.phase}')" = Bound ] && break; sleep 3; done
-# A claim without storageClassName gets the default class when it is created. If there is none
+kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
-# it stays Pending with the field unset - and is given the default as soon as one appears: the
+              cmd(r"""echo %{sc} > {WS}/q123/class.txt""", r"""Marking `standard` would have bound it too - to the wrong class. With two defaults, new claims take the newest; never leave two.""")],
-# claim does not have to be recreated. Marking `standard` instead would have bound it as well,
-# to the wrong class. With two defaults, new claims get the most recently created one - never
-# leave two.
-""",
 },
 # ------------------------------------------------------------------ 124 ----
 {
@@ -937,53 +1129,116 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.nodeName}')" "$W2" || exit 1
 [ -z "$(j -n %{ns} pod %{pod} '{.spec.nodeSelector}{.spec.affinity}')" ] || exit 1
 eq "$($K -n %{ns} exec %{pod} -- cat /disk/disk.id 2>/dev/null | tr -d '\r')" "disk %{tok}" """),
  ],
- "solution": """
+ "solution": [hand(r"""vim local.yaml""",
-cat <<EOF | k apply -f -
+                   r"""The docs page *Storage Classes* (under *Local*) has the class, *Persistent Volumes* a `local` volume with its required node affinity:
+```
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
-metadata: {name: %{sc}}
+metadata:
+  name: %{sc}
 provisioner: kubernetes.io/no-provisioner
 volumeBindingMode: WaitForFirstConsumer
 ---
 apiVersion: v1
 kind: PersistentVolume
-metadata: {name: %{pv}}
+metadata:
+  name: %{pv}
 spec:
-  capacity: {storage: 1Gi}
+  capacity:
-  accessModes: [ReadWriteOnce]
+    storage: 1Gi
-  storageClassName: %{sc}
+  accessModes:
+  - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
-  local: {path: /mnt/disks/%{pv}}
+  storageClassName: %{sc}
-  nodeAffinity:                              # required for a local volume
+  local:
+    path: /mnt/disks/%{pv}
+  nodeAffinity:
     required:
       nodeSelectorTerms:
-        - matchExpressions: [{key: kubernetes.io/hostname, operator: In, values: [{W2}]}]
+      - matchExpressions:
+        - key: kubernetes.io/hostname
+          operator: In
+          values:
+          - {W2}
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
-metadata: {name: %{pvc}, namespace: %{ns}}
+metadata:
+  name: %{pvc}
+  namespace: %{ns}
 spec:
   storageClassName: %{sc}
-  accessModes: [ReadWriteOnce]
+  accessModes:
-  resources: {requests: {storage: 1Gi}}
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+```""",
+                   run=r"""cat > local.yaml <<'EOF'
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: %{sc}
+provisioner: kubernetes.io/no-provisioner
+volumeBindingMode: WaitForFirstConsumer
 ---
 apiVersion: v1
-kind: Pod
+kind: PersistentVolume
-metadata: {name: %{pod}, namespace: %{ns}}
+metadata:
+  name: %{pv}
 spec:
-  volumes: [{name: disk, persistentVolumeClaim: {claimName: %{pvc}}}]
+  capacity:
-  containers:
+    storage: 1Gi
-    - name: prom
+  accessModes:
-      image: busybox:1.36
+  - ReadWriteOnce
-      command: [sleep, "86400"]
+  persistentVolumeReclaimPolicy: Retain
-      volumeMounts: [{name: disk, mountPath: /disk}]
+  storageClassName: %{sc}
-EOF
+  local:
-k -n %{ns} get pod %{pod} -o wide          # on {W2}
+    path: /mnt/disks/%{pv}
-k -n %{ns} exec %{pod} -- cat /disk/disk.id
+  nodeAffinity:
-# hostPath is a directory on whatever node the Pod happens to land on; a `local` volume knows
+    required:
-# which node it is on, and the scheduler follows it. WaitForFirstConsumer delays binding until a
+      nodeSelectorTerms:
-# Pod needs the claim, so that the Pod's other constraints are weighed together with the volume's.
+      - matchExpressions:
-# The page to search for: "Storage Classes" -> Local.
+        - key: kubernetes.io/hostname
-""",
+          operator: In
+          values:
+          - {W2}
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: %{pvc}
+  namespace: %{ns}
+spec:
+  storageClassName: %{sc}
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+EOF"""),
+              cmd(r"""k apply -f local.yaml
+k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
+              hand(r"""vim pod.yaml""",
+                   r"""Add to the container, below `name: %{pod}`:
+```
+    volumeMounts:
+    - name: disk
+      mountPath: /disk
+```
+and to the Pod's `spec:`, right below `spec:`:
+```
+  volumes:
+  - name: disk
+    persistentVolumeClaim:
+      claimName: %{pvc}
+```""",
+                   run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "disk", "mountPath": "/disk"}]
+  | .spec.volumes = [{"name": "disk", "persistentVolumeClaim": {"claimName": "%{pvc}"}}]' pod.yaml"""),
+              cmd(r"""k apply -f pod.yaml"""),
+              hand(r"""k -n %{ns} get pod %{pod} -o wide""",
+                   r"""Again until `Running` - on `{W2}`, sent there by the volume alone. A `hostPath` is a directory on whatever node the Pod lands on; a `local` volume knows its node, and the scheduler follows it.""",
+                   run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
+              cmd(r"""k -n %{ns} exec %{pod} -- cat /disk/disk.id""")],
 },
 ]

cka/questions.py

@@ -667,7 +667,8 @@ _ids = [q["id"] for q in QUESTIONS]
 assert len(_ids) == len(set(_ids)), "duplicate question ids"
 from cka.solutions import S as _SOL
 for _q in QUESTIONS:
-    _q["solution"] = (_q.get("solution") or _SOL.get(_q["id"], "")).strip()
+    _s = _q.get("solution") or _SOL.get(_q["id"], "")
+    _q["solution"] = _s if isinstance(_s, list) else _s.strip()   # a list: parts (sim/bank.py cmd, hand)
     _q.setdefault("set", None)
 _missing = [q["id"] for q in QUESTIONS if not q["solution"]]
 assert not _missing, f"tasks without a solution: {_missing}"

cka/solve.py

@@ -8,6 +8,10 @@ substituted as in the task. A task without a script is only audited for a clean
 A command that reads stdin (ssh, kubectl run -i) would swallow the rest of the script: ssh -n,
 </dev/null.
 
+A task whose reference solution is in parts (sim/bank.py: cmd, hand) has no script here: the audit
+runs the parts - what is typed as it stands, and for each edit by hand its script - so the solution
+the candidate reads is the one that is proven.
+
 SOLVE: task id -> bash
 """
 
@@ -25,142 +29,6 @@ cpwait() {
 
 SOLVE = {
 # ---- 114-124: bank_storage.py
-114: r"""
-cat <<EOF | kubectl apply -f -
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata: {name: %{sc}}
-provisioner: rancher.io/local-path
-reclaimPolicy: Retain
-volumeBindingMode: WaitForFirstConsumer
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata: {name: %{pvc}, namespace: %{ns}}
-spec:
-  storageClassName: %{sc}
-  accessModes: [ReadWriteOnce]
-  resources: {requests: {storage: 300Mi}}
----
-apiVersion: v1
-kind: Pod
-metadata: {name: %{pod}, namespace: %{ns}}
-spec:
-  terminationGracePeriodSeconds: 2
-  volumes: [{name: data, persistentVolumeClaim: {claimName: %{pvc}}}]
-  containers:
-    - name: writer
-      image: busybox:1.36
-      command: [sh, -c, "echo %{text} > /data/keep.txt; sleep 86400"]
-      volumeMounts: [{name: data, mountPath: /data}]
-EOF
-kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
-kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt
-kubectl -n %{ns} delete pod %{pod}
-kubectl -n %{ns} delete pvc %{pvc}
-sleep 4
-kubectl get pv $(cat {WS}/q114/pv.txt) -o jsonpath='{.status.phase}' > {WS}/q114/phase.txt
-""",
-115: r"""
-cat <<EOF | kubectl apply -f -
-apiVersion: v1
-kind: Pod
-metadata: {name: %{pod}, namespace: %{ns}}
-spec:
-  volumes:
-    - name: scratch
-      ephemeral:
-        volumeClaimTemplate:
-          spec:
-            storageClassName: standard
-            accessModes: [ReadWriteOnce]
-            resources: {requests: {storage: 100Mi}}
-    - name: cache
-      emptyDir: {medium: Memory, sizeLimit: 32Mi}
-  containers:
-    - name: encoder
-      image: busybox:1.36
-      command: [sleep, "86400"]
-      volumeMounts:
-        - {name: scratch, mountPath: /scratch}
-        - {name: cache, mountPath: /cache}
-EOF
-kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
-kubectl -n %{ns} get pvc -o jsonpath='{.items[0].metadata.name}' > {WS}/q115/claim.txt
-kubectl -n %{ns} get pvc %{pod}-scratch -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}' > {WS}/q115/owner.txt
-""",
-116: r"""
-kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'
-kubectl -n %{ns} scale sts %{sts} --replicas=1
-for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc --no-headers | wc -l)" = 1 ] && break; sleep 3; done
-kubectl -n %{ns} get pvc -o name | cut -d/ -f2 > {WS}/q116/claims.txt
-""",
-117: r"""
-cat <<EOF | kubectl apply -f -
-apiVersion: v1
-kind: PersistentVolume
-metadata: {name: %{pv}}
-spec:
-  capacity: {storage: 1Gi}
-  accessModes: [ReadWriteOnce]
-  storageClassName: manual
-  persistentVolumeReclaimPolicy: Retain
-  hostPath: {path: /mnt/%{pv}}
-  claimRef: {namespace: %{ns}, name: %{pvc}}
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata: {name: %{pvc}, namespace: %{ns}}
-spec:
-  storageClassName: manual
-  accessModes: [ReadWriteOnce]
-  resources: {requests: {storage: 1Gi}}
-EOF
-sleep 5
-""",
-118: r"""
-kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt
-kubectl -n %{ns} scale deploy %{dep} --replicas=0
-kubectl -n %{ns} wait --for=delete pod -l app=%{dep} --timeout=120s
-kubectl -n %{ns} delete pvc %{pvc}
-cat <<EOF | kubectl apply -f -
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata: {name: %{pvc}, namespace: %{ns}}
-spec:
-  storageClassName: ""
-  volumeName: %{pv}
-  accessModes: [ReadWriteOnce]
-  resources: {requests: {storage: 500Mi}}
-EOF
-kubectl -n %{ns} scale deploy %{dep} --replicas=1
-kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
-""",
-119: r"""
-printf 'reports capacity\narchive accessmode\nmedia class\n' > {WS}/q119/reasons.txt
-kubectl -n %{ns} delete pvc reports archive media
-for c in "reports 1Gi" "archive 2Gi" "media 1Gi"; do set -- $c; cat <<EOF | kubectl apply -f -
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata: {name: $1, namespace: %{ns}}
-spec:
-  storageClassName: manual
-  accessModes: [ReadWriteOnce]
-  selector: {matchLabels: {use: $1}}
-  resources: {requests: {storage: $2}}
-EOF
-done
-sleep 5
-""",
-120: r"""
-for i in $(seq 1 40); do
-  M=$(kubectl -n %{ns} get events --field-selector reason=FailedScheduling -o jsonpath='{.items[-1:].message}')
-  [ -n "$M" ] && break; sleep 3
-done
-echo "$M" > {WS}/q120/event.txt
-kubectl -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
-kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
-""",
 121: r"""
 kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.metadata.finalizers[0]}' > {WS}/q121/finalizer.txt
 echo deployment/%{dep} > {WS}/q121/used-by.txt
@@ -176,55 +44,6 @@ for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.sta
 kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
 echo rancher.io/local-path > {WS}/q122/provisioner.txt
 """,
-123: r"""
-kubectl annotate sc %{sc} storageclass.kubernetes.io/is-default-class=true
-for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.status.phase}')" = Bound ] && break; sleep 3; done
-kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
-kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.storageClassName}' > {WS}/q123/class.txt
-""",
-124: r"""
-cat <<EOF | kubectl apply -f -
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata: {name: %{sc}}
-provisioner: kubernetes.io/no-provisioner
-volumeBindingMode: WaitForFirstConsumer
----
-apiVersion: v1
-kind: PersistentVolume
-metadata: {name: %{pv}}
-spec:
-  capacity: {storage: 1Gi}
-  accessModes: [ReadWriteOnce]
-  storageClassName: %{sc}
-  persistentVolumeReclaimPolicy: Retain
-  local: {path: /mnt/disks/%{pv}}
-  nodeAffinity:
-    required:
-      nodeSelectorTerms:
-        - matchExpressions: [{key: kubernetes.io/hostname, operator: In, values: [{W2}]}]
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata: {name: %{pvc}, namespace: %{ns}}
-spec:
-  storageClassName: %{sc}
-  accessModes: [ReadWriteOnce]
-  resources: {requests: {storage: 1Gi}}
----
-apiVersion: v1
-kind: Pod
-metadata: {name: %{pod}, namespace: %{ns}}
-spec:
-  volumes: [{name: disk, persistentVolumeClaim: {claimName: %{pvc}}}]
-  containers:
-    - name: prom
-      image: busybox:1.36
-      command: [sleep, "86400"]
-      volumeMounts: [{name: disk, mountPath: /disk}]
-EOF
-kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
-""",
 # ---- 125-128: bank_net.py
 125: r"""
 kubectl -n kube-system get cm coredns -o yaml > {WS}/q125/coredns-backup.yaml
@@ -233,36 +52,6 @@ kubectl -n kube-system get cm coredns -o json | jq --arg add "$ADD" '.data.Coref
 kubectl -n kube-system rollout restart deploy coredns
 kubectl -n kube-system rollout status deploy coredns --timeout=240s
 """,
-126: r"""
-cat <<EOF | kubectl apply -f -
-apiVersion: gateway.networking.k8s.io/v1
-kind: Gateway
-metadata: {name: %{gw}, namespace: %{ns}}
-spec:
-  gatewayClassName: %{gc}
-  listeners:
-    - name: https
-      protocol: HTTPS
-      port: 443
-      hostname: %{host}.example.com
-      tls:
-        mode: Terminate
-        certificateRefs: [{kind: Secret, name: %{host}-tls}]
----
-apiVersion: gateway.networking.k8s.io/v1
-kind: HTTPRoute
-metadata: {name: %{host}, namespace: %{ns}}
-spec:
-  parentRefs: [{name: %{gw}}]
-  hostnames: [%{host}.example.com]
-  rules:
-    - matches: [{path: {type: PathPrefix, value: /api}}]
-      backendRefs: [{name: api, port: 8080}]
-    - matches: [{path: {type: PathPrefix, value: /}}]
-      backendRefs: [{name: web, port: 80}]
-EOF
-kubectl -n %{ns} delete ingress %{host}
-""",
 127: r"""
 kubectl annotate ingressclass %{old} ingressclass.kubernetes.io/is-default-class-
 kubectl annotate ingressclass %{cls} ingressclass.kubernetes.io/is-default-class=true
@@ -272,16 +61,6 @@ kubectl -n %{ns} patch ingress %{ing} --type=json -p '[
   {"op": "replace", "path": "/spec/rules/0/http/paths/1/pathType", "value": "Prefix"}]'
 kubectl apply -f {WS}/q127/blog-ingress.yaml
 """,
-128: r"""
-kubectl -n %{ns} patch ds %{ds} --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/ports/0/hostPort", "value": %{hp}}]'
-kubectl -n %{ns} rollout status ds %{ds} --timeout=240s
-yq -i '.spec.dnsPolicy = "ClusterFirstWithHostNet"' {WS}/q128/collector.yaml
-kubectl -n %{ns} delete pod %{pod} --grace-period=1
-kubectl apply -f {WS}/q128/collector.yaml
-kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=180s
-kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt
-sleep 8
-""",
 # ---- 129-131: bank_tshoot.py
 129: r"""
 ssh -n {W1} 'sudo systemctl enable --now containerd; echo "runtime-endpoint: unix:///run/containerd/containerd.sock" | sudo tee /etc/crictl.yaml >/dev/null'
@@ -321,25 +100,10 @@ sudo mkdir -p /opt/cka
 sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
   --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/cka/etcd-snapshot.db
 """,
-5: r"""
-ssh -n {W3} 'sudo tee /etc/kubernetes/manifests/static-web.yaml >/dev/null <<EOF
-apiVersion: v1
-kind: Pod
-metadata: {name: static-web, namespace: default}
-spec:
-  containers: [{name: web, image: "nginx:1.27"}]
-EOF'
-for i in $(seq 1 60); do [ "$(kubectl get pod static-web-{W3} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
-""",
 17: r"""
 ssh -n {W2} 'sudo systemctl enable --now kubelet'
 for i in $(seq 1 40); do [ "$(kubectl get node {W2} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
 """,
-18: r"""
-kubectl -n troubleshoot set image deploy payments payments=nginx:1.27
-kubectl -n troubleshoot patch svc payments -p '{"spec":{"selector":{"app":"payments"},"ports":[{"port":80,"targetPort":80}]}}'
-kubectl -n troubleshoot rollout status deploy payments --timeout=240s
-""",
 20: CPWAIT + r"""
 sudo sed -i 's/kube-schedulerr/kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
 cpwait kube-scheduler
@@ -359,44 +123,6 @@ kubectl -n kube-system rollout status deploy coredns --timeout=240s
 sudo sed -i '/--leader-elect-lease-duration=1s/d' /etc/kubernetes/manifests/kube-controller-manager.yaml
 cpwait kube-controller-manager
 """,
-39: r"""
-ssh -n {W1} "sudo sed -i 's|ca-bundle.crt|ca.crt|' /var/lib/kubelet/config.yaml; sudo systemctl restart kubelet; sudo systemctl enable kubelet"
-for i in $(seq 1 40); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
-""",
-45: CPWAIT + r"""
-sudo etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore
-sudo sed -i 's|path: /var/lib/etcd$|path: /var/lib/etcd-restore|' /etc/kubernetes/manifests/etcd.yaml
-sleep 30
-for i in $(seq 1 80); do kubectl -n default get cm %{cm} >/dev/null 2>&1 && break; sleep 3; done
-for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
-""",
-46: r"""
-N=$(kubectl get nodes -l tier=edge -o jsonpath='{.items[0].metadata.name}')
-D=$(ssh -n $N "sudo awk '/^staticPodPath:/ {print \$2}' /var/lib/kubelet/config.yaml")
-kubectl create ns %{ns} 2>/dev/null
-ssh -n $N "sudo tee $D/%{pod}.yaml >/dev/null <<EOF
-apiVersion: v1
-kind: Pod
-metadata: {name: %{pod}, namespace: %{ns}, labels: {role: edge}}
-spec:
-  containers:
-    - name: web
-      image: nginx:1.27
-      resources: {requests: {cpu: 50m, memory: 32Mi}}
-EOF"
-for i in $(seq 1 60); do [ "$(kubectl -n %{ns} get pod %{pod}-$N -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
-""",
-51: r"""
-for r in $(kubectl -n %{ns} rollout history deploy %{dep} | awk '/^[0-9]/{print $1}'); do
-  kubectl -n %{ns} rollout history deploy %{dep} --revision=$r | grep -q 'nginx:1.25' && N=$r
-done
-kubectl -n %{ns} rollout undo deploy %{dep} --to-revision=$N
-kubectl -n %{ns} scale deploy %{dep} --replicas=3
-kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
-kubectl -n %{ns} rollout pause deploy %{dep}
-kubectl -n %{ns} set image deploy/%{dep} $(kubectl -n %{ns} get deploy %{dep} -o jsonpath='{.spec.template.spec.containers[0].name}')=nginx:1.27
-kubectl -n %{ns} set env deploy/%{dep} RELEASE=next
-""",
 57: r"""
 kubectl -n kube-system delete netpol maintenance-lockdown
 kubectl -n kube-system scale deploy coredns --replicas=2
@@ -404,11 +130,6 @@ kubectl -n kube-system get cm coredns -o json | jq '.data.Corefile |= sub("forwa
 kubectl -n kube-system rollout restart deploy coredns
 kubectl -n kube-system rollout status deploy coredns --timeout=240s
 """,
-60: r"""
-ssh -n {W2} "sudo sed -i 's|ca-wrong.crt|ca.crt|' /var/lib/kubelet/config.yaml; sudo systemctl enable --now kubelet"
-kubectl uncordon {W2}
-for i in $(seq 1 40); do [ "$(kubectl get node {W2} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
-""",
 61: CPWAIT + r"""
 sudo sed -i 's/kube-schedulerr/kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
 sudo sed -i 's|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)-broken|\1|' /etc/kubernetes/manifests/kube-controller-manager.yaml
@@ -421,17 +142,6 @@ for i in $(seq 1 80); do kubectl get nodes >/dev/null 2>&1 && break; sleep 3; do
 for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
 sudo kubeadm certs check-expiration 2>/dev/null | grep '^apiserver ' > {WS}/q65/expiry.txt
 """,
-66: CPWAIT + r"""
-sudo sed -i 's/kube-apiserverr/kube-apiserver/' /etc/kubernetes/manifests/kube-apiserver.yaml
-for i in $(seq 1 80); do kubectl get nodes >/dev/null 2>&1 && break; sleep 3; done
-for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
-""",
-67: r"""
-T=$(sudo kubeadm token create --ttl 2h)
-H=$(sudo cat /etc/kubernetes/pki/ca.crt | openssl x509 -pubkey | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')
-E=$(kubectl get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}')
-printf 'JOIN_TOKEN=%s\nCA_HASH=sha256:%s\nAPI_ENDPOINT=%s\n' "$T" "$H" "$E" > {WS}/q67/node-join.env
-""",
 70: r"""
 kubectl -n driver-onboard set image deploy/onboard-app onboard=nginx:1.25
 kubectl -n driver-onboard patch deploy onboard-app --type=json -p '[
@@ -439,15 +149,6 @@ kubectl -n driver-onboard patch deploy onboard-app --type=json -p '[
   {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/"}]'
 kubectl -n driver-onboard rollout status deploy onboard-app --timeout=240s
 """,
-71: r"""
-kubectl -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
-kubectl -n dispatch-hub rollout status deploy dispatch-api --timeout=240s
-for i in 1 2 3 4 5; do
-  kubectl -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt </dev/null 2>/dev/null
-  grep -qi nginx {WS}/q71/test-result.txt && break; kubectl -n dispatch-hub delete pod probe --ignore-not-found >/dev/null 2>&1; sleep 3
-done
-kubectl -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt
-""",
 76: r"""
 kubectl -n trip-tracker get deploy trip-logger -o json | jq '
   .spec.template.spec.volumes += [{"name": "config-volume", "emptyDir": {}}]
@@ -467,79 +168,23 @@ kubectl -n surge-pricing get deploy surge-engine -o json | jq '
   | .spec.template.spec.volumes += [{"name": "config-vol", "configMap": {"name": "surge-config"}}]' | kubectl apply -f -
 kubectl -n surge-pricing rollout status deploy surge-engine --timeout=240s
 """,
-81: r"""
-kubectl -n fare-engine delete pvc fare-data-pvc
-cat <<EOF | kubectl apply -f -
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata: {name: fare-data-pvc, namespace: fare-engine}
-spec:
-  accessModes: [ReadWriteOnce]
-  resources: {requests: {storage: 100Mi}}
-  storageClassName: standard
-EOF
-kubectl label node {W1} disk=nvme
-kubectl -n fare-engine rollout status deploy fare-calculator --timeout=240s
-""",
 84: r"""
 kubectl -n booking-tls get cm booking-nginx-config -o json | jq '.data |= with_entries(.value |= sub("ssl_protocols TLSv1.2 TLSv1.3;"; "ssl_protocols TLSv1.3;"))' | kubectl apply -f -
 kubectl -n booking-tls rollout restart deploy booking-nginx
 kubectl -n booking-tls rollout status deploy booking-nginx --timeout=240s
 echo 'ssl_protocols TLSv1.3;' > {WS}/q84/ssl-line.txt
 """,
-86: r"""
-cat > {WS}/q86/controlplane.txt <<'EOF'
-kubelet: process
-kube-apiserver: static-pod
-kube-scheduler: static-pod
-kube-controller-manager: static-pod
-etcd: static-pod
-dns: deployment coredns
-EOF
-""",
-87: r"""
-C=$(kubectl get nodes --no-headers -l node-role.kubernetes.io/control-plane | wc -l)
-W=$(kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' | wc -l)
-CIDR=$(kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -o 'service-cluster-ip-range=[^ ]*' | head -1 | cut -d= -f2)
-F=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
-printf '1: %s\n2: %s\n3: %s\n4: kindnet, %s\n5: -{W3}\n' "$C" "$W" "$CIDR" "$F" > {WS}/q87/cluster-info.txt
-""",
 88: r"""
 kubectl -n oslo-web run gatekeeper --image=busybox:1.36 --dry-run=client -o json -- sleep 86400 | jq '.spec.containers[0].readinessProbe = {"exec": {"command": ["sh", "-c", "wget -T2 -qO- http://ledger-api:8080"]}, "initialDelaySeconds": 5, "periodSeconds": 5}' | kubectl apply -f -
 sleep 12
 kubectl -n oslo-web expose deploy ledger --name=ledger-api --port=8080 --target-port=80
 kubectl -n oslo-web wait --for=condition=Ready pod/gatekeeper --timeout=180s
 """,
-90: r"""
-cat > {WS}/q90/cluster-events.sh <<'EOF'
-kubectl get events -A --sort-by=.metadata.creationTimestamp
-EOF
-chmod +x {WS}/q90/cluster-events.sh
-P=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].metadata.name}')
-kubectl -n kube-system delete pod $P
-for i in $(seq 1 40); do
-  [ "$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].status.phase}' 2>/dev/null)" = Running ] && break; sleep 3
-done
-sleep 5
-kubectl -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
-ssh -n {W2} 'sudo crictl stop $(sudo crictl ps --name kube-proxy -q)'
-sleep 25
-kubectl -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
-echo container > {WS}/q90/restarts.txt
-""",
 92: r"""
 ssh -n {W1} "sudo sed -i 's|^ExecStart=/usr/local/bin/kubelet |ExecStart=/usr/bin/kubelet |' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; sudo systemctl daemon-reload; sudo systemctl restart kubelet"
 echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q92/fixed-file.txt
 for i in $(seq 1 40); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
 """,
-94: r"""
-kubectl run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon --dry-run=client -o yaml \
-  | yq '.spec.containers[0].resources = {"requests": {"cpu": "10m", "memory": "20Mi"}}' | sudo tee /etc/kubernetes/manifests/ops-beacon.yaml >/dev/null
-for i in $(seq 1 60); do [ "$(kubectl -n default get pod ops-beacon-{CP} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
-kubectl -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
-kubectl -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt
-sleep 5
-""",
 95: r"""
 CI=$(ssh -n {W2} 'sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -issuer' | sed 's/^issuer= *//')
 SI=$(ssh -n {W2} 'sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -issuer' | sed 's/^issuer= *//')
@@ -587,47 +232,11 @@ kubectl -n %{ns} patch deploy %{dep} --type=json -p '[
   {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/%{ready}"}]'
 kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
 """,
-107: r"""
-ssh -n {W3} "printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/%{name}.conf >/dev/null
-printf 'net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\n' | sudo tee /etc/sysctl.d/%{name}.conf >/dev/null"
-IPF=$(ssh -n {W3} 'sudo sysctl -n net.ipv4.ip_forward')
-SW=off; [ "$(ssh -n {W3} 'sed 1d /proc/swaps | grep -c .')" -gt 0 ] && SW=on
-FS=$(ssh -n {W3} "sudo awk '/^failSwapOn:/ {print \$2}' /var/lib/kubelet/config.yaml"); [ -n "$FS" ] || FS=true
-CG=$(ssh -n {W3} "sudo awk '/^cgroupDriver:/ {print \$2}' /var/lib/kubelet/config.yaml")
-printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s\n' "$IPF" "$SW" "$FS" "$CG" > {WS}/q107/node.txt
-""",
-108: r"""
-V=$(ssh -n {W2} 'sudo crictl version 2>/dev/null')
-RN=$(echo "$V" | awk '/^RuntimeName:/ {print $2}'); RV=$(echo "$V" | awk '/^RuntimeVersion:/ {print $2}')
-SOCK=$(ssh -n {W2} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
-CID=$(kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#^[a-z-]*://##')
-PID=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.info.pid}}' $CID 2>/dev/null")
-LOG=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.status.logPath}}' $CID 2>/dev/null")
-printf 'runtime=%s %s\nsocket=%s\ncontainer-id=%s\npid=%s\nlog=%s\n' "$RN" "$RV" "$SOCK" "$CID" "$PID" "$LOG" > {WS}/q108/runtime.txt
-""",
 110: r"""
 kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/^controlPlaneEndpoint:/{print $2}' | tr -d '"' > {WS}/q110/endpoint.txt
 KEY=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
 sudo kubeadm token create --print-join-command --certificate-key $KEY 2>/dev/null | grep 'kubeadm join' > {WS}/q110/join-control-plane.sh
 """,
-111: r"""
-E='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
-N=$(kubectl get nodes -l node-role.kubernetes.io/control-plane --no-headers | wc -l)
-M=$($E member list | wc -l)
-L=$($E member list | awk -F', ' 'NR==1 {print $3}')
-S=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
-C=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
-printf 'control-plane-nodes=%s\netcd-topology=stacked\netcd-members=%s\netcd-leader=%s\nscheduler-leader=%s\ncontroller-manager-leader=%s\n' "$N" "$M" "$L" "$S" "$C" > {WS}/q111/ha.txt
-""",
-112: r"""
-CONF=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
-PL=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[].type] | join(",")')
-IPAM=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[] | select(.ipam) | .ipam.type][0]')
-SOCK=$(ssh -n {W1} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
-N=$(kubectl get csidrivers --no-headers 2>/dev/null | wc -l)
-PROV=$(kubectl get sc -o json | jq -r '.items[] | select(.metadata.annotations["storageclass.kubernetes.io/is-default-class"] == "true") | .provisioner')
-printf 'cni-config=%s\ncni-plugins=%s\ncni-ipam=%s\ncri-socket=%s\ncsi-drivers=%s\ndefault-provisioner=%s\n' "$CONF" "$PL" "$IPAM" "$SOCK" "$N" "$PROV" > {WS}/q112/interfaces.txt
-""",
 19: r"""
 for i in $(seq 1 40); do kubectl -n troubleshoot logs crasher --previous 2>/dev/null | grep -q DB_HOST && break; sleep 3; done
 kubectl -n troubleshoot logs crasher --previous > {WS}/q19/crasher.log
@@ -644,71 +253,6 @@ KV=$(kubectl get node %{node} -o jsonpath='{.status.nodeInfo.kubeletVersion}')
 R=$(kubectl get node %{node} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}')
 printf 'apiserver=%s\netcd=%s\nkubelet=%s\nruntime=%s\n' "$A" "$E" "$KV" "$R" > {WS}/q41/versions.txt
 """,
-59: r"""
-kubectl -n %{ns} apply -f - <<EOF
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata: {name: new-data, namespace: %{ns}}
-spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 1Gi}}}
----
-apiVersion: batch/v1
-kind: Job
-metadata: {name: migrate, namespace: %{ns}}
-spec:
-  template:
-    spec:
-      restartPolicy: Never
-      volumes:
-        - {name: src, persistentVolumeClaim: {claimName: old-data}}
-        - {name: dst, persistentVolumeClaim: {claimName: new-data}}
-      containers:
-        - name: cp
-          image: busybox:1.36
-          command: [sh, -c, "cp -a /src/. /dst/ && ls /dst"]
-          volumeMounts: [{name: src, mountPath: /src}, {name: dst, mountPath: /dst}]
-EOF
-kubectl -n %{ns} wait --for=condition=complete job/migrate --timeout=240s
-kubectl -n %{ns} get pod legacy -o json | jq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .status)
-  | (.spec.volumes[] | select(.persistentVolumeClaim.claimName == "old-data") | .persistentVolumeClaim.claimName) = "new-data"' > /tmp/legacy.json
-kubectl -n %{ns} delete pod legacy --now
-kubectl apply -f /tmp/legacy.json
-kubectl -n %{ns} delete pod -l job-name=migrate
-kubectl -n %{ns} delete pvc old-data
-kubectl -n %{ns} wait --for=condition=Ready pod/legacy --timeout=180s
-""",
-77: r"""
-kubectl -n kube-system get ds -o name | sed 's|.*/||' | grep -i kindnet > {WS}/q77/cni.txt
-kubectl -n zero-trust run srv --image=nginx:1.27 --labels=app=srv
-kubectl -n zero-trust expose pod srv --port 80
-kubectl -n zero-trust run cli --image=busybox:1.36 -- sleep 3600
-kubectl -n zero-trust wait --for=condition=Ready pod/srv pod/cli --timeout=180s
-kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null && echo reachable
-cat <<EOF | kubectl apply -f -
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata: {name: default-deny-all, namespace: zero-trust}
-spec:
-  podSelector: {}
-  policyTypes: [Ingress, Egress]
-EOF
-sleep 6
-if kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null 2>&1; then echo no; else echo yes; fi > {WS}/q77/enforced.txt
-kubectl -n zero-trust delete pod srv cli --now
-kubectl -n zero-trust delete svc srv
-""",
-91: r"""
-kubectl api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q91/namespaced.txt
-for ns in census-north census-south census-east census-west; do echo "$ns $(kubectl -n $ns get role --no-headers | wc -l)"; done | sort -k2 -n | tail -1 > {WS}/q91/most-roles.txt
-""",
-102: r"""
-kubectl -n oslo-files get pvc shared-files -o json | jq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .metadata.annotations, .metadata.finalizers, .status)
-  | .spec.accessModes = ["ReadWriteOnce"]' > /tmp/pvc.json
-kubectl -n oslo-files delete pvc shared-files
-kubectl apply -f /tmp/pvc.json
-kubectl -n oslo-files rollout restart deploy uploader
-kubectl -n oslo-files rollout status deploy uploader --timeout=240s
-echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
-""",
 93: r"""
 kubectl -n oslo-secure create secret generic bundle --from-file={WS}/q93/tls-bundle.pem
 kubectl -n oslo-secure create secret generic db-login --from-literal=user=svc-ledger --from-literal=pass='%{pw}'
@@ -719,48 +263,14 @@ kubectl -n oslo-secure run vault-reader --image=busybox:1.36 --dry-run=client -o
                                {"name": "APP_PASS", "valueFrom": {"secretKeyRef": {"name": "db-login", "key": "pass"}}}]' | kubectl apply -f -
 kubectl -n oslo-secure wait --for=condition=Ready pod/vault-reader --timeout=180s
 """,
-# ---- 132-145: bank_course.py (the others are solved by their solution as written, below)
-132: r"""
-apt_get() { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq -y -o Dpkg::Options::=--force-confold "$@" </dev/null >/dev/null; }
-sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
-apt_get update
-sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null
-sudo kubeadm upgrade apply v1.37.1 -y </dev/null | tail -2
-kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
-sudo apt-mark unhold kubelet kubectl >/dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null
-sudo systemctl daemon-reload && sudo systemctl restart kubelet
-for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done
-kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data
-ssh {W1} sudo bash <<'EOF'
-export DEBIAN_FRONTEND=noninteractive
-sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
-apt-get update -qq </dev/null
-apt-mark unhold kubeadm kubelet kubectl >/dev/null
-apt-get install -y -qq kubeadm=1.37.1-1.1 </dev/null >/dev/null
-kubeadm upgrade node </dev/null | tail -1
-apt-get install -y -qq kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 </dev/null >/dev/null
-apt-mark hold kubeadm kubelet kubectl >/dev/null
-systemctl daemon-reload && systemctl restart kubelet
-EOF
-kubectl uncordon {W1}
-""",
-139: r"""
-ssh {W1} sudo bash <<'EOF'
-yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
-systemctl restart kubelet
-EOF
-for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
-""",
 }
 
 # ---- 1-113: the banks carried over from cka-sim.
 # For these tasks the reference solution is a script as it stands, so the audit runs exactly the
 # text the candidate is shown - and a solution that stops working is found the next time it runs.
-AS_WRITTEN = [1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
+AS_WRITTEN = [1, 21, 40, 55, 69, 75, 85, 98, 104, 109,
-              40, 42, 43, 44, 47, 48, 49, 50, 52, 53, 54, 55, 56, 58, 62, 63, 64, 68, 69, 72, 73, 74, 75, 78, 80, 82, 83,
-              85, 89, 98, 99, 100, 101, 103, 104, 109, 113,
               # bank_course.py: written to run as they stand
-              133, 134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 145, 146, 147]
+              133]
 
 
 def _as_written():

ksim

@@ -22,7 +22,7 @@ from collections.abc import Callable
 from datetime import datetime
 from typing import Any
 
-VERSION = "2026.10.26.4"
+VERSION = "2026.10.27.1"
 ROOT = os.path.dirname(os.path.abspath(__file__))
 sys.path.insert(0, ROOT)
 from sim import desktop, envs  # noqa: E402
@@ -96,7 +96,11 @@ def load_track(track_id):
         q.setdefault("set", None)
         if q["id"] in workers:
             q["workers"] = workers[q["id"]]
+        if isinstance(q.get("solution"), list):   # a solution in parts: the audit runs what it shows
             if q["id"] in solve:
+                sys.exit(f"{track_id} task {q['id']}: its solution is in parts, which the audit runs - drop its script from {track_id}/solve.py")
+            q["solve"] = show_script({"show": q["solution"]})
+        elif q["id"] in solve:
             q["solve"] = solve[q["id"]]
         # Every task has a cluster to itself, so nothing it breaks can reach another task: the flags
         # that kept such tasks apart on a shared cluster ("breaks", "solo", "disruptive") are void.
@@ -124,7 +128,7 @@ def lesson_task(l, objectives):
     o = next((o for o in objectives if o["id"] == l["objective"]), None)
     return dict(l, lesson=True, domain=o["domain"] if o else "", difficulty="normal", set=None,
                 checks=[{"name": s["title"], "pts": 1, "script": s["check"]} for s in l["steps"]], weight=len(l["steps"]),
-                task=l["intro"], solution="\n\n".join(show_md(s) for s in l["steps"] if show_parts(s)))
+                task=l["intro"], solution=[p for s in l["steps"] for p in show_parts(s)])
 
 
 def show_parts(s):
@@ -143,6 +147,11 @@ def show_script(s):
     """What the audit runs for a step: what you type, and for each part done by hand its script."""
     return "\n".join(p["run"] for p in show_parts(s))
 
+def solution_parts(q):
+    """A task's reference solution as parts, like a lesson step's: a solution in parts (cmd, hand)
+    as it stands, one written as a single script as one part."""
+    return show_parts({"show": q.get("solution") or ""})
+
 
 def set_questions(name):
     return [q for q in QUESTIONS if q.get("set") == name]
@@ -455,7 +464,9 @@ def task_text(q, params=None):
     return prose(q["task"], params if params is not None else session_params(load_state(), q), q)
 
 def solution_text(q, params=None):
-    return prose(str(q.get("solution", "")), params if params is not None else session_params(load_state(), q), q)
+    """A task's reference solution as you read it (Markdown): each thing to type in a block of its
+    own, and beneath it what to do there by hand."""
+    return prose(show_md({"show": solution_parts(q)}), params if params is not None else session_params(load_state(), q), q)
 
 def make_params(q, rng):
     fn = q.get("params")
@@ -765,12 +776,15 @@ def solution_allowed(st: dict[str, Any] | None, qid: int | str) -> bool:
 def cmd_solution(args):
     if not args:
         die("usage: ./ksim solution N   (drills: after grading the task; exams: after submit; --force to peek anyway)")
+    if args[0] == "lint":   # no cluster needed, and no solution shown: only where one has you type a script
+        names = [a.upper() for a in args[1:]]
+        return print_solution_lint(solution_lint([q for q in QUESTIONS if not names or str(q["id"]) in names or q["domain"] in names]))
     force = "--force" in args; args = [a for a in args if a != "--force"]
     q = q_by_id(int(args[0])); st = load_state()
     if not solution_allowed(st, int(q["id"])) and not force:
         die("Not yet: in a drill, grade the task first (./ksim grade N); in an exam, submit first. --force overrides.")
     print(bold(f"Solution - Q{q['id']} {q['title']}")); print("-" * 64)
-    print(solution_text(q, session_params(st, q))); print("-" * 64)
+    print(cli_md(solution_text(q, session_params(st, q)))); print("-" * 64)
     print(dim(f"One correct way, not the only one. $do = --dry-run=client -o yaml, k = kubectl. Run it on the task's host: ssh {host_of(q)}"))
 
 def cmd_progress(args):
@@ -1897,6 +1911,19 @@ def print_lint(problems):
     print(bold(f"{steps} step(s) ask you to type what you would not type in the exam.") if steps
           else green("Nothing a step asks you to type is a script: every step is done as in the exam."))
 
+def solution_lint(qs):
+    """What each task's reference solution has you type, its comments aside, against TYPING_TRAPS.
+    Returns [(task, why, line)]."""
+    return [(q, why, line.strip()) for q in qs for p in solution_parts(q) for line in p["type"].splitlines()
+            if not line.lstrip().startswith("#") for rx, why in TYPING_TRAPS if rx.search(line)]
+
+def print_solution_lint(problems):
+    for q, why, line in problems:
+        print(f"  {yellow('Q' + str(q['id']))}  {why}\n      {dim(line[:150])}")
+    tasks = len({q["id"] for q, _, _ in problems})
+    print(bold(f"{tasks} solution(s) ask you to type what you would not type in the exam.") if tasks
+          else green("Nothing a solution asks you to type is a script: every one is done as in the exam."))
+
 def lesson_audit(args):
     """Prove lessons on real clusters: each is seeded on a host of its own; then every step must
     fail its check, have its commands run as `candidate`, and pass its check - so a lesson can be
@@ -2788,6 +2815,7 @@ def cmd_help(args):
                            whatever else you named: ./ksim next OBS:3
   progress                attempted/passed per domain, normal and hard
   solution N               reference solution (drills: after grading N; exams: after submit)
+  solution lint [N|DOMAIN..]   what the solutions have you type: no scripts, only what you would type in the exam
   sim {'|'.join(SETS) or '<set>'}{' ' * max(1, 20 - len('|'.join(SETS)))}fixed simulator set, exam rules, {TRACK['minutes']} min
   start [--hard]           full exam: {sum(QUOTA.values()) or '?'} tasks sampled by domain ({sum(QUOTA_HARD.values()) or '?'} in hard mode), {TRACK['minutes']} min.
                            Requires every domain drilled once first (--force skips). Hard mode adds traps,

web/index.html

@@ -1030,7 +1030,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
       <div class="qbody">${render(q.task)}</div>
       ${on && sess.mode === 'drill' ? '<div class="checkrow"><button class="btn btn-outline-primary btn-sm" id="btnCheck">Check this task</button><span class="muted">grades it per check; its reference solution opens once it is graded</span></div>' : ''}
       ${show ? `<div class="checks"><b>${r.earned}/${r.total} points</b>${r.checks.map(c => `<div class="${c.pass ? 'ok' : 'no'}">${esc(c.name)} <span class="pts">(${c.pts})</span></div>`).join('')}</div>` : ''}
-      ${q.solution ? `<details class="fold"><summary>Reference solution</summary><pre>${esc(q.solution)}</pre></details>` : ''}
+      ${q.solution ? `<details class="fold"><summary>Reference solution</summary><div class="qbody" style="margin-top:10px">${render(q.solution)}</div></details>` : ''}
       ${obj ? `<details class="fold"><summary>Objective ${obj.id} · ${esc(obj.text)}</summary><div class="notes">${notesHtml(obj)}</div></details>` : ''}`;
     copyable($('#qContent'));
     wireRebuild();
@@ -1316,8 +1316,9 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
       const cls = r.pass ? 't-pass' : (r.earned ? 't-partial' : 't-fail');
       return `<div class="qcard ${cls}"><div class="qcard-h"><h3>Question ${i + 1} · Q${q.id} ${esc(q.title)}</h3><span class="qs">${r.earned} of ${r.total}</span></div>
         <ul class="vitems">${r.checks.map(c => `<li class="vitem"><span>${esc(c.name)} <span class="muted">(${c.pts})</span></span><span class="vstat ${c.pass ? 'ok' : 'no'}">${c.pass ? '✓' : '✗'}</span></li>`).join('')}</ul>
-        ${q.solution ? `<details class="fold"><summary>Reference solution</summary><pre>${esc(q.solution)}</pre></details>` : '<div class="pad"></div>'}</div>`;
+        ${q.solution ? `<details class="fold"><summary>Reference solution</summary><div class="qbody" style="margin-top:10px">${render(q.solution)}</div></details>` : '<div class="pad"></div>'}</div>`;
     }).join('');
+    copyable($('#resQuestions'));
     const missed = S.questions.filter(q => s.results[q.id] && !s.results[q.id].pass).map(q => q.id);
     $('#btnDrillMissed').disabled = !missed.length;
     $('#btnDrillMissed').onclick = () => start('drill', missed);
@@ -1325,8 +1326,9 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
   function showAnswers() {
     const s = S.session;
     $('#answersBody').innerHTML = S.questions.filter(q => q.solution).map((q, i) =>
-      `<h4>Question ${S.questions.indexOf(q) + 1} · Q${q.id} ${esc(q.title)}${s && s.results[q.id] ? ` <span class="muted">(${s.results[q.id].earned}/${s.results[q.id].total})</span>` : ''}</h4><pre>${esc(q.solution)}</pre>`).join('')
+      `<h4>Question ${S.questions.indexOf(q) + 1} · Q${q.id} ${esc(q.title)}${s && s.results[q.id] ? ` <span class="muted">(${s.results[q.id].earned}/${s.results[q.id].total})</span>` : ''}</h4><div class="qbody">${render(q.solution)}</div>`).join('')
       || '<p class="muted">No reference solutions are open yet.</p>';
+    copyable($('#answersBody'));
     openModal('#mAnswers');
   }