fix: repair the CKA bank and harness after a live audit; add Simulator Set D

Every changed task was seeded, graded untouched, solved and re-graded on a
separate kind cluster. The four sim sets were rehearsed as full sittings (real
baseline, seed everything, solve in order, grade at the end): A 150/150,
B 176/176, C 163/163, D 122/122. Every random-pool task was either passed in a
real session or cycled the same way.

Graders that could never pass, or passed for free:
  Q306 (local-path cannot resize; now graded as an in-place resize), Q314 (GNU
  grep prefixed file names), Q305 (plain `get events` has no SOURCE column and
  events expire; now managedFields), Q410 (kindnet DOES enforce NetworkPolicy;
  the expected answer is now derived from the CNI), Q35, Q212 (missing `]`),
  Q101 (`can-i create pods/exec` asks about a pod NAMED exec; multi-type get),
  Q109 (an emptyDir over nginx's /var/log crash-loops it), Q40 (TTL was
  measured at grading time), Q121, Q106, Q104, Q41, Q26, Q403, Q409, Q417,
  Q113, Q22, Q24/Q108 (flaky right after a rollout), and free points on
  untouched Q24/Q102/Q108/Q119/Q122.

Reference solutions: S[43], S[107], S[121], S[106] rewritten; S[116] (a
Completed Job pod still pins its PVC), S[119] (a pods quota equal to replicas
deadlocks the rollout); DNS checks use FQDNs; the explainer and exam guide
are corrected (etcd restore is etcdutl's job on 3.6+).

Cross-session leaks: the pre-session baseline now also repairs Q114 (the
kube-system NetworkPolicy and CoreDNS replicas), Q117/Q39 kubelet CA paths,
Q317, kubelet drop-in binary paths, a moved scheduler manifest, leftover
static pods and the default StorageClass. Q115 releases its fixed PVs from
claims left by earlier sessions.

Conflicts: `breaks` may be a list and `solo` keeps Q102's etcd restore out of
mixed sessions. Q118, Q5/Q103, Q26 vs the W1 taints, set A 203/204/208/212 and
set B 305/309 no longer sabotage each other. Q317 now disables only CSR
signing (the old break killed the whole controller-manager). Q110/Q311 no
longer compare against the whole cluster at submit time. Two delete/recreate
races in setups are fixed.

Harness: the weak list clears after two clean passes in a row, weak entries
show the latest attempt's failures, domain scores use the exam's own tasks,
history entries can be marked synthetic (entry 11 is), the web UI lists
every sim set, `./exam grade` refuses mid-exam (--force), preflight checks W1.

Set D (`./exam sim d`): 17 original killer.sh-style tasks, ids 501-517,
122 points - control-plane inventory, event forensics with crictl, a broken
kubelet unit, kubelet certificates, manual scheduling with the scheduler
stopped, Helm housekeeping with a stuck release, an enforced AND-vs-OR
NetworkPolicy, dnsPolicy and access-mode troubleshooting, and more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CKA_EXAM_GUIDE.md

@@ -51,7 +51,7 @@ EOF
 | **Troubleshoot Node (Kubelet)** | `systemctl status kubelet` <br> `journalctl -u kubelet -e --no-pager` | Check `/etc/kubernetes/kubelet.conf` & `/var/lib/kubelet/config.yaml` |
 | **Control Plane Troubleshooting** | Inspect `/etc/kubernetes/manifests/` | Check `kube-apiserver.yaml`, `kube-scheduler.yaml`, `kube-controller-manager.yaml` |
 | **Etcd Backup** | `etcdctl snapshot save <file>` | `--endpoints=https://127.0.0.1:2379 --cacert=... --cert=... --key=...` |
-| **Etcd Restore** | `etcdctl snapshot restore <file>` | `--data-dir=/var/lib/etcd-backup` (update `etcd.yaml` volume hostPath) |
+| **Etcd Restore** | `etcdutl snapshot restore <file>` | `--data-dir=/var/lib/etcd-backup` (update `etcd.yaml` volume hostPath). `etcdctl snapshot restore` is gone from etcd 3.6+ |
 | **Upgrade Node (Control Plane / Worker)** | `apt-get update && apt-get install -y --allow-change-held-packages kubeadm=1.31.x-1.1` | `kubeadm upgrade plan` -> `kubeadm upgrade apply` -> upgrade `kubelet`/`kubectl` -> `systemctl restart kubelet` |
 
 ---
@@ -98,12 +98,16 @@ ETCDCTL_API=3 etcdctl \
 
 ### Restore:
 ```bash
-ETCDCTL_API=3 etcdctl \
+etcdutl snapshot restore /opt/snapshot-pre-boot.db \
-  --data-dir=/var/lib/etcd-from-backup \
+  --data-dir=/var/lib/etcd-from-backup
-  snapshot restore /opt/snapshot-pre-boot.db
 ```
 Then edit `/etc/kubernetes/manifests/etcd.yaml` to update the hostPath volume for `etcd-data` to `/var/lib/etcd-from-backup`.
 
+Restore is an offline file operation, so it needs no endpoints or certificates - and it lives in
+`etcdutl`, not `etcdctl`: `etcdctl snapshot restore` was deprecated in etcd 3.5 and is gone from
+3.6+ (this cluster runs 3.7, where it fails with `unknown flag: --data-dir`). Check first with
+`etcdctl version`; on a 3.5 node the old `ETCDCTL_API=3 etcdctl snapshot restore` still works.
+
 ---
 
 ## 5. Official Kubernetes Documentation URLs Allowed in CKA

CKA_SOLUTIONS_EXPLAINED.md

@@ -830,7 +830,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
               - labelSelector: {matchLabels: {app: %{dep}}}
                 topologyKey: kubernetes.io/hostname
         tolerations:
-          - {key: %{key}, operator: {Equal, value: "%{val}", effect: NoSchedule}}
+          - {key: %{key}, operator: Equal, value: "%{val}", effect: NoSchedule}
         containers:
           - name: app
             image: nginx:1.27
@@ -838,6 +838,8 @@ The exam-day skill this table is really pointing at, regardless of which platfor
   ```
   ```bash
   k create priorityclass %{dep}-prio --value=50000
+  # the seeded quota (pods: 2, requests.cpu: 250m) cannot hold 3 x 100m - raise BOTH, keep the quota
+  k -n %{ns} patch quota %{ns}-quota -p '{"spec":{"hard":{"pods":"3","requests.cpu":"300m"}}}'
   ```
 - **Gotchas**: with exactly 3 worker nodes and `replicas: 3` under required anti-affinity, there is zero slack — if the quota's `pods` cap is lower than 3, or CPU requests don't fit the quota's `requests.cpu` budget, pods stay `Pending` with no scheduling error beyond an event; check `kubectl -n %{ns} get quota` and size requests accordingly rather than raising replicas blindly.
 
@@ -874,12 +876,12 @@ The exam-day skill this table is really pointing at, regardless of which platfor
       - name: agent
         image: busybox:1.36
         restartPolicy: Always
-        command: [sh, -c, "while true; do date >> /var/log/agent.log; sleep 1; done"]
+        command: [sh, -c, "while true; do date >> /var/log/agent/agent.log; sleep 1; done"]
-        volumeMounts: [{name: logs, mountPath: /var/log}]
+        volumeMounts: [{name: logs, mountPath: /var/log/agent}]
     containers:
       - name: app
         image: nginx:1.27
-        volumeMounts: [{name: logs, mountPath: /var/log}]
+        volumeMounts: [{name: logs, mountPath: /var/log/agent}]   # NOT /var/log - see Gotchas
         readinessProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5}
         livenessProbe: {tcpSocket: {port: 80}}
     volumes: [{name: logs, emptyDir: {}}]
@@ -889,7 +891,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
     | tr ' ' '\n' | grep . | sort > {WS}/q109/containers.txt
   k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' | tr -d '\n' > {WS}/q109/ip.txt
   ```
-- **Gotchas**: `restartPolicy: Always` goes **inside the initContainer entry itself**, not at the pod level — that per-container field is exactly what flips a normal init container into a native sidecar. The report file must list container names sorted, one per line, no header — matching the check's own `sort` byte-for-byte.
+- **Gotchas**: never mount an emptyDir over `/var/log` in an nginx container — it hides nginx's own `/var/log/nginx` and nginx dies with `could not open error log file`, crash-looping forever. A volume mount **replaces** the directory's contents; mount a subdirectory instead. `restartPolicy: Always` goes **inside the initContainer entry itself**, not at the pod level — that per-container field is exactly what flips a normal init container into a native sidecar. The report file must list container names sorted, one per line, no header — matching the check's own `sort` byte-for-byte.
 
 ---
 
@@ -1093,7 +1095,8 @@ The exam-day skill this table is really pointing at, regardless of which platfor
     externalName: %{ext}
   ```
   ```bash
-  k -n %{ns} exec %{sts}-0 -- nslookup %{sts}-0.%{sts} > {WS}/q43/pod0.txt
+  # full name: busybox's nslookup applies no search domains, so the short form returns NXDOMAIN
+  k -n %{ns} exec %{sts}-0 -- nslookup %{sts}-0.%{sts}.%{ns}.svc.cluster.local > {WS}/q43/pod0.txt
   ```
 - **Gotchas**: the headless Service's name **must match** `spec.serviceName` on the StatefulSet — that field is what wires the StatefulSet's per-pod DNS into that specific headless Service; a mismatched name means the stable pod hostnames never resolve, even though both objects individually look fine. `type: ExternalName` Services never get a `ClusterIP` or endpoints at all — that's expected, not a misconfiguration.
 
@@ -1225,7 +1228,8 @@ The exam-day skill this table is really pointing at, regardless of which platfor
   # forward . 10.255.255.254  ->  forward . /etc/resolv.conf
 
   k -n kube-system rollout restart deploy coredns
-  k run dns-check --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
+  # FULL names: busybox's nslookup applies no search domains - `kubernetes.default` fails even when DNS works
+  k run dns-check --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default.svc.cluster.local
   ```
 - **Gotchas**: fixing the Corefile alone looks successful (`kubectl edit` returns cleanly) but does nothing while replicas are still at 0 and/or a NetworkPolicy blocks ingress to the pods — always verify the actual symptom (`nslookup` from a real pod) rather than trusting that each individual fix "should" have worked. `rollout restart` is necessary after editing the ConfigMap because CoreDNS doesn't hot-reload `Corefile` changes on its own.
 
@@ -1392,9 +1396,11 @@ The exam-day skill this table is really pointing at, regardless of which platfor
   # re-point legacy: swap claimName old-data -> new-data, then recreate the pod
   k -n %{ns} delete pod legacy --now
   k -n %{ns} apply -f legacy.yaml
+  # the Job's Completed pod still references old-data and pins it - delete that pod first
+  k -n %{ns} delete pod -l job-name=migrate
   k -n %{ns} delete pvc old-data
   ```
-- **Gotchas**: `cp -a /src/. /dst/` (trailing `/.`) copies the **contents** of `/src` into `/dst`, preserving permissions/timestamps — dropping the trailing dot instead creates a nested `/dst/src/` directory, which silently fails any check expecting files directly under the new mount. Repointing `legacy`'s claim requires deleting and recreating the pod (`volumes[].persistentVolumeClaim.claimName` isn't mutable in place) — but the task requires the *Deployment/PVC* naming to stay the same, not the pod's UID, so this is safe here.
+- **Gotchas**: a pod that references a claim holds it in `pvc-protection` even after it has **Completed** — `delete pvc old-data` hangs in Terminating until the Job's pod is gone (`k describe pvc old-data` names it under "Used By"). Deleting the pod leaves the Job object and its `succeeded: 1` intact. `cp -a /src/. /dst/` (trailing `/.`) copies the **contents** of `/src` into `/dst`, preserving permissions/timestamps — dropping the trailing dot instead creates a nested `/dst/src/` directory, which silently fails any check expecting files directly under the new mount. Repointing `legacy`'s claim requires deleting and recreating the pod (`volumes[].persistentVolumeClaim.claimName` isn't mutable in place) — but the task requires the *Deployment/PVC* naming to stay the same, not the pod's UID, so this is safe here.
 
 ---
 
@@ -1484,7 +1490,8 @@ The exam-day skill this table is really pointing at, regardless of which platfor
   # forward . 10.255.255.1  ->  forward . /etc/resolv.conf
 
   k -n kube-system rollout restart deploy coredns
-  k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
+  # FULL names: busybox's nslookup applies no search domains - `kubernetes.default` fails even when DNS works
+  k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default.svc.cluster.local
   ```
 - **Gotchas**: `forward . /etc/resolv.conf` tells CoreDNS to use the **node's own** resolver config for anything outside the cluster zone — that's the correct, portable default; pointing `forward` at a hardcoded IP address is exactly the kind of change that breaks the moment that IP stops being reachable, which is the fault being simulated here.
 
@@ -1579,7 +1586,9 @@ The exam-day skill this table is really pointing at, regardless of which platfor
   k -n %{ns} describe deploy %{dep}
   k -n %{ns} describe rs -l app=%{dep}          # "exceeded quota: pods, requested: pods=1"
 
-  k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"3"}}}'
+  # 4, not 3: the rolling update needs a surge pod, and with maxUnavailable 0 (25% of 3 rounds
+  # down) the controller will not remove an old pod first - pods=3 deadlocks the rollout
+  k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"4"}}}'
   k -n %{ns} set image deploy %{dep} web=nginx:1.27
 
   k -n %{ns} patch secret %{dep}-env -p '{"stringData":{"DATABASE_URL":"postgres://db"}}'

README.md

@@ -4,8 +4,9 @@ Entry point is `./exam` (renamed from `cka-sim` so it can't collide with the dir
 
 Hands-on exam tasks graded **per check** against a real cluster, an exam clock, a browser
 front end modelled on the real PSI exam, attempt history, and drills that rotate through the
-domains before you sit a full exam. Tracks: **CKA** (65-task bank: 43 normal + 22 hard),
+domains before you sit a full exam. Tracks: **CKA** (136 tasks: a random pool of 43 normal +
-CKAD and CKS (scaffolded — same format, add questions when you get there).
+24 hard, plus four fixed simulator sets A–D of 17–18 tasks each), CKAD and CKS (scaffolded — same
+format, add questions when you get there).
 
 ## How a session is built
 
@@ -65,8 +66,10 @@ The same state is available from the terminal if you prefer:
 
 ```
 ./exam drill                      # next least-drilled domain, 5 tasks (add --hard)
-./exam drill weak | drill 2 17 20 | drill TSHOOT NET
+./exam drill weak | drill 2 17 20 | drill TSHOOT NET | drill setd
 ./exam start [--hard]             # full exam (gated until all domains drilled; --force)
+./exam sim a|b|c|d                # a fixed 17/18-task simulator set, exam rules, 120 min
+./exam study [DOMAIN|weak|N..]    # no clock, solutions visible, nothing recorded
 ./exam list | show N | timer | status                       # during
 ./exam end                                                  # submit
 ./exam weak | history                                       # afterwards
@@ -82,10 +85,12 @@ exam. Every submit appends to `history-<track>.jsonl` and regenerates `weak-spot
 latest domain scores, a "drill next" section listing the *failing checks* per task, and a
 ✓ / n-of-m matrix across all attempts.
 
-A task counts as weak if it failed most recently, has failed twice ever, or was never
+A task counts as weak if it failed on its latest attempt, or has failed at least twice and has
-attempted. Drills seed only those tasks, give 6 minutes per task, and — unlike exam mode —
+not yet been passed twice in a row since, or was never attempted (fixed-set tasks excepted — note
-let you check a task while you work. A task stops being weak only when you pass it again
+that this makes an early `drill weak` very large; name tasks explicitly for a short drill). Drills
-under the clock.
+give 6 minutes per task and — unlike exam mode — let you check a task while you work. History
+entries marked `"synthetic": true` (scripted verification runs, not a person) are kept in the file
+but left out of every statistic.
 
 ## Using your own cluster instead of kind
 
@@ -111,7 +116,7 @@ fails loudly if it cannot. `./etcd-tools.sh` does the same by hand.
 
 ## CKA bank
 
-| Domain | Normal (43) | Hard (22) |
+| Domain | Normal (43) | Hard (24) |
 |---|---|---|
 | Architecture | RBAC, etcd snapshot, Kustomize, CRD, static pod, certs/CSR, Helm, aggregated ClusterRole, `kubectl explain` on a CRD, kubeadm join token, component versions | exact RBAC with an over-grant to remove, **etcd restore**, static pod on a relocated path, user → self-contained kubeconfig, Kustomize generators + patches, CRD validation/defaults, renew all control-plane certs |
 | Workloads | rolling update, sidecar, node affinity + resources, ConfigMap/Secret, metrics-server repair + HPA, PriorityClass, taints, DaemonSet, CronJob, init container, immutable ConfigMap rotation, parallel Job, quota + LimitRange | anti-affinity + toleration + priority under a quota, rollback to a specific revision then pause, native sidecar + probes + exact report, cluster audit to exact files, work in the context's namespace |
@@ -120,12 +125,24 @@ fails loudly if it cannot. `./etcd-tools.sh` does the same by hand.
 | Troubleshooting | node NotReady, broken deployment + service, CrashLoopBackOff, scheduler down, CoreDNS misconfig, controller-manager down, kubelet config drift | worker with three faults (+ uncordon check), two control-plane components down, app down for four reasons fixed in place (UID check), cluster facts exactly |
 
 Disruptive tasks (anything that breaks a node or control-plane component) are seeded last and
-can block other tasks until fixed — read everything first, as in the real exam. kindnet doesn't
+can block other tasks until fixed — read everything first, as in the real exam. Tasks that damage
-enforce NetworkPolicy and no ingress/gateway controller runs, so those are graded on the objects.
+or depend on the same shared state declare it in `breaks` and never share a session, and the etcd
+restore (Q102) is `solo`: restoring rolls the whole cluster back, so it only runs on its own.
+Before every session a baseline pass repairs anything a previous, unfinished session left broken.
+
+The kindnet CNI here **enforces** NetworkPolicy (it embeds a policy engine), so a wrong policy
+really drops traffic; no ingress or Gateway controller runs, so those are graded on the objects.
+
+**Simulator sets** (`./exam sim a|b|c|d`, never drawn into random sessions): A and B are
+killer.sh-grade investigate-and-write-to-file sets, C is modelled on an 18-question commercial
+mock sitting, and D is killer.sh-style — control-plane inventory, event forensics with crictl,
+a broken kubelet unit, kubelet certificates, manual scheduling with the scheduler stopped, Helm
+housekeeping with a stuck release, and an enforced AND-vs-OR NetworkPolicy. All are original
+scenarios, not copies of any commercial simulator.
 
 ## Adding tasks or a new track
 
-Banks live in `tracks/cka/{questions,bank_extra,bank_hard}.py`; a task is:
+Banks live in `tracks/cka/{questions,bank_extra,bank_hard,bank_sim_*}.py`; a task is:
 
 ```python
 {"id": 44, "domain": "NET", "difficulty": "normal",   # or "hard"
@@ -133,7 +150,9 @@ Banks live in `tracks/cka/{questions,bank_extra,bank_hard}.py`; a task is:
  "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
  "setup": "bash that seeds the scenario (may use %{ns})",
  "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
- "disruptive": False}
+ "disruptive": False,          # True: seeded last, after everything else is scheduled
+ "breaks": ["w1-taint"],       # shared state it damages or needs intact; never paired with a task sharing any
+ "solo": False}                # True: only ever drilled on its own
 ```
 
 Scripts get `$K` (pinned to the grading context), `$WS`, `$CP/$W1/$W2/$W3`, `nx NODE 'cmd'`,

exam

@@ -212,8 +212,8 @@ def q_by_id(n: int) -> dict[str, Any]:
     for q in QUESTIONS:
         if q["id"] == n:
             return q
-    # ids are not contiguous (1-43 base, 101-124 hard, 201-217 / 301-317 sim sets), so print the
+    # ids are not contiguous (1-43 base, 101-124 hard, 201-217 / 301-317 / 401-418 / 501-517 sim
-    # real ranges rather than "1-<count>", which named ids that do not exist.
+    # sets), so print the real ranges rather than "1-<count>", which named ids that do not exist.
     ids = sorted(q["id"] for q in QUESTIONS)
     ranges, start, prev = [], ids[0], ids[0]
     for i in ids[1:] + [None]:
@@ -286,11 +286,15 @@ def score(st, results=None):
     return earned, total, (round(100 * earned / total, 1) if total else 0)
 
 # ----------------------------------------------------------- history ----
-def load_history():
+def load_history(include_synthetic=False):
+    """Attempts from history-<track>.jsonl. Entries marked "synthetic" (a scripted verification
+    run, not a person sitting the tasks) stay in the file for the record but are left out of every
+    statistic, weak list and "latest exam" unless explicitly asked for."""
     if not os.path.exists(HISTORY_FILE):
         return []
     with open(HISTORY_FILE) as f:
-        return [json.loads(l) for l in f if l.strip()]
+        entries = [json.loads(l) for l in f if l.strip()]
+    return entries if include_synthetic else [e for e in entries if not e.get("synthetic")]
 
 def record_attempt(st):
     earned, total, pct = score(st)
@@ -313,6 +317,7 @@ class AggEntry:
         self.last_fail_at: str = ""
         self.failed: list[str] = []
         self.last_earned: str = ""
+        self.streak: int = 0          # consecutive full passes at the end of the history
 
     def __getitem__(self, item: str) -> Any:
         return getattr(self, item)
@@ -328,6 +333,7 @@ class AggEntry:
             "last_fail_at": self.last_fail_at,
             "failed": self.failed,
             "last_earned": self.last_earned,
+            "streak": self.streak,
         }
 
 
@@ -341,6 +347,7 @@ def stats() -> dict[int, AggEntry]:
             a.attempts += 1
             a.last = r.get("pass")
             a.last_earned = f"{r.get('earned', 0)}/{r.get('total', 0)}"
+            a.streak = a.streak + 1 if r.get("pass") else 0
             if not r.get("pass"):
                 a.fails += 1
                 a.last_fail_at = str(e.get("at", ""))
@@ -366,27 +373,44 @@ def drilled_domains():
                 out[q["domain"]] += 1
     return out
 
+def breaks_of(q):
+    """A task's "breaks" as a set. It is a string or a list: Q118 damages the scheduler AND the
+    controller-manager, and declaring only one let it share a session with Q38/Q317."""
+    b = q.get("breaks")
+    return set() if not b else ({b} if isinstance(b, str) else set(b))
+
 def no_conflicts(cands, n, taken=()):
-    """Take up to n tasks, never two that break the same thing (or clash with tasks already taken)."""
+    """Take up to n tasks, never two that break the same thing (or clash with tasks already taken).
-    used = {q.get("breaks") for q in taken if q.get("breaks")}
+    A "solo" task (Q102's etcd restore rolls the WHOLE cluster back to its snapshot, wiping every
+    other task's seed and answers) is only ever taken on its own."""
+    used = set().union(*(breaks_of(q) for q in taken)) if taken else set()
     out = []
     for q in cands:
-        b = q.get("breaks")
+        if q.get("solo") and (taken or out):
-        if b and b in used:
+            continue
+        if any(x.get("solo") for x in list(taken) + out):
+            break
+        b = breaks_of(q)
+        if b & used:
             continue
         out.append(q)
-        if b:
+        used |= b
-            used.add(b)
         if len(out) == n:
             break
     return out
 
+def split_solo(qs):
+    """Pull solo tasks out of a multi-task selection; they have to be drilled alone."""
+    if len(qs) <= 1:
+        return list(qs), []
+    return [q for q in qs if not q.get("solo")], [q for q in qs if q.get("solo")]
+
 def pick(domain, n, hard, rng, exclude=(), taken=(), unseen_only=False):
     """n tasks from a domain: unseen first, then least recently seen; hard sessions prefer hard tasks."""
     seen = last_seen()
 
     def build(allow_hard):
-        pool = [q for q in POOL if q["domain"] == domain and q["id"] not in exclude
+        pool = [q for q in POOL if q["domain"] == domain and q["id"] not in exclude and not q.get("solo")
                 and (allow_hard or q["difficulty"] == "normal") and not (unseen_only and q["id"] in seen)]
         rng.shuffle(pool)
         pool.sort(key=lambda q: (0 if (hard and q["difficulty"] == "hard") else 1, seen.get(q["id"], -1)))
@@ -425,7 +449,7 @@ def campaign_batch(hard, rng):
     """'drill all': DRILL_SIZE tasks you have never attempted, round-robin across domains,
     hard tasks included when --hard. When everything has been seen once, falls back to weak tasks."""
     seen = last_seen()
-    pool = [q for q in POOL if q["id"] not in seen and (hard or q["difficulty"] == "normal")]
+    pool = [q for q in POOL if q["id"] not in seen and not q.get("solo") and (hard or q["difficulty"] == "normal")]
     if not pool:
         return None, []
     qs, domains = [], list(DOMAINS)
@@ -475,23 +499,46 @@ def cmd_progress(args):
     print(dim(f"\n{len(QUESTIONS) - len(left)}/{len(QUESTIONS)} tasks attempted at least once." + (f"  Never seen: Q{', Q'.join(map(str, left))}" if left and len(left) <= 25 else "")))
     print(dim("passed = latest attempt was a full pass.  ./exam drill all [--hard] works through what's left."))
 
+def is_weak(q, a):
+    """Weak = failed on the latest attempt, or failed at least twice and not yet passed twice in a
+    row since - a flaky task has to prove itself - or never attempted (outside the fixed sets).
+    The old rule ("failed twice, ever") kept a task weak forever: 13 of 29 listed were passing."""
+    if a.attempts == 0:
+        return not q.get("set")
+    return a.last is False or (a.fails >= 2 and a.streak < 2)
+
 def weak_ids(agg=None):
     agg = agg or stats()
-    return [q["id"] for q in QUESTIONS
+    return [q["id"] for q in QUESTIONS if is_weak(q, agg[q["id"]])]
-            if (agg[q["id"]]["last"] is False or agg[q["id"]]["fails"] >= 2)
+
-            or (agg[q["id"]]["attempts"] == 0 and not q.get("set"))]
+def weak_lines(a):
+    """What to show under a weak task: the failing checks if it failed last time, otherwise how
+    close it is to dropping off the list - not the ✗ list of some older, since-fixed attempt."""
+    if a.last is False:
+        return [f"✗ {n}" for n in (a.failed or [])]
+    return [f"passed last time ({a.last_earned}) - one more clean pass takes it off this list"]
+
+def exam_domain_scores(e):
+    """domain -> (earned, total) over the tasks that were actually IN that attempt. Summing the
+    weight of every task in the bank instead made a 100% run read "ARCH 38/266"."""
+    out = {}
+    for k, r in e.get("results", {}).items():
+        q = QBYID.get(int(k))
+        if q:
+            de, dt = out.get(q["domain"], (0, 0))
+            out[q["domain"]] = (de + r.get("earned", 0), dt + r.get("total", 0))
+    return out
 
 def write_weak_spots():
     agg = stats(); hist = load_history()
     L = [f"# {TRACK_ID.upper()} weak spots", "", f"_Generated by exam from {len(hist)} attempt(s); regenerated on every `end`._", ""]
     exams = [e for e in hist if e["mode"] == "exam"]
     if exams:
-        e = exams[-1]
+        e = exams[-1]; ds = exam_domain_scores(e)
         L += [f"## Domain scores (latest full exam, {e['at']}, {e['score_pct']}%)", ""]
         for d, name in DOMAINS.items():
-            dq = [q for q in QUESTIONS if q["domain"] == d]
+            if d in ds:
-            dt = sum(q["weight"] for q in dq); de = sum(e["results"].get(str(q["id"]), {}).get("earned", 0) for q in dq)
+                L.append(f"- **{d}** {ds[d][0]}/{ds[d][1]} — {name}")
-            L.append(f"- **{d}** {de}/{dt} — {name}")
         L.append("")
     L += ["## Drill next  (`./exam drill weak`)", ""]
     weak = [q for q in QUESTIONS if q["id"] in weak_ids(agg) and agg[q["id"]]["attempts"] > 0]
@@ -499,9 +546,8 @@ def write_weak_spots():
         L.append("Nothing failing right now.")
     for q in weak:
         a = agg[q["id"]]
-        L.append(f"### Q{q['id']:02d} · {q['domain']} · {q['title']}  ({a['fails']}/{a['attempts']} failed, last {a['last_earned']} pts, {a['last_fail_at']})")
+        L.append(f"### Q{q['id']:02d} · {q['domain']} · {q['title']}  ({a['fails']}/{a['attempts']} failed, last {a['last_earned']} pts, last failed {a['last_fail_at']})")
-        failed_list = list(a["failed"]) if isinstance(a["failed"], (list, tuple)) else []
+        L += [f"- {n}" for n in weak_lines(a)]
-        L += [f"- ✗ {n}" for n in failed_list]
         L.append("")
     L += ["## Attempt matrix", "", "| Q | Domain | Title | " + " | ".join(f"{e['mode'][0].upper()}{i + 1}" for i, e in enumerate(hist)) + " |",
           "|---|---|---|" + "---|" * len(hist)]
@@ -517,13 +563,13 @@ def write_weak_spots():
 
 # ------------------------------------------------------------ session ----
 def preflight():
-    out = subprocess.run(["kubectl", "get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"],
+    out = subprocess.run(["kubectl", "--context", CONTEXT, "get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"],
                          capture_output=True, text=True).stdout.split()
     missing = [v for v in NODES.values() if v not in out]
     if missing:
         die(f"Nodes not found in cluster: {missing}\n  cluster has: {out}\n"
             "  Set CKA_CP, CKA_W1, CKA_W2, CKA_W3 to your node names (needs 1 control-plane + 3 workers).")
-    rc, _ = run('nx "$CP" "true" </dev/null && nx "$W2" "true" </dev/null && nx "$W3" "true" </dev/null', timeout=60)
+    rc, _ = run('nx "$CP" "true" </dev/null && nx "$W1" "true" </dev/null && nx "$W2" "true" </dev/null && nx "$W3" "true" </dev/null', timeout=60)
     if rc != 0:
         die(f"Cannot run commands on nodes with CKA_NODE_ACCESS={NODE_ACCESS}.\n"
             "  podman/docker -> node names must be kind container names (kind get nodes --name cka-sim)\n"
@@ -598,17 +644,26 @@ if nx "$CP" 'grep -q "^ *- kube-apiserverr$" /etc/kubernetes/manifests/kube-apis
   for _ in $(seq 1 30); do kubectl --context "${CKA_CONTEXT:-kind-cka-sim}" get --raw=/readyz >/dev/null 2>&1 && break; sleep 2; done
 fi
 
-# scheduler: undo the "kube-scheduler" -> "kube-schedulerr" command typo (Q20/Q118)
+# scheduler: undo the "kube-scheduler" -> "kube-schedulerr" command typo (Q20/Q118), and put the
+# manifest back if a task had the candidate stop the scheduler by moving it out (Q512) and they
+# never moved it back - that setup keeps a pristine copy outside the static pod directory.
 nx "$CP" '
 f=/etc/kubernetes/manifests/kube-scheduler.yaml
+[ -f $f ] || { [ -f /etc/kubernetes/.cka-sim/kube-scheduler.yaml ] && cp /etc/kubernetes/.cka-sim/kube-scheduler.yaml $f; }
 sed -i "s|^\( *\)- kube-schedulerr$|\1- kube-scheduler|" $f 2>/dev/null
 ' </dev/null
 
-# controller-manager: undo Q38's injected flag and Q118's broken image tag
+# controller-manager: undo Q38's injected flag, Q118's broken image tag and Q317's sabotage - both
+# its current form (the CSR signing controller switched off in --controllers) and the older one
+# (--cluster-signing-cert-file pointed at a missing file). That older form crash-looped the WHOLE
+# controller-manager, because a controller that fails to start is fatal there, so every later
+# session silently lost deployments, jobs, endpoints and PV phases, not just CSR signing.
 nx "$CP" '
 f=/etc/kubernetes/manifests/kube-controller-manager.yaml
 sed -i "/--leader-elect-lease-duration=1s/d" $f 2>/dev/null
 sed -i "s|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)-broken|\1|" $f 2>/dev/null
+sed -i "s|--cluster-signing-cert-file=/etc/kubernetes/pki/ca-signing.crt|--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt|" $f 2>/dev/null
+sed -i "s|,-certificatesigningrequest-signing-controller||; s|,-csrsigning||" $f 2>/dev/null
 ' </dev/null
 
 # coredns: restore the default upstream forward (Q37/Q114 point it somewhere bogus)
@@ -618,25 +673,61 @@ if [ -n "$cf" ] && ! printf '%s' "$cf" | grep -Eq '^[[:space:]]*forward \. /etc/
   $K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.restore --dry-run=client -o yaml | $K apply -f - >/dev/null 2>&1
   $K -n kube-system rollout restart deploy coredns >/dev/null 2>&1
 fi
+# ...and Q114's other two faults. kindnet ENFORCES NetworkPolicy, so a leftover lockdown policy on
+# the CoreDNS pods kills name resolution for the whole cluster in every later session, and so does
+# a Deployment left at 0 replicas.
+$K -n kube-system delete netpol maintenance-lockdown --ignore-not-found >/dev/null 2>&1
+r=$($K -n kube-system get deploy coredns -o jsonpath='{.spec.replicas}' 2>/dev/null)
+if [ -n "$r" ] && [ "$r" -lt 2 ]; then $K -n kube-system scale deploy coredns --replicas=2 >/dev/null 2>&1; fi
 
 # w1/w2/w3 kubelet: undo Q17/Q39-style tampering, then make sure it's enabled and running.
 # staticPodPath is normalised unconditionally: Q103 relocates it to /etc/kubernetes/static, and
 # the config.yaml.orig copy below only helps if something made that backup first - nothing does,
 # so a relocated path used to survive every later session and silently break Q05 (whose check
 # looks in /etc/kubernetes/manifests). Only restart the kubelet if we actually changed the file.
+# clientCAFile gets the same treatment: Q39 points it at ca-bundle.crt and Q117 at ca-wrong.crt, and
+# only W1 ever gets a config.yaml.orig (Q39 makes it), so an unfinished Q117 left W2's kubelet
+# crash-looping in every later session - enable/start below cannot help a kubelet that exits.
+# The systemd drop-in is checked too: Q507 points ExecStart at a kubelet binary that does not exist.
 for n in "$W1" "$W2" "$W3"; do
   nx "$n" '
-    [ -f /var/lib/kubelet/config.yaml.orig ] && cp -f /var/lib/kubelet/config.yaml.orig /var/lib/kubelet/config.yaml
+    c=/var/lib/kubelet/config.yaml; d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
-    if grep -q "^staticPodPath:" /var/lib/kubelet/config.yaml 2>/dev/null \
+    [ -f $c.orig ] && cp -f $c.orig $c
-       && ! grep -q "^staticPodPath: /etc/kubernetes/manifests$" /var/lib/kubelet/config.yaml; then
+    restart=0
-      sed -i "s|^staticPodPath: .*|staticPodPath: /etc/kubernetes/manifests|" /var/lib/kubelet/config.yaml
+    if grep -q "^staticPodPath:" $c 2>/dev/null && ! grep -q "^staticPodPath: /etc/kubernetes/manifests$" $c; then
-      systemctl restart kubelet >/dev/null 2>&1
+      sed -i "s|^staticPodPath: .*|staticPodPath: /etc/kubernetes/manifests|" $c; restart=1
     fi
+    ca=$(awk "/clientCAFile:/{print \$2}" $c 2>/dev/null)
+    if [ -n "$ca" ] && [ ! -f "$ca" ]; then
+      sed -i "s|clientCAFile: .*|clientCAFile: /etc/kubernetes/pki/ca.crt|" $c; restart=1
+    fi
+    bin=$(sed -n "s|^ExecStart=\([^ ]*kubelet\) .*|\1|p" $d 2>/dev/null | tail -1)
+    if [ -n "$bin" ] && [ ! -x "$bin" ]; then
+      sed -i "s|^ExecStart=$bin |ExecStart=/usr/bin/kubelet |" $d; systemctl daemon-reload; restart=1
+    fi
+    [ $restart = 1 ] && systemctl restart kubelet >/dev/null 2>&1
     systemctl enable kubelet >/dev/null 2>&1
     systemctl start kubelet  >/dev/null 2>&1
   ' </dev/null
 done
 
+# static pods the bank drops on nodes: Q05's static-web and Q314's edge-cache on W3 (edge-cache
+# sat in ContainerCreating for days, its hostPath deliberately missing), Q103's relocated copies,
+# Q509's ops-beacon on the control plane. Each setup recreates what its own session needs.
+nx "$W3" 'rm -f /etc/kubernetes/manifests/static-web*.yaml /etc/kubernetes/manifests/edge-cache.yaml /etc/kubernetes/static/*.yaml' </dev/null
+nx "$CP" 'rm -f /etc/kubernetes/manifests/ops-beacon*.yaml' </dev/null
+
+# default StorageClass: Q16 makes fast-local the default and strips `standard`. Leave exactly one
+# default - `standard` - so "the default class" means the same thing in every session.
+if $K get sc standard >/dev/null 2>&1; then
+  for sc in $($K get sc -o jsonpath='{.items[*].metadata.name}'); do
+    [ "$sc" = standard ] && continue
+    [ "$($K get sc "$sc" -o jsonpath='{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" = true ] \
+      && $K annotate sc "$sc" storageclass.kubernetes.io/is-default-class- >/dev/null 2>&1
+  done
+  $K annotate sc standard storageclass.kubernetes.io/is-default-class=true --overwrite >/dev/null 2>&1
+fi
+
 # w1-taint: clear every randomized demo taint key the bank can apply. Q25 draws from
 # dedicated/team/workload, Q107 from pool/lane, and Q307 pins maintenance=inprogress - miss one
 # and that node silently stops accepting pods for every later session.
@@ -794,7 +885,7 @@ def cmd_drill(args):
         label, qs = campaign_batch(hard, random.Random())
         if not qs:
             print(bold("Every task in the bank has been attempted." ) + " Switching to weak tasks.")
-            qs = [q for q in QUESTIONS if q["id"] in weak_ids()]
+            qs, _ = split_solo([q for q in QUESTIONS if q["id"] in weak_ids()])
             qs = no_conflicts(qs, DRILL_SIZE)
             if not qs:
                 die("Nothing weak either - you have passed everything on its latest attempt. Time for ./exam start --hard.")
@@ -816,7 +907,10 @@ def cmd_drill(args):
             else:
                 ids.append(q_by_id(int(a))["id"])
     ids = sorted(set(ids))
-    qs = [q for q in QUESTIONS if q["id"] in ids]
+    qs, solo = split_solo([q for q in QUESTIONS if q["id"] in ids])
+    if solo:
+        print(yellow(f"Dropped Q{', Q'.join(str(q['id']) for q in solo)}: restoring it rolls the whole cluster back and would "
+                     f"wipe the other tasks. Drill it on its own: ./exam drill {solo[0]['id']}"))
     kept = no_conflicts(qs, len(qs))
     if len(kept) < len(qs):
         dropped = [q["id"] for q in qs if q not in kept]
@@ -840,10 +934,10 @@ def cmd_study(args):
         qs = pick(domain, n, hard, rng)
         print(bold(f"Study: {DOMAINS[domain]}"))
     elif rest and rest[0] == "weak":
-        qs = no_conflicts([q for q in QUESTIONS if q["id"] in weak_ids()], n)
+        qs = no_conflicts(split_solo([q for q in QUESTIONS if q["id"] in weak_ids()])[0], n)
         print(bold("Study: tasks you have been failing"))
     elif rest:
-        qs = no_conflicts([q_by_id(int(a)) for a in rest], len(rest))
+        qs = no_conflicts(split_solo([q_by_id(int(a)) for a in rest])[0], len(rest))
         print(bold("Study: " + ", ".join(f"Q{q['id']}" for q in qs)))
     else:
         domain, qs = next_drill(hard, rng)
@@ -851,7 +945,7 @@ def cmd_study(args):
     begin_session(qs, "drill", EXAM_SECONDS, force, hard=hard, untimed_session=True)
 
 def cmd_sim(args):
-    """Fixed 17-task simulator set, full exam rules and clock."""
+    """A fixed simulator set, full exam rules and clock."""
     force = "--force" in args
     names = [a.lower() for a in args if not a.startswith("--")]
     if not names or names[0] not in SETS:
@@ -895,8 +989,13 @@ def cmd_show(args):
 
 def cmd_grade(args):
     st = require_started()
+    force = "--force" in args; args = [a for a in args if a != "--force"]
     if not args:
         die("usage: ./exam grade N | all")
+    # The web UI already refused this; the CLI handed out per-check feedback mid-exam, which a sim
+    # meant to rehearse the real thing (no feedback until you submit) must not do by default.
+    if st.get("mode") == "exam" and not st.get("ended_at") and not force:
+        die("No feedback during an exam - the real one gives none. Submit with ./exam end (--force to peek anyway).")
     ids = [q["id"] for q in session_questions(st)] if args[0] == "all" else [int(a) for a in args]
     for n in ids:
         q = q_by_id(n); r = grade_one(q)
@@ -980,7 +1079,7 @@ def cmd_end(args):
     print_score(st)
     if entry is None:
         print(dim("Study session - not added to history or weak-spots.")); return
-    print(dim(f"Time used: {fmt(st['ended_at'] - st['started_at'])}" + ("  (over time)" if st.get("over_time") else "")))
+    print(dim(f"Time used: {fmt(st['ended_at'] - st['started_at'] - paused_total(st))}" + ("  (over time)" if st.get("over_time") else "")))
     failed = sorted(int(k) for k, v in entry["results"].items() if not v["pass"])
     print(dim(f"Attempt #{len(load_history())} logged to {os.path.basename(HISTORY_FILE)}; {os.path.basename(WEAK_FILE)} updated."))
     if failed:
@@ -1012,31 +1111,34 @@ def cmd_weak(args):
         print(green("  none - everything attempted has passed on its latest try"))
     for i in ids:
         q = q_by_id(i); a = agg[i]
-        print(f"\n{red('Q%02d' % i)} {q['domain']:<7} {q['title']}  " + dim(f"{a['fails']}/{a['attempts']} failed, last {a['last_earned']} pts, {a['last_fail_at']}"))
+        print(f"\n{red('Q%02d' % i)} {q['domain']:<7} {q['title']}  " + dim(f"{a['fails']}/{a['attempts']} failed, last {a['last_earned']} pts, last failed {a['last_fail_at']}"))
-        for n in a["failed"]:
+        for n in weak_lines(a):
-            print(dim("    ✗ " + n))
+            print(dim("    " + n))
     never = [q["id"] for q in QUESTIONS if agg[q["id"]]["attempts"] == 0]
     if never:
         print(dim(f"\nNever attempted: Q{', Q'.join(map(str, never))}"))
     print(dim(f"\nFull log: {WEAK_FILE}     Drill them: ./exam drill weak"))
 
 def cmd_history(args):
-    hist = load_history()
+    hist = load_history(include_synthetic=True)
     if not hist:
         die("No history yet.")
     print(bold(f"{'#':>3}  {'when':<17}{'mode':<7}{'score':>7}  {'time':<9} tasks"))
     for i, e in enumerate(hist, 1):
         col = green if e["score_pct"] >= PASS_PCT else red
         tasks = f"all {len(QUESTIONS)}" if len(e["ids"]) == len(QUESTIONS) else "Q" + ",".join(map(str, e["ids"]))
-        print(f"{i:>3}  {e['at']:<17}{(e['mode'] + ('*' if e.get('hard') else '')):<7}{col('%5.1f%%' % e['score_pct']):>16}  {fmt(e['time_used']):<9} {tasks}")
+        tag = dim("  [scripted verification - not counted]") if e.get("synthetic") else ""
-    exams = [e for e in hist if e["mode"] == "exam"]
+        print(f"{i:>3}  {e['at']:<17}{(e['mode'] + ('*' if e.get('hard') else '')):<7}{col('%5.1f%%' % e['score_pct']):>16}  {fmt(e['time_used']):<9} {tasks}{tag}")
+    exams = [e for e in hist if e["mode"] == "exam" and not e.get("synthetic")]
     print(dim("* = hard"))
     if len(exams) >= 2:
-        print(bold("\nDomain trend (full exams, % of points):"))
+        print(bold("\nDomain trend (full exams, % of that exam's points in the domain):"))
         for d in DOMAINS:
-            dq = [q for q in QUESTIONS if q["domain"] == d]; dt = sum(q["weight"] for q in dq)
+            cells = []
-            pts = [round(100 * sum(e["results"].get(str(q["id"]), {}).get("earned", 0) for q in dq) / dt) for e in exams]
+            for e in exams:
-            print(f"  {d:<7} " + " → ".join(f"{p:>3}%" for p in pts))
+                de, dt = exam_domain_scores(e).get(d, (0, 0))
+                cells.append(f"{round(100 * de / dt):>3}%" if dt else "   -")
+            print(f"  {d:<7} " + " → ".join(cells))
 
 def cmd_reset(args):
     if not args:
@@ -1114,12 +1216,12 @@ class Api(http.server.BaseHTTPRequestHandler):
                             elif body.get("campaign"):
                                 _, qs = campaign_batch(hard, rng)
                                 if not qs:
-                                    qs = no_conflicts([q for q in QUESTIONS if q["id"] in weak_ids()], DRILL_SIZE)
+                                    qs = no_conflicts(split_solo([q for q in QUESTIONS if q["id"] in weak_ids()])[0], DRILL_SIZE)
                             elif body.get("next"):
                                 _, qs = next_drill(hard, rng)
                             else:
                                 ids = body.get("ids") or weak_ids()
-                                qs = no_conflicts([q for q in QUESTIONS if q["id"] in ids], len(ids))
+                                qs = no_conflicts(split_solo([q for q in QUESTIONS if q["id"] in ids])[0], len(ids))
                             per = DRILL_SECONDS_PER_TASK_HARD if hard else DRILL_SECONDS_PER_TASK
                             begin_session(qs, "drill", per * len(qs), True, log=ulog, hard=hard,
                                           untimed_session=bool(body.get("study")))
@@ -1269,9 +1371,9 @@ def cmd_help(args):
                            exact-output tasks, compound failures, collateral checks and exam-style contexts.
   list                     tasks in this session, with current + historical results
   show N                   display task N
-  grade N|all              grade one task (or all) per check - drills only, in the UI
+  grade N|all              grade one task (or all) per check - drills only (--force to peek in an exam)
   status / timer           time remaining (timer = live countdown)
-  pause / resume           stop the clock (recorded on the attempt - the real exam has no pause)
+  pause / resume           stop the clock (not recorded anywhere - but the real exam has no pause)
   end                      submit: grade per check, score, log attempt, update weak-spots
   weak                     what you keep getting wrong, with the failing checks
   history                  every attempt, scores, and per-domain trend

history-cka.jsonl

@@ -8,4 +8,4 @@
 {"at": "2026-09-15T20:55", "mode": "drill", "hard": false, "ids": [13, 15, 16, 19, 25, 28, 29, 30, 33, 37, 39, 40, 42, 106, 112, 115, 120, 121], "score_pct": 41.7, "earned": 40, "total": 96, "time_used": 608, "results": {"13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "19": {"pass": true, "earned": 6, "total": 6, "failed": []}, "33": {"pass": true, "earned": 5, "total": 5, "failed": []}, "30": {"pass": true, "earned": 4, "total": 4, "failed": []}, "37": {"pass": true, "earned": 6, "total": 6, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~6h"]}, "42": {"pass": true, "earned": 4, "total": 4, "failed": []}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries team=finance:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "115": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "120": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "121": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-21 in ac-echo (the context's namespace), 2 available, owner=ac", "Service ledger-21 port 9000 in the same namespace", "ns.txt has the namespace"]}}}
 {"at": "2026-09-16T19:40", "mode": "drill", "hard": false, "ids": [15, 16, 25, 28, 29, 40], "score_pct": 50.0, "earned": 14, "total": 28, "time_used": 1258, "results": {"15": {"pass": true, "earned": 6, "total": 6, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "29": {"pass": true, "earned": 4, "total": 4, "failed": []}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~24h"]}}}
 {"at": "2026-09-19T12:27", "mode": "drill", "hard": false, "ids": [106, 217, 34, 202, 102, 301, 39, 13, 32, 204, 25, 2, 8, 112, 4, 109, 313, 9], "score_pct": 39.7, "earned": 48, "total": 121, "time_used": 4197, "results": {"8": {"pass": true, "earned": 4, "total": 4, "failed": []}, "34": {"pass": true, "earned": 4, "total": 4, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "301": {"pass": true, "earned": 9, "total": 9, "failed": []}, "4": {"pass": true, "earned": 4, "total": 4, "failed": []}, "9": {"pass": true, "earned": 4, "total": 4, "failed": []}, "106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "217": {"pass": false, "earned": 0, "total": 10, "failed": ["Pod tiger-probe Running with both labels", "container.txt holds a real container ID for that Pod", "container.txt records the runtimeType", "container.log holds the httpd startup output"]}, "202": {"pass": false, "earned": 3, "total": 8, "failed": ["watchAllNamespaces=true reached the pod env", "Ledger main-ledger has currency USD and retentionDays 30"]}, "102": {"pass": false, "earned": 2, "total": 10, "failed": ["ConfigMap restore-proof-67 exists again with its original data", "etcd uses a restored data directory (manifest changed)"]}, "32": {"pass": false, "earned": 0, "total": 4, "failed": ["ResourceQuota limits pods/6, requests.cpu 1, limits.memory 2Gi", "LimitRange defaults set", "worker pods Running with defaulted requests"]}, "204": {"pass": false, "earned": 0, "total": 8, "failed": ["evicted-first.txt lists exactly the BestEffort pods", "api-qos.txt says Burstable", "guaranteed.txt names orion-ledger"]}, "2": {"pass": false, "earned": 0, "total": 7, "failed": ["/opt/cka/etcd-snapshot.db exists on the control-plane node", "snapshot is a real etcd backup (> 100 KB)"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret secure.lab.local-tls is kubernetes.io/tls with CN secure.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "109": {"pass": false, "earned": 2, "total": 8, "failed": ["native sidecar: initContainer agent with restartPolicy Always", "probes on app as specified", "Running, shared log written by agent, grace period 5"]}, "313": {"pass": false, "earned": 0, "total": 11, "failed": ["Secret classic-tls is a TLS secret with CN classic.r700.io", "both host rules route to the right Services on /", "TLS covers both hosts with classic-tls", "defaultBackend points at web-classic:80, class is nginx"]}}}
-{"at": "2026-09-20T17:32", "mode": "exam", "hard": true, "ids": [401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418], "score_pct": 100.0, "earned": 163, "total": 163, "time_used": 2909, "results": {"402": {"pass": true, "earned": 9, "total": 9, "failed": []}, "404": {"pass": true, "earned": 9, "total": 9, "failed": []}, "405": {"pass": true, "earned": 11, "total": 11, "failed": []}, "401": {"pass": true, "earned": 8, "total": 8, "failed": []}, "403": {"pass": true, "earned": 9, "total": 9, "failed": []}, "406": {"pass": true, "earned": 7, "total": 7, "failed": []}, "407": {"pass": true, "earned": 10, "total": 10, "failed": []}, "408": {"pass": true, "earned": 9, "total": 9, "failed": []}, "409": {"pass": true, "earned": 9, "total": 9, "failed": []}, "410": {"pass": true, "earned": 8, "total": 8, "failed": []}, "411": {"pass": true, "earned": 7, "total": 7, "failed": []}, "412": {"pass": true, "earned": 11, "total": 11, "failed": []}, "413": {"pass": true, "earned": 10, "total": 10, "failed": []}, "414": {"pass": true, "earned": 9, "total": 9, "failed": []}, "415": {"pass": true, "earned": 9, "total": 9, "failed": []}, "416": {"pass": true, "earned": 10, "total": 10, "failed": []}, "417": {"pass": true, "earned": 10, "total": 10, "failed": []}, "418": {"pass": true, "earned": 8, "total": 8, "failed": []}}}
+{"at": "2026-09-20T17:32", "mode": "exam", "hard": true, "ids": [401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418], "score_pct": 100.0, "earned": 163, "total": 163, "time_used": 2909, "results": {"402": {"pass": true, "earned": 9, "total": 9, "failed": []}, "404": {"pass": true, "earned": 9, "total": 9, "failed": []}, "405": {"pass": true, "earned": 11, "total": 11, "failed": []}, "401": {"pass": true, "earned": 8, "total": 8, "failed": []}, "403": {"pass": true, "earned": 9, "total": 9, "failed": []}, "406": {"pass": true, "earned": 7, "total": 7, "failed": []}, "407": {"pass": true, "earned": 10, "total": 10, "failed": []}, "408": {"pass": true, "earned": 9, "total": 9, "failed": []}, "409": {"pass": true, "earned": 9, "total": 9, "failed": []}, "410": {"pass": true, "earned": 8, "total": 8, "failed": []}, "411": {"pass": true, "earned": 7, "total": 7, "failed": []}, "412": {"pass": true, "earned": 11, "total": 11, "failed": []}, "413": {"pass": true, "earned": 10, "total": 10, "failed": []}, "414": {"pass": true, "earned": 9, "total": 9, "failed": []}, "415": {"pass": true, "earned": 9, "total": 9, "failed": []}, "416": {"pass": true, "earned": 10, "total": 10, "failed": []}, "417": {"pass": true, "earned": 10, "total": 10, "failed": []}, "418": {"pass": true, "earned": 8, "total": 8, "failed": []}}, "synthetic": true, "note": "Scripted end-to-end verification of Set C (tasks solved programmatically), not a person sitting it"}

setup.sh

@@ -98,4 +98,4 @@ echo
 echo "Cluster ready. Nodes:"
 kubectl get nodes -o wide
 echo
-echo "Next:  ./exam start     (seeds all 20 tasks and starts the 120-minute clock)"
+echo "Next:  ./exam web   (browser UI)   or   ./exam drill   (5 tasks from your least-drilled domain)"

tracks/cka/bank_extra.py

@@ -169,8 +169,9 @@ for t in $($K -n kube-system get secrets -o jsonpath='{.items[?(@.type=="bootstr
   C("the token in the file exists in the cluster with TTL ~%{ttl}", 3, r"""
 tok=$(grep -o -- '--token [a-z0-9]\{6\}\.[a-z0-9]\{16\}' "$WS/q40/join.sh" 2>/dev/null | awk '{print $2}'); [ -n "$tok" ] || exit 1
 id=${tok%%.*}; exp=$(j -n kube-system secret bootstrap-token-$id '{.data.expiration}' | base64 -d 2>/dev/null); [ -n "$exp" ] || exit 1
-secs=$(( $(date -d "$exp" +%s) - $(date +%s) )); want=$(( $(echo %{ttl} | tr -d h) * 3600 ))
+ct=$(j -n kube-system secret bootstrap-token-$id '{.metadata.creationTimestamp}'); [ -n "$ct" ] || exit 1
-[ $secs -gt $((want - 900)) ] && [ $secs -le $((want + 60)) ]"""),
+secs=$(( $(date -d "$exp" +%s) - $(date -d "$ct" +%s) )); want=$(( $(echo %{ttl} | tr -d h) * 3600 ))
+[ $secs -ge $((want - 120)) ] && [ $secs -le $((want + 120)) ] && [ "$(date -d "$exp" +%s)" -gt "$(date +%s)" ]"""),
  ],
 },
 {
@@ -197,9 +198,9 @@ kub=$(j node %{node} '{.status.nodeInfo.kubeletVersion}')
 grep -qx "apiserver=$api" "$f" && grep -qx "kubelet=$kub" "$f" """),
   C("etcd and runtime versions correct", 2, r"""
 f="$WS/q41/versions.txt"
-etcd=$(j -n kube-system pod -l component=etcd '{.items[0].spec.containers[0].image}' | sed 's/.*://')
+etcd=$(j -n kube-system pod -l component=etcd '{.items[0].spec.containers[0].image}' | sed 's/.*://'); ev=${etcd%-*}
 rt=$(j node %{node} '{.status.nodeInfo.containerRuntimeVersion}')
-grep -qx "etcd=$etcd" "$f" && grep -qx "runtime=$rt" "$f" """),
+grep -qxE "etcd=v?($etcd|$ev)" "$f" && grep -qx "runtime=$rt" "$f" """),
  ],
 },
 # ---------------------------------------------------------------- WORK ----
@@ -244,11 +245,17 @@ c["args"]=[a for a in c["args"] if a!="--kubelet-insecure-tls"]; print(json.dump
  "setup": NS + r"""
 mkns %{ns}
 $K delete priorityclass %{pc} --ignore-not-found >/dev/null
+$K -n %{ns} delete deploy payments --ignore-not-found --wait=true >/dev/null 2>&1
 $K -n %{ns} create deploy payments --image=nginx:1.25 --replicas=2 --dry-run=client -o yaml | $K apply -f - >/dev/null
 """,
  "checks": [
   C("PriorityClass %{pc} value %{val}, not default", 2, r"""eq "$(j priorityclass %{pc} '{.value}')" %{val} && [ "$(j priorityclass %{pc} '{.globalDefault}')" != true ]"""),
-  C("payments pods running with priority %{val}", 2, r"""eq "$(j -n %{ns} deploy payments '{.status.availableReplicas}')" 2 && for p in $(j -n %{ns} pod -l app=payments '{.items[*].spec.priority}'); do eq "$p" %{val} || exit 1; done"""),
+  C("payments pods running with priority %{val}", 2, r"""
+for t in $(seq 1 15); do   # the old ReplicaSet's pods linger for a moment while they terminate
+  ok=1; eq "$(j -n %{ns} deploy payments '{.status.availableReplicas}')" 2 || ok=0
+  for p in $(j -n %{ns} pod -l app=payments '{.items[*].spec.priority}'); do eq "$p" %{val} || ok=0; done
+  [ $ok = 1 ] && exit 0; sleep 2
+done; exit 1"""),
  ],
 },
 {
@@ -275,6 +282,8 @@ $K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null
 },
 {
  "id": 26, "domain": "WORK", "title": "DaemonSet on every node",
+ # needs every node schedulable: Q25/Q107 taint W1 and must keep the taint, Q307 blocks it
+ "breaks": ["w1-taint", "w1-block"],
  "params": lambda r: {"ns": ns_pick(r, "agents"), "ds": name_pick(r, "node-agent")},
  "task": """
 Create a DaemonSet `%{ds}` in namespace `%{ns}` (image `busybox:1.36`, command `sleep 3600`)
@@ -284,7 +293,9 @@ Requests: cpu 10m, memory 16Mi.
  "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete ds %{ds} --ignore-not-found >/dev/null""",
  "checks": [
   C("DaemonSet exists with requests set", 1, r"""eq "$(j -n %{ns} ds %{ds} '{.spec.template.spec.containers[0].resources.requests.cpu}')" 10m"""),
-  C("tolerates the control-plane taint", 1, r"""j -n %{ns} ds %{ds} '{.spec.template.spec.tolerations[*].key}' | grep -q control-plane"""),
+  C("tolerates the control-plane taint", 1, r"""$K -n %{ns} get ds %{ds} -o json 2>/dev/null | python3 -c '
+import sys,json; t=json.load(sys.stdin)["spec"]["template"]["spec"].get("tolerations") or []
+sys.exit(0 if any("control-plane" in (x.get("key") or "") or (x.get("operator")=="Exists" and not x.get("key")) for x in t) else 1)'"""),
   C("one ready pod per node (including control-plane)", 2, r"""n=$($K get nodes --no-headers 2>/dev/null | wc -l); eq "$(j -n %{ns} ds %{ds} '{.status.numberReady}')" "$n" && eq "$(j -n %{ns} ds %{ds} '{.status.desiredNumberScheduled}')" "$n" """),
  ],
 },
@@ -418,7 +429,8 @@ $K -n %{ns} create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o y
 $K -n %{ns} create deploy api --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
 $K -n %{ns} expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
 $K -n %{ns} expose deploy api --port=8080 --target-port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
-$K delete gatewayclass %{gc} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete gateway %{gw} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete httproute %{host}-route --ignore-not-found >/dev/null 2>&1
+# these fail harmlessly when the Gateway API CRDs are not installed yet - installing them is step 1
+$K delete gatewayclass %{gc} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete gateway %{gw} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete httproute %{host}-route --ignore-not-found >/dev/null 2>&1 || true
 """,
  "checks": [
   C("Gateway API CRDs installed (gateways, httproutes)", 1, r"""$K get crd gateways.gateway.networking.k8s.io httproutes.gateway.networking.k8s.io >/dev/null 2>&1"""),
@@ -430,7 +442,7 @@ eq "$($h -o jsonpath='{.spec.parentRefs[0].name}')" %{gw} || exit 1
 $h -o json | python3 -c '
 import sys,json; r=json.load(sys.stdin)["spec"]["rules"]
 api=[x for x in r if any(m.get("path",{}).get("value")=="/api" for m in x.get("matches",[]))]
-web=[x for x in r if x is not api and x.get("backendRefs",[{}])[0].get("name")=="web"]
+web=[x for x in r if x not in api and x.get("backendRefs",[{}])[0].get("name")=="web"]
 ok=api and api[0]["backendRefs"][0]["name"]=="api" and api[0]["backendRefs"][0]["port"]==8080 and web and web[0]["backendRefs"][0]["port"]==80
 sys.exit(0 if ok else 1)'"""),
  ],
@@ -492,8 +504,10 @@ sys.exit(0 if ok else 1)'"""),
 1. In `%{ns}`, StatefulSet `%{sts}` (2 replicas) exists but has no Service. Create the **headless**
    Service `%{sts}` it needs (port 5432) so that `%{sts}-0.%{sts}.%{ns}.svc.cluster.local` resolves.
 2. Create Service `legacy` of type ExternalName pointing at `%{ext}`.
-3. From a pod in the namespace, resolve `%{sts}-0.%{sts}` and write the full nslookup output
+3. From a pod in the namespace, resolve pod 0's stable name
-   to `{WS}/q43/pod0.txt`.
+   `%{sts}-0.%{sts}.%{ns}.svc.cluster.local` and write the full nslookup output to
+   `{WS}/q43/pod0.txt`. (busybox's nslookup applies no search domains, so a short name such as
+   `%{sts}-0.%{sts}` fails even when DNS is healthy.)
 """,
  "setup": NS + r"""
 mkns %{ns}; rm -rf "$WS/q43"; mkdir -p "$WS/q43"
@@ -550,11 +564,18 @@ spec: {accessModes: [ReadWriteOnce], storageClassName: manual, resources: {reque
 EOF
 for i in $(seq 1 20); do [ "$(j -n %{ns} pvc old-claim '{.status.phase}')" = Bound ] && break; sleep 1; done
 $K -n %{ns} delete pvc old-claim >/dev/null
+# every PV that exists now; the "no extra PV" check compares against this, so Retain leftovers from
+# earlier sessions (this task's own, Q15's pv-data, Set C's) can no longer fail it
+$K get pv -o jsonpath='{.items[*].metadata.name}' > "$WS/.q35-pvs"
 """,
  "checks": [
   C("PV %{pv} still exists and was not recreated", 2, r"""[ -n "$(j pv %{pv} '{.metadata.uid}')" ] && eq "$(j pv %{pv} '{.spec.persistentVolumeReclaimPolicy}')" Retain"""),
   C("PVC %{pvc} Bound to %{pv}", 3, r"""eq "$(j -n %{ns} pvc %{pvc} '{.status.phase}')" Bound && eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}')" %{pv}"""),
-  C("no extra PVs were created", 1, r"""[ "$($K get pv --no-headers 2>/dev/null | grep -c manual)" -le 2 ]"""),
+  C("no extra PVs were created", 1, r"""
+before=" $(cat "$WS/.q35-pvs" 2>/dev/null) "; [ "$before" != "  " ] || exit 1
+for pv in $($K get pv -o jsonpath='{range .items[?(@.spec.storageClassName=="manual")]}{.metadata.name}{" "}{end}' 2>/dev/null); do
+  case "$before" in *" $pv "*) ;; *) exit 1 ;; esac
+done"""),
  ],
 },
 {

tracks/cka/bank_hard.py

@@ -32,14 +32,18 @@ $K delete clusterrole %{sa}-nodes --ignore-not-found >/dev/null; $K delete clust
 $K create clusterrolebinding %{sa}-legacy-admin --clusterrole=cluster-admin --serviceaccount=%{ns}:%{sa} >/dev/null
 """,
  "checks": [
-  C("can get/list/watch/patch deployments and replicasets in %{ns}", 2, r"""for res in deployments.apps replicasets.apps; do for v in get list watch patch; do eq "$($K auth can-i $v $res -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes || exit 1; done; done"""),
+  C("can get/list/watch/patch deployments and replicasets in %{ns}", 2, r"""! $K get clusterrolebinding %{sa}-legacy-admin >/dev/null 2>&1 || exit 1   # not via cluster-admin
-  C("can create pods/exec in %{ns}; can list nodes", 2, r"""eq "$($K auth can-i create pods/exec -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes && eq "$($K auth can-i list nodes --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes"""),
+for res in deployments.apps replicasets.apps; do for v in get list watch patch; do eq "$($K auth can-i $v $res -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes || exit 1; done; done"""),
+  C("can create pods/exec in %{ns}; can list nodes", 2, r"""! $K get clusterrolebinding %{sa}-legacy-admin >/dev/null 2>&1 || exit 1
+eq "$($K auth can-i create pods --subresource=exec -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes && eq "$($K auth can-i list nodes --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes"""),
   C("no over-grant: cannot delete deployments, cannot get secrets, cannot get nodes", 3, r"""for t in "delete deployments -n %{ns}" "get secrets -n %{ns}" "get nodes" "create pods -n default"; do eq "$($K auth can-i $t --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null || true)" no || exit 1; done"""),
-  C("the cluster-admin binding is gone and named objects exist", 2, r"""! $K get clusterrolebinding %{sa}-legacy-admin >/dev/null 2>&1 && $K -n %{ns} get role %{sa}-role rolebinding %{sa}-rb >/dev/null 2>&1 && $K get clusterrole %{sa}-nodes clusterrolebinding %{sa}-crb >/dev/null 2>&1"""),
+  C("the cluster-admin binding is gone and named objects exist", 2, r"""! $K get clusterrolebinding %{sa}-legacy-admin >/dev/null 2>&1 && $K -n %{ns} get role/%{sa}-role rolebinding/%{sa}-rb >/dev/null 2>&1 && $K get clusterrole/%{sa}-nodes clusterrolebinding/%{sa}-crb >/dev/null 2>&1"""),
  ],
 },
 {
  "id": 102, "domain": "ARCH", "breaks": "etcd", "difficulty": "hard", "title": "etcd restore",
+ # solo: restoring rolls EVERY API object back to the snapshot - other tasks' seeds and answers too
+ "solo": True,
  "params": lambda r: {"ctx": "k8s-c1-H", "cm": name_pick(r, "restore-proof")},
  "task": CTX + """
 An etcd snapshot taken earlier is at `/opt/cka/backup.db` on the control-plane node.
@@ -49,6 +53,9 @@ been deleted.
 **Restore the cluster's etcd from that snapshot** so the ConfigMap exists again. The cluster
 must be healthy afterwards (API server up, all kube-system pods Running).
 Do not recreate the ConfigMap by hand — its data must come from the snapshot.
+
+A restore rolls every API object back to the moment of the snapshot — on a real exam cluster that
+would include the work you did on other tasks, so a restore is the last thing you do.
 """,
  "setup": r"""
 nx "$CP" 'mkdir -p /opt/cka; rm -f /opt/cka/backup.db'
@@ -58,15 +65,24 @@ $K -n default create cm %{cm} --from-literal=proof=$(date +%s)-%{cm} >/dev/null
 out=$(nx "$CP" 'etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/cka/backup.db' 2>&1) \
   || { echo "snapshot save on $CP failed: $out"; exit 1; }
 sleep 2; $K -n default delete cm %{cm} >/dev/null
+nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' </dev/null > "$WS/.q102-datadir"
+date +%s > "$WS/.q102-t0"
 """,
  "checks": [
   C("ConfigMap %{cm} exists again with its original data", 6, r"""j -n default cm %{cm} '{.data.proof}' | grep -q -- '-%{cm}$'"""),
-  C("etcd uses a restored data directory (manifest changed)", 2, r"""! nx "$CP" 'grep -q "path: /var/lib/etcd$" /etc/kubernetes/manifests/etcd.yaml' """),
+  C("etcd runs on a data directory restored from the snapshot", 2, r"""
-  C("control plane healthy: etcd, apiserver, scheduler, controller-manager Running", 2, r"""for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done"""),
+was=$(cat "$WS/.q102-datadir" 2>/dev/null); t0=$(cat "$WS/.q102-t0" 2>/dev/null); [ -n "$was" ] && [ -n "$t0" ] || exit 1
+now=$(nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' </dev/null)
+[ -n "$now" ] || exit 1
+[ "$now" != "$was" ] && exit 0
+# the same directory name as before is fine if it was rebuilt from the snapshot after seeding
+m=$(nx "$CP" "stat -c %Y $now/member 2>/dev/null" </dev/null); [ -n "$m" ] && [ "$m" -gt "$t0" ]"""),
+  C("control plane healthy: etcd, apiserver, scheduler, controller-manager Running", 2, r"""j -n default cm %{cm} '{.data.proof}' | grep -q -- '-%{cm}$' || exit 1   # only once restored
+for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done"""),
  ],
 },
 {
- "id": 103, "domain": "ARCH", "breaks": "w3-kubelet", "difficulty": "hard", "title": "Static pod on a relocated manifest path",
+ "id": 103, "domain": "ARCH", "breaks": ["w3-kubelet", "w3-staticpath"], "difficulty": "hard", "title": "Static pod on a relocated manifest path",
  "params": lambda r: {"ctx": "k8s-c1-H", "ns": ns_pick(r, "edge"), "pod": name_pick(r, "edge-proxy")},
  "task": CTX + """
 Exactly one worker node is labelled `tier=edge`. On that node create a **static pod**:
@@ -108,7 +124,7 @@ Onboard user `%{user}` (group `developers`) end to end:
   C("CSR %{user} approved with certificate, O=developers", 2, r"""[ -n "$(j csr %{user} '{.status.certificate}')" ] && j csr %{user} '{.status.certificate}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'O *= *developers'"""),
   C("kubeconfig is self-contained and works without flags", 3, r"""f="$WS/q104/%{user}.kubeconfig"; [ -f "$f" ] || exit 1; grep -q 'client-certificate-data' "$f" && grep -q 'certificate-authority-data' "$f" && ! grep -q 'client-certificate:' "$f" && kubectl --kubeconfig "$f" get pods >/dev/null 2>&1"""),
   C("context %{user}@lab defaults to %{ns}", 1, r"""eq "$(kubectl --kubeconfig "$WS/q104/%{user}.kubeconfig" config view -o jsonpath='{.contexts[?(@.name=="%{user}@lab")].context.namespace}')" %{ns}"""),
-  C("edit rights in %{ns}, none elsewhere", 2, r"""eq "$($K auth can-i create deployments -n %{ns} --as=%{user} 2>/dev/null)" yes && eq "$($K auth can-i delete pods -n %{ns} --as=%{user} 2>/dev/null)" yes && eq "$($K auth can-i get pods -n default --as=%{user} 2>/dev/null || true)" no && eq "$($K auth can-i get secrets -n kube-system --as=%{user} 2>/dev/null || true)" no"""),
+  C("edit rights in %{ns}, none elsewhere", 2, r"""eq "$($K auth can-i create deployments -n %{ns} --as=%{user} --as-group=developers 2>/dev/null)" yes && eq "$($K auth can-i delete pods -n %{ns} --as=%{user} --as-group=developers 2>/dev/null)" yes && eq "$($K auth can-i get pods -n default --as=%{user} --as-group=developers 2>/dev/null || true)" no && eq "$($K auth can-i get secrets -n kube-system --as=%{user} --as-group=developers 2>/dev/null || true)" no"""),
  ],
 },
 {
@@ -159,18 +175,22 @@ $K delete ns %{ns} --ignore-not-found --wait=false >/dev/null
  "id": 106, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults",
  "params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])},
  "task": CTX + """
-Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural lowercase+s, shortName `%{kind}`-initials) with schema:
+Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the kind in lowercase plus
+`s`, shortName = the first two letters of the kind in lowercase) with schema:
 - `spec.replicas` integer, **required**, minimum 1, maximum 10,
 - `spec.environment` string, enum `prod|stage|dev`, **default** `dev`,
 - `spec.image` string, required.
 Then:
 1. Create `%{kind}` `good` in `default` with replicas 3, image `nginx:1.27`, environment `%{env}`.
 2. Save the API server's rejection message for a `%{kind}` with `replicas: 0` to `{WS}/q106/rejected.txt`.
-3. Create `%{kind}` `defaulted` with only replicas 2 and image set; its environment must be `dev` after creation.
+3. Create `%{kind}` `defaulted` in `default` with only replicas 2 and image set; its environment
+   must be `dev` after creation.
+
+This context's default namespace is not necessarily `default` — say `-n default` explicitly.
 """,
  "setup": r"""rm -rf "$WS/q106"; mkdir -p "$WS/q106"; k=$(echo %{kind} | tr A-Z a-z); $K delete crd ${k}s.deploy.example.com --ignore-not-found >/dev/null""",
  "checks": [
-  C("CRD exists, Namespaced, replicas has min/max and is required", 2, r"""k=$(echo %{kind} | tr A-Z a-z); c="$K get crd ${k}s.deploy.example.com"; eq "$($c -o jsonpath='{.spec.scope}' 2>/dev/null)" Namespaced && s=$($c -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec}'); echo "$s" | grep -q '"minimum":1' && echo "$s" | grep -q '"maximum":10' && echo "$s" | grep -q '"required":\[' """),
+  C("CRD exists, Namespaced, replicas has min/max and is required", 2, r"""s=""; k=$(echo %{kind} | tr A-Z a-z); c="$K get crd ${k}s.deploy.example.com"; eq "$($c -o jsonpath='{.spec.scope}' 2>/dev/null)" Namespaced && s=$($c -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec}'); echo "$s" | grep -q '"minimum":1' && echo "$s" | grep -q '"maximum":10' && echo "$s" | grep -q '"required":\[' """),
   C("good created with environment %{env}", 1, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com good '{.spec.environment}')" %{env}"""),
   C("replicas 0 is rejected and the message was saved", 2, r"""grep -qiE 'minimum|greater than|>= 1|invalid' "$WS/q106/rejected.txt" 2>/dev/null"""),
   C("defaulting works: defaulted.spec.environment == dev", 2, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com defaulted '{.spec.environment}')" dev"""),
@@ -191,7 +211,9 @@ Write the new `apiserver` certificate's expiry (as printed by `kubeadm certs che
   C("apiserver certificate was renewed (newer notBefore)", 3, r"""b=$(sed 's/notBefore=//' "$WS/.q122-before"); a=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' | sed 's/notBefore=//'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""),
   C("all components restarted and Running with the new certs (apiserver serving cert matches file)", 3, r"""
 for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done
-srv=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial'); f=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial'); [ -n "$srv" ] && eq "$srv" "$f" """),
+srv=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial -startdate'); f=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial -startdate'); [ -n "$srv" ] && eq "$srv" "$f" || exit 1
+# ...and it is the RENEWED certificate: an untouched cluster also serves the file it has on disk
+b=$(sed 's/notBefore=//' "$WS/.q122-before"); a=$(echo "$srv" | sed -n 's/notBefore=//p'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""),
   C("admin.conf uses the renewed client cert", 2, r"""nx "$CP" 'grep client-certificate-data /etc/kubernetes/admin.conf | awk "{print \$2}" | base64 -d | openssl x509 -noout -startdate' | sed 's/notBefore=//' > /tmp/q122a; b=$(sed 's/notBefore=//' "$WS/.q122-before"); [ "$(date -d "$(cat /tmp/q122a)" +%s)" -gt "$(date -d "$b" +%s)" ]"""),
   C("expiry.txt matches kubeadm's report for apiserver", 1, r"""exp=$(nx "$CP" 'kubeadm certs check-expiration 2>/dev/null' | awk '$1=="apiserver"{print $2,$3,$4,$5,$6}'); [ -n "$exp" ] && grep -qF "$exp" "$WS/q122/expiry.txt" 2>/dev/null"""),
  ],
@@ -291,9 +313,15 @@ $K -n %{ns} set image deploy %{dep} nginx=nginx:1.27 >/dev/null; $K -n %{ns} rol
 $K -n %{ns} set image deploy %{dep} nginx=nginx:9.99-broken >/dev/null
 """,
  "checks": [
-  C("running pods use nginx:1.25 and 3 are available", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 && for i in $(j -n %{ns} pod -l app=%{dep} '{.items[*].spec.containers[0].image}'); do eq "$i" nginx:1.25 || exit 1; done"""),
+  C("running pods use nginx:1.25 and 3 are available", 3, r"""
+for t in $(seq 1 15); do   # pods of the replaced ReplicaSets linger for a moment while they terminate
+  ok=1; eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 || ok=0
+  for i in $(j -n %{ns} pod -l app=%{dep} '{.items[*].spec.containers[0].image}'); do eq "$i" nginx:1.25 || ok=0; done
+  [ $ok = 1 ] && exit 0; sleep 2
+done; exit 1"""),
   C("deployment is paused with template image nginx:1.27 and RELEASE=next", 2, r"""d="$K -n %{ns} get deploy %{dep}"; eq "$($d -o jsonpath='{.spec.paused}' 2>/dev/null)" true && eq "$($d -o jsonpath='{.spec.template.spec.containers[0].image}')" nginx:1.27 && eq "$($d -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="RELEASE")].value}')" next"""),
-  C("no ReplicaSet exists for the paused template (1.27 + RELEASE)", 2, r"""! $K -n %{ns} get rs -o json 2>/dev/null | python3 -c '
+  C("no ReplicaSet exists for the paused template (1.27 + RELEASE)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1
+! $K -n %{ns} get rs -o json 2>/dev/null | python3 -c '
 import sys,json
 for rs in json.load(sys.stdin)["items"]:
     c=rs["spec"]["template"]["spec"]["containers"][0]
@@ -307,44 +335,99 @@ sys.exit(1)'"""),
  "task": CTX + """
 Create pod `%{pod}` in `%{ns}`:
 - a **native sidecar** (an init container with `restartPolicy: Always`) named `agent` (busybox:1.36)
-  that appends `$(date)` to `/var/log/agent.log` every second,
+  that appends `$(date)` to `/var/log/agent/agent.log` every second,
 - main container `app` (nginx:1.27) with a readiness probe (HTTP GET / on 80, period 5s)
   and a liveness probe (TCP 80),
-- shared `emptyDir` `logs` mounted at `/var/log` in both,
+- shared `emptyDir` `logs` mounted at `/var/log/agent` in both (not at `/var/log`: nginx keeps its
+  own logs under `/var/log/nginx`, and an emptyDir mounted over `/var/log` hides that directory),
 - `terminationGracePeriodSeconds: 5`.
 When it is Running, write the names of **all** containers of the pod (init and regular), sorted,
 one per line, to `{WS}/q109/containers.txt`, and the pod's `.status.podIP` to `{WS}/q109/ip.txt`.
 """,
- "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q109"; mkdir -p "$WS/q109"; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""",
+ "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q109"; mkdir -p "$WS/q109"; $K -n %{ns} delete pod %{pod} --ignore-not-found --force --grace-period=0 --wait=true >/dev/null 2>&1""",
  "checks": [
   C("native sidecar: initContainer agent with restartPolicy Always", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.spec.initContainers[?(@.name=="agent")].restartPolicy}')" Always"""),
   C("probes on app as specified", 2, r"""c='{.spec.containers[?(@.name=="app")]'; eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.httpGet.path}")" / && eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.periodSeconds}")" 5 && eq "$(j -n %{ns} pod %{pod} "$c.livenessProbe.tcpSocket.port}")" 80"""),
-  C("Running, shared log written by agent, grace period 5", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.terminationGracePeriodSeconds}')" 5 && [ -n "$($K -n %{ns} exec %{pod} -c app -- sh -c 'tail -1 /var/log/agent.log' 2>/dev/null)" ]"""),
+  C("Running, shared log written by agent, grace period 5", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.terminationGracePeriodSeconds}')" 5 && [ -n "$($K -n %{ns} exec %{pod} -c app -- sh -c 'tail -1 /var/log/agent/agent.log' 2>/dev/null)" ]"""),
-  C("report files exact", 2, r"""want=$( (j -n %{ns} pod %{pod} '{.spec.initContainers[*].name}'; echo; j -n %{ns} pod %{pod} '{.spec.containers[*].name}') | tr ' ' '\n' | grep . | sort); [ "$(grep . "$WS/q109/containers.txt" 2>/dev/null | sort)" = "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q109/ip.txt" 2>/dev/null)" "$(j -n %{ns} pod %{pod} '{.status.podIP}')" """),
+  C("report files exact", 2, r"""want=$( (j -n %{ns} pod %{pod} '{.spec.initContainers[*].name}'; echo; j -n %{ns} pod %{pod} '{.spec.containers[*].name}') | tr ' ' '\n' | grep . | sort); [ -n "$want" ] && [ "$(grep . "$WS/q109/containers.txt" 2>/dev/null | sort)" = "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q109/ip.txt" 2>/dev/null)" "$(j -n %{ns} pod %{pod} '{.status.podIP}')" """),
  ],
 },
 {
- "id": 110, "domain": "WORK", "difficulty": "hard", "title": "Cluster audit to exact files",
+ # Scoped to its own namespace: the cluster-wide version compared a file written mid-exam with the
- "params": lambda r: {"ctx": "k8s-c1-H"},
+ # whole cluster at submit time, so every pod another task (or a grader probe) created or deleted
+ # in between failed it. killer.sh-style exact-output tasks work on data the task owns.
+ "id": 110, "domain": "WORK", "difficulty": "hard", "title": "Namespace audit to exact files",
+ "params": lambda r: {"ctx": "k8s-c1-H", "ns": ns_pick(r, "audit")},
  "task": CTX + """
-Produce three files, contents **exactly** as specified (no headers, sorted with `sort`, one per line):
+Namespace `%{ns}` runs a mix of workloads. Produce three files with contents **exactly** as
-1. `{WS}/q110/no-readiness.txt` — `<namespace>/<pod>` for every pod in the cluster whose first
+specified — no headers, one entry per line, sorted with `sort`:
-   container has **no** readiness probe. Exclude namespace `kube-system`.
+1. `{WS}/q110/no-readiness.txt` — the name of every Pod in `%{ns}` whose **first** container has
-2. `{WS}/q110/per-node.txt` — `<node> <count>` of Running pods per node, all namespaces.
+   no readiness probe (whatever the Pod's phase).
-3. `{WS}/q110/images.txt` — every distinct container image running in namespace `kube-system`.
+2. `{WS}/q110/per-node.txt` — `<node> <count>`: how many Pods in `%{ns}` are **Running** on each
+   node, listing only nodes that have at least one.
+3. `{WS}/q110/images.txt` — every distinct container image used by Pods in `kube-system`.
+""",
+ "setup": NS + r"""
+mkns %{ns}; rm -rf "$WS/q110"; mkdir -p "$WS/q110"
+$K -n %{ns} delete pod --all --force --grace-period=0 --wait=true >/dev/null 2>&1
+$K apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: Pod
+metadata: {name: web-a, namespace: %{ns}}
+spec:
+  nodeSelector: {kubernetes.io/hostname: $W1}
+  containers: [{name: web, image: nginx:1.27, readinessProbe: {httpGet: {path: /, port: 80}}}]
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: web-b, namespace: %{ns}}
+spec:
+  nodeSelector: {kubernetes.io/hostname: $W2}
+  containers: [{name: web, image: nginx:1.27, readinessProbe: {tcpSocket: {port: 80}}}]
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: batch-a, namespace: %{ns}}
+spec:
+  nodeSelector: {kubernetes.io/hostname: $W1}
+  containers: [{name: batch, image: busybox:1.36, command: [sleep, "86400"]}]
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: cache-a, namespace: %{ns}}
+spec:
+  nodeSelector: {kubernetes.io/hostname: $W3}
+  containers:
+    - {name: cache, image: redis:7-alpine}
+    - {name: exporter, image: nginx:1.27, readinessProbe: {httpGet: {path: /, port: 80}}}
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: once-a, namespace: %{ns}}
+spec:
+  restartPolicy: Never
+  nodeSelector: {kubernetes.io/hostname: $W3}
+  containers: [{name: job, image: busybox:1.36, command: ["true"]}]
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: idle-a, namespace: %{ns}}
+spec:
+  nodeSelector: {kubernetes.io/hostname: $W2}
+  containers: [{name: idle, image: busybox:1.36, command: [sleep, "86400"]}]
+EOF
+for i in $(seq 1 60); do [ "$($K -n %{ns} get pod once-a -o jsonpath='{.status.phase}' 2>/dev/null)" = Succeeded ] && break; sleep 2; done
 """,
- "setup": r"""rm -rf "$WS/q110"; mkdir -p "$WS/q110" """,
  "checks": [
-  C("no-readiness.txt exact", 3, r"""want=$($K get pods -A -o json 2>/dev/null | python3 -c '
+  C("no-readiness.txt exact (first container only, every phase)", 3, r"""want=$($K -n %{ns} get pods -o json 2>/dev/null | python3 -c '
 import sys,json
 for p in json.load(sys.stdin)["items"]:
-    if p["metadata"]["namespace"]=="kube-system": continue
+    if not p["spec"]["containers"][0].get("readinessProbe"): print(p["metadata"]["name"])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q110/no-readiness.txt" 2>/dev/null)" = "$want" ]"""),
-    if not p["spec"]["containers"][0].get("readinessProbe"): print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort); [ "$(grep . "$WS/q110/no-readiness.txt" 2>/dev/null | sort)" = "$want" ]"""),
+  C("per-node.txt exact (Running pods only)", 2, r"""want=$($K -n %{ns} get pods -o json 2>/dev/null | python3 -c '
-  C("per-node.txt exact", 2, r"""want=$($K get pods -A -o json 2>/dev/null | python3 -c '
 import sys,json,collections
 c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"] if p["status"]["phase"]=="Running" and p["spec"].get("nodeName"))
-for n in sorted(c): print(n, c[n])'); [ "$(grep . "$WS/q110/per-node.txt" 2>/dev/null | sort)" = "$(echo "$want" | sort)" ]"""),
+for n in c: print(n, c[n])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q110/per-node.txt" 2>/dev/null)" = "$want" ]"""),
-  C("images.txt exact", 2, r"""want=$(j -n kube-system pods '{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u); [ "$(grep . "$WS/q110/images.txt" 2>/dev/null | sort -u)" = "$want" ]"""),
+  C("images.txt exact", 2, r"""want=$(j -n kube-system pods '{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u); [ -n "$want" ] && [ "$(grep . "$WS/q110/images.txt" 2>/dev/null)" = "$want" ]"""),
  ],
 },
 {
@@ -361,7 +444,7 @@ by `kubectl config view --minify`, to `{WS}/q121/ns.txt`.
 """,
  "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q121"; mkdir -p "$WS/q121"; $K -n %{ns} delete deploy,svc %{dep} --ignore-not-found >/dev/null; $K -n default delete deploy,svc %{dep} --ignore-not-found >/dev/null""",
  "checks": [
-  C("Deployment %{dep} in %{ns} (the context's namespace), 2 available, owner=ac", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && eq "$(j -n %{ns} deploy %{dep} '{.spec.template.metadata.labels.owner}')" ac"""),
+  C("Deployment %{dep} in %{ns} (the context's namespace), 2 available, owner=ac", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && { eq "$(j -n %{ns} deploy %{dep} '{.metadata.labels.owner}')" ac || eq "$(j -n %{ns} deploy %{dep} '{.spec.template.metadata.labels.owner}')" ac; }"""),
   C("Service %{dep} port 9000 in the same namespace", 1, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].port}')" 9000"""),
   C("nothing was created in namespace default by mistake", 1, r"""! $K -n default get deploy %{dep} >/dev/null 2>&1"""),
   C("ns.txt has the namespace", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q121/ns.txt" 2>/dev/null)" %{ns}"""),
@@ -461,7 +544,7 @@ ok=any(any(any(h.get("name","").lower()=="x-canary" and h.get("value")=="always"
 sys.exit(0 if ok else 1)'"""),
   C("default rule weights v1 %{w2} / v2 %{w}", 2, r"""$K -n %{ns} get httproute %{host}-route -o json 2>/dev/null | python3 -c '
 import sys,json; r=json.load(sys.stdin)["spec"]["rules"]
-ok=any(not x.get("matches") or all(not m.get("headers") for m in x["matches"]) and {b["name"]:b.get("weight") for b in x["backendRefs"]}=={"v1":%{w2},"v2":%{w}} for x in r if len(x.get("backendRefs",[]))==2)
+ok=any((not x.get("matches") or all(not m.get("headers") for m in x["matches"])) and {b["name"]:b.get("weight") for b in x["backendRefs"]}=={"v1":%{w2},"v2":%{w}} for x in r if len(x.get("backendRefs",[]))==2)
 sys.exit(0 if ok else 1)'"""),
  ],
 },
@@ -513,7 +596,18 @@ mkns %{ns}
 for n in "$W1" "$W2" "$W3"; do $K label node $n disk- >/dev/null 2>&1 || true; done
 $K label node "$W2" disk=nvme --overwrite >/dev/null
 $K -n %{ns} delete pod consumer --ignore-not-found --wait=false >/dev/null; $K -n %{ns} delete pvc %{pvc} --ignore-not-found >/dev/null
-$K delete pv lv-hdd lv-nvme --ignore-not-found >/dev/null
+# The PV names are fixed but every session draws a new namespace, so a claim from an earlier run
+# (another namespace, often with its pod still running) keeps lv-hdd/lv-nvme Bound. pv-protection
+# then holds `kubectl delete pv` forever and the seed timed out. Release them from whoever has them.
+for pv in lv-hdd lv-nvme; do
+  cns=$(j pv $pv '{.spec.claimRef.namespace}'); cn=$(j pv $pv '{.spec.claimRef.name}')
+  [ -n "$cns" ] && [ -n "$cn" ] || continue
+  for p in $($K -n "$cns" get pod -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}{end}' 2>/dev/null | awk -v c="$cn" '{for(i=2;i<=NF;i++) if($i==c) print $1}'); do
+    $K -n "$cns" delete pod "$p" --grace-period=0 --force >/dev/null 2>&1
+  done
+  $K -n "$cns" delete pvc "$cn" --ignore-not-found --wait=false >/dev/null 2>&1
+done
+$K delete pv lv-hdd lv-nvme --ignore-not-found --timeout=90s >/dev/null 2>&1 || true
 for pair in "lv-hdd:$W1:hdd:2Gi" "lv-nvme:$W2:nvme:1Gi"; do IFS=: read name node kind size <<<"$pair"
 nx "$node" "mkdir -p /mnt/$name"
 $K apply -f - >/dev/null <<EOF
@@ -576,7 +670,7 @@ echo "$pv" > "$WS/.q116-pv"
 },
 # ============================================================== TSHOOT ====
 {
- "id": 117, "domain": "TSHOOT", "breaks": "w2-kubelet", "difficulty": "hard", "title": "Worker node: three things wrong", "disruptive": True,
+ "id": 117, "domain": "TSHOOT", "breaks": ["w2-kubelet", "w2-drain"], "difficulty": "hard", "title": "Worker node: three things wrong", "disruptive": True,
  "params": lambda r: {"ctx": "k8s-c1-H"},
  "task": CTX + """
 Node `{W2}` is unusable. Make it fully usable again: `Ready`, schedulable, kubelet running and
@@ -600,7 +694,7 @@ $K -n default delete pod node-probe --wait=false >/dev/null 2>&1; exit $ok"""),
  ],
 },
 {
- "id": 118, "domain": "TSHOOT", "breaks": "scheduler", "difficulty": "hard", "title": "Two control-plane components down", "disruptive": True,
+ "id": 118, "domain": "TSHOOT", "breaks": ["scheduler", "controller-manager"], "difficulty": "hard", "title": "Two control-plane components down", "disruptive": True,
  "params": lambda r: {"ctx": "k8s-c1-H"},
  "task": CTX + """
 New Deployments create no pods, and pods that do get created stay `Pending`. Two different
@@ -673,7 +767,8 @@ j -n %{ns} deploy %{dep} '{.metadata.uid}' > "$WS/.q119-uid"; j -n %{ns} svc %{d
  "checks": [
   C("3 replicas Available", 3, r"""for i in $(seq 1 6); do [ "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" = 3 ] && exit 0; sleep 5; done; exit 1"""),
   C("Service routes: endpoints ready, targetPort 80", 2, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].targetPort}')" 80 && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""),
-  C("Deployment UID and Service ClusterIP unchanged (fixed in place)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.metadata.uid}')" "$(cat "$WS/.q119-uid")" && eq "$(j -n %{ns} svc %{dep} '{.spec.clusterIP}')" "$(cat "$WS/.q119-ip")" """),
+  C("Deployment UID and Service ClusterIP unchanged (fixed in place)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 || exit 1   # only once it is actually fixed
+eq "$(j -n %{ns} deploy %{dep} '{.metadata.uid}')" "$(cat "$WS/.q119-uid")" && eq "$(j -n %{ns} svc %{dep} '{.spec.clusterIP}')" "$(cat "$WS/.q119-ip")" """),
   C("quota still present and permits the 3 pods", 1, r"""p=$(j -n %{ns} quota tight '{.spec.hard.pods}'); [ -n "$p" ] && [ "$p" -ge 3 ]"""),
  ],
 },

tracks/cka/bank_sim_a.py

@@ -243,12 +243,15 @@ Namespace `project-orion` is close to filling its nodes.
     setup=NS + r"""
 mkns project-orion
 rm -rf "$WS/q204"; mkdir -p "$WS/q204"
-$K -n project-orion delete deploy --all >/dev/null 2>&1; $K -n project-orion delete pod orion-api-0 --ignore-not-found --wait=false >/dev/null
+# the pod is re-created below under the same name: a delete that does not wait lets the apply land on
-$K apply -f - >/dev/null <<'EOF'
+# the terminating old pod, which then vanishes and leaves the task without orion-api-0
+$K -n project-orion delete deploy --all >/dev/null 2>&1; $K -n project-orion delete pod orion-api-0 --ignore-not-found --force --grace-period=0 --wait=true >/dev/null 2>&1
+$K apply -f - >/dev/null <<EOF
 apiVersion: v1
 kind: Pod
 metadata: {name: orion-api-0, namespace: project-orion, labels: {app: orion-api}}
 spec:
+  affinity: {nodeAffinity: {requiredDuringSchedulingIgnoredDuringExecution: {nodeSelectorTerms: [{matchExpressions: [{key: kubernetes.io/hostname, operator: NotIn, values: [$W2]}]}]}}}
   containers:
     - name: api
       image: busybox:1.36
@@ -264,6 +267,7 @@ spec:
   template:
     metadata: {labels: {app: orion-batch}}
     spec:
+      affinity: {nodeAffinity: {requiredDuringSchedulingIgnoredDuringExecution: {nodeSelectorTerms: [{matchExpressions: [{key: kubernetes.io/hostname, operator: NotIn, values: [$W2]}]}]}}}
       containers: [{name: batch, image: busybox:1.36, command: [sleep, "86400"]}]
 ---
 apiVersion: apps/v1
@@ -275,6 +279,7 @@ spec:
   template:
     metadata: {labels: {app: orion-ledger}}
     spec:
+      affinity: {nodeAffinity: {requiredDuringSchedulingIgnoredDuringExecution: {nodeSelectorTerms: [{matchExpressions: [{key: kubernetes.io/hostname, operator: NotIn, values: [$W2]}]}]}}}
       containers:
         - name: ledger
           image: busybox:1.36
@@ -283,11 +288,18 @@ spec:
 EOF
 """,
     checks=[
-        C("evicted-first.txt lists exactly the BestEffort pods", 4, r"""
+        # Every BestEffort pod this task seeded must be listed and no Burstable/Guaranteed one may be.
-want=$($K -n project-orion get pod -o json 2>/dev/null | python3 -c '
+        # BestEffort pods other tasks add to the namespace later (212's spread-check has no
+        # resources at all) are tolerated either way - they did not exist when the file was written.
+        C("evicted-first.txt lists the BestEffort pods and nothing else", 4, r"""
+f="$WS/q204/evicted-first.txt"; [ -s "$f" ] || exit 1
+$K -n project-orion get pod -o json 2>/dev/null | python3 -c '
 import sys,json
-print("\n".join(sorted(p["metadata"]["name"] for p in json.load(sys.stdin)["items"] if p["status"].get("qosClass")=="BestEffort")))')
+items=json.load(sys.stdin)["items"]; got={l.strip() for l in open(sys.argv[1]) if l.strip()}
-[ -n "$want" ] && [ "$(grep . "$WS/q204/evicted-first.txt" 2>/dev/null | sort)" = "$want" ]"""),
+qos={p["metadata"]["name"]:p["status"].get("qosClass") for p in items}
+seeded={p["metadata"]["name"] for p in items if p["metadata"].get("labels",{}).get("app")=="orion-batch"}
+ok=seeded and seeded<=got and not any(qos.get(n) in ("Burstable","Guaranteed") for n in got)
+sys.exit(0 if ok else 1)' "$f" """),
         C("api-qos.txt says Burstable", 2, r"""grep -qi burstable "$WS/q204/api-qos.txt" 2>/dev/null"""),
         C("guaranteed.txt names orion-ledger", 2, r"""grep -q 'orion-ledger' "$WS/q204/guaranteed.txt" 2>/dev/null"""),
     ],
@@ -504,6 +516,11 @@ chmod +x {WS}/q207/usage-nodes.sh {WS}/q207/usage-pods.sh
 )
 
 # ------------------------------------------------------------------ 208 ----
+# Own namespace (project-vega): it used to share project-atlas with 203, whose "running pods"
+# count then dropped by one when this drain deleted the bare pod. And the grader checks the pods
+# that were on the node when it was seeded, not "nothing on the node at grading time" - after the
+# required uncordon the scheduler is free to put new pods there (212's one-per-node spread always
+# does), which failed this task no matter what the candidate did.
 task(
     id=208, domain="ARCH", title="Drain a node for maintenance", breaks="w2-drain",
     task="""
@@ -512,23 +529,28 @@ Node `{W2}` needs kernel maintenance.
 1. Evict every non-DaemonSet Pod from it. Pods with `emptyDir` volumes must not stop you,
    and neither must Pods that aren't managed by a controller.
 2. Write the exact command you used to `{WS}/q208/drain.sh`.
-3. Deployment `atlas-web` in Namespace `project-atlas` must still have all 3 replicas Available afterwards.
+3. Deployment `vega-web` in Namespace `project-vega` must still have all 3 replicas Available afterwards.
 4. When you're done, make the node schedulable again.
+
+A Pod without a controller is **deleted** by a forced drain, not rescheduled. Before you drain,
+look at what else runs on the node (`k get pods -A -o wide --field-selector spec.nodeName={W2}`) —
+anything you created there for another task has to be recreated afterwards.
 """,
     setup=NS + r"""
-mkns project-atlas
+mkns project-vega
 rm -rf "$WS/q208"; mkdir -p "$WS/q208"
-$K -n project-atlas delete deploy atlas-web --ignore-not-found >/dev/null
+$K -n project-vega delete deploy vega-web --ignore-not-found >/dev/null
+$K -n project-vega delete pod vega-orphan --ignore-not-found --force --grace-period=0 --wait=true >/dev/null 2>&1
 $K uncordon "$W2" >/dev/null 2>&1 || true
 $K apply -f - >/dev/null <<EOF
 apiVersion: apps/v1
 kind: Deployment
-metadata: {name: atlas-web, namespace: project-atlas}
+metadata: {name: vega-web, namespace: project-vega}
 spec:
   replicas: 3
-  selector: {matchLabels: {app: atlas-web}}
+  selector: {matchLabels: {app: vega-web}}
   template:
-    metadata: {labels: {app: atlas-web}}
+    metadata: {labels: {app: vega-web}}
     spec:
       containers:
         - name: web
@@ -536,33 +558,41 @@ spec:
           resources: {requests: {cpu: 10m, memory: 16Mi}}
           volumeMounts: [{name: scratch, mountPath: /scratch}]
       volumes: [{name: scratch, emptyDir: {}}]
+      # at least one replica on the node being drained, so the emptyDir flag really matters
+      topologySpreadConstraints: [{maxSkew: 1, topologyKey: kubernetes.io/hostname, whenUnsatisfiable: ScheduleAnyway, labelSelector: {matchLabels: {app: vega-web}}}]
 EOF
-$K -n project-atlas rollout status deploy atlas-web --timeout=120s >/dev/null 2>&1
+$K -n project-vega rollout status deploy vega-web --timeout=120s >/dev/null 2>&1
-$K -n project-atlas run atlas-orphan --image=busybox:1.36 --restart=Never --overrides="{\"spec\":{\"nodeName\":\"$W2\"}}" -- sleep 86400 >/dev/null 2>&1
+$K -n project-vega run vega-orphan --image=busybox:1.36 --restart=Never --overrides="{\"spec\":{\"nodeName\":\"$W2\"}}" -- sleep 86400 >/dev/null 2>&1
+$K -n project-vega wait --for=condition=Ready pod/vega-orphan --timeout=90s >/dev/null 2>&1
+# what the drain has to clear: every non-DaemonSet, non-mirror, non-kube-system pod on the node now
+$K get pods -A -o json | python3 -c '
+import sys,json
+n=sys.argv[1]
+for p in json.load(sys.stdin)["items"]:
+    m=p["metadata"]; o=(m.get("ownerReferences") or [{}])[0].get("kind")
+    if p["spec"].get("nodeName")==n and m["namespace"]!="kube-system" and o not in ("DaemonSet","Node"):
+        print(m["uid"])' "$W2" > "$WS/.q208-uids"
 """,
     checks=[
-        C("no non-DaemonSet workload pods remain on the node", 4, r"""
+        C("every pod that was on the node when it was seeded has been evicted", 4, r"""
-left=$($K get pods -A -o json 2>/dev/null | python3 -c '
+[ -s "$WS/.q208-uids" ] || exit 1
-import sys,json,os
+now=$($K get pods -A -o jsonpath='{range .items[*]}{.metadata.uid}{" "}{.spec.nodeName}{"\n"}{end}' 2>/dev/null)
-n=os.environ["CKA_W2"]
+while read -r u; do echo "$now" | grep -q "^$u $W2\$" && exit 1; done < "$WS/.q208-uids"
-print(sum(1 for p in json.load(sys.stdin)["items"]
+! $K -n project-vega get pod vega-orphan >/dev/null 2>&1"""),
-  if p["spec"].get("nodeName")==n and p["status"]["phase"]=="Running"
-  and (p["metadata"].get("ownerReferences") or [{}])[0].get("kind") not in ("DaemonSet",)
-  and p["metadata"]["namespace"] not in ("kube-system",)))')
-eq "$left" 0"""),
         C("drain.sh records a drain with --ignore-daemonsets and --delete-emptydir-data", 2, r"""
 f="$WS/q208/drain.sh"; grep -q 'drain' "$f" 2>/dev/null && grep -q -- '--ignore-daemonsets' "$f" && grep -q -- '--delete-emptydir-data' "$f" """),
-        C("atlas-web still has 3 replicas Available", 2, r"""eq "$(j -n project-atlas deploy atlas-web '{.status.availableReplicas}')" 3"""),
+        C("vega-web still has 3 replicas Available", 2, r"""eq "$(j -n project-vega deploy vega-web '{.status.availableReplicas}')" 3"""),
         C("node is schedulable again", 2, r"""[ "$(j node "$W2" '{.spec.unschedulable}')" != true ]"""),
     ],
     solution="""
+k get pods -A -o wide --field-selector spec.nodeName={W2}      # what is about to be evicted or deleted
 k drain {W2} --ignore-daemonsets --delete-emptydir-data --force
 #   --ignore-daemonsets      DaemonSet pods can't be evicted, skip them
-#   --delete-emptydir-data   required because atlas-web mounts an emptyDir
+#   --delete-emptydir-data   required because vega-web mounts an emptyDir (its data is lost)
-#   --force                  required for the bare pod atlas-orphan (no controller)
+#   --force                  required for the bare pod vega-orphan - which is DELETED, not moved
 echo 'kubectl drain {W2} --ignore-daemonsets --delete-emptydir-data --force' > {WS}/q208/drain.sh
-k get pods -A -o wide --field-selector spec.nodeName={W2}
+k get pods -A -o wide --field-selector spec.nodeName={W2}      # only DaemonSet pods left
-k -n project-atlas rollout status deploy atlas-web
+k -n project-vega rollout status deploy vega-web
 k uncordon {W2}
 """,
 )
@@ -657,7 +687,7 @@ k -n project-atlas auth can-i get    secret --as=system:serviceaccount:project-a
 
 # ------------------------------------------------------------------ 211 ----
 task(
-    id=211, domain="WORK", title="DaemonSet on every node",
+    id=211, domain="WORK", title="DaemonSet on every node", breaks=["w1-taint", "w1-block"],
     task="""
 In Namespace `project-orion` create a DaemonSet `node-collector`:
 
@@ -697,7 +727,7 @@ k -n project-orion get ds node-collector -o wide
 
 # ------------------------------------------------------------------ 212 ----
 task(
-    id=212, domain="WORK", title="One Pod per node, on purpose",
+    id=212, domain="WORK", title="One Pod per node, on purpose", breaks=["w1-taint", "w1-block"],
     task="""
 In Namespace `project-orion` create Deployment `spread-check`:
 
@@ -722,7 +752,7 @@ echo "$t" | grep -q '"topologyKey":"kubernetes.io/hostname"' && { echo "$t" | gr
         C("exactly 3 Pods Running, one per node, 1 Pending", 3, r"""
 r=$($K -n project-orion get pod -l id=spread-check --no-headers 2>/dev/null | grep -c Running)
 p=$($K -n project-orion get pod -l id=spread-check --no-headers 2>/dev/null | grep -c Pending)
-eq "$r" 3 && eq "$p" 1 && [ "$(j -n project-orion pod -l id=spread-check '{.items[*].spec.nodeName}' | tr ' ' '\n' | grep . | sort -u | wc -l)" = 3"""),
+eq "$r" 3 && eq "$p" 1 && [ "$(j -n project-orion pod -l id=spread-check '{.items[*].spec.nodeName}' | tr ' ' '\n' | grep . | sort -u | wc -l)" = 3 ]"""),
         C("pending.txt names the Pending pod", 2, r"""
 want=$($K -n project-orion get pod -l id=spread-check --no-headers 2>/dev/null | awk '$3=="Pending"{print $1}')
 [ -n "$want" ] && grep -qF "$want" "$WS/q212/pending.txt" 2>/dev/null"""),

tracks/cka/bank_sim_b.py

@@ -233,14 +233,16 @@ k -n project-token get secret reporter-legacy -o yaml     # the controller fills
 )
 
 # ------------------------------------------------------------------ 305 ----
+# {W2}, not {W3}: 309 drains {W3} in this same set, and a drain without --force refuses to touch
+# bare Pods - while one with --force deletes them, taking this task's answer with it.
 task(
     id=305, domain="WORK", title="Schedule without the scheduler",
     task="""
 In Namespace `project-orion`:
 
 1. Create Pod `direct-assign` (image `busybox:1.36`, command `sleep 86400`) that runs on node
-   `{W3}` **without the scheduler being involved at all**.
+   `{W2}` **without the scheduler being involved at all**.
-2. Create Pod `scheduled` (same image and command) that lands on `{W3}` **through** the scheduler.
+2. Create Pod `scheduled` (same image and command) that lands on `{W2}` **through** the scheduler.
 3. Write to `{WS}/q305/proof.txt` the value of the field you used for Pod 1, in the form
    `field=<name>`, and on a second line `scheduler=<the schedulerName of the second Pod>`.
 """,
@@ -248,16 +250,26 @@ In Namespace `project-orion`:
 mkns project-orion
 rm -rf "$WS/q305"; mkdir -p "$WS/q305"
 $K -n project-orion delete pod direct-assign scheduled --ignore-not-found --wait=false >/dev/null
-$K label node "$W3" tier- >/dev/null 2>&1 || true
 """,
+    # Who set spec.nodeName is recorded permanently in managedFields: a client that submits it
+    # (kubectl create/apply/run) owns f:nodeName, while a scheduler binding goes through the
+    # pods/binding subresource and leaves no managedFields entry for it (verified on 1.37). Events
+    # were the old signal, but plain `get events` has no SOURCE column and they expire after 1h.
     checks=[
-        C("direct-assign Running on the right node with spec.nodeName set", 3, r"""
+        C("direct-assign Running on {W2}", 3, r"""
-eq "$(j -n project-orion pod direct-assign '{.status.phase}')" Running && eq "$(j -n project-orion pod direct-assign '{.spec.nodeName}')" "$W3" """),
+eq "$(j -n project-orion pod direct-assign '{.status.phase}')" Running && eq "$(j -n project-orion pod direct-assign '{.spec.nodeName}')" "$W2" """),
-        C("direct-assign was never scheduled (no Scheduled event from default-scheduler)", 3, r"""
+        C("direct-assign bypassed the scheduler (spec.nodeName came from your manifest)", 3, r"""
-! $K -n project-orion get events --field-selector involvedObject.name=direct-assign 2>/dev/null | grep -q 'default-scheduler'"""),
+$K -n project-orion get pod direct-assign -o json --show-managed-fields 2>/dev/null | python3 -c '
-        C("scheduled Running on the same node via the default scheduler", 3, r"""
+import sys,json
-eq "$(j -n project-orion pod scheduled '{.status.phase}')" Running && eq "$(j -n project-orion pod scheduled '{.spec.nodeName}')" "$W3" \
+mf=json.load(sys.stdin)["metadata"].get("managedFields",[])
- && $K -n project-orion get events --field-selector involvedObject.name=scheduled 2>/dev/null | grep -q 'default-scheduler'"""),
+sys.exit(0 if any("f:nodeName" in json.dumps(m.get("fieldsV1",{})) for m in mf if m.get("subresource","")!="status") else 1)'"""),
+        C("scheduled Running on {W2}, placed there by default-scheduler (no nodeName in its manifest)", 3, r"""
+eq "$(j -n project-orion pod scheduled '{.status.phase}')" Running && eq "$(j -n project-orion pod scheduled '{.spec.nodeName}')" "$W2" \
+ && eq "$(j -n project-orion pod scheduled '{.spec.schedulerName}')" default-scheduler || exit 1
+$K -n project-orion get pod scheduled -o json --show-managed-fields 2>/dev/null | python3 -c '
+import sys,json
+mf=json.load(sys.stdin)["metadata"].get("managedFields",[])
+sys.exit(1 if any("f:nodeName" in json.dumps(m.get("fieldsV1",{})) for m in mf) else 0)'"""),
         C("proof.txt names nodeName and default-scheduler", 2, r"""
 grep -q 'field= *nodeName' "$WS/q305/proof.txt" 2>/dev/null && grep -q 'scheduler= *default-scheduler' "$WS/q305/proof.txt" """),
     ],
@@ -267,7 +279,7 @@ apiVersion: v1
 kind: Pod
 metadata: {name: direct-assign, namespace: project-orion}
 spec:
-  nodeName: {W3}            # bypasses the scheduler entirely - the kubelet just picks it up
+  nodeName: {W2}            # bypasses the scheduler entirely - the kubelet just picks it up
   containers: [{name: c, image: busybox:1.36, command: [sleep, "86400"]}]
 EOF
 cat <<EOF | k apply -f -
@@ -275,12 +287,15 @@ apiVersion: v1
 kind: Pod
 metadata: {name: scheduled, namespace: project-orion}
 spec:
-  nodeSelector: {kubernetes.io/hostname: {W3}}     # the scheduler still makes the decision
+  nodeSelector: {kubernetes.io/hostname: {W2}}     # the scheduler still makes the decision
   containers: [{name: c, image: busybox:1.36, command: [sleep, "86400"]}]
 EOF
-k -n project-orion get events --field-selector involvedObject.name=direct-assign   # no Scheduled event
+k -n project-orion describe pod scheduled | grep -A3 Events      # Scheduled  default-scheduler  Successfully assigned...
-k -n project-orion get events --field-selector involvedObject.name=scheduled       # Scheduled by default-scheduler
+k -n project-orion describe pod direct-assign | grep -A3 Events  # no Scheduled event at all
 printf 'field=nodeName\\nscheduler=default-scheduler\\n' > {WS}/q305/proof.txt
+# `k get events` hides who reported an event; -o wide (or describe) shows the SOURCE column.
+# Events expire after an hour, so for lasting proof look at who owns spec.nodeName:
+#   k -n project-orion get pod direct-assign -o yaml --show-managed-fields | grep -B2 -A2 nodeName
 """,
 )
 
@@ -294,8 +309,13 @@ In Namespace `project-orbit`:
    `volumeBindingMode: WaitForFirstConsumer`, and allowing volume expansion.
 2. Create PVC `growing` (1Gi, RWO) using that class and Pod `grower` (busybox:1.36, `sleep 86400`)
    mounting it at `/data`, so the claim binds.
-3. Then grow the claim to **2Gi** without deleting it, and write the claim's
+3. Then grow the claim to **2Gi** without deleting it, and write both sizes the claim reports
-   `.status.capacity.storage` after the resize to `{WS}/q306/capacity.txt`.
+   afterwards to `{WS}/q306/capacity.txt`, one per line:
+   `requested=<.spec.resources.requests.storage>` and `status=<.status.capacity.storage>`.
+
+(The local-path provisioner has no volume resizer, so on this cluster the status will not catch up
+with the request — that gap is what step 3 asks you to observe. With a CSI driver that supports
+expansion, the status would follow within seconds.)
 """,
     setup=NS + r"""
 mkns project-orbit
@@ -312,10 +332,17 @@ eq "$($s -o jsonpath='{.provisioner}' 2>/dev/null)" rancher.io/local-path && eq
         C("PVC growing Bound on that class with grower Running", 3, r"""
 eq "$(j -n project-orbit pvc growing '{.spec.storageClassName}')" expandable && eq "$(j -n project-orbit pvc growing '{.status.phase}')" Bound \
  && eq "$(j -n project-orbit pod grower '{.status.phase}')" Running"""),
-        C("claim requests 2Gi and reports at least 2Gi capacity", 4, r"""
+        # local-path cannot resize (verified: ExternalExpanding, capacity never moves), so a bound PV
+        # that is still 1Gi under a claim requesting 2Gi is exactly what "grown in place" looks like
+        # here. A claim deleted and recreated at 2Gi would have been provisioned a 2Gi volume instead.
+        C("claim grown in place: requests 2Gi on the original 1Gi volume", 4, r"""
 eq "$(j -n project-orbit pvc growing '{.spec.resources.requests.storage}')" 2Gi || exit 1
-for i in $(seq 1 12); do c=$(j -n project-orbit pvc growing '{.status.capacity.storage}'); [ "$c" = 2Gi ] && exit 0; sleep 5; done; exit 1"""),
+pv=$(j -n project-orbit pvc growing '{.spec.volumeName}'); [ -n "$pv" ] || exit 1
-        C("capacity.txt records the new capacity", 1, r"""grep -q '2Gi' "$WS/q306/capacity.txt" 2>/dev/null"""),
+eq "$(j pv "$pv" '{.spec.capacity.storage}')" 1Gi"""),
+        C("capacity.txt records requested and status sizes as the cluster reports them", 1, r"""
+f="$WS/q306/capacity.txt"
+eq "$(sed -n 's/^ *requested *= *//p' "$f" 2>/dev/null | tr -d '[:space:]')" 2Gi \
+ && eq "$(sed -n 's/^ *status *= *//p' "$f" | tr -d '[:space:]')" "$(j -n project-orbit pvc growing '{.status.capacity.storage}')" """),
     ],
     solution="""
 cat <<EOF | k apply -f -
@@ -342,15 +369,19 @@ spec:
   containers: [{name: c, image: busybox:1.36, command: [sleep, "86400"], volumeMounts: [{name: d, mountPath: /data}]}]
 EOF
 k -n project-orbit patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
-k -n project-orbit get pvc growing -w        # spec 2Gi first, then status.capacity follows
+k -n project-orbit describe pvc growing      # event ExternalExpanding: waiting for an external controller
-k -n project-orbit get pvc growing -o jsonpath='{.status.capacity.storage}' > {WS}/q306/capacity.txt
+k -n project-orbit get pvc growing -o jsonpath='requested={.spec.resources.requests.storage}{"\\n"}status={.status.capacity.storage}{"\\n"}' > {WS}/q306/capacity.txt
-# Shrinking is never allowed. Without allowVolumeExpansion the patch is rejected outright.
+cat {WS}/q306/capacity.txt                   # requested=2Gi  status=1Gi
+# allowVolumeExpansion is the gate: without it the API rejects the patch outright, and shrinking is
+# never allowed. Accepting the request is all the API does - a resizer (the CSI driver's
+# external-resizer sidecar) then grows the volume and the kubelet the filesystem, and only then does
+# .status.capacity change. local-path has no resizer, so here the status stays at 1Gi forever.
 """,
 )
 
 # ------------------------------------------------------------------ 307 ----
 task(
-    id=307, domain="TSHOOT", title="Node accepts nothing new", breaks="w1-block",
+    id=307, domain="TSHOOT", title="Node accepts nothing new", breaks=["w1-block", "w1-taint"],
     task="""
 Node `{W1}` is `Ready` and its kubelet is healthy, but no new Pod will schedule onto it.
 
@@ -451,6 +482,10 @@ at all times.
 2. Drain `{W3}` (DaemonSet Pods aside), then make it schedulable again.
 3. Write the value of the PDB's `.status.disruptionsAllowed` after your change to
    `{WS}/q309/allowed.txt`.
+
+Look before you drain: `k get pods -A -o wide --field-selector spec.nodeName={W3}`. A Pod without a
+controller blocks a plain drain, and `--force` deletes it for good — if it belongs to another
+task, recreate it once the node is back.
 """,
     setup=NS + r"""
 mkns project-atlas
@@ -498,6 +533,7 @@ k -n project-atlas get pdb atlas-api
 #   a drain would hang forever waiting for the eviction API.
 k -n project-atlas patch pdb atlas-api -p '{"spec":{"minAvailable":1}}'
 k -n project-atlas get pdb atlas-api        # ALLOWED DISRUPTIONS: 1
+k get pods -A -o wide --field-selector spec.nodeName={W3}    # bare pods here? they need --force and are then gone
 k drain {W3} --ignore-daemonsets --delete-emptydir-data
 k -n project-atlas rollout status deploy atlas-api
 k uncordon {W3}
@@ -583,24 +619,33 @@ Then write the number of **distinct** images in use across those Deployments to
 """,
     setup=r"""rm -rf "$WS/q311"; mkdir -p "$WS/q311" """,
     checks=[
-        C("images.txt matches the cluster exactly", 5, r"""
+        # Graded against the Deployments that existed when the file was written (its mtime): other
-want=$($K get deploy -A -o json 2>/dev/null | python3 -c '
+        # set-B tasks create Deployments later (303's cidr-demo), and an exact cluster-wide
-import sys,json
+        # comparison at submit time failed a correct answer whenever one of them came after this.
-out=[]
+        C("images.txt matches the cluster as it was when you wrote it", 5, r"""
+f="$WS/q311/images.txt"; [ -s "$f" ] || exit 1
+$K get deploy -A -o json 2>/dev/null | python3 -c '
+import sys,json,os,datetime
+f=sys.argv[1]; T=os.path.getmtime(f)
+def ts(d): return datetime.datetime.strptime(d["metadata"]["creationTimestamp"],"%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=datetime.timezone.utc).timestamp()
+want=set(); later=set()
 for d in json.load(sys.stdin)["items"]:
-    s=d["spec"]["template"]["spec"]
+    key="%s/%s"%(d["metadata"]["namespace"],d["metadata"]["name"]); s=d["spec"]["template"]["spec"]
-    imgs=[c["image"] for c in s.get("initContainers",[])]+[c["image"] for c in s["containers"]]
+    if ts(d) > T: later.add(key); continue
-    out.append("%s/%s %s"%(d["metadata"]["namespace"],d["metadata"]["name"],",".join(imgs)))
+    want.add("%s %s"%(key,",".join([c["image"] for c in s.get("initContainers",[])]+[c["image"] for c in s["containers"]])))
-print("\n".join(sorted(out)))')
+got={l.strip() for l in open(f) if l.strip()}
-[ "$(grep . "$WS/q311/images.txt" 2>/dev/null | sort)" = "$want" ]"""),
+extra={g for g in got-want if g.split(" ")[0] in {w.split(" ")[0] for w in want}}   # a real deployment, wrong images
+sys.exit(0 if want<=got and not extra else 1)' "$f" """),
         C("distinct.txt has the right count", 3, r"""
+f="$WS/q311/images.txt"; [ -s "$f" ] || exit 1
 want=$($K get deploy -A -o json 2>/dev/null | python3 -c '
-import sys,json
+import sys,json,os,datetime
-s=set()
+T=os.path.getmtime(sys.argv[1]); s=set()
 for d in json.load(sys.stdin)["items"]:
+    if datetime.datetime.strptime(d["metadata"]["creationTimestamp"],"%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=datetime.timezone.utc).timestamp() > T: continue
     sp=d["spec"]["template"]["spec"]
     for c in sp.get("initContainers",[])+sp["containers"]: s.add(c["image"])
-print(len(s))')
+print(len(s))' "$f")
 eq "$(tr -dc '0-9' < "$WS/q311/distinct.txt" 2>/dev/null)" "$want" """),
     ],
     solution="""
@@ -784,8 +829,10 @@ for i in $(seq 1 12); do
   [ "$(j -n default pod "edge-cache-$W3" '{.status.phase}')" = Running ] && [ "$(j -n default pod "edge-cache-$W3" '{.spec.containers[0].image}')" = nginx:1.27 ] && exit 0
   sleep 5
 done; exit 1"""),
+        # -h: with two file operands GNU grep prefixes every line with the file name, which made
+        # awk's $2 the literal word "path:" and this check unpassable.
         C("the hostPath the manifest references now exists on the node", 3, r"""
-p=$(nx "$W3" 'grep -A2 hostPath /etc/kubernetes/manifests/edge-cache.yaml /etc/kubernetes/static/edge-cache.yaml 2>/dev/null | grep "path:" | head -1' | awk '{print $2}')
+p=$(nx "$W3" 'grep -h -A2 hostPath /etc/kubernetes/manifests/edge-cache.yaml /etc/kubernetes/static/edge-cache.yaml 2>/dev/null | grep "path:" | head -1' | awk '{print $2}')
 [ -n "$p" ] && nx "$W3" "test -d $p" """),
         C("reason.txt mentions the missing host directory", 2, r"""grep -Eqi 'director|hostpath|/opt/edge-cache|not exist|missing' "$WS/q314/reason.txt" 2>/dev/null"""),
     ],
@@ -916,11 +963,16 @@ EOF
 )
 
 # ------------------------------------------------------------------ 317 ----
+# The sabotage switches off ONE controller. It used to point --cluster-signing-cert-file at a
+# missing file instead, but a controller that fails to start is fatal to kube-controller-manager
+# (verified on 1.37: "Error building controllers" -> exit), so the whole KCM crash-looped and every
+# other set-B task that needs a controller (Deployments, Jobs, EndpointSlices, PV phases, PDB
+# status, token Secrets) silently stopped working until this task - the last one - was fixed.
 task(
     id=317, domain="TSHOOT", title="Certificates stop being issued", breaks="controller-manager",
     task="""
 Certificate signing requests are approved but never get a certificate: `kubectl get csr` shows
-them `Approved` with an empty `CONDITION`-issued certificate forever.
+them `Approved` and never `Approved,Issued`. Everything else in the cluster works.
 
 Find the responsible control-plane component, repair it on the control-plane node, and prove it
 works: a CertificateSigningRequest named `issue-probe` (CN `probe`, signer
@@ -931,14 +983,14 @@ receive a certificate.
 $K delete csr issue-probe --ignore-not-found >/dev/null 2>&1
 nx "$CP" '
 f=/etc/kubernetes/manifests/kube-controller-manager.yaml
-grep -q "cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt" $f && sed -i "s|--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt|--cluster-signing-cert-file=/etc/kubernetes/pki/ca-signing.crt|" $f
+grep -q -- "-certificatesigningrequest-signing-controller" $f || sed -i "s|^\( *- --controllers=.*\)$|\1,-certificatesigningrequest-signing-controller|" $f
 '
 """,
     checks=[
-        C("kube-controller-manager points at a signing certificate that exists", 4, r"""
+        C("kube-controller-manager runs the CSR signing controller again", 4, r"""
-p=$(nx "$CP" 'grep -o "\-\-cluster-signing-cert-file=[^ ]*" /etc/kubernetes/manifests/kube-controller-manager.yaml' | cut -d= -f2)
+c=$(nx "$CP" 'grep -o -- "--controllers=[^ ]*" /etc/kubernetes/manifests/kube-controller-manager.yaml')
-[ -n "$p" ] && nx "$CP" "test -f $p" """),
+[ -n "$c" ] && ! echo "$c" | grep -Eq -- '-(certificatesigningrequest-signing-controller|csrsigning)(,|$)'"""),
-        C("kube-controller-manager pod is Running", 3, r"""
+        C("kube-controller-manager still Running after your edit", 3, r"""
 for i in $(seq 1 12); do [ "$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].status.phase}')" = Running ] && exit 0; sleep 5; done; exit 1"""),
         C("an approved CSR now receives a certificate", 5, r"""
 $K get csr issue-probe >/dev/null 2>&1 || exit 1
@@ -946,15 +998,18 @@ $K get csr issue-probe -o jsonpath='{.status.conditions[*].type}' | grep -q Appr
 for i in $(seq 1 15); do [ -n "$(j csr issue-probe '{.status.certificate}')" ] && exit 0; sleep 4; done; exit 1"""),
     ],
     solution="""
-k get csr                                     # Approved but never Issued
+k get csr                                     # Approved - but never Approved,Issued
-k -n kube-system logs kube-controller-manager-{CP} | tail -20
+# Approving is the API's job; ISSUING is a controller's job, and controllers live in
-#   ... unable to load client CA file .../ca-signing.crt: no such file or directory
+# kube-controller-manager. So: is it running, and is the right controller in it running?
+k -n kube-system get pod -l component=kube-controller-manager      # Running - so not a crash
+k -n kube-system logs kube-controller-manager-{CP} | grep -i disabled
+#   "Warning: controller is disabled" controller="certificatesigningrequest-signing-controller"
 {SHELL_CP}
-grep signing /etc/kubernetes/manifests/kube-controller-manager.yaml
+grep -- --controllers /etc/kubernetes/manifests/kube-controller-manager.yaml
-ls /etc/kubernetes/pki/ca.crt                 # the real file
+#   --controllers=*,bootstrapsigner,tokencleaner,-certificatesigningrequest-signing-controller
-vi /etc/kubernetes/manifests/kube-controller-manager.yaml
+vi /etc/kubernetes/manifests/kube-controller-manager.yaml    # remove ,-certificatesigningrequest-signing-controller
-#   --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
 exit
+k -n kube-system get pod -l component=kube-controller-manager -w   # the kubelet restarts the static pod
 openssl genrsa -out probe.key 2048
 openssl req -new -key probe.key -subj "/CN=probe" -out probe.csr
 cat <<EOF | k apply -f -
@@ -964,9 +1019,9 @@ metadata: {name: issue-probe}
 spec: {request: $(base64 -w0 probe.csr), signerName: kubernetes.io/kube-apiserver-client, usages: [client auth]}
 EOF
 k certificate approve issue-probe
-k get csr issue-probe                          # certificate now populated
+k get csr issue-probe                          # Approved,Issued
-# The csrsigning controller lives in kube-controller-manager: approval is the API's job,
+# `--controllers`: '*' = every default-on controller, 'name' adds one, '-name' removes one.
-# but issuing the certificate is the controller's.
+# kubeadm itself adds bootstrapsigner and tokencleaner, which are off by default.
 """,
 )
 

tracks/cka/bank_sim_c.py

@@ -202,8 +202,11 @@ i=$(j -n driver-onboard deploy onboard-app '{.spec.template.spec.containers[0].i
 echo "$i" | grep -q '^nginx:' && ! echo "$i" | grep -q ngnix"""),
         C("liveness probe targets port 80", 2, r"""
 eq "$(j -n driver-onboard deploy onboard-app '{.spec.template.spec.containers[0].livenessProbe.httpGet.port}')" 80"""),
-        C("readiness probe uses a path nginx actually serves", 2, r"""
+        # any readiness probe nginx can satisfy counts (/, /index.html, a tcpSocket on 80...); the
-eq "$(j -n driver-onboard deploy onboard-app '{.spec.template.spec.containers[0].readinessProbe.httpGet.path}')" /"""),
+        # "all replicas Ready" check below proves it actually passes
+        C("readiness probe no longer targets the missing /healthz", 2, r"""
+rp=$(j -n driver-onboard deploy onboard-app '{.spec.template.spec.containers[0].readinessProbe}')
+[ -n "$rp" ] && ! echo "$rp" | grep -q '/healthz'"""),
         C("all replicas Running and Ready", 2, r"""
 for i in $(seq 1 24); do [ "$(j -n driver-onboard deploy onboard-app '{.status.readyReplicas}')" = 2 ] && exit 0; sleep 5; done; exit 1"""),
     ],
@@ -637,7 +640,7 @@ eq "$(j -n trip-tracker deploy trip-logger '{.spec.template.spec.initContainers[
         C("it uses busybox:1.36 and targets /config/app-config.json", 3, r"""
 c='{.spec.template.spec.initContainers[?(@.name=="config-init")]'
 eq "$(j -n trip-tracker deploy trip-logger "$c.image}")" busybox:1.36 \
- && j -n trip-tracker deploy trip-logger "$c.command[*]}" | grep -q '/config/app-config.json'"""),
+ && { j -n trip-tracker deploy trip-logger "$c.command[*]}"; j -n trip-tracker deploy trip-logger "$c.args[*]}"; } | grep -q '/config/app-config.json'"""),
         C("emptyDir config-volume mounted at /config in both containers", 2, r"""
 d="$K -n trip-tracker get deploy trip-logger"
 [ -n "$($d -o jsonpath='{.spec.template.spec.volumes[?(@.name=="config-volume")].emptyDir}')" ] \
@@ -678,8 +681,8 @@ Before rolling out NetworkPolicies, confirm what the cluster's CNI can actually
 3. In Namespace `zero-trust`, create NetworkPolicy `default-deny-all` that denies **all** ingress
    and egress for every Pod in the Namespace, with no allow rules of any kind.
 
-Note which CNI is installed here before answering step 2 — several popular CNIs accept
+Answer step 2 from evidence, not memory — several popular CNIs accept NetworkPolicy objects
-NetworkPolicy objects happily and never enforce them.
+happily and never enforce them, and a CNI's behaviour can change between releases.
 """,
     setup=NS + r"""
 mkns zero-trust
@@ -691,8 +694,17 @@ $K -n zero-trust delete netpol default-deny-all --ignore-not-found >/dev/null 2>
 want=$($K -n kube-system get ds -o name 2>/dev/null | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1)
 [ -n "$want" ] || want=kindnet
 grep -qi "$want" "$WS/q410/cni.txt" 2>/dev/null"""),
-        C("enforced.txt correctly says this CNI does not enforce NetworkPolicy", 2, r"""
+        # The right answer depends on the build, so derive it: recent kindnet embeds a policy engine
-grep -qix 'no' "$WS/q410/enforced.txt" 2>/dev/null"""),
+        # (nftables table `kindnet-network-policies` on every node - verified enforcing on this
+        # cluster), calico/cilium/weave enforce, flannel does not.
+        C("enforced.txt gives the right answer for this cluster's CNI", 2, r"""
+ds=$($K -n kube-system get ds -o name 2>/dev/null | sed 's|.*/||' | tr '\n' ' ')
+case "$ds" in
+  *calico*|*cilium*|*weave*|*antrea*) want=yes ;;
+  *kindnet*) nx "$W1" 'nft list table inet kindnet-network-policies >/dev/null 2>&1' </dev/null && want=yes || want=no ;;
+  *) want=no ;;
+esac
+grep -qix "$want" "$WS/q410/enforced.txt" 2>/dev/null"""),
         C("default-deny-all selects every pod in both directions", 4, r"""
 n="$K -n zero-trust get netpol default-deny-all"
 eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \
@@ -702,7 +714,10 @@ eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \
     ],
     solution="""
 k -n kube-system get ds                       # kindnet  -> {WS}/q410/cni.txt
-echo no > {WS}/q410/enforced.txt              # kindnet has no policy engine
+# Don't answer from memory - prove it. Two pods, one deny policy, one wget:
+k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv; k -n zero-trust expose pod srv --port 80
+k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600
+k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null && echo reachable
 cat <<EOF | k apply -f -
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
@@ -711,10 +726,14 @@ spec:
   podSelector: {}
   policyTypes: [Ingress, Egress]
 EOF
+k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null || echo BLOCKED   # -> BLOCKED here
+echo yes > {WS}/q410/enforced.txt
+k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv                # leave only the policy
 # podSelector {} means every pod in the namespace; naming both policyTypes with no ingress/egress
-# blocks listed means "deny everything". kindnet (like flannel) has no policy engine, so the API
+# blocks listed means "deny everything". Recent kindnet builds (this cluster's included) embed a
-# server stores these objects and nothing enforces them - a real trap in a hardening review.
+# policy engine - look for `nft list table inet kindnet-network-policies` on a node. Flannel stores
-# Calico and Cilium do enforce; the exam expects you to know which is which.
+# NetworkPolicies and enforces nothing; Calico and Cilium enforce. A policy that "works" on a
+# non-enforcing CNI is the classic hardening-review trap.
 """,
 )
 
@@ -1066,7 +1085,8 @@ In Namespace `safety-monitor`, create NetworkPolicy `safety-api-policy` for the
 - **egress**: DNS on port 53, both UDP and TCP, to any destination,
 - everything else denied in both directions.
 
-(kindnet does not enforce policies; you are graded on the object.)
+(This cluster's CNI enforces policies, so a wrong policy really breaks traffic — but you are graded on
+the object.)
 """,
     setup=NS + r"""
 mkns safety-monitor
@@ -1195,8 +1215,13 @@ $K -n booking-tls rollout status deploy booking-nginx --timeout=120s >/dev/null
         C("the ConfigMap allows TLSv1.3 only", 3, r"""
 c=$(j -n booking-tls cm booking-nginx-config '{.data.default\.conf}')
 echo "$c" | grep -q 'ssl_protocols.*TLSv1\.3' && ! echo "$c" | grep -q 'TLSv1\.2'"""),
+        # rollout restart (restartedAt annotation) or deleting the pods - either way every running
+        # pod must be younger than the last write to the ConfigMap
         C("the pods were restarted to pick the change up", 2, r"""
-j -n booking-tls deploy booking-nginx '{.spec.template.metadata.annotations}' | grep -q 'restartedAt'"""),
+j -n booking-tls deploy booking-nginx '{.spec.template.metadata.annotations}' | grep -q 'restartedAt' && exit 0
+cm_t=$($K -n booking-tls get cm booking-nginx-config -o json --show-managed-fields 2>/dev/null | python3 -c 'import sys,json; print(max(m.get("time","") for m in json.load(sys.stdin)["metadata"].get("managedFields",[])))')
+ts=$(j -n booking-tls pod -l app=booking-nginx '{.items[*].metadata.creationTimestamp}'); [ -n "$cm_t" ] && [ -n "$ts" ] || exit 1
+for t in $ts; do [[ "$t" > "$cm_t" ]] || exit 1; done"""),
         C("ssl-line.txt records the directive", 2, r"""
 f="$WS/q417/ssl-line.txt"
 grep -q 'ssl_protocols' "$f" 2>/dev/null && grep -q 'TLSv1\.3' "$f" && ! grep -q 'TLSv1\.2' "$f" """),

tracks/cka/bank_sim_d.py

@@ -0,0 +1,988 @@
+"""Simulator Set D - 17 tasks, fixed set, killer.sh-style.
+
+Original scenarios, not copies of any commercial simulator. They exercise the techniques the
+killer.sh CKA environment is known for that sets A-C do not already cover: saying how every
+control-plane piece is run, cluster facts to an exact file, readiness gated on a Service,
+`--sort-by` scripts, event forensics around kube-proxy with crictl, API-resource and RBAC census,
+a kubelet whose systemd unit points at the wrong binary, Secrets mounted read-only and renamed into
+env vars, a static Pod on the control plane behind a Service, kubelet client/serving certificate
+forensics, control-plane-only scheduling, stopping the scheduler and scheduling by hand, Helm
+release management (including a release stuck in pending-install), StatefulSet storage, the
+namespaceSelector-AND-podSelector NetworkPolicy trap (enforced here - kindnet enforces), a
+dnsPolicy that silently breaks name resolution, and a claim Pending on an access mode.
+
+Every value a grader compares against is derived from the live cluster, never hardcoded, and every
+task was seeded, solved and graded on an isolated kind cluster before it went in.
+
+Run with:  ./exam sim d
+"""
+from tracks.cka.common import C, NS
+
+Q = []
+CTX = "Use context: `kubectl config use-context %{ctx}`\n\n"
+
+
+def task(**kw):
+    kw.setdefault("difficulty", "hard")
+    kw.setdefault("params", lambda r: {"ctx": "k8s-c1-H"})
+    kw["task"] = CTX + kw["task"].lstrip("\n")
+    kw["set"] = "d"
+    Q.append(kw)
+
+
+# ------------------------------------------------------------------ 501 ----
+task(
+    id=501, domain="ARCH", title="How the control plane is run",
+    task="""
+The operations handbook has to say how each part of this cluster's control plane is run. Write
+`{WS}/q501/controlplane.txt` with exactly these six lines, each value one of `process`,
+`static-pod`, `pod`, `deployment` or `daemonset`:
+
+```
+kubelet: <value>
+kube-apiserver: <value>
+kube-scheduler: <value>
+kube-controller-manager: <value>
+etcd: <value>
+dns: <value> <name of that object>
+```
+
+`dns` is whatever serves cluster DNS, for example `dns: daemonset some-name`.
+""",
+    setup=r"""rm -rf "$WS/q501"; mkdir -p "$WS/q501" """,
+    checks=[
+        C("kubelet is a process", 1, r"""
+f="$WS/q501/controlplane.txt"; [ -s "$f" ] || exit 1
+want=process; [ "$(nx "$CP" 'systemctl is-active kubelet' </dev/null)" = active ] || want=unknown
+eq "$(sed -n 's/^ *kubelet *: *//p' "$f" | head -1 | tr -d '[:space:]' | tr A-Z a-z)" "$want" """),
+        C("apiserver, scheduler, controller-manager and etcd are static pods", 3, r"""
+f="$WS/q501/controlplane.txt"
+# a kubeadm control plane: the manifests live in the kubelet's staticPodPath on the node
+nx "$CP" 'ls /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null >/dev/null 2>&1 || exit 1
+for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do
+  eq "$(sed -n "s/^ *$c *: *//p" "$f" | head -1 | tr -d '[:space:]' | tr A-Z a-z)" static-pod || exit 1
+done"""),
+        C("dns names the right kind of object and its name", 2, r"""
+f="$WS/q501/controlplane.txt"
+sel=$(j -n kube-system svc kube-dns '{.spec.selector.k8s-app}')
+own=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].kind}')
+name=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].name}')
+if [ "$own" = ReplicaSet ]; then want="deployment $(j -n kube-system rs "$name" '{.metadata.ownerReferences[0].name}')"
+else want="$(echo "$own" | tr A-Z a-z) $name"; fi
+eq "$(sed -n 's/^ *dns *: *//p' "$f" | head -1 | tr -s ' ' | tr A-Z a-z | sed 's/ *$//')" "$want" """),
+    ],
+    solution="""
+{SHELL_CP}
+systemctl status kubelet | head -3          # a systemd service: a plain process on the node
+ls /etc/kubernetes/manifests                # etcd, kube-apiserver, kube-controller-manager, kube-scheduler
+exit
+k -n kube-system get pod -o wide            # static pods carry the node name as a suffix...
+k -n kube-system get pod kube-apiserver-{CP} -o jsonpath='{.metadata.ownerReferences[0].kind}{"\\n"}'   # ...and are owned by the Node
+k -n kube-system get deploy,ds              # coredns is a Deployment (kube-proxy, kindnet are DaemonSets)
+cat > {WS}/q501/controlplane.txt <<'EOF'
+kubelet: process
+kube-apiserver: static-pod
+kube-scheduler: static-pod
+kube-controller-manager: static-pod
+etcd: static-pod
+dns: deployment coredns
+EOF
+# The Service is called kube-dns for historical reasons; what actually answers is the coredns
+# Deployment. Follow the chain svc selector -> pod -> ownerReferences when unsure.
+""",
+)
+
+# ------------------------------------------------------------------ 502 ----
+task(
+    id=502, domain="TSHOOT", title="Cluster facts to an exact file",
+    task="""
+Write `{WS}/q502/cluster-info.txt` — answers only, exactly this format:
+
+```
+1: <number of control-plane nodes>
+2: <number of worker nodes>
+3: <the Service CIDR>
+4: <name of the CNI plugin>, <full path of its configuration file on the nodes>
+5: <the suffix the names of static Pods running on node {W3} end with>
+```
+""",
+    setup=r"""rm -rf "$WS/q502"; mkdir -p "$WS/q502" """,
+    checks=[
+        C("node counts (lines 1 and 2)", 2, r"""
+f="$WS/q502/cluster-info.txt"; [ -s "$f" ] || exit 1
+cp=$($K get nodes -l node-role.kubernetes.io/control-plane --no-headers 2>/dev/null | wc -l); all=$($K get nodes --no-headers | wc -l)
+eq "$(sed -n 's/^ *1 *: *//p' "$f" | tr -d '[:space:]')" "$cp" && eq "$(sed -n 's/^ *2 *: *//p' "$f" | tr -d '[:space:]')" "$((all - cp))" """),
+        C("Service CIDR (line 3)", 2, r"""
+sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2)
+[ -n "$sc" ] && eq "$(sed -n 's/^ *3 *: *//p' "$WS/q502/cluster-info.txt" | tr -d '[:space:]')" "$sc" """),
+        C("CNI plugin and its config file (line 4)", 2, r"""
+l=$(sed -n 's/^ *4 *: *//p' "$WS/q502/cluster-info.txt" | tr -d '[:space:]')
+cni=$($K -n kube-system get ds -o name | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1)
+cfg=$(nx "$W1" 'ls /etc/cni/net.d/*.conflist /etc/cni/net.d/*.conf 2>/dev/null | head -1' </dev/null)
+[ -n "$cni" ] && [ -n "$cfg" ] && echo "$l" | grep -qi "$cni" && echo "$l" | grep -qF "$cfg" """),
+        C("static Pod name suffix on {W3} (line 5)", 1, r"""
+v=$(sed -n 's/^ *5 *: *//p' "$WS/q502/cluster-info.txt" | tr -d '[:space:]'); [ "$v" = "-$W3" ] || [ "$v" = "$W3" ]"""),
+    ],
+    solution="""
+k get nodes                                   # ROLES shows control-plane; count the rest
+k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
+k -n kube-system get ds                       # kindnet
+{SHELL_W1}
+ls /etc/cni/net.d/                            # 10-kindnet.conflist
+exit
+cat > {WS}/q502/cluster-info.txt <<'EOF'
+1: 1
+2: 3
+3: 10.96.0.0/16
+4: kindnet, /etc/cni/net.d/10-kindnet.conflist
+5: -{W3}
+EOF
+# A static Pod's mirror is named <pod>-<node name>, e.g. kube-apiserver-{CP}. Values come from THIS
+# cluster - check each one instead of trusting the ones shown here.
+""",
+)
+
+# ------------------------------------------------------------------ 503 ----
+task(
+    id=503, domain="NET", title="Ready only when its dependency answers",
+    task="""
+In Namespace `oslo-web`:
+
+1. Create Pod `gatekeeper` (image `busybox:1.36`, command `sleep 86400`) with an **exec readiness
+   probe** that runs `wget -T2 -qO- http://ledger-api:8080`, initial delay 5 seconds, period 5
+   seconds. For now it must stay **not ready** — confirm that it does.
+2. Deployment `ledger` serves HTTP on port 80, but nothing exposes it. Create the Service
+   `ledger-api` (ClusterIP, port **8080**) for it, so that `gatekeeper` turns Ready without you
+   touching the Pod again.
+""",
+    setup=NS + r"""
+mkns oslo-web
+$K -n oslo-web delete pod gatekeeper --ignore-not-found --force --grace-period=0 >/dev/null 2>&1
+$K -n oslo-web delete svc ledger-api --ignore-not-found >/dev/null
+$K -n oslo-web create deploy ledger --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n oslo-web rollout status deploy ledger --timeout=120s >/dev/null 2>&1
+""",
+    checks=[
+        C("gatekeeper has the exec readiness probe (delay 5, period 5)", 2, r"""
+p='{.spec.containers[0].readinessProbe'
+j -n oslo-web pod gatekeeper "$p.exec.command}" | grep -q 'ledger-api:8080' \
+ && eq "$(j -n oslo-web pod gatekeeper "$p.initialDelaySeconds}")" 5 && eq "$(j -n oslo-web pod gatekeeper "$p.periodSeconds}")" 5"""),
+        C("Service ledger-api: 8080 -> 80 with ready endpoints", 2, r"""
+eq "$(j -n oslo-web svc ledger-api '{.spec.ports[0].port}')" 8080 && eq "$(j -n oslo-web svc ledger-api '{.spec.ports[0].targetPort}')" 80 \
+ && [ -n "$(j -n oslo-web endpointslices -l kubernetes.io/service-name=ledger-api '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""),
+        C("gatekeeper is Ready", 3, r"""
+for i in $(seq 1 20); do [ "$(j -n oslo-web pod gatekeeper '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 3; done; exit 1"""),
+    ],
+    solution="""
+k -n oslo-web run gatekeeper --image=busybox:1.36 $do -- sleep 86400 > gk.yaml
+# add to the container:
+#   readinessProbe:
+#     exec:
+#       command: ["sh", "-c", "wget -T2 -qO- http://ledger-api:8080"]
+#     initialDelaySeconds: 5
+#     periodSeconds: 5
+k apply -f gk.yaml
+k -n oslo-web get pod gatekeeper                  # 0/1 Running - the probe cannot resolve ledger-api
+k -n oslo-web describe pod gatekeeper | tail -3   # Readiness probe failed: wget: bad address 'ledger-api:8080'
+k -n oslo-web expose deploy ledger --name=ledger-api --port=8080 --target-port=80
+k -n oslo-web get pod gatekeeper -w               # 1/1 after the next probe
+# Readiness removes a Pod from Service endpoints and blocks rollouts; liveness restarts it.
+# A dependency check belongs in readiness - failing liveness here would just restart-loop the Pod.
+""",
+)
+
+# ------------------------------------------------------------------ 504 ----
+task(
+    id=504, domain="WORK", title="Sorting done by kubectl",
+    task="""
+Write two executable scripts that call `kubectl` (spelled out — aliases do not exist in a fresh
+shell) and let **kubectl itself** do the sorting with `--sort-by`:
+
+1. `{WS}/q504/pods-by-age.sh` — every Pod in every Namespace, oldest first (by creation time).
+2. `{WS}/q504/pods-by-uid.sh` — the Pods in Namespace `kube-system`, sorted by `metadata.uid`.
+""",
+    setup=r"""rm -rf "$WS/q504"; mkdir -p "$WS/q504" """,
+    checks=[
+        C("pods-by-age.sh: all namespaces, --sort-by creationTimestamp, runs", 2, r"""
+f="$WS/q504/pods-by-age.sh"; [ -x "$f" ] || exit 1
+grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \
+ && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.creationTimestamp" "$f" || exit 1
+out=$(timeout 60 bash "$f" 2>/dev/null); [ "$(echo "$out" | wc -l)" -ge 5 ] && echo "$out" | grep -q kube-system"""),
+        C("pods-by-uid.sh: kube-system, --sort-by metadata.uid, runs", 2, r"""
+f="$WS/q504/pods-by-uid.sh"; [ -x "$f" ] || exit 1
+grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-n|--namespace)[= ]*kube-system' "$f" \
+ && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.uid" "$f" || exit 1
+timeout 60 bash "$f" 2>/dev/null | grep -q etcd"""),
+        C("no aliases in either script", 1, r"""[ -s "$WS/q504/pods-by-age.sh" ] && [ -s "$WS/q504/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q504/pods-by-age.sh" "$WS/q504/pods-by-uid.sh" """),
+    ],
+    solution="""
+cat > {WS}/q504/pods-by-age.sh <<'EOF'
+kubectl get pods -A --sort-by=.metadata.creationTimestamp
+EOF
+cat > {WS}/q504/pods-by-uid.sh <<'EOF'
+kubectl -n kube-system get pods --sort-by=.metadata.uid
+EOF
+chmod +x {WS}/q504/*.sh
+bash {WS}/q504/pods-by-age.sh | head -3        # the control-plane pods come first - they are oldest
+# --sort-by takes a JSONPath into each item; ascending only, so "newest first" is `| tac` or
+# sorting by a field that grows the other way.
+""",
+)
+
+# ------------------------------------------------------------------ 505 ----
+task(
+    id=505, domain="TSHOOT", title="Event forensics around kube-proxy", breaks="w2-kube-proxy",
+    task="""
+1. Write `{WS}/q505/cluster-events.sh`: a script that prints the latest events of the **whole**
+   cluster (all Namespaces) sorted by creation time, using `kubectl`. Make it executable.
+2. Delete the kube-proxy Pod that runs on node `{W2}`. Write the events this caused to
+   `{WS}/q505/pod-deleted.log`.
+3. On node `{W2}`, stop **and remove** the kube-proxy **container** with `crictl` — leave the Pod
+   object alone. Write the events this caused to `{WS}/q505/container-killed.log`.
+4. Which of the two actions made the Pod's `RESTARTS` counter go up? Write `pod` or `container` to
+   `{WS}/q505/restarts.txt`.
+
+Do step 2 before step 3.
+""",
+    setup=r"""
+rm -rf "$WS/q505"; mkdir -p "$WS/q505"
+$K -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 -o jsonpath='{.items[0].metadata.uid}' > "$WS/.q505-uid"
+""",
+    checks=[
+        C("cluster-events.sh: all namespaces, sorted by time, runs", 2, r"""
+f="$WS/q505/cluster-events.sh"; [ -x "$f" ] || exit 1
+grep -Eq 'kubectl\b.*\bget\b.*\b(ev|events?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \
+ && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?(metadata\.creationTimestamp|lastTimestamp|eventTime|firstTimestamp)" "$f" || exit 1
+timeout 60 bash "$f" >/dev/null 2>&1"""),
+        C("the kube-proxy Pod on {W2} was deleted and replaced", 2, r"""
+old=$(cat "$WS/.q505-uid" 2>/dev/null); [ -n "$old" ] || exit 1
+now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}')
+[ -n "$now" ] && [ "$now" != "$old" ] && eq "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.phase}')" Running"""),
+        # counted on the REPLACEMENT pod, which starts at 0 - after a node reboot the original pod
+        # often already shows a restart, which would otherwise hand these points out for free
+        C("then its container was killed (restart counter >= 1)", 2, r"""
+old=$(cat "$WS/.q505-uid" 2>/dev/null); now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}')
+[ -n "$old" ] && [ -n "$now" ] && [ "$now" != "$old" ] || exit 1
+[ "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null"""),
+        C("pod-deleted.log and container-killed.log hold the events", 2, r"""
+for f in pod-deleted.log container-killed.log; do
+  grep -q 'kube-proxy' "$WS/q505/$f" 2>/dev/null && grep -Eq 'Killing|SuccessfulCreate|SuccessfulDelete|Scheduled|Pulled|Created|Started|BackOff' "$WS/q505/$f" || exit 1
+done"""),
+        C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q505/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""),
+    ],
+    solution="""
+cat > {WS}/q505/cluster-events.sh <<'EOF'
+kubectl get events -A --sort-by=.metadata.creationTimestamp
+EOF
+chmod +x {WS}/q505/cluster-events.sh
+
+k -n kube-system get pod -l k8s-app=kube-proxy -o wide              # the one on {W2}
+k -n kube-system delete pod <kube-proxy-on-{W2}>
+k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q505/pod-deleted.log
+#   Killing (old pod), SuccessfulCreate (DaemonSet), Scheduled, Pulled, Created, Started (new pod)
+
+{SHELL_W2}
+crictl ps --name kube-proxy                 # container id
+crictl stop <id> && crictl rm <id>
+exit
+k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q505/container-killed.log
+#   Pulled, Created, Started - same Pod, new container
+k -n kube-system get pod -l k8s-app=kube-proxy -o wide              # RESTARTS 1 on {W2}
+echo container > {WS}/q505/restarts.txt
+# Deleting the Pod makes the DaemonSet create a brand-new Pod (RESTARTS 0). Killing only the
+# container makes the kubelet start a new container inside the SAME Pod - that is a restart.
+""",
+)
+
+# ------------------------------------------------------------------ 506 ----
+def _census(r):
+    n = r.sample(range(2, 10), 4)
+    return {"ctx": "k8s-c1-H", "north": n[0], "south": n[1], "east": n[2], "west": n[3]}
+
+
+task(
+    id=506, domain="ARCH", title="API resources and a Role census", params=_census,
+    task="""
+1. Write the names of every **namespaced** resource type this API server offers to
+   `{WS}/q506/namespaced.txt`, one per line — plural names, as `kubectl api-resources` prints them.
+2. Of the Namespaces `census-north`, `census-south`, `census-east` and `census-west`, which one holds
+   the most Roles? Write `<namespace> <number of Roles>` to `{WS}/q506/most-roles.txt`.
+""",
+    setup=NS + r"""
+rm -rf "$WS/q506"; mkdir -p "$WS/q506"
+for pair in north:%{north} south:%{south} east:%{east} west:%{west}; do
+  ns=census-${pair%%:*}; n=${pair##*:}
+  mkns $ns
+  $K -n $ns delete role --all >/dev/null 2>&1
+  for i in $(seq 1 $n); do $K -n $ns create role reader-$i --verb=get,list --resource=configmaps >/dev/null; done
+done
+""",
+    checks=[
+        C("namespaced.txt lists exactly the namespaced resource types", 3, r"""
+f="$WS/q506/namespaced.txt"; [ -s "$f" ] || exit 1
+want=$($K api-resources --namespaced=true -o name 2>/dev/null | sed 's/\..*//' | sort -u)
+got=$(awk '{print $1}' "$f" | sed 's/\..*//' | grep . | sort -u)
+[ -n "$want" ] && [ "$got" = "$want" ]"""),
+        C("most-roles.txt names the namespace and its Role count", 3, r"""
+best=$(for ns in census-north census-south census-east census-west; do echo "$ns $($K -n $ns get role --no-headers 2>/dev/null | wc -l)"; done | sort -k2 -n | tail -1)
+eq "$(tr -s ' ' < "$WS/q506/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """),
+    ],
+    solution="""
+k api-resources --namespaced=true -o name > /tmp/r.txt       # deployments.apps, ... (group suffix)
+k api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q506/namespaced.txt
+for ns in census-north census-south census-east census-west; do echo "$ns $(k -n $ns get role --no-headers | wc -l)"; done
+#   pick the largest, e.g.  echo "census-east 8" > {WS}/q506/most-roles.txt
+# --namespaced=false gives the cluster-scoped half (nodes, pv, storageclasses, clusterroles...).
+# `k get role -A --no-headers | awk '{print $1}' | sort | uniq -c | sort -n` counts every
+# namespace at once.
+""",
+)
+
+# ------------------------------------------------------------------ 507 ----
+task(
+    id=507, domain="TSHOOT", title="Kubelet unit points at nothing", breaks="w1-kubelet", disruptive=True,
+    task="""
+Node `{W1}` went `NotReady` after someone "tidied up" its kubelet installation. Bring it back to
+`Ready`; the kubelet has to run as the systemd service it always was. Then write the full path of the
+file you had to fix to `{WS}/q507/fixed-file.txt`.
+
+Reach the node with `{SHELL_W1}`. Pods on a node that stays NotReady are evicted after five minutes.
+""",
+    setup=r"""
+rm -rf "$WS/q507"; mkdir -p "$WS/q507"
+nx "$W1" '
+d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
+grep -q "^ExecStart=/usr/bin/kubelet " $d && sed -i "s|^ExecStart=/usr/bin/kubelet |ExecStart=/usr/local/bin/kubelet |" $d
+systemctl daemon-reload; systemctl restart kubelet' </dev/null >/dev/null 2>&1
+exit 0
+""",
+    checks=[
+        C("{W1} is Ready", 4, r"""for i in $(seq 1 15); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 4; done; exit 1"""),
+        C("kubelet active, and its unit runs a kubelet binary that exists", 2, r"""
+nx "$W1" 'systemctl is-active --quiet kubelet && b=$(sed -n "s|^ExecStart=\([^ ]*kubelet\) .*|\1|p" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | tail -1) && [ -x "$b" ]' </dev/null"""),
+        C("fixed-file.txt names the drop-in", 2, r"""grep -q '/etc/systemd/system/kubelet.service.d/10-kubeadm.conf' "$WS/q507/fixed-file.txt" 2>/dev/null"""),
+    ],
+    solution="""
+k get nodes                                   # {W1} NotReady
+{SHELL_W1}
+systemctl status kubelet
+#   Process: ... ExecStart=/usr/local/bin/kubelet ... (code=exited, status=203/EXEC)
+journalctl -u kubelet -n 20 --no-pager        # Unable to locate executable '/usr/local/bin/kubelet'
+which kubelet                                 # /usr/bin/kubelet - the binary was never moved
+systemctl cat kubelet                         # shows the unit AND every drop-in, with file names
+vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf     # ExecStart=/usr/bin/kubelet ...
+systemctl daemon-reload                       # systemd caches unit files - without this nothing changes
+systemctl restart kubelet
+systemctl status kubelet                      # active (running)
+exit
+echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q507/fixed-file.txt
+k get nodes -w
+# status=203/EXEC means systemd could not exec the binary at all - a path problem, not a kubelet
+# configuration problem. `ExecStart=` (empty) in a drop-in clears the unit's own ExecStart first.
+""",
+)
+
+# ------------------------------------------------------------------ 508 ----
+task(
+    id=508, domain="WORK", title="Secrets as a read-only volume and renamed env vars",
+    params=lambda r: {"ctx": "k8s-c1-H", "pw": "ledger-" + "".join(r.choice("abcdefghjkmnpqrstuvwxyz23456789") for _ in range(10))},
+    task="""
+In Namespace `oslo-secure`:
+
+1. Create Secret `bundle` from the file `{WS}/q508/tls-bundle.pem`; the key must be the file's name.
+2. Create Secret `db-login` with `user=svc-ledger` and `pass=%{pw}`.
+3. Create Pod `vault-reader` (image `busybox:1.36`, command `sleep 86400`) that
+   - mounts Secret `bundle` **read-only** at `/etc/bundle`,
+   - receives `db-login`'s keys as environment variables named **`APP_USER`** and **`APP_PASS`**.
+""",
+    setup=NS + r"""
+mkns oslo-secure
+rm -rf "$WS/q508"; mkdir -p "$WS/q508"
+openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=ledger.oslo.internal" -keyout /dev/null -out "$WS/q508/tls-bundle.pem" >/dev/null 2>&1
+$K -n oslo-secure delete pod vault-reader --ignore-not-found --force --grace-period=0 >/dev/null 2>&1
+$K -n oslo-secure delete secret bundle db-login --ignore-not-found >/dev/null
+""",
+    checks=[
+        C("Secrets bundle (key tls-bundle.pem, the file's content) and db-login", 2, r"""
+[ "$(j -n oslo-secure secret bundle '{.data.tls-bundle\.pem}' | base64 -d 2>/dev/null)" = "$(cat "$WS/q508/tls-bundle.pem")" ] \
+ && eq "$(j -n oslo-secure secret db-login '{.data.user}' | base64 -d)" svc-ledger && eq "$(j -n oslo-secure secret db-login '{.data.pass}' | base64 -d)" '%{pw}'"""),
+        C("vault-reader Running with bundle mounted read-only at /etc/bundle", 3, r"""
+p="$K -n oslo-secure get pod vault-reader"
+eq "$($p -o jsonpath='{.status.phase}' 2>/dev/null)" Running || exit 1
+v=$($p -o jsonpath='{.spec.volumes[?(@.secret.secretName=="bundle")].name}'); [ -n "$v" ] || exit 1
+eq "$($p -o jsonpath="{.spec.containers[0].volumeMounts[?(@.name==\"$v\")].mountPath}")" /etc/bundle \
+ && eq "$($p -o jsonpath="{.spec.containers[0].volumeMounts[?(@.name==\"$v\")].readOnly}")" true \
+ && $K -n oslo-secure exec vault-reader -- cat /etc/bundle/tls-bundle.pem 2>/dev/null | grep -q 'BEGIN CERTIFICATE'"""),
+        C("APP_USER / APP_PASS come from db-login", 3, r"""
+e=$(j -n oslo-secure pod vault-reader '{.spec.containers[0].env}')
+echo "$e" | grep -q '"name":"APP_USER"' && echo "$e" | grep -q '"name":"APP_PASS"' && echo "$e" | grep -q 'db-login' || exit 1
+env=$($K -n oslo-secure exec vault-reader -- env 2>/dev/null)
+echo "$env" | grep -qx 'APP_USER=svc-ledger' && echo "$env" | grep -qx 'APP_PASS=%{pw}'"""),
+    ],
+    solution="""
+k -n oslo-secure create secret generic bundle --from-file={WS}/q508/tls-bundle.pem   # key = file name
+k -n oslo-secure create secret generic db-login --from-literal=user=svc-ledger --from-literal=pass='%{pw}'
+k -n oslo-secure run vault-reader --image=busybox:1.36 $do -- sleep 86400 > vr.yaml
+# in vr.yaml, pod level:
+#   volumes: [{name: bundle, secret: {secretName: bundle}}]
+# in the container:
+#   volumeMounts: [{name: bundle, mountPath: /etc/bundle, readOnly: true}]
+#   env:
+#     - name: APP_USER
+#       valueFrom: {secretKeyRef: {name: db-login, key: user}}
+#     - name: APP_PASS
+#       valueFrom: {secretKeyRef: {name: db-login, key: pass}}
+k apply -f vr.yaml
+k -n oslo-secure exec vault-reader -- env | grep APP_
+k -n oslo-secure exec vault-reader -- ls /etc/bundle
+# envFrom would inject the keys under their own names (user, pass); renaming needs env + secretKeyRef.
+# A Secret volume updates in place when the Secret changes; env vars never do.
+""",
+)
+
+# ------------------------------------------------------------------ 509 ----
+task(
+    id=509, domain="ARCH", title="Static Pod on the control plane behind a Service",
+    task="""
+1. On the control-plane node (`{SHELL_CP}`) create a **static Pod** `ops-beacon` in Namespace
+   `default`: image `nginx:1.27`, requests `cpu: 10m` and `memory: 20Mi`, label `app=ops-beacon`.
+2. Expose it with a Service `ops-beacon-svc` of type **NodePort** (port 80 → 80) whose endpoints
+   contain that static Pod.
+3. Write the node port the Service was given to `{WS}/q509/nodeport.txt`.
+""",
+    setup=r"""
+rm -rf "$WS/q509"; mkdir -p "$WS/q509"
+nx "$CP" 'rm -f /etc/kubernetes/manifests/ops-beacon*.yaml' </dev/null
+$K -n default delete svc ops-beacon-svc --ignore-not-found >/dev/null
+# the kubelet drops the mirror pod on its next sync of the manifest directory
+for i in $(seq 1 30); do $K -n default get pod "ops-beacon-$CP" >/dev/null 2>&1 || break; sleep 2; done
+""",
+    checks=[
+        C("manifest in the control-plane node's staticPodPath", 1, r"""
+nx "$CP" 'd=$(awk "/^staticPodPath:/{print \$2}" /var/lib/kubelet/config.yaml); grep -ls "ops-beacon" "$d"/*.yaml' </dev/null >/dev/null 2>&1"""),
+        C("mirror Pod ops-beacon-{CP} Running with the requests and label", 3, r"""
+p="$K -n default get pod ops-beacon-$CP"
+eq "$($p -o jsonpath='{.status.phase}' 2>/dev/null)" Running && eq "$($p -o jsonpath='{.metadata.labels.app}')" ops-beacon \
+ && eq "$($p -o jsonpath='{.spec.containers[0].resources.requests.cpu}')" 10m && eq "$($p -o jsonpath='{.spec.containers[0].resources.requests.memory}')" 20Mi"""),
+        C("NodePort Service ops-beacon-svc 80 -> 80 with the static Pod as endpoint", 3, r"""
+s="$K -n default get svc ops-beacon-svc"
+eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" NodePort && eq "$($s -o jsonpath='{.spec.ports[0].port}')" 80 && eq "$($s -o jsonpath='{.spec.ports[0].targetPort}')" 80 || exit 1
+ip=$(j -n default pod "ops-beacon-$CP" '{.status.podIP}'); [ -n "$ip" ] || exit 1
+j -n default endpointslices -l kubernetes.io/service-name=ops-beacon-svc '{.items[*].endpoints[*].addresses[*]}' | tr ' ' '\n' | grep -qx "$ip" """),
+        C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q509/nodeport.txt" 2>/dev/null)" "$np" """),
+    ],
+    solution="""
+k run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon $do > beacon.yaml
+# add to the container:  resources: {requests: {cpu: 10m, memory: 20Mi}}
+{SHELL_CP}
+grep staticPodPath /var/lib/kubelet/config.yaml          # /etc/kubernetes/manifests
+vi /etc/kubernetes/manifests/ops-beacon.yaml             # paste beacon.yaml
+exit
+k -n default get pod ops-beacon-{CP} -o wide             # mirror pod, named <pod>-<node>
+k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
+k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc   # the mirror pod's IP
+k -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q509/nodeport.txt
+# The control-plane taint only keeps the SCHEDULER from placing pods there; the kubelet runs its
+# static pods regardless. `expose pod` copies the pod's labels into the selector.
+""",
+)
+
+# ------------------------------------------------------------------ 510 ----
+task(
+    id=510, domain="ARCH", title="The kubelet's two certificates",
+    task="""
+On node `{W2}` find the certificate the kubelet uses as a **client** (to talk to the API server)
+and the one it **serves** its own API with (port 10250). Write exactly this to
+`{WS}/q510/kubelet-certs.txt`:
+
+```
+client-issuer: <issuer, as openssl prints it>
+client-eku: <the extended key usage>
+server-issuer: <issuer>
+server-eku: <the extended key usage>
+```
+
+Reach the node with `{SHELL_W2}`.
+""",
+    setup=r"""rm -rf "$WS/q510"; mkdir -p "$WS/q510" """,
+    checks=[
+        C("client certificate: issuer and extended key usage", 3, r"""
+f="$WS/q510/kubelet-certs.txt"; [ -s "$f" ] || exit 1
+iss=$(nx "$W2" 'openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -issuer' </dev/null | sed 's/^issuer= *//' | tr -d ' ' | tr A-Z a-z)
+[ -n "$iss" ] && sed -n 's/^ *client-issuer *: *//p' "$f" | tr -d ' ' | tr A-Z a-z | grep -qF "$iss" \
+ && sed -n 's/^ *client-eku *: *//p' "$f" | grep -qi 'client auth'"""),
+        C("serving certificate: issuer and extended key usage", 3, r"""
+f="$WS/q510/kubelet-certs.txt"
+iss=$(nx "$W2" 'openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -issuer' </dev/null | sed 's/^issuer= *//' | tr -d ' ' | tr A-Z a-z)
+[ -n "$iss" ] && sed -n 's/^ *server-issuer *: *//p' "$f" | tr -d ' ' | tr A-Z a-z | grep -qF "$iss" \
+ && sed -n 's/^ *server-eku *: *//p' "$f" | grep -qi 'server auth'"""),
+    ],
+    solution="""
+{SHELL_W2}
+grep -A3 users: /etc/kubernetes/kubelet.conf      # client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
+ls /var/lib/kubelet/pki/                          # kubelet.crt / kubelet.key = the serving pair
+openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text | grep -E 'Issuer|Extended Key Usage' -A1
+#   Issuer: CN = kubernetes          Extended Key Usage: TLS Web Client Authentication
+openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -text | grep -E 'Issuer|Extended Key Usage' -A1
+#   Issuer: CN = {W2}-ca@<timestamp>   Extended Key Usage: TLS Web Server Authentication
+exit
+# write the four lines with the values YOUR openssl printed:
+#   client-issuer: CN=kubernetes
+#   client-eku: TLS Web Client Authentication
+#   server-issuer: CN={W2}-ca@<timestamp>
+#   server-eku: TLS Web Server Authentication
+# The client cert is signed by the cluster CA and rotated by the kubelet through CSRs. The serving
+# cert here is self-signed by the kubelet at startup (no serverTLSBootstrap) - which is exactly why
+# metrics-server needs --kubelet-insecure-tls on this cluster.
+""",
+)
+
+# ------------------------------------------------------------------ 511 ----
+task(
+    id=511, domain="WORK", title="A Pod for the control plane only",
+    task="""
+Create Pod `cp-watch` in Namespace `default`: image `httpd:2-alpine`, container name `watch`.
+
+It must be able to run **only** on control-plane nodes. Select them by the node label kubeadm puts
+on control-plane nodes — not by host name, and not with `nodeName` — and tolerate whatever keeps
+ordinary Pods off them. It must be Running.
+""",
+    setup=r"""$K -n default delete pod cp-watch --ignore-not-found --force --grace-period=0 >/dev/null 2>&1""",
+    checks=[
+        C("cp-watch Running on the control-plane node, container named watch", 2, r"""
+eq "$(j -n default pod cp-watch '{.status.phase}')" Running && eq "$(j -n default pod cp-watch '{.spec.nodeName}')" "$CP" && eq "$(j -n default pod cp-watch '{.spec.containers[0].name}')" watch"""),
+        C("selected by the node-role.kubernetes.io/control-plane label", 2, r"""
+s=$(j -n default pod cp-watch '{.spec.nodeSelector}{.spec.affinity.nodeAffinity}')
+echo "$s" | grep -q 'node-role.kubernetes.io/control-plane' && ! echo "$s" | grep -q 'kubernetes.io/hostname'"""),
+        C("tolerates the control-plane taint, no nodeName in the manifest", 2, r"""
+$K -n default get pod cp-watch -o json 2>/dev/null | python3 -c '
+import sys,json; t=json.load(sys.stdin)["spec"].get("tolerations") or []
+sys.exit(0 if any("control-plane" in (x.get("key") or "") or (x.get("operator")=="Exists" and not x.get("key")) for x in t) else 1)' || exit 1
+$K -n default get pod cp-watch -o json --show-managed-fields | python3 -c '
+import sys,json
+mf=json.load(sys.stdin)["metadata"].get("managedFields",[])
+sys.exit(1 if any("f:nodeName" in json.dumps(m.get("fieldsV1",{})) for m in mf) else 0)'"""),
+    ],
+    solution="""
+k get node {CP} --show-labels | tr ',' '\\n' | grep node-role     # node-role.kubernetes.io/control-plane=
+k describe node {CP} | grep Taints                                 # node-role.kubernetes.io/control-plane:NoSchedule
+k -n default run cp-watch --image=httpd:2-alpine $do > cp.yaml
+# rename the container to watch, then add at pod level:
+#   nodeSelector:
+#     node-role.kubernetes.io/control-plane: ""
+#   tolerations:
+#     - {key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule}
+k apply -f cp.yaml
+k -n default get pod cp-watch -o wide
+# The toleration only ALLOWS the control plane; the nodeSelector is what REQUIRES it. You need both.
+""",
+)
+
+# ------------------------------------------------------------------ 512 ----
+task(
+    id=512, domain="TSHOOT", title="Stop the scheduler, schedule by hand", breaks="scheduler",
+    task="""
+1. Stop the kube-scheduler **temporarily** — it runs as a static Pod on the control-plane node — and
+   confirm it is gone.
+2. Create Pod `manual-1` (image `busybox:1.36`, command `sleep 86400`) in Namespace `default` and
+   confirm it stays `Pending`.
+3. Schedule `manual-1` **by hand** onto the control-plane node and get it Running.
+4. Start the kube-scheduler again, then create Pod `manual-2` (same image and command) in `default`
+   and confirm that the scheduler places it.
+""",
+    setup=r"""
+# a pristine copy outside the static pod directory, so the pre-session baseline can put the
+# scheduler back if this task is abandoned half-done
+nx "$CP" 'mkdir -p /etc/kubernetes/.cka-sim && [ -f /etc/kubernetes/manifests/kube-scheduler.yaml ] && cp /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/.cka-sim/kube-scheduler.yaml' </dev/null
+$K -n default delete pod manual-1 manual-2 --ignore-not-found --force --grace-period=0 >/dev/null 2>&1
+exit 0
+""",
+    checks=[
+        C("kube-scheduler is back and Running", 2, r"""
+$K -n default get pod manual-1 >/dev/null 2>&1 || exit 1          # only once the task was actually done
+nx "$CP" 'test -f /etc/kubernetes/manifests/kube-scheduler.yaml' </dev/null || exit 1
+for i in $(seq 1 15); do [ "$(j -n kube-system pod -l component=kube-scheduler '{.items[0].status.containerStatuses[0].ready}')" = true ] && exit 0; sleep 4; done; exit 1"""),
+        C("manual-1 Running on the control-plane node", 3, r"""
+eq "$(j -n default pod manual-1 '{.status.phase}')" Running && eq "$(j -n default pod manual-1 '{.spec.nodeName}')" "$CP" """),
+        C("manual-2 Running, placed by the scheduler (no nodeName in its manifest)", 3, r"""
+eq "$(j -n default pod manual-2 '{.status.phase}')" Running && [ -n "$(j -n default pod manual-2 '{.spec.nodeName}')" ] || exit 1
+$K -n default get pod manual-2 -o json --show-managed-fields | python3 -c '
+import sys,json
+mf=json.load(sys.stdin)["metadata"].get("managedFields",[])
+sys.exit(1 if any("f:nodeName" in json.dumps(m.get("fieldsV1",{})) for m in mf) else 0)'"""),
+    ],
+    solution="""
+{SHELL_CP}
+mv /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/     # the kubelet stops what it no longer sees
+exit
+k -n kube-system get pod -l component=kube-scheduler                   # gone (give it ~20s)
+k -n default run manual-1 --image=busybox:1.36 -- sleep 86400
+k -n default get pod manual-1                                          # Pending, no node
+# spec.nodeName is only settable on creation, so recreate it with the node filled in:
+k -n default get pod manual-1 -o yaml > m1.yaml
+#   add under spec:  nodeName: {CP}
+k replace --force -f m1.yaml
+k -n default get pod manual-1 -o wide                                  # Running on {CP}
+{SHELL_CP}
+mv /etc/kubernetes/kube-scheduler.yaml /etc/kubernetes/manifests/
+exit
+k -n kube-system get pod -l component=kube-scheduler -w                # Running again
+k -n default run manual-2 --image=busybox:1.36 -- sleep 86400
+k -n default get pod manual-2 -o wide                                  # a worker, chosen by the scheduler
+# nodeName skips the scheduler entirely - including its taint checks, which is why manual-1 can
+# sit on the tainted control plane. (The Binding API does the same without recreating the pod.)
+""",
+)
+
+# ------------------------------------------------------------------ 513 ----
+_CHART = r'''
+mkchart() {  # dir name version image replicas
+  mkdir -p "$1/templates"
+  printf 'apiVersion: v2\nname: %s\nversion: %s\nappVersion: "%s"\n' "$2" "$3" "${4##*:}" > "$1/Chart.yaml"
+  printf 'replicaCount: %s\nimage: %s\n' "$5" "$4" > "$1/values.yaml"
+  cat > "$1/templates/deployment.yaml" <<'YAML'
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: {{ .Release.Name }}, labels: {app: {{ .Release.Name }}}}
+spec:
+  replicas: {{ .Values.replicaCount }}
+  selector: {matchLabels: {app: {{ .Release.Name }}}}
+  template:
+    metadata: {labels: {app: {{ .Release.Name }}}}
+    spec:
+      containers: [{name: main, image: "{{ .Values.image }}"}]
+YAML
+}
+'''
+
+task(
+    id=513, domain="ARCH", title="Helm release housekeeping",
+    task="""
+Namespace `oslo-helm` holds several Helm releases; local charts are under `{WS}/q513/charts`.
+
+1. Delete the release `web-legacy`.
+2. Upgrade the release `web-main` to chart version **1.1.0** (`{WS}/q513/charts/nginx-lite-1.1.0`).
+3. Install a new release `cache` from `{WS}/q513/charts/redis-lite` with **2** replicas (the chart's
+   `replicaCount` value).
+4. One release in the Namespace never finished installing. Find it and remove it.
+""",
+    setup=NS + _CHART + r"""
+mkns oslo-helm
+C="$WS/q513/charts"; rm -rf "$WS/q513"; mkdir -p "$C"
+mkchart "$C/nginx-lite-1.0.0" nginx-lite 1.0.0 nginx:1.25 1
+mkchart "$C/nginx-lite-1.1.0" nginx-lite 1.1.0 nginx:1.27 1
+mkchart "$C/redis-lite" redis-lite 0.3.0 redis:7-alpine 1
+mkchart /tmp/q513-stuck cache-edge 0.1.0 busybox:0.0-does-not-exist 1
+for r in $(helm -n oslo-helm list -q 2>/dev/null); do helm -n oslo-helm uninstall "$r" >/dev/null 2>&1; done
+helm -n oslo-helm install web-legacy "$C/nginx-lite-1.0.0" >/dev/null
+helm -n oslo-helm install web-main "$C/nginx-lite-1.0.0" >/dev/null
+# helm records the release as pending-install BEFORE it starts waiting; kill it mid-wait and it
+# stays that way - the classic result of an interrupted install or a CI job that timed out
+timeout -s KILL 6 helm -n oslo-helm install cache-broken /tmp/q513-stuck --wait --timeout 120s >/dev/null 2>&1
+rm -rf /tmp/q513-stuck
+helm -n oslo-helm status cache-broken -o json 2>/dev/null | grep -q '"status":"pending-install"' || echo "warn: cache-broken did not end up pending-install"
+""",
+    checks=[
+        C("web-legacy is gone", 1, r"""! helm -n oslo-helm status web-legacy >/dev/null 2>&1 && helm -n oslo-helm status web-main >/dev/null 2>&1"""),
+        C("web-main deployed at chart version 1.1.0", 2, r"""
+helm -n oslo-helm list -o json 2>/dev/null | python3 -c '
+import sys,json
+r={x["name"]:x for x in json.load(sys.stdin)}.get("web-main") or {}
+sys.exit(0 if r.get("chart")=="nginx-lite-1.1.0" and r.get("status")=="deployed" and int(r.get("revision",0))>=2 else 1)'"""),
+        C("cache deployed from redis-lite with 2 available replicas", 2, r"""
+helm -n oslo-helm list -o json 2>/dev/null | python3 -c '
+import sys,json
+r={x["name"]:x for x in json.load(sys.stdin)}.get("cache") or {}
+sys.exit(0 if r.get("chart","").startswith("redis-lite-") and r.get("status")=="deployed" else 1)' || exit 1
+for i in $(seq 1 20); do [ "$(j -n oslo-helm deploy cache '{.status.availableReplicas}')" = 2 ] && exit 0; sleep 3; done; exit 1"""),
+        C("the stuck release is removed; nothing pending remains", 2, r"""
+! helm -n oslo-helm status cache-broken >/dev/null 2>&1 && [ -z "$(helm -n oslo-helm list --pending -q 2>/dev/null)" ] && ! $K -n oslo-helm get deploy cache-broken >/dev/null 2>&1"""),
+    ],
+    solution="""
+helm -n oslo-helm list                         # Helm 4 shows every status; Helm 3 hides pending ones
+helm -n oslo-helm list --pending               # works in both: cache-broken   pending-install
+helm -n oslo-helm uninstall web-legacy
+helm -n oslo-helm upgrade web-main {WS}/q513/charts/nginx-lite-1.1.0
+helm -n oslo-helm install cache {WS}/q513/charts/redis-lite --set replicaCount=2
+helm -n oslo-helm uninstall cache-broken       # also removes the Deployment it half-created
+helm -n oslo-helm list                         # web-main (1.1.0, rev 2) and cache, both deployed
+# A pending-install release blocks a plain re-install under the same name ("cannot re-use a name
+# that is still in use"). `helm history <rel>` shows when it got stuck; uninstall clears it.
+""",
+)
+
+# ------------------------------------------------------------------ 514 ----
+task(
+    id=514, domain="STOR", title="Per-replica storage for a StatefulSet",
+    task="""
+In Namespace `lima-data`:
+
+1. Create a headless Service `ledger-db` (port 5432) and a StatefulSet `ledger-db` with **3**
+   replicas: image `busybox:1.36`, command `sh -c 'hostname > /data/id.txt && sleep 86400'`, and a
+   `volumeClaimTemplates` entry `data` — 100Mi, ReadWriteOnce, StorageClass `standard` — mounted at
+   `/data`.
+2. Once all 3 are Ready, scale the StatefulSet down to **2**.
+3. Write the names of the PersistentVolumeClaims that belong to the StatefulSet and still exist
+   after the scale-down to `{WS}/q514/pvcs.txt`, one per line, sorted.
+""",
+    setup=NS + r"""
+mkns lima-data
+rm -rf "$WS/q514"; mkdir -p "$WS/q514"
+$K -n lima-data delete sts ledger-db --ignore-not-found --wait=true >/dev/null 2>&1
+$K -n lima-data delete svc ledger-db --ignore-not-found >/dev/null
+$K -n lima-data delete pvc -l app=ledger-db --ignore-not-found >/dev/null 2>&1
+for i in 0 1 2 3 4; do $K -n lima-data delete pvc data-ledger-db-$i --ignore-not-found --wait=false >/dev/null 2>&1; done
+""",
+    checks=[
+        C("headless Service and a StatefulSet with the right claim template", 2, r"""
+eq "$(j -n lima-data svc ledger-db '{.spec.clusterIP}')" None && eq "$(j -n lima-data sts ledger-db '{.spec.serviceName}')" ledger-db || exit 1
+t='{.spec.volumeClaimTemplates[?(@.metadata.name=="data")]'
+eq "$(j -n lima-data sts ledger-db "$t.spec.resources.requests.storage}")" 100Mi && eq "$(j -n lima-data sts ledger-db "$t.spec.storageClassName}")" standard \
+ && j -n lima-data sts ledger-db "$t.spec.accessModes}" | grep -q ReadWriteOnce"""),
+        C("scaled to 2, both Ready", 2, r"""eq "$(j -n lima-data sts ledger-db '{.spec.replicas}')" 2 && eq "$(j -n lima-data sts ledger-db '{.status.readyReplicas}')" 2"""),
+        C("each replica wrote its own identity to its own volume", 2, r"""
+for i in 0 1; do eq "$($K -n lima-data exec ledger-db-$i -- cat /data/id.txt 2>/dev/null | tr -d '[:space:]')" "ledger-db-$i" || exit 1; done
+[ "$(j -n lima-data pod ledger-db-0 '{.spec.volumes[?(@.name=="data")].persistentVolumeClaim.claimName}')" = data-ledger-db-0 ]"""),
+        C("pvcs.txt lists every claim that survived the scale-down", 2, r"""
+want=$($K -n lima-data get pvc --no-headers -o custom-columns=N:.metadata.name 2>/dev/null | grep '^data-ledger-db-' | sort)
+[ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q514/pvcs.txt" 2>/dev/null)" = "$want" ]"""),
+    ],
+    solution="""
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Service
+metadata: {name: ledger-db, namespace: lima-data}
+spec: {clusterIP: None, selector: {app: ledger-db}, ports: [{port: 5432}]}
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata: {name: ledger-db, namespace: lima-data}
+spec:
+  serviceName: ledger-db
+  replicas: 3
+  selector: {matchLabels: {app: ledger-db}}
+  template:
+    metadata: {labels: {app: ledger-db}}
+    spec:
+      containers:
+        - name: db
+          image: busybox:1.36
+          command: [sh, -c, "hostname > /data/id.txt && sleep 86400"]
+          volumeMounts: [{name: data, mountPath: /data}]
+  volumeClaimTemplates:
+    - metadata: {name: data}
+      spec: {accessModes: [ReadWriteOnce], storageClassName: standard, resources: {requests: {storage: 100Mi}}}
+EOF
+k -n lima-data rollout status sts ledger-db
+k -n lima-data scale sts ledger-db --replicas=2
+k -n lima-data get pvc -o custom-columns=N:.metadata.name --no-headers | grep '^data-ledger-db-' | sort > {WS}/q514/pvcs.txt
+cat {WS}/q514/pvcs.txt           # all THREE: data-ledger-db-0, -1, -2
+# Claims are named <template>-<statefulset>-<ordinal> and outlive their pods: scaling down (or
+# deleting the StatefulSet) keeps them unless spec.persistentVolumeClaimRetentionPolicy says
+# otherwise, so scaling back up re-attaches ledger-db-2 to its old data.
+""",
+)
+
+# ------------------------------------------------------------------ 515 ----
+task(
+    id=515, domain="NET", title="Namespace AND pod selector, enforced",
+    task="""
+Pods labelled `app=db` in Namespace `oslo-db` must accept connections on TCP port 80 **only** from
+Pods labelled `role=backend` that run in Namespaces labelled `env=prod`. Every other source must be
+refused. Do not restrict egress.
+
+Create the NetworkPolicy `db-ingress` in `oslo-db`. This cluster's CNI enforces NetworkPolicy, so
+test what you built: the `api` Pods in `oslo-api` must get through to Service `db`, the `debug`
+Pods in `oslo-api` must not, and neither must the `api` Pods in `lima-api`.
+""",
+    setup=NS + r"""
+mkns oslo-db; mkns oslo-api; mkns lima-api
+$K label ns oslo-api env=prod --overwrite >/dev/null; $K label ns lima-api env=dev --overwrite >/dev/null
+$K -n oslo-db delete netpol --all >/dev/null 2>&1
+$K -n oslo-db create deploy db --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n oslo-db expose deploy db --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+mk() {  # ns name role
+  cat <<EOF | $K apply -f - >/dev/null
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: $2, namespace: $1}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: $2}}
+  template:
+    metadata: {labels: {app: $2, role: $3}}
+    spec: {containers: [{name: c, image: busybox:1.36, command: [sleep, "86400"]}]}
+EOF
+}
+mk oslo-api api backend; mk oslo-api debug debug; mk lima-api api backend
+for d in oslo-db/db oslo-api/api oslo-api/debug lima-api/api; do $K -n ${d%%/*} rollout status deploy ${d##*/} --timeout=120s >/dev/null 2>&1; done
+""",
+    checks=[
+        C("selects app=db, Ingress only", 1, r"""
+n="$K -n oslo-db get netpol db-ingress"
+eq "$($n -o jsonpath='{.spec.podSelector.matchLabels.app}' 2>/dev/null)" db && eq "$($n -o jsonpath='{.spec.policyTypes}')" '["Ingress"]'"""),
+        C("one peer holding BOTH namespaceSelector env=prod AND podSelector role=backend, on TCP 80", 3, r"""
+$K -n oslo-db get netpol db-ingress -o json 2>/dev/null | python3 -c '
+import sys,json
+ing=json.load(sys.stdin)["spec"].get("ingress",[])
+ok=len(ing)==1 and len(ing[0].get("from",[]))==1
+if ok:
+    f=ing[0]["from"][0]
+    ok=f.get("namespaceSelector",{}).get("matchLabels",{}).get("env")=="prod" and f.get("podSelector",{}).get("matchLabels",{}).get("role")=="backend"
+    ok=ok and [(str(p.get("port")),p.get("protocol","TCP")) for p in ing[0].get("ports",[])]==[("80","TCP")]
+sys.exit(0 if ok else 1)'"""),
+        C("oslo-api/api reaches db", 2, r"""
+$K -n oslo-db get netpol db-ingress >/dev/null 2>&1 || exit 1     # no policy yet: nothing earned
+for i in 1 2 3 4 5; do $K -n oslo-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1 && exit 0; sleep 2; done; exit 1"""),
+        C("oslo-api/debug and lima-api/api are refused", 2, r"""
+$K -n oslo-db get netpol db-ingress >/dev/null 2>&1 || exit 1
+! $K -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1 \
+ && ! $K -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1"""),
+    ],
+    solution="""
+k get ns --show-labels | grep -E 'oslo-api|lima-api'    # env=prod / env=dev
+cat <<EOF | k apply -f -
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: db-ingress, namespace: oslo-db}
+spec:
+  podSelector: {matchLabels: {app: db}}
+  policyTypes: [Ingress]
+  ingress:
+    - from:
+        - namespaceSelector: {matchLabels: {env: prod}}
+          podSelector: {matchLabels: {role: backend}}      # SAME list item = AND
+      ports: [{protocol: TCP, port: 80}]
+EOF
+k -n oslo-api exec deploy/api   -- wget -qO- -T3 http://db.oslo-db | head -1     # <!DOCTYPE html>
+k -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db               # download timed out
+k -n lima-api exec deploy/api   -- wget -qO- -T3 http://db.oslo-db               # download timed out
+# The trap is one dash: "- namespaceSelector: ...\\n  - podSelector: ..." (two list items) means OR -
+# any pod in a prod namespace, OR a backend pod in oslo-db's own namespace. One item with both
+# selectors means AND.
+""",
+)
+
+# ------------------------------------------------------------------ 516 ----
+task(
+    id=516, domain="NET", title="Resolves everything except the cluster",
+    task="""
+Deployment `reporter` in Namespace `lima-client` cannot reach Service `ledger` — its log says
+`bad address`. The Service, its endpoints and cluster DNS are all fine: other Pods in the Namespace
+resolve `ledger` without trouble.
+
+Fix the `reporter` Deployment (not the Service), without changing its image or command, so that its
+requests succeed. Write the field you changed and its new value, as `<field>=<value>`, to
+`{WS}/q516/fix.txt`.
+""",
+    setup=NS + r"""
+mkns lima-client
+rm -rf "$WS/q516"; mkdir -p "$WS/q516"
+$K -n lima-client create deploy ledger --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n lima-client expose deploy ledger --port=8080 --target-port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n lima-client delete deploy reporter --ignore-not-found --wait=true >/dev/null 2>&1
+$K apply -f - >/dev/null <<'EOF'
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: reporter, namespace: lima-client}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: reporter}}
+  template:
+    metadata: {labels: {app: reporter}}
+    spec:
+      dnsPolicy: Default
+      containers:
+        - name: reporter
+          image: busybox:1.36
+          command: [sh, -c, 'while true; do if wget -qO- -T2 http://ledger:8080/ >/dev/null 2>&1; then echo "$(date +%T) ledger ok"; else wget -qO- -T2 http://ledger:8080/ 2>&1 | head -1; fi; sleep 3; done']
+EOF
+$K -n lima-client rollout status deploy ledger --timeout=120s >/dev/null 2>&1
+""",
+    checks=[
+        C("reporter uses the cluster DNS policy again", 3, r"""
+p=$(j -n lima-client deploy reporter '{.spec.template.spec.dnsPolicy}'); [ -z "$p" ] || [ "$p" = ClusterFirst ] || [ "$p" = ClusterFirstWithHostNet ]"""),
+        C("reporter's requests now succeed", 3, r"""
+for i in $(seq 1 15); do
+  p=$($K -n lima-client get pod -l app=reporter --field-selector status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
+  [ -n "$p" ] && $K -n lima-client logs "$p" --tail=3 2>/dev/null | grep -q 'ledger ok' && exit 0; sleep 3
+done; exit 1"""),
+        C("fix.txt names dnsPolicy and its new value", 1, r"""grep -Eqi '^ *dnsPolicy *= *ClusterFirst *$' "$WS/q516/fix.txt" 2>/dev/null"""),
+    ],
+    solution="""
+k -n lima-client logs deploy/reporter --tail=3            # wget: bad address 'ledger:8080'
+k -n lima-client get endpointslices                       # the Service has endpoints - not the problem
+k -n lima-client get deploy reporter -o yaml | grep -n dns
+#   dnsPolicy: Default   <- "Default" means: use the NODE's resolver, NOT cluster DNS
+k -n lima-client exec deploy/reporter -- cat /etc/resolv.conf   # the podman/host resolver, no cluster search domains
+k -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'
+k -n lima-client logs deploy/reporter -f                  # "ledger ok"
+echo dnsPolicy=ClusterFirst > {WS}/q516/fix.txt
+# The trap is the name: the DEFAULT dnsPolicy is ClusterFirst, while the policy literally called
+# "Default" inherits the node's /etc/resolv.conf. ClusterFirstWithHostNet is for hostNetwork pods.
+""",
+)
+
+# ------------------------------------------------------------------ 517 ----
+task(
+    id=517, domain="STOR", title="A claim that can never bind",
+    task="""
+Deployment `uploader` in Namespace `oslo-files` has no running Pod. Find out why and fix it, so its
+Pod runs with the claim mounted at `/files`. Keep the claim's name (`shared-files`), its size and its
+StorageClass. Write the root cause, in one line, to `{WS}/q517/reason.txt`.
+""",
+    setup=NS + r"""
+mkns oslo-files
+rm -rf "$WS/q517"; mkdir -p "$WS/q517"
+$K -n oslo-files delete deploy uploader --ignore-not-found --wait=true >/dev/null 2>&1
+$K -n oslo-files delete pvc shared-files --ignore-not-found --wait=true >/dev/null 2>&1
+$K apply -f - >/dev/null <<'EOF'
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: shared-files, namespace: oslo-files}
+spec: {accessModes: [ReadWriteMany], storageClassName: standard, resources: {requests: {storage: 1Gi}}}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: uploader, namespace: oslo-files}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: uploader}}
+  template:
+    metadata: {labels: {app: uploader}}
+    spec:
+      volumes: [{name: files, persistentVolumeClaim: {claimName: shared-files}}]
+      containers: [{name: up, image: busybox:1.36, command: [sleep, "86400"], volumeMounts: [{name: files, mountPath: /files}]}]
+EOF
+""",
+    checks=[
+        C("shared-files Bound: 1Gi, class standard, an access mode local-path supports", 3, r"""
+eq "$(j -n oslo-files pvc shared-files '{.status.phase}')" Bound && eq "$(j -n oslo-files pvc shared-files '{.spec.resources.requests.storage}')" 1Gi \
+ && eq "$(j -n oslo-files pvc shared-files '{.spec.storageClassName}')" standard \
+ && ! j -n oslo-files pvc shared-files '{.spec.accessModes}' | grep -q ReadWriteMany"""),
+        C("uploader available with the claim at /files", 2, r"""
+for i in $(seq 1 20); do [ "$(j -n oslo-files deploy uploader '{.status.availableReplicas}')" = 1 ] && break; sleep 3; done
+eq "$(j -n oslo-files deploy uploader '{.status.availableReplicas}')" 1 \
+ && eq "$(j -n oslo-files deploy uploader '{.spec.template.spec.volumes[?(@.persistentVolumeClaim.claimName=="shared-files")].name}')" files \
+ && j -n oslo-files deploy uploader '{.spec.template.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /files"""),
+        C("reason.txt names the unsupported access mode", 2, r"""grep -Eqi 'ReadWriteMany|RWX|access ?mode' "$WS/q517/reason.txt" 2>/dev/null"""),
+    ],
+    solution="""
+k -n oslo-files get pod                          # Pending
+k -n oslo-files describe pod -l app=uploader | tail -4    # unbound immediate PersistentVolumeClaims / waiting
+k -n oslo-files describe pvc shared-files | tail -4
+#   ProvisioningFailed ... NodePath only supports ReadWriteOnce and ReadWriteOncePod (1.22+) access modes
+# accessModes is immutable on a PVC, so recreate it. The Pending pod was never scheduled, so it
+# does not hold the claim (pvc-protection only counts scheduled pods) and the delete goes through.
+k -n oslo-files get pvc shared-files -o yaml > pvc.yaml     # change ReadWriteMany -> ReadWriteOnce, strip status/uid
+k -n oslo-files delete pvc shared-files
+k apply -f pvc.yaml
+k -n oslo-files rollout restart deploy uploader            # or just wait - the pending pod retries
+k -n oslo-files get pvc,pod
+echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q517/reason.txt
+# RWX needs a filesystem many nodes can mount at once (NFS, CephFS, a cloud file service).
+# ReadWriteOnce is per NODE, not per pod - ReadWriteOncePod is the per-pod one.
+""",
+)
+
+QUESTIONS = Q

tracks/cka/questions.py

@@ -23,8 +23,8 @@ TRACK = {
     # tasks per domain drawn into a session (weights follow the exam blueprint)
     "quota":      {"ARCH": 5, "WORK": 4, "NET": 4, "STOR": 2, "TSHOOT": 5},      # 20 tasks
     "quota_hard": {"ARCH": 6, "WORK": 5, "NET": 5, "STOR": 3, "TSHOOT": 6},      # 25 tasks, hard ones preferred
-    # fixed simulator sets (./exam sim a|b|c) - excluded from random sampling
+    # fixed simulator sets (./exam sim a|b|c|d) - excluded from random sampling
-    "sets": {"a": "Simulator Set A", "b": "Simulator Set B", "c": "Simulator Set C"},
+    "sets": {"a": "Simulator Set A", "b": "Simulator Set B", "c": "Simulator Set C", "d": "Simulator Set D (killer.sh-style)"},
     "domains": {
         "ARCH":   "Cluster Architecture, Installation & Configuration (25%)",
         "WORK":   "Workloads & Scheduling (15%)",
@@ -177,7 +177,7 @@ $K delete ns store-prod --ignore-not-found --wait=false >/dev/null
  ],
 },
 {
- "id": 5, "domain": "ARCH", "title": "Static pod",
+ "id": 5, "domain": "ARCH", "title": "Static pod", "breaks": ["w3-staticpath"],   # Q103 relocates W3's staticPodPath
  "task": """
 Create a **static pod** named `static-web` running image `nginx:1.27`
 on node `{W3}` (namespace `default`).
@@ -362,8 +362,8 @@ Create a NetworkPolicy `backend-policy` in `netpol` so that pods with `app=backe
   on TCP port **8080**,
 - reject all other ingress.
 
-Do not restrict egress. (The default kind CNI does not enforce policies;
+Do not restrict egress. (This cluster's kindnet enforces NetworkPolicy, so a wrong policy really
-you are graded on the policy spec.)
+blocks traffic; you are graded on the policy spec.)
 """,
  "setup": NS + r"""
 mkns netpol
@@ -631,10 +631,10 @@ $K -n default delete pod sched-probe --wait=false >/dev/null 2>&1
 ]
 
 # ---- aggregate the banks: 1-20 here, 21-43 in bank_extra, 101-124 (hard) in bank_hard,
-# ---- 201-217 / 301-317 / 401-418 in the three fixed simulator sets
+# ---- 201-217 / 301-317 / 401-418 / 501-517 in the four fixed simulator sets
-from tracks.cka import bank_extra, bank_hard, bank_sim_a, bank_sim_b, bank_sim_c
+from tracks.cka import bank_extra, bank_hard, bank_sim_a, bank_sim_b, bank_sim_c, bank_sim_d
 QUESTIONS = (QUESTIONS + bank_extra.QUESTIONS + bank_hard.QUESTIONS
-             + bank_sim_a.QUESTIONS + bank_sim_b.QUESTIONS + bank_sim_c.QUESTIONS)
+             + bank_sim_a.QUESTIONS + bank_sim_b.QUESTIONS + bank_sim_c.QUESTIONS + bank_sim_d.QUESTIONS)
 QUESTIONS.sort(key=lambda q: q["id"])
 _ids = [q["id"] for q in QUESTIONS]
 assert len(_ids) == len(set(_ids)), "duplicate question ids"

tracks/cka/solutions.py

@@ -262,8 +262,10 @@ k -n %{ns} create rolebinding %{sa}-rb --role=%{sa}-role --serviceaccount=%{ns}:
 k create clusterrole %{sa}-nodes --verb=list --resource=nodes
 k create clusterrolebinding %{sa}-crb --clusterrole=%{sa}-nodes --serviceaccount=%{ns}:%{sa}
 
-# Step 6: Verify effective permissions
+# Step 6: Verify effective permissions. For a SUBRESOURCE use --subresource: `can-i create
+# pods/exec` means "create the pod named exec" and says no even when the Role is right.
 k auth can-i --list -n %{ns} --as=system:serviceaccount:%{ns}:%{sa}
+k auth can-i create pods --subresource=exec -n %{ns} --as=system:serviceaccount:%{ns}:%{sa}   # yes
 """
 
 S[102] = """# Step 1: Access the control plane node
@@ -381,7 +383,10 @@ k create ns %{ns}
 k apply -k .
 """
 
-S[106] = """# Step 1: Create CRD with validation rules (required fields, enum, integer bounds)
+S[106] = """# Step 0: This context does not default to the `default` namespace - check, then pass -n default
+k config view --minify -o jsonpath='{..namespace}{"\\n"}'
+
+# Step 1: CRD with validation (required fields, integer bounds, enum) and a server-side default
 cat <<EOF | k apply -f -
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
@@ -389,7 +394,11 @@ metadata: {name: $(echo %{kind} | tr A-Z a-z)s.deploy.example.com}
 spec:
   group: deploy.example.com
   scope: Namespaced
-  names: {kind: %{kind}, plural: $(echo %{kind} | tr A-Z a-z)s, singular: $(echo %{kind} | tr A-Z a-z)}
+  names:
+    kind: %{kind}
+    plural: $(echo %{kind} | tr A-Z a-z)s
+    singular: $(echo %{kind} | tr A-Z a-z)
+    shortNames: [$(echo %{kind} | tr A-Z a-z | cut -c1-2)]
   versions:
     - name: v1
       served: true
@@ -400,26 +409,28 @@ spec:
           properties:
             spec:
               type: object
-              required: [replicas, image]
+              required: [replicas, image]          # sits NEXT TO properties, not inside a field
               properties:
                 replicas: {type: integer, minimum: 1, maximum: 10}
                 environment: {type: string, enum: [prod, stage, dev], default: dev}
                 image: {type: string}
 EOF
 
-# Step 2: Apply valid custom resource
+# Step 2: A valid object
-k apply -f - <<EOF
+k -n default apply -f - <<EOF
 apiVersion: deploy.example.com/v1
 kind: %{kind}
 metadata: {name: good}
 spec: {replicas: 3, image: nginx:1.27, environment: %{env}}
 EOF
 
-# Step 3: Test schema rejection of invalid resource (0 replicas)
+# Step 3: The rejection goes to STDERR - redirect 2>, not >, or the file stays empty
-printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: bad}\\nspec: {replicas: 0, image: x}\\n' | k apply -f - 2> {WS}/q106/rejected.txt
+printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: bad}\\nspec: {replicas: 0, image: x}\\n' | k -n default apply -f - 2> {WS}/q106/rejected.txt
+cat {WS}/q106/rejected.txt          # ...spec.replicas in body should be greater than or equal to 1
 
-# Step 4: Verify default values on omitted fields
+# Step 4: Omit environment and let the API server fill in the default
-printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: defaulted}\\nspec: {replicas: 2, image: nginx:1.27}\\n' | k apply -f -
+printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: defaulted}\\nspec: {replicas: 2, image: nginx:1.27}\\n' | k -n default apply -f -
+k -n default get $(echo %{kind} | tr A-Z a-z) defaulted -o jsonpath='{.spec.environment}{"\\n"}'   # dev
 """
 
 S[122] = """# Step 1: Access control plane node
@@ -782,40 +793,47 @@ k -n %{ns} get pod -l app=worker -o jsonpath='{.items[0].spec.containers[0].reso
 """
 
 # Hard Workload Tasks
-S[107] = """# Step 1: Inspect namespace quota constraints
+S[107] = """# Step 1: Read the constraints before writing anything
-k -n %{ns} get quota
+k get node {W1} -o jsonpath='{.spec.taints}{"\\n"}'      # %{key}=%{val}:NoSchedule
+k -n %{ns} describe quota                               # pods: 2, requests.cpu: 250m
 
-# Step 2: Deploy pods with podAntiAffinity, tolerations, PriorityClass, and fitted resources
+# Step 2: The quota is the trap twice over: 3 pods do not fit "pods: 2", and 3 x 100m = 300m does
+# not fit 250m either. Raise both - the quota has to stay and still cap pods.
+k -n %{ns} patch quota %{ns}-quota -p '{"spec":{"hard":{"pods":"3","requests.cpu":"300m"}}}'
+
+# Step 3: The PriorityClass
+k create priorityclass %{dep}-prio --value=50000
+
+# Step 4: One pod per worker (required anti-affinity on hostname) that tolerates {W1}'s taint
 cat <<EOF | k apply -f -
 apiVersion: apps/v1
 kind: Deployment
-metadata: {name: anti-workload, namespace: %{ns}}
+metadata: {name: %{dep}, namespace: %{ns}}
 spec:
   replicas: 3
-  selector: {matchLabels: {app: anti-workload}}
+  selector: {matchLabels: {app: %{dep}}}
   template:
-    metadata: {labels: {app: anti-workload}}
+    metadata: {labels: {app: %{dep}}}
     spec:
-      priorityClassName: high-priority
+      priorityClassName: %{dep}-prio
       affinity:
         podAntiAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
-            - labelSelector: {matchLabels: {app: anti-workload}}
+            - labelSelector: {matchLabels: {app: %{dep}}}
               topologyKey: kubernetes.io/hostname
       tolerations:
-        - key: dedicated
+        - {key: %{key}, operator: Equal, value: "%{val}", effect: NoSchedule}
-          operator: Equal
-          value: special
-          effect: NoSchedule
       containers:
-        - name: app
+        - name: nginx
-          image: busybox:1.36
+          image: nginx:1.27
-          command: [sleep, "3600"]
+          resources: {requests: {cpu: 100m, memory: 64Mi}}
-          resources: {requests: {cpu: 50m, memory: 32Mi}, limits: {cpu: 100m, memory: 64Mi}}
 EOF
 
-# Step 3: Confirm each replica landed on a distinct node
+# Step 5: Verify: 3 Running on 3 different nodes. The control-plane is tainted too, so with three
-k -n %{ns} get pods -o wide -l app=anti-workload
+# workers there is zero slack - a missed quota field or toleration shows up as a Pending pod, and
+# `k -n %{ns} describe rs` tells you which (FailedCreate: exceeded quota).
+k -n %{ns} get pod -l app=%{dep} -o wide
+k -n %{ns} describe quota %{ns}-quota      # pods 3/3, requests.cpu 300m/300m
 """
 
 S[108] = """# Step 1: Find WHICH revision used nginx:1.25 - the task asks for that one specifically, which
@@ -846,7 +864,9 @@ k -n %{ns} get rs -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spe
 """
 
 S[109] = """# Step 1: Deploy Pod with a native sidecar (restartPolicy: Always inside initContainers),
-# a shared emptyDir the sidecar writes into and the main container reads from
+# a shared emptyDir the sidecar writes into and the main container reads from. Mount it at
+# /var/log/agent, NOT /var/log: an emptyDir over /var/log hides nginx's /var/log/nginx and
+# nginx crash-loops with "could not open error log file"
 cat <<EOF | k apply -f -
 apiVersion: v1
 kind: Pod
@@ -860,9 +880,9 @@ spec:
     - name: agent
       image: busybox:1.36
       restartPolicy: Always
-      command: [sh, -c, "while true; do date >> /var/log/agent.log; sleep 1; done"]
+      command: [sh, -c, "while true; do date >> /var/log/agent/agent.log; sleep 1; done"]
       volumeMounts:
-        - {name: logs, mountPath: /var/log}
+        - {name: logs, mountPath: /var/log/agent}
   containers:
     - name: app
       image: nginx:1.27
@@ -872,7 +892,7 @@ spec:
       livenessProbe:
         tcpSocket: {port: 80}
       volumeMounts:
-        - {name: logs, mountPath: /var/log}
+        - {name: logs, mountPath: /var/log/agent}
 EOF
 
 # Step 2: Once Running, write the exact report files
@@ -881,23 +901,40 @@ k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name} {.spec.con
 k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' > {WS}/q109/ip.txt
 """
 
-S[110] = r"""# Step 1: pods outside kube-system whose FIRST container has no readiness probe
+S[110] = r"""# Step 0: look before you script - which pods, which phases, which nodes
-k get pods -A -o json | jq -r '.items[] | select(.metadata.namespace!="kube-system") | select(.spec.containers[0].readinessProbe == null) | "\(.metadata.namespace)/\(.metadata.name)"' | sort > {WS}/q110/no-readiness.txt
+k -n %{ns} get pods -o wide
 
-# Step 2: count of Running pods per node ("<node> <count>", node first)
+# Step 1: FIRST container without a readinessProbe, any phase (the Completed pod counts too).
-k get pods -A --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort | uniq -c | awk '{print $2, $1}' | sort > {WS}/q110/per-node.txt
+# jsonpath prints nothing for a missing field, so emit "<name> <probe-or-empty>" and keep the bare names.
+k -n %{ns} get pods -o jsonpath='{range .items[*]}{.metadata.name} {.spec.containers[0].readinessProbe}{"\n"}{end}' \
+  | awk 'NF==1{print $1}' | sort > {WS}/q110/no-readiness.txt
 
-# Step 3: sorted unique container images used across kube-system
+# Step 2: Running pods per node, "<node> <count>". uniq -c prints "count node", so swap the columns.
+k -n %{ns} get pods --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' \
+  | sort | uniq -c | awk '{print $2, $1}' | sort > {WS}/q110/per-node.txt
+
+# Step 3: distinct images in kube-system
 k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u > {WS}/q110/images.txt
+
+# Traps: cache-a has a readiness probe - on its SECOND container, so it belongs in file 1;
+# once-a is Completed, so it is in file 1 but not in file 2. `cat -A` shows stray spaces or \r.
 """
 
-S[121] = """# Step 1: Confirm current context namespace
+S[121] = """# Step 1: Switch context and find out which namespace it defaults to
-k config view --minify -o jsonpath='{.contexts[0].context.namespace}'
+k config use-context %{ctx}
+k config view --minify -o jsonpath='{..namespace}{"\\n"}'        # -> %{ns}
 
-# Step 2: Perform all required operations without specifying -n, relying on context namespace
+# Step 2: Everything WITHOUT -n, so it lands in the context's namespace
-k create deploy %{dep} --image=nginx:1.27 --replicas=2
+k create deploy %{dep} --image=busybox:1.36 --replicas=2 -- sleep 3600
-k set env deploy/%{dep} APP_ENV=production
+k label deploy %{dep} owner=ac
-k expose deploy %{dep} --port=80 --target-port=80
+k expose deploy %{dep} --port=9000 --target-port=9000             # ClusterIP is the default type
+
+# Step 3: Record the namespace exactly as kubectl reports it
+k config view --minify -o jsonpath='{..namespace}' > {WS}/q121/ns.txt
+
+# Step 4: Verify it all landed there - and nothing in default
+k get deploy,svc %{dep}
+k -n default get deploy %{dep}          # NotFound is the right answer
 """
 
 
@@ -1090,27 +1127,32 @@ EOF
 k -n %{ns} describe netpol %{app}-egress
 """
 
-S[43] = """# Step 1: Create Headless Service (clusterIP: None)
+S[43] = """# Step 1: The headless Service must be named EXACTLY like the StatefulSet's spec.serviceName -
+# that name is what makes CoreDNS publish one record per pod: <pod>.<service>.<ns>.svc.cluster.local
+k -n %{ns} get sts %{sts} -o jsonpath='{.spec.serviceName}{"\\n"}'      # -> %{sts}
 cat <<EOF | k apply -f -
 apiVersion: v1
 kind: Service
-metadata: {name: db-headless, namespace: %{ns}}
+metadata: {name: %{sts}, namespace: %{ns}}
 spec:
   clusterIP: None
-  selector: {app: db}
+  selector: {app: %{sts}}
   ports: [{port: 5432, targetPort: 5432}]
 ---
-# Step 2: Create ExternalName Service pointing to external domain
 apiVersion: v1
 kind: Service
-metadata: {name: external-api, namespace: %{ns}}
+metadata: {name: legacy, namespace: %{ns}}
 spec:
   type: ExternalName
-  externalName: api.partner.com
+  externalName: %{ext}
 EOF
 
-# Step 3: Verify DNS resolution for headless endpoints and external CNAME
+# Step 2: Resolve pod 0 by its FULL name. busybox's nslookup applies no search domains, so the
-k -n %{ns} get svc
+# short form %{sts}-0.%{sts} returns NXDOMAIN even though DNS is perfectly healthy.
+k -n %{ns} exec %{sts}-0 -- nslookup %{sts}-0.%{sts}.%{ns}.svc.cluster.local > {WS}/q43/pod0.txt
+cat {WS}/q43/pod0.txt                   # Address: <the pod's IP>
+k -n %{ns} get pod %{sts}-0 -o wide     # same IP
+k -n %{ns} get svc legacy               # EXTERNAL-IP column shows %{ext} - it is only a CNAME
 """
 
 # Hard Networking Tasks
@@ -1223,7 +1265,9 @@ k -n kube-system edit cm coredns
 
 # Step 4: Restart CoreDNS deployment and test DNS resolution
 k -n kube-system rollout restart deploy coredns
-k run dns-check --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
+# busybox's nslookup applies no search domains, so use FULL names (a short one fails even when DNS works)
+k run dns-check --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default.svc.cluster.local
+k run dns-ext --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.io
 """
 
 
@@ -1419,8 +1463,13 @@ k -n %{ns} get pod legacy -o yaml > legacy.yaml
 # Edit legacy.yaml: replace claimName: old-data with claimName: new-data
 k -n %{ns} delete pod legacy --now && k apply -f legacy.yaml
 
-# Step 5: Clean up old PVC (PV remains Retained)
+# Step 5: Delete the old claim. The Job's finished pod still references old-data, and a pod that
+# references a claim - even a Completed one - holds it in pvc-protection: `delete pvc` just sits
+# there (describe pvc -> "Used By: migrate-xxxxx"). Delete that pod first; the Job itself stays,
+# still reporting succeeded: 1.
+k -n %{ns} delete pod -l job-name=migrate
 k -n %{ns} delete pvc old-data
+k get pv | grep old-data                                    # STATUS Released - Retain kept the data
 """
 
 
@@ -1518,7 +1567,9 @@ k -n kube-system edit cm coredns
 k -n kube-system rollout restart deploy coredns
 
 # Step 5: Verify DNS resolution from a test pod
-k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
+# busybox's nslookup applies no search domains, so use FULL names (a short one fails even when DNS works)
+k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default.svc.cluster.local
+k run test-ext --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.io
 """
 
 S[38] = """# Step 1: Check control-plane pod status
@@ -1609,8 +1660,11 @@ k -n %{ns} get all
 k -n %{ns} describe deploy %{dep}
 k -n %{ns} describe rs -l app=%{dep}               # Quota exceeded: pods limit is 1
 
-# Step 2: Fix quota to accommodate required replicas
+# Step 2: Fix the quota - with room for the rolling update, not just the end state. The default
-k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"3"}}}'
+# strategy (maxSurge 25% -> 1, maxUnavailable 25% of 3 -> 0) needs a 4th pod mid-rollout, and with
+# maxUnavailable 0 the controller will not remove an old pod first either: at pods=3 the rollout
+# deadlocks at "0 out of 3 new replicas have been updated".
+k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"4"}}}'
 
 # Step 3: Fix deployment image tag
 k -n %{ns} set image deploy %{dep} web=nginx:1.27

tracks/ckad/questions.py

@@ -535,7 +535,8 @@ Namespace `%{ns}` runs Pods labelled `app=db` and `app=%{client}`.
 Create NetworkPolicy `db-allow` so `app=db` Pods accept ingress **only** from `app=%{client}`
 Pods in the same Namespace on TCP 5432, and nothing else. Do not restrict egress.
 
-(kindnet doesn't enforce policies here, so you are graded on the object.)
+(kindnet enforces NetworkPolicy here, so a wrong policy really blocks traffic; you are graded
+on the object.)
 """,
  "setup": NS + r"""
 mkns %{ns}

tracks/cks/questions.py

@@ -1,8 +1,8 @@
 """CKS track. Starter bank.
 
 Caveat worth knowing before you rely on this: several CKS objectives need infrastructure the
-default kind cluster does not have - kindnet does not enforce NetworkPolicy, and there is no
+default kind cluster does not have - there is no Falco, AppArmor profile loader, gVisor runtime
-Falco, AppArmor profile loader, gVisor runtime or image scanner installed. Tasks here either
+or image scanner installed. (NetworkPolicy IS enforced: recent kindnet embeds a policy engine.) Tasks here either
 stick to what a stock kubeadm/kind cluster can really do (PSA, RBAC, admission policies, audit
 logging, seccomp, token hygiene) or are graded on the object rather than on enforcement, and say so.
 """
@@ -43,7 +43,8 @@ A NetworkPolicy has exactly one `podSelector`, so the egress half (selecting `we
 ingress half (selecting `db`) cannot live in the same object. Once a default deny is in place
 both halves are required — forgetting one is the classic CKS mistake.
 
-(kindnet does not enforce policies; you are graded on the objects.)
+(This cluster's kindnet enforces NetworkPolicy, so a gap really drops traffic; you are graded
+on the objects.)
 """,
  "setup": NS + r"""
 mkns %{ns}

weak-spots-cka.md

@@ -1,149 +1,121 @@
 # CKA weak spots
 
-_Generated by exam from 11 attempt(s); regenerated on every `end`._
+_Generated by exam from 10 attempt(s); regenerated on every `end`._
 
-## Domain scores (latest full exam, 2026-09-20T17:32, 100.0%)
+## Domain scores (latest full exam, 2026-09-08T19:59, 14.3%)
 
-- **ARCH** 38/266 — Cluster Architecture, Installation & Configuration (25%)
+- **ARCH** 12/26 — Cluster Architecture, Installation & Configuration (25%)
-- **WORK** 36/187 — Workloads & Scheduling (15%)
+- **WORK** 1/16 — Workloads & Scheduling (15%)
-- **NET** 37/179 — Services & Networking (20%)
+- **NET** 0/17 — Services & Networking (20%)
-- **STOR** 19/85 — Storage (10%)
+- **STOR** 0/11 — Storage (10%)
-- **TSHOOT** 33/171 — Troubleshooting (30%)
+- **TSHOOT** 2/35 — Troubleshooting (30%)
 
 ## Drill next  (`./exam drill weak`)
 
-### Q02 · ARCH · etcd snapshot  (1/2 failed, last 0/7 pts, 2026-09-19T12:27)
+### Q02 · ARCH · etcd snapshot  (1/2 failed, last 0/7 pts, last failed 2026-09-19T12:27)
 - ✗ /opt/cka/etcd-snapshot.db exists on the control-plane node
 - ✗ snapshot is a real etcd backup (> 100 KB)
 
-### Q06 · ARCH · Certificates: expiry check and user CSR  (2/3 failed, last 6/6 pts, 2026-09-12T22:17)
+### Q06 · ARCH · Certificates: expiry check and user CSR  (2/3 failed, last 6/6 pts, last failed 2026-09-12T22:17)
-- ✗ CSR jane exists with signer kubernetes.io/kube-apiserver-client
+- passed last time (6/6) - one more clean pass takes it off this list
-- ✗ CSR jane approved and certificate issued
 
-### Q13 · NET · Ingress routing  (2/4 failed, last 5/5 pts, 2026-09-14T16:49)
+### Q15 · STOR · PersistentVolume, claim and pod  (3/4 failed, last 6/6 pts, last failed 2026-09-15T20:55)
-- ✗ ingressClassName nginx and host shop.example.com
+- passed last time (6/6) - one more clean pass takes it off this list
-- ✗ /api (Prefix) -> api:8080
-- ✗ / (Prefix) -> web:80
 
-### Q15 · STOR · PersistentVolume, claim and pod  (3/4 failed, last 6/6 pts, 2026-09-15T20:55)
+### Q16 · STOR · Default StorageClass and dynamic provisioning  (4/4 failed, last 0/5 pts, last failed 2026-09-16T19:40)
-- ✗ PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual
-- ✗ PVC pvc-data Bound to pv-data
-- ✗ data-pod Running with claim mounted at /usr/share/data
-
-### Q16 · STOR · Default StorageClass and dynamic provisioning  (4/4 failed, last 0/5 pts, 2026-09-16T19:40)
 - ✗ StorageClass fast-local: local-path, WaitForFirstConsumer, Delete
 - ✗ fast-local is default and standard is not
 - ✗ fast-claim picked up fast-local and is Bound
 - ✗ fast-pod Running with claim at /data
 
-### Q17 · TSHOOT · Node NotReady  (3/4 failed, last 8/8 pts, 2026-09-13T01:06)
+### Q17 · TSHOOT · Node NotReady  (3/4 failed, last 8/8 pts, last failed 2026-09-13T01:06)
-- ✗ kubelet is active
+- passed last time (8/8) - one more clean pass takes it off this list
-- ✗ kubelet is enabled (survives reboot)
 
-### Q19 · TSHOOT · CrashLoopBackOff diagnosis  (4/5 failed, last 6/6 pts, 2026-09-14T16:49)
+### Q19 · TSHOOT · CrashLoopBackOff diagnosis  (4/5 failed, last 6/6 pts, last failed 2026-09-14T16:49)
-- ✗ crasher.log captured and contains the DB_HOST failure
+- passed last time (6/6) - one more clean pass takes it off this list
 
-### Q25 · WORK · Taints and tolerations  (3/5 failed, last 4/4 pts, 2026-09-15T20:55)
+### Q28 · WORK · Init container prepares content  (4/4 failed, last 0/4 pts, last failed 2026-09-16T19:40)
-- ✗ cka-sim-worker carries team=finance:NoSchedule
-- ✗ deployment tolerates the taint
-- ✗ both pods Running on cka-sim-worker
-
-### Q28 · WORK · Init container prepares content  (4/4 failed, last 0/4 pts, 2026-09-16T19:40)
 - ✗ pod Running, init container named setup completed
 - ✗ emptyDir 'content' shared by both containers
 - ✗ nginx serves the init-written content
 
-### Q29 · WORK · Immutable ConfigMap rotation  (3/4 failed, last 4/4 pts, 2026-09-15T20:55)
+### Q29 · WORK · Immutable ConfigMap rotation  (3/4 failed, last 4/4 pts, last failed 2026-09-15T20:55)
-- ✗ deployment now references a different ConfigMap
+- passed last time (4/4) - one more clean pass takes it off this list
-- ✗ all pods available and see mode=new
 
-### Q30 · NET · Named ports  (2/3 failed, last 4/4 pts, 2026-09-14T16:49)
+### Q30 · NET · Named ports  (2/3 failed, last 4/4 pts, last failed 2026-09-14T16:49)
-- ✗ container port 80 named svc-http
+- passed last time (4/4) - one more clean pass takes it off this list
-- ✗ Service port 8080 -> targetPort svc-http (by name)
-- ✗ readiness probe uses the named port; endpoints ready
 
-### Q32 · WORK · ResourceQuota and LimitRange  (1/1 failed, last 0/4 pts, 2026-09-19T12:27)
+### Q32 · WORK · ResourceQuota and LimitRange  (1/1 failed, last 0/4 pts, last failed 2026-09-19T12:27)
 - ✗ ResourceQuota limits pods/6, requests.cpu 1, limits.memory 2Gi
 - ✗ LimitRange defaults set
 - ✗ worker pods Running with defaulted requests
 
-### Q33 · ARCH · Aggregated ClusterRole  (2/3 failed, last 5/5 pts, 2026-09-14T16:49)
+### Q33 · ARCH · Aggregated ClusterRole  (2/3 failed, last 5/5 pts, last failed 2026-09-14T16:49)
-- ✗ ops-19 has an aggregationRule with the right label
+- passed last time (5/5) - one more clean pass takes it off this list
-- ✗ aggregation populated rules from the two labelled roles
-- ✗ sam can list pods and services cluster-wide
 
-### Q37 · TSHOOT · Cluster DNS is broken  (4/5 failed, last 6/6 pts, 2026-09-14T16:49)
+### Q37 · TSHOOT · Cluster DNS is broken  (4/5 failed, last 6/6 pts, last failed 2026-09-14T16:49)
-- ✗ CoreDNS Corefile forwards to the node resolver again
+- passed last time (6/6) - one more clean pass takes it off this list
-- ✗ in-cluster and external names resolve from a pod
 
-### Q38 · TSHOOT · Deployments never create pods  (2/3 failed, last 7/7 pts, 2026-09-13T01:06)
+### Q38 · TSHOOT · Deployments never create pods  (2/3 failed, last 7/7 pts, last failed 2026-09-13T01:06)
-- ✗ kube-controller-manager manifest repaired and pod Running
+- passed last time (7/7) - one more clean pass takes it off this list
-- ✗ a new Deployment gets a ReplicaSet and running pods
 
-### Q39 · TSHOOT · kubelet misconfiguration  (3/5 failed, last 7/7 pts, 2026-09-14T16:49)
+### Q40 · ARCH · kubeadm join token  (4/4 failed, last 0/5 pts, last failed 2026-09-16T19:40)
-- ✗ kubelet config points at a CA file that exists
-- ✗ node Ready
-- ✗ kubelet active and enabled
-
-### Q40 · ARCH · kubeadm join token  (4/4 failed, last 0/5 pts, 2026-09-16T19:40)
 - ✗ join.sh contains a kubeadm join with token and CA hash
 - ✗ the token in the file exists in the cluster with TTL ~24h
 
-### Q42 · NET · Egress-only NetworkPolicy  (2/3 failed, last 4/4 pts, 2026-09-14T16:49)
+### Q42 · NET · Egress-only NetworkPolicy  (2/3 failed, last 4/4 pts, last failed 2026-09-14T16:49)
-- ✗ applies to app=client-64, Egress only
+- passed last time (4/4) - one more clean pass takes it off this list
-- ✗ DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP
-- ✗ TCP 443 to 172.16.0.0/16 only
 
-### Q102 · ARCH · etcd restore  (1/1 failed, last 2/10 pts, 2026-09-19T12:27)
+### Q102 · ARCH · etcd restore  (1/1 failed, last 2/10 pts, last failed 2026-09-19T12:27)
 - ✗ ConfigMap restore-proof-67 exists again with its original data
 - ✗ etcd uses a restored data directory (manifest changed)
 
-### Q106 · ARCH · CRD with validation and defaults  (4/4 failed, last 0/7 pts, 2026-09-19T12:27)
+### Q106 · ARCH · CRD with validation and defaults  (4/4 failed, last 0/7 pts, last failed 2026-09-19T12:27)
 - ✗ CRD exists, Namespaced, replicas has min/max and is required
 - ✗ good created with environment prod
 - ✗ replicas 0 is rejected and the message was saved
 - ✗ defaulting works: defaulted.spec.environment == dev
 
-### Q109 · WORK · Native sidecar with probes and an exact report  (1/1 failed, last 2/8 pts, 2026-09-19T12:27)
+### Q109 · WORK · Native sidecar with probes and an exact report  (1/1 failed, last 2/8 pts, last failed 2026-09-19T12:27)
 - ✗ native sidecar: initContainer agent with restartPolicy Always
 - ✗ probes on app as specified
 - ✗ Running, shared log written by agent, grace period 5
 
-### Q112 · NET · TLS Ingress with a generated certificate  (4/4 failed, last 0/7 pts, 2026-09-19T12:27)
+### Q112 · NET · TLS Ingress with a generated certificate  (4/4 failed, last 0/7 pts, last failed 2026-09-19T12:27)
 - ✗ Secret secure.lab.local-tls is kubernetes.io/tls with CN secure.lab.local
 - ✗ Ingress web: class nginx, host, TLS secret, ssl-redirect annotation
 - ✗ fp.txt matches the secret's certificate fingerprint
 
-### Q115 · STOR · Bind the right local volume  (3/3 failed, last 1/6 pts, 2026-09-15T20:55)
+### Q115 · STOR · Bind the right local volume  (3/3 failed, last 1/6 pts, last failed 2026-09-15T20:55)
 - ✗ PVC uses a selector and is Bound to lv-nvme
 - ✗ consumer Running on the nvme node with /data mounted
 
-### Q120 · TSHOOT · Cluster facts, exactly  (3/3 failed, last 0/5 pts, 2026-09-15T20:55)
+### Q120 · TSHOOT · Cluster facts, exactly  (3/3 failed, last 0/5 pts, last failed 2026-09-15T20:55)
 - ✗ pod-cidr and service-cidr
 - ✗ dns-ip and proxy-mode
 - ✗ cka-sim-worker3 pod CIDR
 
-### Q121 · WORK · The context's namespace  (3/3 failed, last 1/6 pts, 2026-09-15T20:55)
+### Q121 · WORK · The context's namespace  (3/3 failed, last 1/6 pts, last failed 2026-09-15T20:55)
 - ✗ Deployment ledger-21 in ac-echo (the context's namespace), 2 available, owner=ac
 - ✗ Service ledger-21 port 9000 in the same namespace
 - ✗ ns.txt has the namespace
 
-### Q202 · ARCH · Helm chart with CRDs, then a custom resource  (1/1 failed, last 3/8 pts, 2026-09-19T12:27)
+### Q202 · ARCH · Helm chart with CRDs, then a custom resource  (1/1 failed, last 3/8 pts, last failed 2026-09-19T12:27)
 - ✗ watchAllNamespaces=true reached the pod env
 - ✗ Ledger main-ledger has currency USD and retentionDays 30
 
-### Q204 · WORK · Which Pods get evicted first  (1/1 failed, last 0/8 pts, 2026-09-19T12:27)
+### Q204 · WORK · Which Pods get evicted first  (1/1 failed, last 0/8 pts, last failed 2026-09-19T12:27)
 - ✗ evicted-first.txt lists exactly the BestEffort pods
 - ✗ api-qos.txt says Burstable
 - ✗ guaranteed.txt names orion-ledger
 
-### Q217 · TSHOOT · Inspect a container with crictl  (1/1 failed, last 0/10 pts, 2026-09-19T12:27)
+### Q217 · TSHOOT · Inspect a container with crictl  (1/1 failed, last 0/10 pts, last failed 2026-09-19T12:27)
 - ✗ Pod tiger-probe Running with both labels
 - ✗ container.txt holds a real container ID for that Pod
 - ✗ container.txt records the runtimeType
 - ✗ container.log holds the httpd startup output
 
-### Q313 · NET · Two hosts, TLS, and a default backend  (1/1 failed, last 0/11 pts, 2026-09-19T12:27)
+### Q313 · NET · Two hosts, TLS, and a default backend  (1/1 failed, last 0/11 pts, last failed 2026-09-19T12:27)
 - ✗ Secret classic-tls is a TLS secret with CN classic.r700.io
 - ✗ both host rules route to the right Services on /
 - ✗ TLS covers both hosts with classic-tls
@@ -151,126 +123,126 @@ _Generated by exam from 11 attempt(s); regenerated on every `end`._
 
 ## Attempt matrix
 
-| Q | Domain | Title | D1 | E2 | D3 | D4 | D5 | D6 | D7 | D8 | D9 | D10 | E11 |
+| Q | Domain | Title | D1 | E2 | D3 | D4 | D5 | D6 | D7 | D8 | D9 | D10 |
-|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
-| 1 | ARCH | RBAC for a deployment bot | 2/6 | · | · | ✓ | · | · | · | · | · | · | · |
+| 1 | ARCH | RBAC for a deployment bot | 2/6 | · | · | ✓ | · | · | · | · | · | · |
-| 2 | ARCH | etcd snapshot | · | ✓ | · | · | · | · | · | · | · | 0/7 | · |
+| 2 | ARCH | etcd snapshot | · | ✓ | · | · | · | · | · | · | · | 0/7 |
-| 3 | ARCH | Kustomize overlay | · | 0/5 | · | · | · | · | ✓ | · | · | · | · |
+| 3 | ARCH | Kustomize overlay | · | 0/5 | · | · | · | · | ✓ | · | · | · |
-| 4 | ARCH | CustomResourceDefinition | · | ✓ | · | · | · | · | · | · | · | ✓ | · |
+| 4 | ARCH | CustomResourceDefinition | · | ✓ | · | · | · | · | · | · | · | ✓ |
-| 5 | ARCH | Static pod | 0/5 | · | · | · | · | · | ✓ | · | · | · | · |
+| 5 | ARCH | Static pod | 0/5 | · | · | · | · | · | ✓ | · | · | · |
-| 6 | ARCH | Certificates: expiry check and user CSR | 0/6 | · | · | 3/6 | · | · | ✓ | · | · | · | · |
+| 6 | ARCH | Certificates: expiry check and user CSR | 0/6 | · | · | 3/6 | · | · | ✓ | · | · | · |
-| 7 | WORK | Rolling update & scaling | · | · | · | · | · | · | · | · | · | · | · |
+| 7 | WORK | Rolling update & scaling | · | · | · | · | · | · | · | · | · | · |
-| 8 | WORK | Multi-container pod with shared volume | · | 0/4 | · | · | · | · | ✓ | · | · | ✓ | · |
+| 8 | WORK | Multi-container pod with shared volume | · | 0/4 | · | · | · | · | ✓ | · | · | ✓ |
-| 9 | WORK | Scheduling with node affinity and resources | · | · | · | · | · | · | · | · | · | ✓ | · |
+| 9 | WORK | Scheduling with node affinity and resources | · | · | · | · | · | · | · | · | · | ✓ |
-| 10 | WORK | ConfigMap and Secret consumption | · | · | · | · | · | · | · | · | · | · | · |
+| 10 | WORK | ConfigMap and Secret consumption | · | · | · | · | · | · | · | · | · | · |
-| 11 | NET | NetworkPolicy | · | · | · | · | · | · | · | · | · | · | · |
+| 11 | NET | NetworkPolicy | · | · | · | · | · | · | · | · | · | · |
-| 12 | NET | NodePort Service | · | 0/4 | · | · | · | · | ✓ | · | · | · | · |
+| 12 | NET | NodePort Service | · | 0/4 | · | · | · | · | ✓ | · | · | · |
-| 13 | NET | Ingress routing | · | 0/5 | · | · | · | · | 0/5 | ✓ | · | ✓ | · |
+| 13 | NET | Ingress routing | · | 0/5 | · | · | · | · | 0/5 | ✓ | · | ✓ |
-| 14 | NET | Cluster DNS | · | · | · | · | · | · | · | · | · | · | · |
+| 14 | NET | Cluster DNS | · | · | · | · | · | · | · | · | · | · |
-| 15 | STOR | PersistentVolume, claim and pod | · | 0/6 | · | · | · | · | 0/6 | 0/6 | ✓ | · | · |
+| 15 | STOR | PersistentVolume, claim and pod | · | 0/6 | · | · | · | · | 0/6 | 0/6 | ✓ | · |
-| 16 | STOR | Default StorageClass and dynamic provisioning | · | 0/5 | · | · | · | · | 0/5 | 0/5 | 0/5 | · | · |
+| 16 | STOR | Default StorageClass and dynamic provisioning | · | 0/5 | · | · | · | · | 0/5 | 0/5 | 0/5 | · |
-| 17 | TSHOOT | Node NotReady | · | 0/8 | · | · | 4/8 | 4/8 | ✓ | · | · | · | · |
+| 17 | TSHOOT | Node NotReady | · | 0/8 | · | · | 4/8 | 4/8 | ✓ | · | · | · |
-| 18 | TSHOOT | Broken deployment and service | · | 0/7 | · | · | · | · | ✓ | · | · | · | · |
+| 18 | TSHOOT | Broken deployment and service | · | 0/7 | · | · | · | · | ✓ | · | · | · |
-| 19 | TSHOOT | CrashLoopBackOff diagnosis | · | 1/6 | · | · | 1/6 | 1/6 | 4/6 | ✓ | · | · | · |
+| 19 | TSHOOT | CrashLoopBackOff diagnosis | · | 1/6 | · | · | 1/6 | 1/6 | 4/6 | ✓ | · | · |
-| 20 | TSHOOT | Control plane component failure | · | 0/8 | · | · | · | · | ✓ | · | · | · | · |
+| 20 | TSHOOT | Control plane component failure | · | 0/8 | · | · | · | · | ✓ | · | · | · |
-| 21 | ARCH | Helm release install and upgrade | · | · | · | ✓ | · | · | · | · | · | · | · |
+| 21 | ARCH | Helm release install and upgrade | · | · | · | ✓ | · | · | · | · | · | · |
-| 22 | NET | Gateway API HTTPRoute | · | · | · | · | · | · | · | · | · | · | · |
+| 22 | NET | Gateway API HTTPRoute | · | · | · | · | · | · | · | · | · | · |
-| 23 | WORK | metrics-server repair and HPA | · | · | · | · | · | · | · | · | · | · | · |
+| 23 | WORK | metrics-server repair and HPA | · | · | · | · | · | · | · | · | · | · |
-| 24 | WORK | PriorityClass | · | · | · | · | · | · | · | · | · | · | · |
+| 24 | WORK | PriorityClass | · | · | · | · | · | · | · | · | · | · |
-| 25 | WORK | Taints and tolerations | · | 0/4 | · | · | · | · | 0/4 | 0/4 | ✓ | ✓ | · |
+| 25 | WORK | Taints and tolerations | · | 0/4 | · | · | · | · | 0/4 | 0/4 | ✓ | ✓ |
-| 26 | WORK | DaemonSet on every node | · | · | · | · | · | · | · | · | · | · | · |
+| 26 | WORK | DaemonSet on every node | · | · | · | · | · | · | · | · | · | · |
-| 27 | WORK | CronJob with a manual run | · | · | · | · | · | · | · | · | · | · | · |
+| 27 | WORK | CronJob with a manual run | · | · | · | · | · | · | · | · | · | · |
-| 28 | WORK | Init container prepares content | · | 0/4 | · | · | · | · | 0/4 | 0/4 | 0/4 | · | · |
+| 28 | WORK | Init container prepares content | · | 0/4 | · | · | · | · | 0/4 | 0/4 | 0/4 | · |
-| 29 | WORK | Immutable ConfigMap rotation | · | 1/4 | · | · | · | · | 1/4 | 1/4 | ✓ | · | · |
+| 29 | WORK | Immutable ConfigMap rotation | · | 1/4 | · | · | · | · | 1/4 | 1/4 | ✓ | · |
-| 30 | NET | Named ports | · | 0/4 | · | · | · | · | 0/4 | ✓ | · | · | · |
+| 30 | NET | Named ports | · | 0/4 | · | · | · | · | 0/4 | ✓ | · | · |
-| 31 | WORK | Parallel Job | · | · | · | · | · | · | · | · | · | · | · |
+| 31 | WORK | Parallel Job | · | · | · | · | · | · | · | · | · | · |
-| 32 | WORK | ResourceQuota and LimitRange | · | · | · | · | · | · | · | · | · | 0/4 | · |
+| 32 | WORK | ResourceQuota and LimitRange | · | · | · | · | · | · | · | · | · | 0/4 |
-| 33 | ARCH | Aggregated ClusterRole | · | 1/5 | · | · | · | · | 1/5 | ✓ | · | · | · |
+| 33 | ARCH | Aggregated ClusterRole | · | 1/5 | · | · | · | · | 1/5 | ✓ | · | · |
-| 34 | ARCH | Discover a CRD field with kubectl explain | 0/4 | · | · | ✓ | · | · | · | · | · | ✓ | · |
+| 34 | ARCH | Discover a CRD field with kubectl explain | 0/4 | · | · | ✓ | · | · | · | · | · | ✓ |
-| 35 | STOR | Re-bind a Released PersistentVolume | · | · | · | · | · | · | · | · | · | · | · |
+| 35 | STOR | Re-bind a Released PersistentVolume | · | · | · | · | · | · | · | · | · | · |
-| 36 | STOR | Data survives the pod | · | · | · | · | · | · | · | · | · | · | · |
+| 36 | STOR | Data survives the pod | · | · | · | · | · | · | · | · | · | · |
-| 37 | TSHOOT | Cluster DNS is broken | · | 1/6 | · | · | 1/6 | 1/6 | 1/6 | ✓ | · | · | · |
+| 37 | TSHOOT | Cluster DNS is broken | · | 1/6 | · | · | 1/6 | 1/6 | 1/6 | ✓ | · | · |
-| 38 | TSHOOT | Deployments never create pods | · | · | · | · | 0/7 | 0/7 | ✓ | · | · | · | · |
+| 38 | TSHOOT | Deployments never create pods | · | · | · | · | 0/7 | 0/7 | ✓ | · | · | · |
-| 39 | TSHOOT | kubelet misconfiguration | · | · | · | · | 4/7 | 4/7 | 0/7 | ✓ | · | ✓ | · |
+| 39 | TSHOOT | kubelet misconfiguration | · | · | · | · | 4/7 | 4/7 | 0/7 | ✓ | · | ✓ |
-| 40 | ARCH | kubeadm join token | · | 0/5 | · | · | · | · | 0/5 | 0/5 | 0/5 | · | · |
+| 40 | ARCH | kubeadm join token | · | 0/5 | · | · | · | · | 0/5 | 0/5 | 0/5 | · |
-| 41 | ARCH | Component versions | 0/4 | · | · | ✓ | · | · | · | · | · | · | · |
+| 41 | ARCH | Component versions | 0/4 | · | · | ✓ | · | · | · | · | · | · |
-| 42 | NET | Egress-only NetworkPolicy | · | 0/4 | · | · | · | · | 0/4 | ✓ | · | · | · |
+| 42 | NET | Egress-only NetworkPolicy | · | 0/4 | · | · | · | · | 0/4 | ✓ | · | · |
-| 43 | NET | Headless and ExternalName services | · | · | · | · | · | · | · | · | · | · | · |
+| 43 | NET | Headless and ExternalName services | · | · | · | · | · | · | · | · | · | · |
-| 101 | ARCH | Exact RBAC with an existing over-grant | · | · | · | · | · | · | · | · | · | · | · |
+| 101 | ARCH | Exact RBAC with an existing over-grant | · | · | · | · | · | · | · | · | · | · |
-| 102 | ARCH | etcd restore | · | · | · | · | · | · | · | · | · | 2/10 | · |
+| 102 | ARCH | etcd restore | · | · | · | · | · | · | · | · | · | 2/10 |
-| 103 | ARCH | Static pod on a relocated manifest path | · | · | · | · | · | · | · | · | · | · | · |
+| 103 | ARCH | Static pod on a relocated manifest path | · | · | · | · | · | · | · | · | · | · |
-| 104 | ARCH | User onboarding to a working kubeconfig | · | · | · | · | · | · | · | · | · | · | · |
+| 104 | ARCH | User onboarding to a working kubeconfig | · | · | · | · | · | · | · | · | · | · |
-| 105 | ARCH | Kustomize with generators and patches | · | · | · | · | · | · | · | · | · | · | · |
+| 105 | ARCH | Kustomize with generators and patches | · | · | · | · | · | · | · | · | · | · |
-| 106 | ARCH | CRD with validation and defaults | · | · | 0/7 | · | · | · | 0/7 | 0/7 | · | 0/7 | · |
+| 106 | ARCH | CRD with validation and defaults | · | · | 0/7 | · | · | · | 0/7 | 0/7 | · | 0/7 |
-| 107 | WORK | Spread, tolerate, prioritise — under a quota | · | · | · | · | · | · | · | · | · | · | · |
+| 107 | WORK | Spread, tolerate, prioritise — under a quota | · | · | · | · | · | · | · | · | · | · |
-| 108 | WORK | Rollback to a specific revision, then pause | · | · | · | · | · | · | · | · | · | · | · |
+| 108 | WORK | Rollback to a specific revision, then pause | · | · | · | · | · | · | · | · | · | · |
-| 109 | WORK | Native sidecar with probes and an exact report | · | · | · | · | · | · | · | · | · | 2/8 | · |
+| 109 | WORK | Native sidecar with probes and an exact report | · | · | · | · | · | · | · | · | · | 2/8 |
-| 110 | WORK | Cluster audit to exact files | · | · | · | · | · | · | · | · | · | · | · |
+| 110 | WORK | Namespace audit to exact files | · | · | · | · | · | · | · | · | · | · |
-| 111 | NET | Zero-trust namespace | · | · | · | · | · | · | · | · | · | · | · |
+| 111 | NET | Zero-trust namespace | · | · | · | · | · | · | · | · | · | · |
-| 112 | NET | TLS Ingress with a generated certificate | · | · | 0/7 | · | · | · | 0/7 | 0/7 | · | 0/7 | · |
+| 112 | NET | TLS Ingress with a generated certificate | · | · | 0/7 | · | · | · | 0/7 | 0/7 | · | 0/7 |
-| 113 | NET | Gateway API canary with header match | · | · | · | · | · | · | · | · | · | · | · |
+| 113 | NET | Gateway API canary with header match | · | · | · | · | · | · | · | · | · | · |
-| 114 | NET | DNS down: three faults | · | · | · | · | · | · | · | · | · | · | · |
+| 114 | NET | DNS down: three faults | · | · | · | · | · | · | · | · | · | · |
-| 115 | STOR | Bind the right local volume | · | · | 1/6 | · | · | · | 1/6 | 1/6 | · | · | · |
+| 115 | STOR | Bind the right local volume | · | · | 1/6 | · | · | · | 1/6 | 1/6 | · | · |
-| 116 | STOR | Migrate data between claims | · | · | · | · | · | · | · | · | · | · | · |
+| 116 | STOR | Migrate data between claims | · | · | · | · | · | · | · | · | · | · |
-| 117 | TSHOOT | Worker node: three things wrong | · | · | · | · | · | · | · | · | · | · | · |
+| 117 | TSHOOT | Worker node: three things wrong | · | · | · | · | · | · | · | · | · | · |
-| 118 | TSHOOT | Two control-plane components down | · | · | · | · | · | · | · | · | · | · | · |
+| 118 | TSHOOT | Two control-plane components down | · | · | · | · | · | · | · | · | · | · |
-| 119 | TSHOOT | Application down for four reasons | · | · | · | · | · | · | · | · | · | · | · |
+| 119 | TSHOOT | Application down for four reasons | · | · | · | · | · | · | · | · | · | · |
-| 120 | TSHOOT | Cluster facts, exactly | · | · | 0/5 | · | · | · | 0/5 | 0/5 | · | · | · |
+| 120 | TSHOOT | Cluster facts, exactly | · | · | 0/5 | · | · | · | 0/5 | 0/5 | · | · |
-| 121 | WORK | The context's namespace | · | · | 1/6 | · | · | · | 1/6 | 1/6 | · | · | · |
+| 121 | WORK | The context's namespace | · | · | 1/6 | · | · | · | 1/6 | 1/6 | · | · |
-| 122 | ARCH | Renew control-plane certificates | · | · | · | · | · | · | · | · | · | · | · |
+| 122 | ARCH | Renew control-plane certificates | · | · | · | · | · | · | · | · | · | · |
-| 123 | TSHOOT | Nothing works: kube-apiserver itself is down | · | · | · | · | · | · | · | · | · | · | · |
+| 123 | TSHOOT | Nothing works: kube-apiserver itself is down | · | · | · | · | · | · | · | · | · | · |
-| 124 | ARCH | Prepare a node join without shell access to the control plane | · | · | · | · | · | · | · | · | · | · | · |
+| 124 | ARCH | Prepare a node join without shell access to the control plane | · | · | · | · | · | · | · | · | · | · |
-| 201 | ARCH | kubeconfig forensics | · | · | · | · | · | · | · | · | · | · | · |
+| 201 | ARCH | kubeconfig forensics | · | · | · | · | · | · | · | · | · | · |
-| 202 | ARCH | Helm chart with CRDs, then a custom resource | · | · | · | · | · | · | · | · | · | 3/8 | · |
+| 202 | ARCH | Helm chart with CRDs, then a custom resource | · | · | · | · | · | · | · | · | · | 3/8 |
-| 203 | WORK | Find the controller, then scale it | · | · | · | · | · | · | · | · | · | · | · |
+| 203 | WORK | Find the controller, then scale it | · | · | · | · | · | · | · | · | · | · |
-| 204 | WORK | Which Pods get evicted first | · | · | · | · | · | · | · | · | · | 0/8 | · |
+| 204 | WORK | Which Pods get evicted first | · | · | · | · | · | · | · | · | · | 0/8 |
-| 205 | ARCH | Kustomize: drop a ConfigMap, add an HPA, override in prod | · | · | · | · | · | · | · | · | · | · | · |
+| 205 | ARCH | Kustomize: drop a ConfigMap, add an HPA, override in prod | · | · | · | · | · | · | · | · | · | · |
-| 206 | STOR | PV, PVC and a Deployment that mounts it | · | · | · | · | · | · | · | · | · | · | · |
+| 206 | STOR | PV, PVC and a Deployment that mounts it | · | · | · | · | · | · | · | · | · | · |
-| 207 | TSHOOT | Resource usage scripts | · | · | · | · | · | · | · | · | · | · | · |
+| 207 | TSHOOT | Resource usage scripts | · | · | · | · | · | · | · | · | · | · |
-| 208 | ARCH | Drain a node for maintenance | · | · | · | · | · | · | · | · | · | · | · |
+| 208 | ARCH | Drain a node for maintenance | · | · | · | · | · | · | · | · | · | · |
-| 209 | ARCH | Query the API from inside a Pod | · | · | · | · | · | · | · | · | · | · | · |
+| 209 | ARCH | Query the API from inside a Pod | · | · | · | · | · | · | · | · | · | · |
-| 210 | ARCH | ServiceAccount limited to two resources | · | · | · | · | · | · | · | · | · | · | · |
+| 210 | ARCH | ServiceAccount limited to two resources | · | · | · | · | · | · | · | · | · | · |
-| 211 | WORK | DaemonSet on every node | · | · | · | · | · | · | · | · | · | · | · |
+| 211 | WORK | DaemonSet on every node | · | · | · | · | · | · | · | · | · | · |
-| 212 | WORK | One Pod per node, on purpose | · | · | · | · | · | · | · | · | · | · | · |
+| 212 | WORK | One Pod per node, on purpose | · | · | · | · | · | · | · | · | · | · |
-| 213 | NET | Convert an Ingress to Gateway API, then split on a header | · | · | · | · | · | · | · | · | · | · | · |
+| 213 | NET | Convert an Ingress to Gateway API, then split on a header | · | · | · | · | · | · | · | · | · | · |
-| 214 | ARCH | Certificate expiry, two ways | · | · | · | · | · | · | · | · | · | · | · |
+| 214 | ARCH | Certificate expiry, two ways | · | · | · | · | · | · | · | · | · | · |
-| 215 | NET | Egress policy with paired destinations | · | · | · | · | · | · | · | · | · | · | · |
+| 215 | NET | Egress policy with paired destinations | · | · | · | · | · | · | · | · | · | · |
-| 216 | NET | CoreDNS: back up, extend, verify | · | · | · | · | · | · | · | · | · | · | · |
+| 216 | NET | CoreDNS: back up, extend, verify | · | · | · | · | · | · | · | · | · | · |
-| 217 | TSHOOT | Inspect a container with crictl | · | · | · | · | · | · | · | · | · | 0/10 | · |
+| 217 | TSHOOT | Inspect a container with crictl | · | · | · | · | · | · | · | · | · | 0/10 |
-| 301 | ARCH | etcd configuration facts | · | · | · | · | · | · | · | · | · | ✓ | · |
+| 301 | ARCH | etcd configuration facts | · | · | · | · | · | · | · | · | · | ✓ |
-| 302 | NET | Service rules on the node | · | · | · | · | · | · | · | · | · | · | · |
+| 302 | NET | Service rules on the node | · | · | · | · | · | · | · | · | · | · |
-| 303 | NET | A second Service CIDR | · | · | · | · | · | · | · | · | · | · | · |
+| 303 | NET | A second Service CIDR | · | · | · | · | · | · | · | · | · | · |
-| 304 | ARCH | Bound tokens versus legacy tokens | · | · | · | · | · | · | · | · | · | · | · |
+| 304 | ARCH | Bound tokens versus legacy tokens | · | · | · | · | · | · | · | · | · | · |
-| 305 | WORK | Schedule without the scheduler | · | · | · | · | · | · | · | · | · | · | · |
+| 305 | WORK | Schedule without the scheduler | · | · | · | · | · | · | · | · | · | · |
-| 306 | STOR | Grow a volume in place | · | · | · | · | · | · | · | · | · | · | · |
+| 306 | STOR | Grow a volume in place | · | · | · | · | · | · | · | · | · | · |
-| 307 | TSHOOT | Node accepts nothing new | · | · | · | · | · | · | · | · | · | · | · |
+| 307 | TSHOOT | Node accepts nothing new | · | · | · | · | · | · | · | · | · | · |
-| 308 | ARCH | API resource reference card | · | · | · | · | · | · | · | · | · | · | · |
+| 308 | ARCH | API resource reference card | · | · | · | · | · | · | · | · | · | · |
-| 309 | WORK | A PodDisruptionBudget that blocks maintenance | · | · | · | · | · | · | · | · | · | · | · |
+| 309 | WORK | A PodDisruptionBudget that blocks maintenance | · | · | · | · | · | · | · | · | · | · |
-| 310 | NET | Route to something outside the cluster | · | · | · | · | · | · | · | · | · | · | · |
+| 310 | NET | Route to something outside the cluster | · | · | · | · | · | · | · | · | · | · |
-| 311 | ARCH | Image inventory across the cluster | · | · | · | · | · | · | · | · | · | · | · |
+| 311 | ARCH | Image inventory across the cluster | · | · | · | · | · | · | · | · | · | · |
-| 312 | WORK | A Job that fails, then one that doesn't | · | · | · | · | · | · | · | · | · | · | · |
+| 312 | WORK | A Job that fails, then one that doesn't | · | · | · | · | · | · | · | · | · | · |
-| 313 | NET | Two hosts, TLS, and a default backend | · | · | · | · | · | · | · | · | · | 0/11 | · |
+| 313 | NET | Two hosts, TLS, and a default backend | · | · | · | · | · | · | · | · | · | 0/11 |
-| 314 | TSHOOT | A static pod that never starts | · | · | · | · | · | · | · | · | · | · | · |
+| 314 | TSHOOT | A static pod that never starts | · | · | · | · | · | · | · | · | · | · |
-| 315 | STOR | Keep the data when the claim goes | · | · | · | · | · | · | · | · | · | · | · |
+| 315 | STOR | Keep the data when the claim goes | · | · | · | · | · | · | · | · | · | · |
-| 316 | ARCH | Plan a control-plane upgrade | · | · | · | · | · | · | · | · | · | · | · |
+| 316 | ARCH | Plan a control-plane upgrade | · | · | · | · | · | · | · | · | · | · |
-| 317 | TSHOOT | Certificates stop being issued | · | · | · | · | · | · | · | · | · | · | · |
+| 317 | TSHOOT | Certificates stop being issued | · | · | · | · | · | · | · | · | · | · |
-| 401 | STOR | Provision geo-fence persistent storage | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 401 | STOR | Provision geo-fence persistent storage | · | · | · | · | · | · | · | · | · | · |
-| 402 | ARCH | Render a chart, then install it | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 402 | ARCH | Render a chart, then install it | · | · | · | · | · | · | · | · | · | · |
-| 403 | TSHOOT | Fix the driver onboarding app | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 403 | TSHOOT | Fix the driver onboarding app | · | · | · | · | · | · | · | · | · | · |
-| 404 | NET | Expose and verify the dispatch API | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 404 | NET | Expose and verify the dispatch API | · | · | · | · | · | · | · | · | · | · |
-| 405 | STOR | Migrate ride history to a Retain volume | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 405 | STOR | Migrate ride history to a Retain volume | · | · | · | · | · | · | · | · | · | · |
-| 406 | TSHOOT | Fix ETA service connectivity | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 406 | TSHOOT | Fix ETA service connectivity | · | · | · | · | · | · | · | · | · | · |
-| 407 | NET | Fleet dashboard canary with Gateway API | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 407 | NET | Fleet dashboard canary with Gateway API | · | · | · | · | · | · | · | · | · | · |
-| 408 | ARCH | Cluster-wide read access for the auth team | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 408 | ARCH | Cluster-wide read access for the auth team | · | · | · | · | · | · | · | · | · | · |
-| 409 | WORK | Add an init container to the trip logger | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 409 | WORK | Add an init container to the trip logger | · | · | · | · | · | · | · | · | · | · |
-| 410 | NET | Audit the CNI, then lock a namespace down | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 410 | NET | Audit the CNI, then lock a namespace down | · | · | · | · | · | · | · | · | · | · |
-| 411 | WORK | Priority for the rider support portal | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 411 | WORK | Priority for the rider support portal | · | · | · | · | · | · | · | · | · | · |
-| 412 | WORK | Surge pricing configuration | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 412 | WORK | Surge pricing configuration | · | · | · | · | · | · | · | · | · | · |
-| 413 | ARCH | Plan the control-plane upgrade | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 413 | ARCH | Plan the control-plane upgrade | · | · | · | · | · | · | · | · | · | · |
-| 414 | TSHOOT | Fix the fare calculation pods | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 414 | TSHOOT | Fix the fare calculation pods | · | · | · | · | · | · | · | · | · | · |
-| 415 | WORK | Autoscale the route planner on memory | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 415 | WORK | Autoscale the route planner on memory | · | · | · | · | · | · | · | · | · | · |
-| 416 | NET | NetworkPolicy for the safety monitor | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 416 | NET | NetworkPolicy for the safety monitor | · | · | · | · | · | · | · | · | · | · |
-| 417 | ARCH | Enforce TLS 1.3 on the booking portal | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 417 | ARCH | Enforce TLS 1.3 on the booking portal | · | · | · | · | · | · | · | · | · | · |
-| 418 | TSHOOT | Investigate vehicle tracker health | · | · | · | · | · | · | · | · | · | · | ✓ |
+| 418 | TSHOOT | Investigate vehicle tracker health | · | · | · | · | · | · | · | · | · | · |
 
 _E = full exam, D = drill, · = not in that session, n/m = partial credit._

web/index.html

@@ -186,8 +186,7 @@
       <button class="opt" id="optNext"><b>Drill next domain</b><span id="nextDesc"></span></button>
       <button class="opt" id="optAll"><b>Drill the whole bank</b><span id="allDesc"></span></button>
       <button class="opt" id="optDrill"><b>Drill: weak / pick</b><span>Tasks you're weak on, or choose your own. Feedback per task while you work.</span></button>
-      <button class="opt" id="optSimA"><b>Simulator Set A</b><span id="simADesc"></span></button>
+      <span id="simOpts" style="display:contents"></span>
-      <button class="opt" id="optSimB"><b>Simulator Set B</b><span id="simBDesc"></span></button>
       <button class="opt" id="optExam" style="grid-column:1/3"><b>Full exam</b><span id="examDesc"></span></button>
     </div>
     <div id="gate" class="hidden" style="color:var(--warn);margin-top:10px;font-size:13px"></div>
@@ -348,11 +347,11 @@
     const eligible = S.bank.filter(q => hard || q.difficulty !== 'hard'), seenN = eligible.filter(q => (S.seen || []).includes(q.id)).length;
     $('#allDesc').textContent = `${seenN}/${eligible.length} tasks attempted. 5 unseen tasks per session, round-robin across domains${hard ? ', hard first' : ''}; then whatever you failed.`;
     $('#nextDesc').textContent = `${S.track.domains[nxt]} — 5 tasks, ${hard ? 5 : 6} min each. Drills rotate through all domains before the exam.`;
-    for (const s of ['a', 'b']) {
+    // one button per fixed set the track defines (Set C and later were unreachable from here when
-      const info = (S.sets || {})[s];
+    // the buttons were hard-coded to A and B)
-      const el = $('#sim' + s.toUpperCase() + 'Desc');
+    $('#simOpts').innerHTML = Object.entries(S.sets || {}).map(([k, info]) =>
-      if (el) el.textContent = info ? `${info.count} fixed tasks, ${S.track.minutes} min, exam rules. Hardest set in the bank — closest to a real sitting.` : '';
+      `<button class="opt" data-sim="${k}"><b>${esc(info.name)}</b><span>${info.count} fixed tasks, ${S.track.minutes} min, exam rules, no feedback until you submit.</span></button>`).join('');
-    }
+    $('#simOpts').querySelectorAll('[data-sim]').forEach(el => el.onclick = () => startSim(el.dataset.sim));
     $('#gate').classList.toggle('hidden', !undone.length);
     $('#gate').innerHTML = undone.length ? `Drill every domain once before the full exam. Not yet drilled: ${undone.join(', ')}. <a href="#" id="forceExam">Start the exam anyway</a>` : '';
     $('#optExam').disabled = !!undone.length; $('#optExam').style.opacity = undone.length ? .5 : 1;
@@ -451,8 +450,6 @@
   $('#optNext').onclick = () => start('drill', null, true);
   $('#optStudy').onclick = () => start('drill', null, true, false, null, true);
   $('#optAll').onclick = () => start('drill', null, false, true);
-  $('#optSimA').onclick = () => startSim('a');
-  $('#optSimB').onclick = () => startSim('b');
   $('#hardMode').onchange = () => load();
   $('#optDrill').onclick = async () => {
     const h = await api('/api/history');