feat(ckad): cover the whole curriculum - 37 tasks, 16-task mocks

The CKAD track was a 12-task starter bank whose quota (3/3/2/2/2) drew the
whole bank, so every mock was the same 12 tasks reshuffled, and about half
of the curriculum had no task at all.

Tasks 13-37 (tracks/ckad/bank_extra.py) give every missing competency at
least one task: build and export an image with podman (OCI archive), Job
completions and parallelism, a DaemonSet on every node, a native sidecar
that lets a Job finish, a PVC and an in-memory emptyDir, a canary by replica
ratio, rolling-update limits, a Kustomize overlay, a Helm values file and an
upgrade that keeps values, manifests on removed APIs, kubectl top, logs of a
crashing container, two stacked faults, OOMKilled, a custom resource, RBAC
for a ServiceAccount, Pod Security admission, ResourceQuota with LimitRange
defaults, Guaranteed QoS, ConfigMaps, Secrets, ServiceAccount tokens, an
egress NetworkPolicy tested with real traffic, a Service with two faults and
an Ingress with TLS.

A mock now draws 16 tasks weighted like the exam (DESIGN 3, DEPLOY 3, OBS 3,
ENV 4, NET 3) from a pool of 8/7/7/10/5. New ensure_metrics_server helper so
`kubectl top` works on a freshly built cluster. Starter tasks now delete
their Pods with --wait=true, so a re-seed cannot apply onto a Pod that is
still terminating, and setups that create bare Pods fail loudly if one is
missing. README gains a CKAD section.

Verified on a throwaway cluster: all 12 starter and 25 new tasks score 0
untouched and full when solved; four full 16-task mocks that together
include all 37 tasks scored 122/122, 118/118, 124/124 and 119/119.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mhxm4rsUfuE5HoytYbu2YW

README.md

@@ -5,8 +5,9 @@ Entry point is `./exam` (renamed from `cka-sim` so it can't collide with the dir
 Hands-on exam tasks graded **per check** against a real cluster, an exam clock, a browser
 front end modelled on the real PSI exam, attempt history, and drills that rotate through the
 domains before you sit a full exam. Tracks: **CKA** (102 tasks: a random pool of 43 normal +
-24 hard, plus two fixed simulator sets, C and D, of 18 and 17 tasks), CKAD and CKS (scaffolded —
+24 hard, plus two fixed simulator sets, C and D, of 18 and 17 tasks), **CKAD** (37 tasks, at
-same format, add questions when you get there).
+least one for every competency in the curriculum) and CKS (scaffolded — same format, add
+questions when you get there).
 
 ## How a session is built
 
@@ -73,10 +74,25 @@ The same state is available from the terminal if you prefer:
 ./exam list | show N | timer | status                       # during
 ./exam end                                                  # submit
 ./exam weak | history                                       # afterwards
-./exam --track ckad web                                     # other tracks
+./exam --track ckad web                                     # other tracks (or CKA_TRACK=ckad)
 ./teardown.sh                                                  # delete cluster + state (history kept)
 ```
 
+## CKAD track
+
+`./exam --track ckad ...` (every command takes it). A mock draws **16** tasks in 120 minutes,
+weighted like the exam: Design and Build 3, Deployment 3, Observability 3, Environment and
+Security 4, Networking 3, from a pool of 37 (8/7/7/10/5), so no two mocks are the same set. Every
+line of the curriculum has at least one task: image builds, Jobs, DaemonSets, native sidecars,
+volumes; blue/green, canary, rolling-update limits, Helm, Kustomize; API deprecations, probes,
+`kubectl top`, logs, debugging (CrashLoopBackOff, OOMKilled, config errors); CRDs, RBAC, Pod
+Security admission, quotas and LimitRanges, QoS, ConfigMaps, Secrets, ServiceAccounts;
+NetworkPolicy (tested with real traffic), Services, Ingress with TLS.
+
+A few tasks need more than kubectl on this machine: `podman` (Q13 builds an image), `helm`
+(Q6, Q21), and network access the first time Q23 runs on a fresh cluster (it installs
+metrics-server). Drill a domain with `./exam --track ckad drill ENV`, as for CKA.
+
 ## Scoring, history, drills
 
 Each task is a list of named, weighted checks (`tracks/cka/questions.py`). You get the points
@@ -158,5 +174,5 @@ Banks live in `tracks/cka/{questions,bank_extra,bank_hard,bank_sim_*}.py`; a tas
 Scripts get `$K` (pinned to the grading context), `$WS`, `$CP/$W1/$W2/$W3`, `nx NODE 'cmd'`,
 and helpers `j <get args> '{jsonpath}'` and `eq a b`. Task text may use `{WS}`, `{CP}`, `{W2}`,
 `{W3}`, `{SHELL_CP}` etc. `TRACK["quota"]` / `["quota_hard"]` set how many tasks per domain a
-session draws. For CKAD/CKS, fill in `QUESTIONS` in the
+session draws. For CKS, fill in `QUESTIONS` in the
 existing placeholder track; `TRACK` already carries the domain table and pass mark.

exam

@@ -136,6 +136,20 @@ ensure_gateway_crds() {
     && $K wait --for=condition=Established --timeout=60s crd/gatewayclasses.gateway.networking.k8s.io \
          crd/gateways.gateway.networking.k8s.io crd/httproutes.gateway.networking.k8s.io >/dev/null 2>&1
 }
+# ensure_metrics_server -> a working metrics API for `kubectl top` (CKAD Q23). kind ships none, and
+# kind's kubelet serving certs have no IP SANs, so it only becomes Ready with --kubelet-insecure-tls.
+# CKA Q23 strips that flag on purpose; the baseline puts it back before every session.
+ensure_metrics_server() {
+  if ! $K -n kube-system get deploy metrics-server >/dev/null 2>&1; then
+    curl -sfL --max-time 60 https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -o /tmp/ms.yaml \
+      && $K apply -f /tmp/ms.yaml >/dev/null 2>&1 || return 1
+  fi
+  if ! $K -n kube-system get deploy metrics-server -o jsonpath='{.spec.template.spec.containers[0].args}' | grep -q kubelet-insecure-tls; then
+    $K -n kube-system patch deploy metrics-server --type=json \
+      -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' >/dev/null 2>&1
+  fi
+  $K -n kube-system rollout status deploy metrics-server --timeout=150s >/dev/null 2>&1
+}
 """
 
 def _pick_track_arg(argv):
@@ -644,11 +658,22 @@ def _log_msg(*args: Any, **kwargs: Any) -> None:
 # restores every known "breaks" target to a good baseline before each session seeds, so
 # leftover damage never leaks across sessions regardless of which questions get sampled.
 RESTORE_BASELINE = r"""
+# Static-pod manifests are only ever rewritten when an edit actually changes them, and through a
+# hidden temp file the kubelet ignores. `sed -i` writes its temp file (sedXXXXXX) into the SAME
+# directory, the kubelet loads that as one more static pod and kills the running one: every session
+# start used to knock the scheduler and controller-manager into CrashLoopBackOff for up to minutes
+# while the session's tasks were being seeded (new Namespaces got no ServiceAccount, so bare Pods
+# were rejected; Pods sat unscheduled). The user's cluster had 10 such restarts by 2026-09-24.
+MEDIT='medit() { f=$1; shift; t=$(mktemp "${f%/*}/.medit.XXXXXX") || return 1
+  if sed "$@" "$f" > "$t" 2>/dev/null && [ "$(md5sum < "$t")" != "$(md5sum < "$f")" ]; then
+    chmod --reference="$f" "$t" 2>/dev/null; mv -f "$t" "$f"; else rm -f "$t"; fi; }'
+
 # apiserver FIRST: Q66 renames the command to "kube-apiserverr", which takes the whole API
 # down - every kubectl-based step below would fail until this is undone. The repair goes over
 # podman/ssh (nx), not kubectl, so it works precisely when the cluster is unreachable.
 if nx "$CP" 'grep -q "^ *- kube-apiserverr$" /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null 2>/dev/null; then
-  nx "$CP" 'sed -i "s|^\( *\)- kube-apiserverr$|\1- kube-apiserver|" /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null
+  nx "$CP" "$MEDIT"'
+medit /etc/kubernetes/manifests/kube-apiserver.yaml -e "s|^\( *\)- kube-apiserverr$|\1- kube-apiserver|"' </dev/null
   # the kubelet re-reads the manifest on its own; wait for the API to answer again
   for _ in $(seq 1 30); do kubectl --context "${CKA_CONTEXT:-kind-cka-sim}" get --raw=/readyz >/dev/null 2>&1 && break; sleep 2; done
 fi
@@ -656,22 +681,22 @@ fi
 # scheduler: undo the "kube-scheduler" -> "kube-schedulerr" command typo (Q20/Q61), and put the
 # manifest back if a task had the candidate stop the scheduler by moving it out (Q97) and they
 # never moved it back - that setup keeps a pristine copy outside the static pod directory.
-nx "$CP" '
+nx "$CP" "$MEDIT"'
 f=/etc/kubernetes/manifests/kube-scheduler.yaml
 [ -f $f ] || { [ -f /etc/kubernetes/.cka-sim/kube-scheduler.yaml ] && cp /etc/kubernetes/.cka-sim/kube-scheduler.yaml $f; }
-sed -i "s|^\( *\)- kube-schedulerr$|\1- kube-scheduler|" $f 2>/dev/null
+medit $f -e "s|^\( *\)- kube-schedulerr$|\1- kube-scheduler|"
 ' </dev/null
 
 # controller-manager: undo Q38's injected flag and Q61's broken image tag. The last two lines
 # undo the retired set B's Q317 (the CSR signing controller switched off in --controllers, or in
 # its older form --cluster-signing-cert-file pointed at a missing file, which crash-looped the WHOLE
 # controller-manager); a cluster that ever ran that set may still carry either.
-nx "$CP" '
+nx "$CP" "$MEDIT"'
-f=/etc/kubernetes/manifests/kube-controller-manager.yaml
+medit /etc/kubernetes/manifests/kube-controller-manager.yaml \
-sed -i "/--leader-elect-lease-duration=1s/d" $f 2>/dev/null
+  -e "/--leader-elect-lease-duration=1s/d" \
-sed -i "s|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)-broken|\1|" $f 2>/dev/null
+  -e "s|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)-broken|\1|" \
-sed -i "s|--cluster-signing-cert-file=/etc/kubernetes/pki/ca-signing.crt|--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt|" $f 2>/dev/null
+  -e "s|--cluster-signing-cert-file=/etc/kubernetes/pki/ca-signing.crt|--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt|" \
-sed -i "s|,-certificatesigningrequest-signing-controller||; s|,-csrsigning||" $f 2>/dev/null
+  -e "s|,-certificatesigningrequest-signing-controller||; s|,-csrsigning||"
 ' </dev/null
 
 # coredns: restore the default upstream forward (Q37/Q57 point it somewhere bogus)

tracks/cka/common.py

@@ -8,7 +8,11 @@ mkns() {
   #   error looking up service account <ns>/default: serviceaccount "default" not found
   # Setups that create a namespace and immediately apply Pods lose that race under load,
   # which fails the seed and costs the candidate a whole task. It belongs here, not in each task.
-  for _ in $(seq 1 30); do $K -n "$1" get sa default >/dev/null 2>&1 && return 0; sleep 1; done
+  # 90 s, not 30: the ServiceAccount controller runs in kube-controller-manager, and a manager that
+  # is restarting (it takes ~60 s to be back and leading) left bare Pods rejected with no trace.
+  for _ in $(seq 1 90); do $K -n "$1" get sa default >/dev/null 2>&1 && return 0; sleep 1; done
+  echo "namespace $1 still has no default ServiceAccount after 90 s - is kube-controller-manager running?" >&2
+  return 1
 }
 """
 

tracks/ckad/questions.py

@@ -1,4 +1,5 @@
-"""CKAD track. Starter bank - same format as the CKA banks, extend it the same way.
+"""CKAD track: 37 tasks, at least one per competency in the CKAD curriculum (1-12 here, 13-37 in
+bank_extra.py). Same format as the CKA banks.
 
 CKAD is a build-things exam: less cluster surgery, more "produce the right object, fast".
 The clock is the adversary, so these are mostly single-object tasks graded exactingly.
@@ -17,8 +18,10 @@ TRACK = {
         "ENV":    "Application Environment, Configuration and Security (25%)",
         "NET":    "Services and Networking (20%)",
     },
-    "quota":      {"DESIGN": 3, "DEPLOY": 3, "OBS": 2, "ENV": 2, "NET": 2},
+    # 16 tasks in 120 minutes, weighted like the exam (20/20/15/25/20 %). The pool per domain is
-    "quota_hard": {"DESIGN": 3, "DEPLOY": 3, "OBS": 2, "ENV": 2, "NET": 2},
+    # 8/7/7/10/5, so no two mocks are the same set.
+    "quota":      {"DESIGN": 3, "DEPLOY": 3, "OBS": 3, "ENV": 4, "NET": 3},
+    "quota_hard": {"DESIGN": 3, "DEPLOY": 3, "OBS": 3, "ENV": 4, "NET": 3},
 }
 
 QUESTIONS = [
@@ -35,7 +38,7 @@ mounted at `/spool` in both:
 
 `kubectl logs %{pod} -c adapter` must show the writer's lines.
 """,
- "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true --grace-period=1 >/dev/null""",
  "checks": [
   C("pod Running with containers writer and adapter", 2, r"""
 p="$K -n %{ns} get pod %{pod}"
@@ -86,7 +89,7 @@ connection instead.
 """,
  "setup": NS + r"""
 mkns %{ns}
-$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null
+$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true --grace-period=1 >/dev/null
 $K -n %{ns} delete deploy backend --ignore-not-found >/dev/null
 $K apply -f - >/dev/null <<EOF
 apiVersion: v1
@@ -363,6 +366,7 @@ spec:
       image: busybox:1.99-doesnotexist
       command: [sh, -c, "echo working; sleep 86400"]
 EOF
+$K -n %{ns} get pod %{pod} >/dev/null
 """,
  "checks": [
   C("reason.txt records the image pull failure", 3, r"""grep -Eqi 'ImagePull|ErrImagePull|ImagePullBackOff' "$WS/q8/reason.txt" 2>/dev/null"""),
@@ -394,7 +398,7 @@ Create Pod `%{pod}` in `%{ns}` (image `busybox:1.36`, command `sleep 86400`) tha
 - drops **all** capabilities and adds back only `NET_BIND_SERVICE`,
 - cannot gain new privileges.
 """,
- "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true --grace-period=1 >/dev/null""",
  "checks": [
   C("runs as UID %{uid} with runAsNonRoot", 3, r"""
 p="$K -n %{ns} get pod %{pod}"
@@ -444,7 +448,7 @@ In `%{ns}`:
 """,
  "setup": NS + r"""
 mkns %{ns}
-$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null
+$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true --grace-period=1 >/dev/null
 $K -n %{ns} delete cm app-cfg --ignore-not-found >/dev/null; $K -n %{ns} delete secret app-sec --ignore-not-found >/dev/null
 """,
  "checks": [
@@ -573,3 +577,8 @@ EOF
 """,
 },
 ]
+
+# 13-37: the rest of the curriculum, one or more tasks per competency the starter bank left out.
+from tracks.ckad import bank_extra  # noqa: E402
+
+QUESTIONS = QUESTIONS + bank_extra.QUESTIONS