refactor: fixed some errors while deploying the cluster

`./exam sim c` - 18 tasks, 163 points, 120 minutes, exam rules. Built after a
23% result on a paid mock, so reruns cost nothing and the weak topics can be
drilled as often as needed.

Original scenarios, not copies, exercising the same techniques as an 18-question
sitting: hostPath PV/PVC bound by volumeName, data migration between claims onto
a Retain volume, Helm render-then-install, Gateway API header canary, RBAC with
the answer written to a file, probe/selector/targetPort repair, HPA v2 behavior
blocks, and three "write the output to an exact path" data-extraction tasks.

Ids 401-418, set="c", difficulty="hard". cmd_sim is already generic over SETS so
no harness change was needed; set-tagged tasks stay out of POOL and cannot leak
into `./exam start` or drills. CKA bank goes 101 -> 119 tasks.

Two scenarios are deliberately adapted, because the originals would destroy the
practice cluster:

  - 410 audits the installed CNI and applies a default-deny, rather than
    installing Calico over the running kindnet. The Flannel-vs-Calico
    "does this CNI actually enforce NetworkPolicy" judgement is kept as a
    written answer.
  - 413 plans a control-plane upgrade - versions, ordered command sequence,
    `kubeadm upgrade node` for workers - and never runs `kubeadm upgrade apply`.
    The grader checks the ordering, modelled on Q316.

Verified end to end: every task seeded, solved with its own reference solution
and graded, 163/163 across all 18, in four batches. Checks were also confirmed
to fail cleanly against an unsolved cluster rather than erroring.

Three environment facts cost a round of debugging and are now encoded in the
tasks and their solutions:

  - The StorageClass on this kind cluster is named `standard`, not `local-path`,
    and is WaitForFirstConsumer - a PVC stays Pending until its pod can be
    scheduled, which is what interlocks the two faults in 414.
  - Under rootless podman the host cannot route to node IPs, so a host-side
    openssl against a NodePort just hangs (rc=124). 417 probes TLS from inside a
    node via nx against 127.0.0.1:<nodePort>. Nodes have openssl and curl but
    not nc or wget.
  - hostPath PV data must be seeded on every node (401, 405), since nothing
    pins the pod to the node holding the directory.

Seeding all 18 creates ~24 pods, which on this host needs
kernel.keys.maxkeys raised above the 200 default or container starts fail with
"unable to join session keyring: disk quota exceeded".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NziZdvBvy7CpEAAbZNgp8i

exam

@@ -201,7 +201,12 @@ def session_questions(st: dict[str, Any] | None) -> list[dict[str, Any]]:
     if not st:
         return list(QUESTIONS)
     ids = st.get("ids") or [q["id"] for q in QUESTIONS]
-    return [q for q in QUESTIONS if q["id"] in ids]
+    # Preserve the session's own order. Filtering QUESTIONS (which is sorted by id) threw it away,
+    # so compose_exam's rng.shuffle never reached the candidate and every exam arrived grouped by
+    # domain and difficulty - base tasks first, then hard, then the sim sets. Real exams are not
+    # ordered by difficulty, and practising triage on a sorted list teaches the wrong instinct.
+    by_id = {q["id"]: q for q in QUESTIONS}
+    return [by_id[i] for i in ids if i in by_id]
 
 def q_by_id(n: int) -> dict[str, Any]:
     for q in QUESTIONS:
@@ -380,16 +385,40 @@ def no_conflicts(cands, n, taken=()):
 def pick(domain, n, hard, rng, exclude=(), taken=(), unseen_only=False):
     """n tasks from a domain: unseen first, then least recently seen; hard sessions prefer hard tasks."""
     seen = last_seen()
+
+    def build(allow_hard):
         pool = [q for q in POOL if q["domain"] == domain and q["id"] not in exclude
-            and (hard or q["difficulty"] == "normal") and not (unseen_only and q["id"] in seen)]
+                and (allow_hard or q["difficulty"] == "normal") and not (unseen_only and q["id"] in seen)]
         rng.shuffle(pool)
         pool.sort(key=lambda q: (0 if (hard and q["difficulty"] == "hard") else 1, seen.get(q["id"], -1)))
-    return no_conflicts(pool, n, taken)
+        return pool
+
+    got = no_conflicts(build(hard), n, taken)
+    if len(got) < n and not hard:
+        # A domain whose only tasks are "hard" contributes NOTHING to a normal session and drops
+        # out of the blueprint entirely - CKS SUPPLY and RUNTIME (20% each) were both unreachable
+        # via ./exam start. Better to hand over a hard task than to silently skip the domain.
+        got = no_conflicts(build(True), n, taken)
+    return got
 
 def compose_exam(hard, rng):
-    qs = []
+    quota = QUOTA_HARD if hard else QUOTA
-    for d, n in (QUOTA_HARD if hard else QUOTA).items():
+    # Scarcest domain first. Two tasks that "breaks" the same thing can't share a session, so a
-        qs += pick(d, n, hard, rng, taken=qs)
+    # one-task domain loses every such tie to a bigger domain that claimed the shared slot earlier
+    # and then contributes nothing: CKS RUNTIME (only Q9, apiserver-flag) was never drawn in ANY
+    # mode because SETUP was iterated first and took that slot with Q2.
+    order = sorted(quota, key=lambda d: len([q for q in POOL if q["domain"] == d]))
+    qs, short = [], {}
+    for d in order:
+        got = pick(d, quota[d], hard, rng, taken=qs)
+        if len(got) < quota[d]:
+            short[d] = (len(got), quota[d])
+        qs += got
+    if short:
+        # Never truncate in silence - a short session still gets scored out of its own total, so
+        # the missing domains look like they were simply not on the blueprint.
+        print(yellow("  warning: bank too small to fill the blueprint - "
+                     + ", ".join(f"{d} {g}/{w}" for d, (g, w) in short.items())))
     rng.shuffle(qs)
     return qs
 
@@ -612,7 +641,24 @@ done
 # w1-taint: clear every randomized demo taint key the bank can apply. Q25 draws from
 # dedicated/team/workload, Q107 from pool/lane, and Q307 pins maintenance=inprogress - miss one
 # and that node silently stops accepting pods for every later session.
-$K taint node "$W1" dedicated- workload- team- pool- lane- maintenance- >/dev/null 2>&1 || true
+# One key per call: `kubectl taint node N a- b-` aborts on the first key that is not present and
+# removes NOTHING, so the single taint that is actually set survives the "cleanup" and W1 keeps
+# rejecting pods in every later session - the exact failure this block exists to prevent.
+for _t in dedicated workload team pool lane maintenance; do
+  $K taint node "$W1" "$_t-" >/dev/null 2>&1 || true
+done
+
+# metrics-server: Q23 seeds its scenario by stripping --kubelet-insecure-tls, and kind's kubelet
+# serving certs have no IP SANs, so without that flag metrics-server never becomes Ready. Left
+# broken it does far more than disable `kubectl top`: the v1beta1.metrics.k8s.io APIService has no
+# endpoints, aggregated discovery goes stale, and then EVERY namespace deletion in the cluster
+# hangs on NamespaceDeletionDiscoveryFailure. Baseline runs before seeding, so a session that
+# actually contains Q23 re-breaks it a moment later, exactly as intended.
+if $K -n kube-system get deploy metrics-server >/dev/null 2>&1 \
+   && ! $K -n kube-system get deploy metrics-server -o jsonpath='{.spec.template.spec.containers[0].args}' | grep -q 'kubelet-insecure-tls'; then
+  $K -n kube-system patch deploy metrics-server --type=json \
+    -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' >/dev/null 2>&1 || true
+fi
 
 # w1-block/w2-drain/w3-drain: Q307 cordons W1 and Q117 cordons W2, and Q208/Q309 have the
 # candidate drain W2/W3. A cordon left behind fails any later task that counts pods per node
@@ -776,6 +822,10 @@ def cmd_drill(args):
     if len(kept) < len(qs):
         dropped = [q["id"] for q in qs if q not in kept]
         print(yellow(f"Dropped Q{', Q'.join(map(str, dropped))}: they break the same thing as another task in this set. Drill them separately."))
+    # Shuffle AFTER resolving conflicts, so which task gets dropped stays deterministic while the
+    # order you meet them in does not. A drill listed by id walks you from easy to hard in
+    # lockstep, which is the one ordering the real exam will never give you.
+    random.Random().shuffle(kept)
     begin_session(kept, "drill", (minutes * 60) if minutes else per_task * len(kept), force, hard=hard)
 
 def cmd_study(args):
@@ -806,7 +856,8 @@ def cmd_sim(args):
     force = "--force" in args
     names = [a.lower() for a in args if not a.startswith("--")]
     if not names or names[0] not in SETS:
-        die("usage: ./exam sim a|b [--force]\n  " + "\n  ".join(f"{k}: {v} ({len(set_questions(k))} tasks)" for k, v in SETS.items()))
+        die(f"usage: ./exam sim {'|'.join(SETS) or '<set>'} [--force]\n  "
+            + "\n  ".join(f"{k}: {v} ({len(set_questions(k))} tasks)" for k, v in SETS.items()))
     name = names[0]
     qs = set_questions(name)
     print(bold(f"{SETS[name]} - {len(qs)} tasks, {TRACK['minutes']} minutes, exam rules (no feedback until you submit)"))
@@ -1213,7 +1264,7 @@ def cmd_help(args):
   drill weak|N..|DOMAIN..  chosen tasks  (--time MIN, --force, --hard)
   progress                 attempted/passed per domain, normal and hard
   solution N               reference solution (drills: after grading N; exams: after submit)
-  sim a|b                  fixed 17-task simulator set, exam rules, {TRACK['minutes']} min
+  sim {'|'.join(SETS) or '<set>'}{' ' * max(1, 20 - len('|'.join(SETS)))}fixed simulator set, exam rules, {TRACK['minutes']} min
   start [--hard]           full exam: {sum(QUOTA.values()) or '?'} tasks sampled by domain ({sum(QUOTA_HARD.values()) or '?'} in hard mode), {TRACK['minutes']} min.
                            Requires every domain drilled once first (--force skips). Hard mode adds traps,
                            exact-output tasks, compound failures, collateral checks and exam-style contexts.

tracks/cka/bank_extra.py

@@ -204,7 +204,7 @@ grep -qx "etcd=$etcd" "$f" && grep -qx "runtime=$rt" "$f" """),
 },
 # ---------------------------------------------------------------- WORK ----
 {
- "id": 23, "domain": "WORK", "title": "metrics-server repair and HPA",
+ "id": 23, "domain": "WORK", "title": "metrics-server repair and HPA", "breaks": "metrics-server",
  "params": lambda r: {"ns": ns_pick(r, "scale"), "dep": name_pick(r, "api"), "max": r.choice([5, 6, 8]), "cpu": r.choice([60, 70, 75])},
  "task": """
 `metrics-server` was installed in `kube-system` but `kubectl top nodes` does not work.

tracks/cka/common.py

@@ -1,7 +1,15 @@
 """Shared helpers for CKA task banks."""
 
 NS = r"""
-mkns() { $K create ns "$1" --dry-run=client -o yaml | $K apply -f - >/dev/null; }
+mkns() {
+  $K create ns "$1" --dry-run=client -o yaml | $K apply -f - >/dev/null
+  # Wait for the namespace's `default` ServiceAccount before returning. The controller creates it
+  # asynchronously, and a Pod applied in the gap is rejected outright:
+  #   error looking up service account <ns>/default: serviceaccount "default" not found
+  # Setups that create a namespace and immediately apply Pods lose that race under load (Q204),
+  # which fails the seed and costs the candidate a whole task. It belongs here, not in each task.
+  for _ in $(seq 1 30); do $K -n "$1" get sa default >/dev/null 2>&1 && return 0; sleep 1; done
+}
 """
 
 def C(name, pts, script):

tracks/cka/questions.py

@@ -23,8 +23,8 @@ TRACK = {
     # tasks per domain drawn into a session (weights follow the exam blueprint)
     "quota":      {"ARCH": 5, "WORK": 4, "NET": 4, "STOR": 2, "TSHOOT": 5},      # 20 tasks
     "quota_hard": {"ARCH": 6, "WORK": 5, "NET": 5, "STOR": 3, "TSHOOT": 6},      # 25 tasks, hard ones preferred
-    # fixed simulator sets (./exam sim a|b) - excluded from random sampling
+    # fixed simulator sets (./exam sim a|b|c) - excluded from random sampling
-    "sets": {"a": "Simulator Set A", "b": "Simulator Set B"},
+    "sets": {"a": "Simulator Set A", "b": "Simulator Set B", "c": "Simulator Set C"},
     "domains": {
         "ARCH":   "Cluster Architecture, Installation & Configuration (25%)",
         "WORK":   "Workloads & Scheduling (15%)",
@@ -630,9 +630,11 @@ $K -n default delete pod sched-probe --wait=false >/dev/null 2>&1
 },
 ]
 
-# ---- aggregate the banks: 1-20 here, 21-43 in bank_extra, 101-122 (hard) in bank_hard
+# ---- aggregate the banks: 1-20 here, 21-43 in bank_extra, 101-124 (hard) in bank_hard,
-from tracks.cka import bank_extra, bank_hard, bank_sim_a, bank_sim_b
+# ---- 201-217 / 301-317 / 401-418 in the three fixed simulator sets
-QUESTIONS = QUESTIONS + bank_extra.QUESTIONS + bank_hard.QUESTIONS + bank_sim_a.QUESTIONS + bank_sim_b.QUESTIONS
+from tracks.cka import bank_extra, bank_hard, bank_sim_a, bank_sim_b, bank_sim_c
+QUESTIONS = (QUESTIONS + bank_extra.QUESTIONS + bank_hard.QUESTIONS
+             + bank_sim_a.QUESTIONS + bank_sim_b.QUESTIONS + bank_sim_c.QUESTIONS)
 QUESTIONS.sort(key=lambda q: q["id"])
 _ids = [q["id"] for q in QUESTIONS]
 assert len(_ids) == len(set(_ids)), "duplicate question ids"

tracks/ckad/questions.py

@@ -73,11 +73,16 @@ k -n %{ns} logs %{pod} -c adapter
  "task": """
 Namespace `%{ns}` has a Service `backend` with no endpoints yet.
 
-1. Create Pod `%{pod}` with an init container `wait-for-backend` (busybox:1.36) that loops until
+1. Create Pod `%{pod}` with an init container `wait-for-backend` (busybox:1.36) that blocks until
-   the DNS name `backend` resolves, and a main container `app` (nginx:1.27).
+   Service `backend` actually **answers on port 80**, and a main container `app` (nginx:1.27).
    It will sit in `Init:0/1`.
 2. Then create Deployment `backend` (nginx:1.27, 1 replica, Pod label `app=backend`) so the
    Service gets an endpoint and your Pod starts.
+
+Two traps here: a ClusterIP Service gets a DNS record the moment it is created, endpoints or not,
+so waiting on name resolution alone never blocks. And busybox `nslookup` exits non-zero even when
+it succeeds, because the search-domain attempts after the hit return NXDOMAIN. Wait on a real
+connection instead.
 """,
  "setup": NS + r"""
 mkns %{ns}
@@ -106,13 +111,17 @@ spec:
   initContainers:
     - name: wait-for-backend
       image: busybox:1.36
-      command: [sh, -c, "until nslookup backend; do echo waiting; sleep 2; done"]
+      command: [sh, -c, "until wget -T 2 -q -O- http://backend:80 >/dev/null 2>&1; do echo waiting; sleep 2; done"]
   containers: [{name: app, image: nginx:1.27}]
 EOF
 k -n %{ns} get pod %{pod}            # Init:0/1
 k -n %{ns} create deploy backend --image=nginx:1.27
 k -n %{ns} patch deploy backend -p '{"spec":{"template":{"metadata":{"labels":{"app":"backend"}}}}}'
-# A ClusterIP Service only resolves once it has endpoints - that is what unblocks the init container.
+# A ClusterIP Service resolves in DNS from the moment it exists - endpoints change nothing about
+# that - so `until nslookup backend` would not block even before you add the Deployment. Worse,
+# busybox nslookup returns 1 even on a successful lookup, because it walks the whole search list
+# and the trailing NXDOMAINs set the exit status, so that loop never terminates either way.
+# With no endpoints, kube-proxy REJECTs the connection, so wget fails fast and the loop is real.
 """,
 },
 {
@@ -365,10 +374,12 @@ k -n %{ns} get pod %{pod}
 k -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].state.waiting.reason}'
 #   ImagePullBackOff  -> {WS}/q8/reason.txt
 k -n %{ns} describe pod %{pod} | tail -15      # "Failed to pull image ... not found"
-k -n %{ns} get pod %{pod} -o yaml > fix.yaml   # set image: busybox:1.36, strip status/uid/resourceVersion
+k -n %{ns} set image pod/%{pod} worker=busybox:1.36
-k -n %{ns} delete pod %{pod} --now && k apply -f fix.yaml
+# `image` is one of the very few mutable fields on a running Pod, so no delete/recreate is needed
-# A Pod's image can't be changed in place, so delete + recreate is the fix (a Deployment could
+# here - the kubelet just pulls the new image and restarts the container. Most other spec fields
-# have been patched with `set image`).
+# (command, volumes, securityContext) really are immutable, and for those the fix is:
+#   k -n %{ns} get pod %{pod} -o yaml > fix.yaml   # edit, strip status/uid/resourceVersion
+#   k -n %{ns} delete pod %{pod} --now && k apply -f fix.yaml
 """,
 },
 # ------------------------------------------------------------------- ENV ----

tracks/cks/questions.py

@@ -33,10 +33,15 @@ QUESTIONS = [
  "task": """
 Namespace `%{ns}` runs `app=web` (port 8080) and `app=db` (port 5432).
 
-Create three NetworkPolicies:
+Create four NetworkPolicies:
 1. `default-deny` — denies all ingress **and** egress for every Pod in the Namespace.
 2. `allow-dns` — lets every Pod reach DNS (UDP and TCP 53) in `kube-system`.
-3. `web-to-db` — lets `app=web` reach `app=db` on 5432, and lets `app=db` accept that ingress.
+3. `web-to-db` — **egress**: lets `app=web` reach `app=db` on 5432.
+4. `db-from-web` — **ingress**: lets `app=db` accept that traffic from `app=web` on 5432.
+
+A NetworkPolicy has exactly one `podSelector`, so the egress half (selecting `web`) and the
+ingress half (selecting `db`) cannot live in the same object. Once a default deny is in place
+both halves are required — forgetting one is the classic CKS mistake.
 
 (kindnet does not enforce policies; you are graded on the objects.)
 """,
@@ -221,11 +226,16 @@ eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.serviceAccountName}')" %{de
 p=$(j -n %{ns} pod -l app=%{dep} '{.items[0].metadata.name}'); [ -n "$p" ] || exit 1
 $K -n %{ns} exec "$p" -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null"""),
   C("a pod on the default ServiceAccount has no token", 3, r"""
+cleanup() { $K -n %{ns} delete pod plain --wait=false >/dev/null 2>&1; }
 $K -n %{ns} delete pod plain --ignore-not-found --wait=true >/dev/null 2>&1
 $K -n %{ns} run plain --image=busybox:1.36 --restart=Never -- sleep 120 >/dev/null 2>&1
-for i in $(seq 1 15); do [ "$(j -n %{ns} pod plain '{.status.phase}')" = Running ] && break; sleep 3; done
+ok=0; for i in $(seq 1 20); do [ "$(j -n %{ns} pod plain '{.status.phase}')" = Running ] && { ok=1; break; }; sleep 3; done
-r=1; $K -n %{ns} exec plain -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null || r=0
+# A probe pod that never ran proves nothing. Treating a failed exec as "no token mounted" handed
-$K -n %{ns} delete pod plain --wait=false >/dev/null 2>&1; exit $r"""),
+# out these points for doing absolutely nothing whenever pod creation was blocked or slow.
+[ "$ok" = 1 ] || { cleanup; exit 1; }
+$K -n %{ns} exec plain -- true >/dev/null 2>&1 || { cleanup; exit 1; }
+r=1; $K -n %{ns} exec plain -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token >/dev/null 2>&1 || r=0
+cleanup; exit $r"""),
  ],
  "solution": """
 k -n %{ns} patch sa default -p '{"automountServiceAccountToken": false}'