fix: repair apiserver/staticPodPath leaks, 3 drifted solutions, workspace tracking

Remaining findings from the repo audit (1, 2 and 4 landed in df04c56).

restore_baseline now undoes two more leaks, both of which cost real debugging
time this week:

  - apiserver (Q123 renames the command to "kube-apiserverr"). This runs FIRST
    and over podman/ssh rather than kubectl, because it is exactly the failure
    that makes kubectl unusable - every later step in the block depends on the
    API being up. It waits for /readyz before continuing.
  - staticPodPath (Q103 relocates it to /etc/kubernetes/static on the tier=edge
    node). The existing kubelet repair only fired when
    /var/lib/kubelet/config.yaml.orig existed, and nothing ever creates that
    backup, so a relocated path survived every later session and silently broke
    Q05, whose check hardcodes /etc/kubernetes/manifests. Normalised
    unconditionally, restarting the kubelet only when the file actually changed.

Verified against the live cluster: with both breaks applied the API was
unreachable, and one restore_baseline run brought all four nodes Ready, restored
the manifest, reset staticPodPath and let the kubelet re-adopt the Q05 mirror
pod. Re-running takes 1.1s versus 14.6s when repairing, confirming it is
idempotent and does not restart anything it has not changed.

etcd (Q102) and certs (Q122) are deliberately left alone, now documented in the
block: neither setup damages anything, so any damage comes from a half-finished
candidate attempt, and "mid-attempt" is indistinguishable from "done correctly"
without guessing - guessing wrong rewrites etcd's data-dir or re-issues certs on
a healthy cluster. proxy-rules (Q302) needs no repair at all; its "breaks" key
exists for no_conflicts() bookkeeping, not because it touches shared state.

Also:
  - S[30], S[42], S[108] referenced params that do not exist (%{svc}/%{app},
    %{np}/%{port}, %{rev}), so they rendered placeholders verbatim. Rewritten
    against the real task text - S[42] was missing the DNS egress rule entirely
    and S[108] stopped before the scale/pause/set-image steps, so both would
    have failed 2 of 3 checks if followed literally. A param-reference scan over
    the whole bank is now clean.
  - Q124's check and S[124] used the deprecated v1 Endpoints API; switched to
    EndpointSlice, verified both forms return 10.89.1.3:6443 on this cluster.
  - q_by_id printed "Valid: 1-101" for non-contiguous ids; it now derives the
    real ranges (1-43, 101-124, 201-217, 301-317).
  - Untracked workspace/q03. Both q03 and q21 setups start with rm -rf and
    rewrite every file from heredocs, so they are generated output, not seed
    templates: tracking them only produced phantom diffs, since a re-seed
    reverted edits to base/ and deleted anything left in overlays/. With nothing
    tracked there, teardown.sh can go back to wiping workspace/ wholesale.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fqcio9TXkEXH35UHebTgoL

.gitignore

@@ -4,9 +4,9 @@ __pycache__/
 .DS_Store
 
 # workspace/ is where task setup scripts write scratch files for whatever question you're
-# currently drilling - regenerated (and wiped) every time that question is re-seeded, so it
-# doesn't belong in the repo. q03 and q21 are the two exceptions already tracked as seed
-# templates.
-/workspace/*
-!/workspace/q03
-!/workspace/q21
+# currently drilling - regenerated (and wiped) every time that question is re-seeded, so none
+# of it belongs in the repo. q03 and q21 used to be excepted as "seed templates", but both
+# setups start with `rm -rf "$WS/qNN"` and rewrite every file from heredocs, so they are
+# generated output too: tracking them only produced phantom diffs (a re-seed reverted any edit
+# to base/ and deleted any answer left in overlays/).
+/workspace/

exam

@@ -207,7 +207,16 @@ def q_by_id(n: int) -> dict[str, Any]:
     for q in QUESTIONS:
         if q["id"] == n:
             return q
-    die(f"No question {n}. Valid: 1-{len(QUESTIONS)}")
+    # ids are not contiguous (1-43 base, 101-124 hard, 201-217 / 301-317 sim sets), so print the
+    # real ranges rather than "1-<count>", which named ids that do not exist.
+    ids = sorted(q["id"] for q in QUESTIONS)
+    ranges, start, prev = [], ids[0], ids[0]
+    for i in ids[1:] + [None]:
+        if i != prev + 1:
+            ranges.append(str(start) if start == prev else f"{start}-{prev}")
+            start = i
+        prev = i
+    die(f"No question {n}. Valid: {', '.join(ranges)}")
     return {}
 
 def render(text, params):
@@ -552,6 +561,15 @@ def _log_msg(*args: Any, **kwargs: Any) -> None:
 # restores every known "breaks" target to a good baseline before each session seeds, so
 # leftover damage never leaks across sessions regardless of which questions get sampled.
 RESTORE_BASELINE = r"""
+# apiserver FIRST: Q123 renames the command to "kube-apiserverr", which takes the whole API
+# down - every kubectl-based step below would fail until this is undone. The repair goes over
+# podman/ssh (nx), not kubectl, so it works precisely when the cluster is unreachable.
+if nx "$CP" 'grep -q "^ *- kube-apiserverr$" /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null 2>/dev/null; then
+  nx "$CP" 'sed -i "s|^\( *\)- kube-apiserverr$|\1- kube-apiserver|" /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null
+  # the kubelet re-reads the manifest on its own; wait for the API to answer again
+  for _ in $(seq 1 30); do kubectl --context "${CKA_CONTEXT:-kind-cka-sim}" get --raw=/readyz >/dev/null 2>&1 && break; sleep 2; done
+fi
+
 # scheduler: undo the "kube-scheduler" -> "kube-schedulerr" command typo (Q20/Q118)
 nx "$CP" '
 f=/etc/kubernetes/manifests/kube-scheduler.yaml
@@ -573,10 +591,19 @@ if [ -n "$cf" ] && ! printf '%s' "$cf" | grep -Eq '^[[:space:]]*forward \. /etc/
   $K -n kube-system rollout restart deploy coredns >/dev/null 2>&1
 fi
 
-# w1/w2/w3 kubelet: undo Q17/Q39-style tampering, then make sure it's enabled and running
+# w1/w2/w3 kubelet: undo Q17/Q39-style tampering, then make sure it's enabled and running.
+# staticPodPath is normalised unconditionally: Q103 relocates it to /etc/kubernetes/static, and
+# the config.yaml.orig copy below only helps if something made that backup first - nothing does,
+# so a relocated path used to survive every later session and silently break Q05 (whose check
+# looks in /etc/kubernetes/manifests). Only restart the kubelet if we actually changed the file.
 for n in "$W1" "$W2" "$W3"; do
   nx "$n" '
     [ -f /var/lib/kubelet/config.yaml.orig ] && cp -f /var/lib/kubelet/config.yaml.orig /var/lib/kubelet/config.yaml
+    if grep -q "^staticPodPath:" /var/lib/kubelet/config.yaml 2>/dev/null \
+       && ! grep -q "^staticPodPath: /etc/kubernetes/manifests$" /var/lib/kubelet/config.yaml; then
+      sed -i "s|^staticPodPath: .*|staticPodPath: /etc/kubernetes/manifests|" /var/lib/kubelet/config.yaml
+      systemctl restart kubelet >/dev/null 2>&1
+    fi
     systemctl enable kubelet >/dev/null 2>&1
     systemctl start kubelet  >/dev/null 2>&1
   ' </dev/null
@@ -592,10 +619,18 @@ $K taint node "$W1" dedicated- workload- team- pool- lane- maintenance- >/dev/nu
 # (Q26/Q211 DaemonSet coverage, Q212's one-pod-per-node spread).
 for n in "$W1" "$W2" "$W3"; do $K uncordon "$n" >/dev/null 2>&1 || true; done
 
-# NOTE: etcd/certs/apiserver corruption (hard-mode only: Q102, Q118 apiserver half, Q122,
-# Q123) isn't restored here yet - those are rarer and each needs a more bespoke repair.
-# Q103's staticPodPath relocation also only reverts if /var/lib/kubelet/config.yaml.orig
-# exists, which it does not unless something made that backup first.
+# Deliberately NOT restored, and why:
+#   etcd (Q102) and certs (Q122). Neither setup damages anything - Q102 only takes a snapshot
+#   and deletes a ConfigMap, Q122 only records the current notBefore. Any damage comes from a
+#   half-finished candidate attempt (etcd.yaml left pointing at a restored data-dir, static pod
+#   manifests moved to /tmp and never moved back), which is not deterministic: there is no way
+#   to tell "mid-attempt" from "correctly completed" without guessing, and guessing wrong here
+#   rewrites etcd's data-dir or re-issues certs on a healthy cluster. Re-run ./setup.sh (or
+#   ./teardown.sh && ./setup.sh) if one of those leaves the control plane wedged.
+#
+#   proxy-rules (Q302) needs nothing: its setup only creates a namespace and deletes a pod and
+#   service. The "breaks" key is there for no_conflicts() bookkeeping, not because it damages
+#   shared state.
 """
 
 def restore_baseline(log: Callable[..., Any] = _log_msg):

teardown.sh

@@ -5,5 +5,7 @@ cd "$(dirname "$0")"
 RT="${CONTAINER_RUNTIME:-$(cat .state/runtime 2>/dev/null || echo podman)}"
 [ "$RT" = podman ] && export KIND_EXPERIMENTAL_PROVIDER=podman
 kind delete cluster --name cka-sim || true
+# Safe to wipe wholesale: everything under workspace/ is generated by task setup scripts and
+# nothing there is tracked (see .gitignore). Anything you want to keep, copy out first.
 rm -rf .state workspace
 echo "cka-sim cluster and local state removed."

tracks/cka/bank_hard.py

@@ -233,7 +233,7 @@ want=$(nx "$CP" "openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl r
 got=$(grep -oE '^CA_HASH=sha256:[0-9a-f]{64}' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2 | cut -d: -f2)
 [ -n "$want" ] && eq "$got" "$want" """),
   C("API_ENDPOINT is the real in-cluster address, not the local kubeconfig's", 2, r"""
-want=$($K get endpoints kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}:{.subsets[0].ports[0].port}' 2>/dev/null)
+want=$($K get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}' 2>/dev/null)
 got=$(grep -oE '^API_ENDPOINT=.*' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2 | tr -d '[:space:]')
 [ -n "$want" ] && eq "$got" "$want" """),
  ],

tracks/cka/solutions.py

@@ -456,7 +456,7 @@ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | \\
 # kind maps the API server to 127.0.0.1:<random-host-port> for YOUR machine only; a node
 # joining over the cluster network needs the address the apiserver actually advertises.
 exit
-kubectl get endpoints kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}:{.subsets[0].ports[0].port}{"\\n"}'
+kubectl get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}{"\\n"}'
 
 # Step 4: Assemble the handoff file
 cat > {WS}/q124/node-join.env <<EOF
@@ -818,17 +818,31 @@ EOF
 k -n %{ns} get pods -o wide -l app=anti-workload
 """
 
-S[108] = """# Step 1: Inspect rollout revision history
+S[108] = """# Step 1: Find WHICH revision used nginx:1.25 - the task asks for that one specifically, which
+# is not necessarily the previous one. --revision=N shows a single revision's pod template.
 k -n %{ns} rollout history deploy %{dep}
+for r in $(k -n %{ns} rollout history deploy %{dep} | awk '/^[0-9]/{print $1}'); do
+  echo "revision $r: $(k -n %{ns} rollout history deploy %{dep} --revision=$r | grep -i 'image:')"
+done
 
-# Step 2: Roll back Deployment to specified target revision
-k -n %{ns} rollout undo deploy %{dep} --to-revision=%{rev}
+# Step 2: Roll back to that revision (substitute the number you found for N)
+k -n %{ns} rollout undo deploy %{dep} --to-revision=N
 
-# Step 3: Immediately pause the rollout
+# Step 3: Scale to 3 and wait until they are actually available
+k -n %{ns} scale deploy %{dep} --replicas=3
+k -n %{ns} rollout status deploy %{dep}
+
+# Step 4: Pause BEFORE editing. While paused the controller does not reconcile the template, so
+# no new ReplicaSet is created and the running pods stay on nginx:1.25 - that gap between
+# spec.template and what is running is exactly what the task checks.
 k -n %{ns} rollout pause deploy %{dep}
+k -n %{ns} set image deploy/%{dep} $(k -n %{ns} get deploy %{dep} -o jsonpath='{.spec.template.spec.containers[0].name}')=nginx:1.27
+k -n %{ns} set env deploy/%{dep} RELEASE=next
 
-# Step 4: Verify deployment status is paused
-k -n %{ns} get deploy %{dep}
+# Step 5: Verify - paused true, template 1.27, pods still 1.25, and no RS for the new template
+k -n %{ns} get deploy %{dep} -o jsonpath='{.spec.paused}{"\\n"}'
+k -n %{ns} get pod -l app=%{dep} -o jsonpath='{.items[*].spec.containers[0].image}{"\\n"}'
+k -n %{ns} get rs -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image
 """
 
 S[109] = """# Step 1: Deploy Pod with a native sidecar (restartPolicy: Always inside initContainers),
@@ -1008,43 +1022,72 @@ EOF
 k -n %{ns} get gateway,httproute
 """
 
-S[30] = """# Step 1: Update pod/deployment container definition to include named ports
-# spec.containers[0].ports: [{name: http-web, containerPort: 80}]
-# Step 2: Create Service referencing the container port by name
+S[30] = """# Step 1: Deployment whose container port 80 carries the name %{pname}, with a readiness
+# probe that references that port BY NAME rather than by number.
+cat <<EOF | k apply -f -
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: %{dep}, namespace: %{ns}}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: %{dep}}}
+  template:
+    metadata: {labels: {app: %{dep}}}
+    spec:
+      containers:
+        - name: web
+          image: nginx:1.25
+          ports:
+            - name: %{pname}
+              containerPort: 80
+          readinessProbe:
+            httpGet: {path: /, port: %{pname}}
+EOF
+
+# Step 2: Service whose targetPort is the port NAME, not 80
 cat <<EOF | k apply -f -
 apiVersion: v1
 kind: Service
-metadata: {name: %{svc}, namespace: %{ns}}
+metadata: {name: %{dep}, namespace: %{ns}}
 spec:
-  selector: {app: %{app}}
+  selector: {app: %{dep}}
   ports:
-    - name: http
-      port: 8080
-      targetPort: http-web
+    - port: 8080
+      targetPort: %{pname}
 EOF
 
-# Step 3: Verify endpoints are registered
-k -n %{ns} get endpoints %{svc}
+# Step 3: Verify endpoints are registered (empty means the selector or the probe is wrong)
+k -n %{ns} get endpointslices -l kubernetes.io/service-name=%{dep}
 """
 
-S[42] = """# Step 1: Create egress-only NetworkPolicy restricting outbound traffic to external CIDR
+S[42] = """# Step 1: Egress-only policy with TWO rules. Listing Egress alone in policyTypes is what
+# denies everything else - there is no explicit "deny" rule in a NetworkPolicy. Leaving Ingress
+# out of policyTypes is what keeps inbound traffic unrestricted.
+# The DNS rule needs both UDP and TCP on 53: resolvers fall back to TCP for large answers.
 cat <<EOF | k apply -f -
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
-metadata: {name: %{np}, namespace: %{ns}}
+metadata: {name: %{app}-egress, namespace: %{ns}}
 spec:
-  podSelector: {matchLabels: {role: crawler}}
+  podSelector: {matchLabels: {app: %{app}}}
   policyTypes: [Egress]
   egress:
+    - to:
+        - namespaceSelector:
+            matchLabels: {kubernetes.io/metadata.name: kube-system}
+          podSelector:
+            matchLabels: {k8s-app: kube-dns}
+      ports:
+        - {protocol: UDP, port: 53}
+        - {protocol: TCP, port: 53}
     - to:
         - ipBlock: {cidr: %{cidr}}
       ports:
-        - protocol: TCP
-          port: %{port}
+        - {protocol: TCP, port: 443}
 EOF
 
-# Step 2: Describe policy to confirm egress rules
-k -n %{ns} describe netpol %{np}
+# Step 2: Describe policy to confirm both egress rules
+k -n %{ns} describe netpol %{app}-egress
 """
 
 S[43] = """# Step 1: Create Headless Service (clusterIP: None)
@@ -1581,7 +1624,7 @@ k -n %{ns} patch svc %{dep} -p '{"spec":{"selector":{"app":"%{dep}"},"ports":[{"
 
 # Step 6: Verify all deployment replicas are Available with endpoints
 k -n %{ns} get deploy %{dep}
-k -n %{ns} get endpoints %{dep}
+k -n %{ns} get endpointslices -l kubernetes.io/service-name=%{dep}
 """
 
 S[120] = r"""# Gather each fact and append it directly to the report file, one key=value line at a time

workspace/q03/base/deployment.yaml

@@ -1,19 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: store-api
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: store-api
-  template:
-    metadata:
-      labels:
-        app: store-api
-    spec:
-      containers:
-        - name: api
-          image: nginx:1.25
-          ports:
-            - containerPort: 80

workspace/q03/base/kustomization.yaml

@@ -1,5 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - deployment.yaml
-  - service.yaml

workspace/q03/base/service.yaml

@@ -1,10 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: store-api
-spec:
-  selector:
-    app: store-api
-  ports:
-    - port: 80
-      targetPort: 80

workspace/q03/overlays/prod/kustomization.yaml

@@ -1,13 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - ../../base
-  - namespace.yaml
-namespace: store-prod
-labels:
-  - pairs:
-      env: prod
-    includeSelectors: false
-replicas:
-  - name: store-api
-    count: 3

workspace/q03/overlays/prod/namespace.yaml

@@ -1,4 +0,0 @@
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: store-prod