feat: add CKA/CKAD/CKS exam simulator and question banks

Co-authored-by: Junie <junie@jetbrains.com>

.gitignore

@@ -0,0 +1,4 @@
+.state/
+*.pyc
+__pycache__/
+.DS_Store

README.md

@@ -1,2 +1,137 @@
-# cka-sim
+# cka-sim — Kubernetes certification exam simulator
 
+Entry point is `./exam` (renamed from `cka-sim` so it can't collide with the directory name).
+
+Hands-on exam tasks graded **per check** against a real cluster, an exam clock, a browser
+front end modelled on the real PSI exam, attempt history, and drills that rotate through the
+domains before you sit a full exam. Tracks: **CKA** (65-task bank: 43 normal + 22 hard),
+CKAD and CKS (scaffolded — same format, add questions when you get there).
+
+## How a session is built
+
+Every session draws **fresh** tasks: `start` samples per domain in the exam's proportions
+(ARCH 5, WORK 4, NET 4, STOR 2, TSHOOT 5 = 20), preferring tasks you have never seen, then the
+least-recently seen. Most tasks are parameterized — namespaces, names, replica counts, ports,
+which node is broken — so even a repeat isn't the same task. `drill` with no arguments picks the
+domain you've drilled least and gives you 5 tasks from it; a full exam is gated until every
+domain has been drilled at least once (`--force` skips that).
+
+**Hard mode** (`--hard`, or the toggle in the UI lobby) is modelled on what people report about
+killer.sh, not on its content: 25 tasks in the same 120 minutes, hard tasks preferred, no hints
+about paths or nodes, traps (an over-granting binding you must find, an allow-all NetworkPolicy
+left behind, a relocated `staticPodPath`, two PVs where only one is right), exact-output tasks
+graded byte-for-byte, compound failures (three faults on one node, two control-plane components
+down for different reasons, DNS broken three ways), collateral-damage checks (still cordoned,
+recreated instead of fixed in place, extra PVs), and exam-style contexts `k8s-c1-H` /
+`k8s-c2-AC` / `k8s-c3-CCC` with different default namespaces — each task says which to use.
+Grading always runs against the real cluster context regardless of which one you're on.
+
+## Why kind and not k3s / hosted labs
+
+kind is kubeadm-based: real etcd, static-pod control plane, systemd kubelet on every node,
+`kubeadm certs` on the control plane. That is what lets the troubleshooting tasks break real
+things (kubelet, scheduler manifest) the way the exam does. Hosted labs and CK-X give you no
+node access, which rules out exactly those tasks. `podman exec -it <node> bash` plays the role
+of `ssh <node>` here; use the Vagrant/ssh mode below if you want a real ssh session.
+
+## Requirements
+
+podman, kind, kubectl, python3, **ttyd** (for the terminal pane in the browser UI).
+~4 GB RAM for the 4-node cluster. Rootless podman needs cgroup v2; otherwise `sudo -E ./setup.sh`.
+Docker still works: `CONTAINER_RUNTIME=docker ./setup.sh`.
+
+## Run
+
+```
+./setup.sh          # create the kind cluster, preload images (idempotent)
+./exam web       # open http://127.0.0.1:8080 — start an exam or drill from there
+```
+
+Both the built-in terminal and your own shell can have the exam aliases:
+
+```
+source ./examrc          # bash: k=kubectl, $do, $now, kubectl completion (also for k), vim defaults
+source ./examrc.fish     # fish equivalent
+```
+
+`./exam web` sources the right one automatically for its terminal pane.
+
+That's the whole loop. The UI seeds the scenarios, runs the clock, lets you flag questions and
+keep notes, submits (or auto-submits at 00:00), shows the score with every check, and offers
+"Drill what I missed". The terminal pane is a real `bash` via ttyd with kubectl already pointed
+at the cluster (`--shell fish` if you'd rather practise in fish; the real exam is bash).
+
+The same state is available from the terminal if you prefer:
+
+```
+./exam drill                      # next least-drilled domain, 5 tasks (add --hard)
+./exam drill weak | drill 2 17 20 | drill TSHOOT NET
+./exam start [--hard]             # full exam (gated until all domains drilled; --force)
+./exam list | show N | timer | status                       # during
+./exam end                                                  # submit
+./exam weak | history                                       # afterwards
+./exam --track ckad web                                     # other tracks
+./teardown.sh                                                  # delete cluster + state (history kept)
+```
+
+## Scoring, history, drills
+
+Each task is a list of named, weighted checks (`tracks/cka/questions.py`). You get the points
+for every check that passes, so a task can score 4/6 — the same partial-credit model as the
+exam. Every submit appends to `history-<track>.jsonl` and regenerates `weak-spots-<track>.md`:
+latest domain scores, a "drill next" section listing the *failing checks* per task, and a
+✓ / n-of-m matrix across all attempts.
+
+A task counts as weak if it failed most recently, has failed twice ever, or was never
+attempted. Drills seed only those tasks, give 6 minutes per task, and — unlike exam mode —
+let you check a task while you work. A task stops being weak only when you pass it again
+under the clock.
+
+## Using your own cluster instead of kind
+
+Any kubeadm-style cluster with 1 control-plane + 3 workers works. Skip `setup.sh`, set (Fish):
+
+```
+set -x CKA_CONTEXT     kubernetes-admin@kubernetes
+set -x CKA_NODE_ACCESS ssh        # podman | docker | ssh | vagrant
+set -x CKA_CP cp1;  set -x CKA_W1 node1;  set -x CKA_W2 node2;  set -x CKA_W3 node3
+./exam web
+```
+
+`ssh` needs passwordless `ssh <node-name>` (for Vagrant: `vagrant ssh-config >> ~/.ssh/config`);
+`vagrant` runs `vagrant ssh` from the Vagrantfile directory. Tasks 2 and 20 write to
+`/opt/cka` and `/etc/kubernetes/manifests`, so the login must be able to. Managed clusters
+(EKS/AKS/GKE) won't work — no node or control-plane access. Startup preflights all of this.
+
+## CKA bank
+
+| Domain | Normal (43) | Hard (22) |
+|---|---|---|
+| Architecture | RBAC, etcd snapshot, Kustomize, CRD, static pod, certs/CSR, Helm, aggregated ClusterRole, `kubectl explain` on a CRD, kubeadm join token, component versions | exact RBAC with an over-grant to remove, **etcd restore**, static pod on a relocated path, user → self-contained kubeconfig, Kustomize generators + patches, CRD validation/defaults, renew all control-plane certs |
+| Workloads | rolling update, sidecar, node affinity + resources, ConfigMap/Secret, metrics-server repair + HPA, PriorityClass, taints, DaemonSet, CronJob, init container, immutable ConfigMap rotation, parallel Job, quota + LimitRange | anti-affinity + toleration + priority under a quota, rollback to a specific revision then pause, native sidecar + probes + exact report, cluster audit to exact files, work in the context's namespace |
+| Networking | NetworkPolicy, NodePort, Ingress, cluster DNS, Gateway API, named ports, egress-only policy, headless + ExternalName | zero-trust namespace (4 policies + trap), TLS Ingress with generated cert + fingerprint, Gateway canary with header match and weights, DNS down three ways |
+| Storage | PV/PVC/pod, default StorageClass, re-bind a Released PV, data survives the pod | bind the right local volume by selector, migrate data between claims via a Job |
+| Troubleshooting | node NotReady, broken deployment + service, CrashLoopBackOff, scheduler down, CoreDNS misconfig, controller-manager down, kubelet config drift | worker with three faults (+ uncordon check), two control-plane components down, app down for four reasons fixed in place (UID check), cluster facts exactly |
+
+Disruptive tasks (anything that breaks a node or control-plane component) are seeded last and
+can block other tasks until fixed — read everything first, as in the real exam. kindnet doesn't
+enforce NetworkPolicy and no ingress/gateway controller runs, so those are graded on the objects.
+
+## Adding tasks or a new track
+
+Banks live in `tracks/cka/{questions,bank_extra,bank_hard}.py`; a task is:
+
+```python
+{"id": 44, "domain": "NET", "difficulty": "normal",   # or "hard"
+ "params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])},
+ "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
+ "setup": "bash that seeds the scenario (may use %{ns})",
+ "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
+ "disruptive": False}
+```
+
+Scripts get `$K` (pinned to the grading context), `$WS`, `$CP/$W1/$W2/$W3`, `nx NODE 'cmd'`,
+and helpers `j <get args> '{jsonpath}'` and `eq a b`. Task text may use `{WS}`, `{CP}`, `{W2}`,
+`{W3}`, `{SHELL_CP}` etc. `TRACK["quota"]` / `["quota_hard"]` set how many tasks per domain a
+session draws. For CKAD/CKS, fill in `QUESTIONS` in the
+existing placeholder track; `TRACK` already carries the domain table and pass mark.

examrc

@@ -0,0 +1,17 @@
+# cka-sim shell setup. Sourced automatically by the terminal in ./exam web,
+# and safe to source yourself:   source ./examrc
+[ -f ~/.bashrc ] && . ~/.bashrc
+
+alias k=kubectl
+export do='--dry-run=client -o yaml'      # k create deploy x --image=nginx $do > x.yaml
+export now='--force --grace-period 0'     # k delete pod x $now
+export KUBE_EDITOR="${KUBE_EDITOR:-vi}"
+
+# completion, including for the alias
+source <(kubectl completion bash) 2>/dev/null
+complete -o default -F __start_kubectl k 2>/dev/null
+
+# vim defaults that make pasted YAML behave
+if [ ! -f ~/.vimrc ]; then
+  printf 'set tabstop=2\nset expandtab\nset shiftwidth=2\n' > ~/.vimrc
+fi

examrc.fish

@@ -0,0 +1,7 @@
+# cka-sim shell setup for fish:   source ./examrc.fish
+alias k kubectl
+set -gx KUBE_EDITOR (test -n "$KUBE_EDITOR"; and echo $KUBE_EDITOR; or echo vi)
+# lists, so they expand to separate arguments
+set -g do --dry-run=client -o yaml
+set -g now --force --grace-period 0
+kubectl completion fish 2>/dev/null | source

kind-config.yaml

@@ -0,0 +1,8 @@
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+name: cka-sim
+nodes:
+  - role: control-plane
+  - role: worker
+  - role: worker
+  - role: worker

node-tools.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# Install vim/nano/less/jq on every kind node (the real exam has an editor on each node).
+# Re-run this after ./setup.sh if the install was skipped or the network was down.
+set -euo pipefail
+cd "$(dirname "$0")"
+RT="${CONTAINER_RUNTIME:-$(cat .state/runtime 2>/dev/null || echo podman)}"
+[ "$RT" = podman ] && export KIND_EXPERIMENTAL_PROVIDER=podman
+for n in $(kind get nodes --name cka-sim); do
+  printf '%s: ' "$n"
+  if "$RT" exec "$n" sh -c 'command -v vim >/dev/null'; then echo "already has vim"; continue; fi
+  if "$RT" exec "$n" sh -c 'DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq vim nano less jq >/dev/null 2>&1'; then
+    "$RT" exec "$n" sh -c 'printf "set tabstop=2\nset expandtab\nset shiftwidth=2\n" > /root/.vimrc'
+    echo ok
+  else
+    echo "FAILED (no outbound network?)"
+  fi
+done

setup.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# Bootstraps the cka-sim kind cluster (1 control-plane + 3 workers) on podman and preloads images.
+# Override with CONTAINER_RUNTIME=docker if you ever want docker instead.
+set -euo pipefail
+cd "$(dirname "$0")"
+
+RT="${CONTAINER_RUNTIME:-podman}"
+
+need() {
+  command -v "$1" >/dev/null 2>&1 || { echo "ERROR: '$1' is not installed. $2"; exit 1; }
+}
+need "$RT"   "https://podman.io/docs/installation"
+need kind    "https://kind.sigs.k8s.io/docs/user/quick-start/#installation"
+need kubectl "https://kubernetes.io/docs/tasks/tools/"
+need python3 ""
+
+if [ "$RT" = podman ]; then
+  export KIND_EXPERIMENTAL_PROVIDER=podman
+  podman info >/dev/null 2>&1 || { echo "ERROR: podman is not working (try: podman info)."; exit 1; }
+  # kind needs cgroup v2 with controller delegation when podman is rootless.
+  if [ "$(id -u)" != 0 ] && [ "$(podman info --format '{{.Host.CgroupsVersion}}' 2>/dev/null)" != "v2" ]; then
+    echo "ERROR: rootless podman needs cgroup v2 for kind. Either enable cgroup v2 or run setup as root (sudo -E ./setup.sh)."
+    exit 1
+  fi
+  # inotify limits are the #1 cause of pods stuck in ContainerCreating on multi-node kind
+  for k in fs.inotify.max_user_watches:524288 fs.inotify.max_user_instances:512; do
+    key=${k%%:*}; want=${k##*:}; have=$(sysctl -n "$key" 2>/dev/null || echo 0)
+    [ "$have" -ge "$want" ] || echo "warn: $key=$have (recommend >= $want: sudo sysctl $key=$want)"
+  done
+else
+  docker info >/dev/null 2>&1 || { echo "ERROR: docker daemon is not running/reachable."; exit 1; }
+fi
+
+if kind get clusters 2>/dev/null | grep -qx cka-sim; then
+  echo "==> kind cluster 'cka-sim' already exists, reusing it."
+else
+  echo "==> Creating kind cluster 'cka-sim' on $RT (1 control-plane + 3 workers)..."
+  kind create cluster --config kind-config.yaml ${KIND_NODE_IMAGE:+--image "$KIND_NODE_IMAGE"} --wait 180s
+fi
+
+kubectl config use-context kind-cka-sim >/dev/null
+echo "==> Waiting for nodes..."
+kubectl wait --for=condition=Ready nodes --all --timeout=180s
+
+echo "==> Preloading images into the cluster (non-fatal if a pull fails)..."
+for img in nginx:1.25 nginx:1.27 busybox:1.36 redis:7-alpine; do
+  if "$RT" pull -q "docker.io/library/$img" >/dev/null 2>&1; then
+    # podman stores the fully-qualified name; kind load accepts either
+    kind load docker-image --name cka-sim "docker.io/library/$img" >/dev/null 2>&1 \
+      && echo "    loaded $img" || echo "    warn: could not load $img"
+  else
+    echo "    warn: could not pull $img (tasks will pull at runtime)"
+  fi
+done
+
+echo "==> Making sure the exam workdir exists on the control-plane node..."
+"$RT" exec cka-sim-control-plane mkdir -p /opt/cka
+
+# The real exam gives you vim (and less/jq) on every node; the kind node image is minimal, so
+# install them once here. Skip with CKA_SKIP_NODE_TOOLS=1.
+if [ -z "${CKA_SKIP_NODE_TOOLS:-}" ]; then
+  echo "==> Installing editors and tools on the nodes (vim, nano, less, jq)..."
+  for n in $(kind get nodes --name cka-sim); do
+    if "$RT" exec "$n" sh -c 'command -v vim >/dev/null'; then
+      echo "    $n already has vim"
+      continue
+    fi
+    if "$RT" exec "$n" sh -c 'DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq vim nano less jq >/dev/null 2>&1'; then
+      "$RT" exec "$n" sh -c 'printf "set tabstop=2\nset expandtab\nset shiftwidth=2\n" > /root/.vimrc'
+      echo "    $n ok"
+    else
+      echo "    warn: could not install tools on $n (no outbound network?). Fall back to:"
+      echo "          $RT cp $n:/path/file /tmp/f && \$EDITOR /tmp/f && $RT cp /tmp/f $n:/path/file"
+    fi
+  done
+fi
+
+# remember the runtime so the CLI uses the matching exec command
+mkdir -p .state && echo "$RT" > .state/runtime
+
+chmod +x exam
+echo
+echo "Cluster ready. Nodes:"
+kubectl get nodes -o wide
+echo
+echo "Next:  ./exam start     (seeds all 20 tasks and starts the 120-minute clock)"

teardown.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+# Deletes the kind cluster and all local exam state.
+set -euo pipefail
+cd "$(dirname "$0")"
+RT="${CONTAINER_RUNTIME:-$(cat .state/runtime 2>/dev/null || echo podman)}"
+[ "$RT" = podman ] && export KIND_EXPERIMENTAL_PROVIDER=podman
+kind delete cluster --name cka-sim || true
+rm -rf .state workspace
+echo "cka-sim cluster and local state removed."

tracks/cka/bank_extra.py

@@ -0,0 +1,646 @@
+"""Extra normal-difficulty CKA tasks (ids 21-43). Parameters (%{...}) are drawn per session."""
+from tracks.cka.common import C, NS, ns_pick, name_pick
+
+QUESTIONS = [
+# ---------------------------------------------------------------- ARCH ----
+{
+ "id": 21, "domain": "ARCH", "title": "Helm release install and upgrade",
+ "params": lambda r: {"ns": ns_pick(r, "helm"), "rel": name_pick(r, "web"), "n": r.choice([3, 4, 5])},
+ "task": """
+A Helm chart is at `{WS}/q21/chart` (values: `replicaCount`, `image.*`, `service.type`, `service.port`).
+
+1. Install it as release `%{rel}` into namespace `%{ns}` (create the namespace) with
+   `replicaCount=%{n}` overridden.
+2. Then **upgrade** the same release so the Service type becomes `NodePort`
+   (value `service.type`). The release must be at revision 2 or higher.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+rm -rf "$WS/q21"; mkdir -p "$WS/q21/chart/templates"
+cat > "$WS/q21/chart/Chart.yaml" <<'EOF'
+apiVersion: v2
+name: chart
+description: A small web chart for the exam simulator
+type: application
+version: 0.1.0
+appVersion: "1.27"
+EOF
+cat > "$WS/q21/chart/values.yaml" <<'EOF'
+replicaCount: 1
+image:
+  repository: nginx
+  tag: "1.27"
+service:
+  type: ClusterIP
+  port: 80
+EOF
+cat > "$WS/q21/chart/templates/_helpers.tpl" <<'EOF'
+{{- define "chart.fullname" -}}
+{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+EOF
+cat > "$WS/q21/chart/templates/deployment.yaml" <<'EOF'
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "chart.fullname" . }}
+  labels: {app: {{ include "chart.fullname" . }}}
+spec:
+  replicas: {{ .Values.replicaCount }}
+  selector:
+    matchLabels: {app: {{ include "chart.fullname" . }}}
+  template:
+    metadata:
+      labels: {app: {{ include "chart.fullname" . }}}
+    spec:
+      containers:
+        - name: web
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+          ports: [{containerPort: 80}]
+EOF
+cat > "$WS/q21/chart/templates/service.yaml" <<'EOF'
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ include "chart.fullname" . }}
+spec:
+  type: {{ .Values.service.type }}
+  selector: {app: {{ include "chart.fullname" . }}}
+  ports: [{port: {{ .Values.service.port }}, targetPort: 80}]
+EOF
+command -v helm >/dev/null || echo "warn: helm is not installed on this machine - Q21 cannot be completed or graded without it"
+helm -n %{ns} uninstall %{rel} >/dev/null 2>&1 || true
+""",
+ "checks": [
+  C("release %{rel} deployed in %{ns}", 1, r"""helm -n %{ns} status %{rel} -o json 2>/dev/null | python3 -c 'import sys,json;sys.exit(0 if json.load(sys.stdin)["info"]["status"]=="deployed" else 1)'"""),
+  C("replicaCount=%{n} applied", 1, r"""eq "$(j -n %{ns} deploy %{rel}-chart '{.spec.replicas}')" %{n}"""),
+  C("upgraded: revision >= 2 and Service is NodePort", 2, r"""
+rev=$(helm -n %{ns} history %{rel} -o json 2>/dev/null | python3 -c 'import sys,json;print(max(r["revision"] for r in json.load(sys.stdin)))' 2>/dev/null)
+[ -n "$rev" ] && [ "$rev" -ge 2 ] && eq "$(j -n %{ns} svc %{rel}-chart '{.spec.type}')" NodePort"""),
+ ],
+},
+{
+ "id": 33, "domain": "ARCH", "title": "Aggregated ClusterRole",
+ "params": lambda r: {"cr": name_pick(r, "ops"), "user": r.choice(["dana", "lee", "sam"])},
+ "task": """
+1. Create a ClusterRole `%{cr}` that has **no rules of its own** but aggregates every
+   ClusterRole labelled `rbac.example.com/aggregate-to-%{cr}=true`.
+2. Create two ClusterRoles carrying that label: `%{cr}-pods` (get/list/watch pods) and
+   `%{cr}-services` (get/list/watch services).
+3. Bind `%{cr}` cluster-wide to user `%{user}` with ClusterRoleBinding `%{cr}-binding`.
+
+`kubectl auth can-i list pods --as=%{user} -A` must say yes; deleting pods must say no.
+""",
+ "setup": r"""
+for x in %{cr} %{cr}-pods %{cr}-services; do $K delete clusterrole $x --ignore-not-found >/dev/null; done
+$K delete clusterrolebinding %{cr}-binding --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("%{cr} has an aggregationRule with the right label", 1, r"""j clusterrole %{cr} '{.spec.aggregationRule.clusterRoleSelectors[*].matchLabels}' | grep -q 'aggregate-to-%{cr}":"true'"""),
+  C("aggregation populated rules from the two labelled roles", 2, r"""j clusterrole %{cr} '{.rules[*].resources}' | grep -q pods && j clusterrole %{cr} '{.rules[*].resources}' | grep -q services"""),
+  C("%{user} can list pods and services cluster-wide", 1, r"""eq "$($K auth can-i list pods -A --as=%{user} 2>/dev/null)" yes && eq "$($K auth can-i list services -A --as=%{user} 2>/dev/null)" yes"""),
+  C("%{user} cannot delete pods", 1, r"""eq "$($K auth can-i delete pods -A --as=%{user} 2>/dev/null || true)" no"""),
+ ],
+},
+{
+ "id": 34, "domain": "ARCH", "title": "Discover a CRD field with kubectl explain",
+ "params": lambda r: {"kind": r.choice(["Widget", "Gadget", "Gizmo"]), "field": r.choice(["retries", "attempts", "tries"]), "n": r.randint(2, 7)},
+ "task": """
+A CRD for kind `%{kind}` (group `demo.io`) is installed. Somewhere in its schema is an
+integer field whose description is exactly **"Number of retry attempts"**.
+
+1. Using `kubectl explain`, find the field's full path (e.g. `spec.foo.bar`) and write it
+   on one line to `{WS}/q34/path.txt`.
+2. Create a `%{kind}` named `sample` in namespace `default` with that field set to `%{n}`.
+""",
+ "setup": r"""
+rm -rf "$WS/q34"; mkdir -p "$WS/q34"
+K_LOWER=$(echo %{kind} | tr A-Z a-z)
+$K delete crd ${K_LOWER}s.demo.io --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata: {name: ${K_LOWER}s.demo.io}
+spec:
+  group: demo.io
+  scope: Namespaced
+  names: {kind: %{kind}, plural: ${K_LOWER}s, singular: ${K_LOWER}}
+  versions:
+    - name: v1
+      served: true
+      storage: true
+      schema:
+        openAPIV3Schema:
+          type: object
+          properties:
+            spec:
+              type: object
+              properties:
+                mode: {type: string, description: "Operating mode"}
+                tuning:
+                  type: object
+                  description: "Performance tuning knobs"
+                  properties:
+                    %{field}: {type: integer, description: "Number of retry attempts"}
+                    backoffSeconds: {type: integer, description: "Delay between attempts"}
+EOF
+""",
+ "checks": [
+  C("path.txt has the correct field path", 2, r"""[ "$(tr -d '[:space:]' < "$WS/q34/path.txt" 2>/dev/null)" = "spec.tuning.%{field}" ]"""),
+  C("%{kind}/sample created with the field = %{n}", 2, r"""K_LOWER=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${K_LOWER}s.demo.io sample '{.spec.tuning.%{field}}')" %{n}"""),
+ ],
+},
+{
+ "id": 40, "domain": "ARCH", "title": "kubeadm join token",
+ "params": lambda r: {"ttl": r.choice(["2h", "6h", "24h"])},
+ "task": """
+A new worker is going to be added to the cluster.
+
+On the control-plane node (`{SHELL_CP}`), create a **new** bootstrap token with TTL `%{ttl}`
+and save the complete `kubeadm join` command for it (with `--discovery-token-ca-cert-hash`)
+to `{WS}/q40/join.sh`.
+""",
+ "setup": r"""
+rm -rf "$WS/q40"; mkdir -p "$WS/q40"
+for t in $($K -n kube-system get secrets -o jsonpath='{.items[?(@.type=="bootstrap.kubernetes.io/token")].metadata.name}'); do $K -n kube-system delete secret $t >/dev/null; done
+""",
+ "checks": [
+  C("join.sh contains a kubeadm join with token and CA hash", 2, r"""grep -q 'kubeadm join' "$WS/q40/join.sh" 2>/dev/null && grep -q -- '--token' "$WS/q40/join.sh" && grep -q -- '--discovery-token-ca-cert-hash sha256:' "$WS/q40/join.sh" """),
+  C("the token in the file exists in the cluster with TTL ~%{ttl}", 3, r"""
+tok=$(grep -o -- '--token [a-z0-9]\{6\}\.[a-z0-9]\{16\}' "$WS/q40/join.sh" 2>/dev/null | awk '{print $2}'); [ -n "$tok" ] || exit 1
+id=${tok%%.*}; exp=$(j -n kube-system secret bootstrap-token-$id '{.data.expiration}' | base64 -d 2>/dev/null); [ -n "$exp" ] || exit 1
+secs=$(( $(date -d "$exp" +%s) - $(date +%s) )); want=$(( $(echo %{ttl} | tr -d h) * 3600 ))
+[ $secs -gt $((want - 900)) ] && [ $secs -le $((want + 60)) ]"""),
+ ],
+},
+{
+ "id": 41, "domain": "ARCH", "title": "Component versions",
+ "params": lambda r: {"node": r.choice(["{W1}", "{W2}", "{W3}"])},
+ "task": """
+Write the following to `{WS}/q41/versions.txt`, one per line, in this exact format:
+
+```
+apiserver=<version>
+etcd=<version>
+kubelet=<version>
+runtime=<containerRuntimeVersion of node %{node}>
+```
+
+Use the values the cluster itself reports (`kubectl`, not what you remember).
+""",
+ "setup": r"""rm -rf "$WS/q41"; mkdir -p "$WS/q41" """,
+ "checks": [
+  C("apiserver and kubelet versions correct", 2, r"""
+f="$WS/q41/versions.txt"; [ -f "$f" ] || exit 1
+api=$($K version -o json 2>/dev/null | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])')
+kub=$(j node %{node} '{.status.nodeInfo.kubeletVersion}')
+grep -qx "apiserver=$api" "$f" && grep -qx "kubelet=$kub" "$f" """),
+  C("etcd and runtime versions correct", 2, r"""
+f="$WS/q41/versions.txt"
+etcd=$(j -n kube-system pod -l component=etcd '{.items[0].spec.containers[0].image}' | sed 's/.*://')
+rt=$(j node %{node} '{.status.nodeInfo.containerRuntimeVersion}')
+grep -qx "etcd=$etcd" "$f" && grep -qx "runtime=$rt" "$f" """),
+ ],
+},
+# ---------------------------------------------------------------- WORK ----
+{
+ "id": 23, "domain": "WORK", "title": "metrics-server repair and HPA",
+ "params": lambda r: {"ns": ns_pick(r, "scale"), "dep": name_pick(r, "api"), "max": r.choice([5, 6, 8]), "cpu": r.choice([60, 70, 75])},
+ "task": """
+`metrics-server` was installed in `kube-system` but `kubectl top nodes` does not work.
+
+1. Fix metrics-server so `kubectl top nodes` returns data (hint: in this cluster the kubelet
+   serving certificates are not signed by the cluster CA).
+2. Create an HPA `%{dep}` for Deployment `%{dep}` in namespace `%{ns}`:
+   min 2, max %{max}, target average CPU utilisation %{cpu}%.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} create deploy %{dep} --image=nginx:1.25 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n %{ns} set resources deploy %{dep} --requests=cpu=50m >/dev/null
+$K -n %{ns} delete hpa %{dep} --ignore-not-found >/dev/null
+if ! $K -n kube-system get deploy metrics-server >/dev/null 2>&1; then
+  curl -sfL --max-time 60 https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -o /tmp/ms.yaml && $K apply -f /tmp/ms.yaml >/dev/null || echo "warn: could not download metrics-server manifest"
+fi
+# make it broken (kind kubelets use self-signed serving certs)
+$K -n kube-system get deploy metrics-server -o json 2>/dev/null | python3 -c '
+import sys,json; d=json.load(sys.stdin); c=d["spec"]["template"]["spec"]["containers"][0]
+c["args"]=[a for a in c["args"] if a!="--kubelet-insecure-tls"]; print(json.dumps(d))' | $K apply -f - >/dev/null 2>&1 || true
+""",
+ "checks": [
+  C("kubectl top nodes works", 2, r"""for i in $(seq 1 12); do $K top nodes >/dev/null 2>&1 && exit 0; sleep 5; done; exit 1"""),
+  C("HPA %{dep}: min 2, max %{max}", 1, r"""eq "$(j -n %{ns} hpa %{dep} '{.spec.minReplicas}')" 2 && eq "$(j -n %{ns} hpa %{dep} '{.spec.maxReplicas}')" %{max}"""),
+  C("HPA targets CPU %{cpu}% on the deployment", 1, r"""eq "$(j -n %{ns} hpa %{dep} '{.spec.scaleTargetRef.name}')" %{dep} && j -n %{ns} hpa %{dep} '{.spec.metrics[*].resource}' | grep -q '"averageUtilization":%{cpu}'"""),
+ ],
+},
+{
+ "id": 24, "domain": "WORK", "title": "PriorityClass",
+ "params": lambda r: {"ns": ns_pick(r, "prio"), "pc": name_pick(r, "critical"), "val": r.choice([100000, 500000, 900000])},
+ "task": """
+1. Create a PriorityClass `%{pc}` with value %{val}, not the global default, description "%{pc} workloads".
+2. In namespace `%{ns}`, Deployment `payments` already exists. Make its pods use `%{pc}` and
+   ensure a rollout completes (pods must show priority %{val}).
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K delete priorityclass %{pc} --ignore-not-found >/dev/null
+$K -n %{ns} create deploy payments --image=nginx:1.25 --replicas=2 --dry-run=client -o yaml | $K apply -f - >/dev/null
+""",
+ "checks": [
+  C("PriorityClass %{pc} value %{val}, not default", 2, r"""eq "$(j priorityclass %{pc} '{.value}')" %{val} && [ "$(j priorityclass %{pc} '{.globalDefault}')" != true ]"""),
+  C("payments pods running with priority %{val}", 2, r"""eq "$(j -n %{ns} deploy payments '{.status.availableReplicas}')" 2 && for p in $(j -n %{ns} pod -l app=payments '{.items[*].spec.priority}'); do eq "$p" %{val} || exit 1; done"""),
+ ],
+},
+{
+ "id": 25, "domain": "WORK", "breaks": "w1-taint", "title": "Taints and tolerations",
+ "params": lambda r: {"ns": ns_pick(r, "sched"), "key": r.choice(["dedicated", "workload", "team"]), "val": r.choice(["gpu", "batch", "finance"]), "dep": name_pick(r, "worker")},
+ "task": """
+1. Taint node `{W1}` with `%{key}=%{val}:NoSchedule`.
+2. Create Deployment `%{dep}` in namespace `%{ns}` (image `busybox:1.36`, command `sleep 3600`,
+   2 replicas) that tolerates that taint **and** is pinned to `{W1}` with a nodeSelector on
+   `kubernetes.io/hostname`.
+
+Both pods must be Running on `{W1}`. Do not remove the taint.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K taint node "$W1" %{key}- >/dev/null 2>&1 || true
+$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("{W1} carries %{key}=%{val}:NoSchedule", 1, r"""j node "$W1" '{.spec.taints}' | grep -q '"key":"%{key}"' && j node "$W1" '{.spec.taints}' | grep -q '"value":"%{val}"' """),
+  C("deployment tolerates the taint", 1, r"""j -n %{ns} deploy %{dep} '{.spec.template.spec.tolerations}' | grep -q '"key":"%{key}"' """),
+  C("both pods Running on {W1}", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && for n in $(j -n %{ns} pod -l app=%{dep} '{.items[*].spec.nodeName}'); do eq "$n" "$W1" || exit 1; done"""),
+ ],
+},
+{
+ "id": 26, "domain": "WORK", "title": "DaemonSet on every node",
+ "params": lambda r: {"ns": ns_pick(r, "agents"), "ds": name_pick(r, "node-agent")},
+ "task": """
+Create a DaemonSet `%{ds}` in namespace `%{ns}` (image `busybox:1.36`, command `sleep 3600`)
+that runs on **every** node, including the control-plane.
+Requests: cpu 10m, memory 16Mi.
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete ds %{ds} --ignore-not-found >/dev/null""",
+ "checks": [
+  C("DaemonSet exists with requests set", 1, r"""eq "$(j -n %{ns} ds %{ds} '{.spec.template.spec.containers[0].resources.requests.cpu}')" 10m"""),
+  C("tolerates the control-plane taint", 1, r"""j -n %{ns} ds %{ds} '{.spec.template.spec.tolerations[*].key}' | grep -q control-plane"""),
+  C("one ready pod per node (including control-plane)", 2, r"""n=$($K get nodes --no-headers 2>/dev/null | wc -l); eq "$(j -n %{ns} ds %{ds} '{.status.numberReady}')" "$n" && eq "$(j -n %{ns} ds %{ds} '{.status.desiredNumberScheduled}')" "$n" """),
+ ],
+},
+{
+ "id": 27, "domain": "WORK", "title": "CronJob with a manual run",
+ "params": lambda r: {"ns": ns_pick(r, "jobs"), "cj": name_pick(r, "report"), "every": r.choice([5, 10, 15])},
+ "task": """
+1. Create CronJob `%{cj}` in `%{ns}`: runs every %{every} minutes, image `busybox:1.36`,
+   command `date`, `concurrencyPolicy: Forbid`, keep 3 successful and 1 failed job,
+   each run limited to 30 seconds (`activeDeadlineSeconds`), restartPolicy Never.
+2. Trigger it manually once as a Job named `%{cj}-manual` and let it complete.
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete cj %{cj} --ignore-not-found >/dev/null; $K -n %{ns} delete job %{cj}-manual --ignore-not-found >/dev/null""",
+ "checks": [
+  C("schedule every %{every} minutes, Forbid, history limits", 2, r"""c="$K -n %{ns} get cj %{cj}"; eq "$($c -o jsonpath='{.spec.schedule}' 2>/dev/null)" "*/%{every} * * * *" && eq "$($c -o jsonpath='{.spec.concurrencyPolicy}')" Forbid && eq "$($c -o jsonpath='{.spec.successfulJobsHistoryLimit}')" 3 && eq "$($c -o jsonpath='{.spec.failedJobsHistoryLimit}')" 1"""),
+  C("activeDeadlineSeconds 30, restartPolicy Never", 1, r"""eq "$(j -n %{ns} cj %{cj} '{.spec.jobTemplate.spec.activeDeadlineSeconds}')" 30 && eq "$(j -n %{ns} cj %{cj} '{.spec.jobTemplate.spec.template.spec.restartPolicy}')" Never"""),
+  C("manual job %{cj}-manual completed", 1, r"""for i in $(seq 1 10); do [ "$(j -n %{ns} job %{cj}-manual '{.status.succeeded}')" = 1 ] && exit 0; sleep 3; done; exit 1"""),
+ ],
+},
+{
+ "id": 28, "domain": "WORK", "title": "Init container prepares content",
+ "params": lambda r: {"ns": ns_pick(r, "web"), "pod": name_pick(r, "site"), "text": r.choice(["hello from init", "prepared by init", "init was here"])},
+ "task": """
+Create pod `%{pod}` in namespace `%{ns}`:
+- init container `setup` (busybox:1.36) writes the text `%{text}` into `/work/index.html`,
+- main container `web` (nginx:1.25) serves that file from `/usr/share/nginx/html`,
+- both share an `emptyDir` volume named `content`.
+
+`curl` from inside the cluster to the pod on port 80 must return the text.
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""",
+ "checks": [
+  C("pod Running, init container named setup completed", 1, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.initContainers[0].name}')" setup"""),
+  C("emptyDir 'content' shared by both containers", 1, r"""[ -n "$(j -n %{ns} pod %{pod} '{.spec.volumes[?(@.name=="content")].emptyDir}')" ] && j -n %{ns} pod %{pod} '{.spec.containers[0].volumeMounts[*].name}' | grep -qw content"""),
+  C("nginx serves the init-written content", 2, r"""$K -n %{ns} exec %{pod} -c web -- cat /usr/share/nginx/html/index.html 2>/dev/null | grep -q '%{text}'"""),
+ ],
+},
+{
+ "id": 29, "domain": "WORK", "title": "Immutable ConfigMap rotation",
+ "params": lambda r: {"ns": ns_pick(r, "cfg"), "dep": name_pick(r, "app"), "cm": name_pick(r, "settings")},
+ "task": """
+Deployment `%{dep}` in `%{ns}` mounts ConfigMap `%{cm}` at `/etc/cfg`. The ConfigMap is
+**immutable** and its key `mode` is `old`.
+
+The application must now read `mode=new`. Without deleting `%{cm}` and without editing it
+(it cannot be edited), make the running pods see `new` at `/etc/cfg/mode`.
+Hint: version the ConfigMap.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null
+$K -n %{ns} delete cm -l q=29 --ignore-not-found >/dev/null; $K -n %{ns} delete cm %{cm} --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: ConfigMap
+metadata: {name: %{cm}, namespace: %{ns}, labels: {q: "29"}}
+immutable: true
+data: {mode: old}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: %{dep}, namespace: %{ns}}
+spec:
+  replicas: 2
+  selector: {matchLabels: {app: %{dep}}}
+  template:
+    metadata: {labels: {app: %{dep}}}
+    spec:
+      containers:
+        - name: app
+          image: busybox:1.36
+          command: [sleep, "3600"]
+          volumeMounts: [{name: cfg, mountPath: /etc/cfg}]
+      volumes: [{name: cfg, configMap: {name: %{cm}}}]
+EOF
+""",
+ "checks": [
+  C("original ConfigMap %{cm} still exists and is immutable", 1, r"""eq "$(j -n %{ns} cm %{cm} '{.immutable}')" true && eq "$(j -n %{ns} cm %{cm} '{.data.mode}')" old"""),
+  C("deployment now references a different ConfigMap", 1, r"""n=$(j -n %{ns} deploy %{dep} '{.spec.template.spec.volumes[0].configMap.name}'); [ -n "$n" ] && [ "$n" != %{cm} ]"""),
+  C("all pods available and see mode=new", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && for p in $(j -n %{ns} pod -l app=%{dep} '{.items[*].metadata.name}'); do eq "$($K -n %{ns} exec $p -- cat /etc/cfg/mode 2>/dev/null)" new || exit 1; done"""),
+ ],
+},
+{
+ "id": 31, "domain": "WORK", "title": "Parallel Job",
+ "params": lambda r: {"ns": ns_pick(r, "batch"), "job": name_pick(r, "crunch"), "c": r.choice([6, 8, 9]), "p": r.choice([2, 3])},
+ "task": """
+Create Job `%{job}` in `%{ns}`: image `busybox:1.36`, command `sh -c "echo done; sleep 2"`,
+%{c} completions, %{p} in parallel, `backoffLimit: 2`, restartPolicy Never. Let it finish.
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete job %{job} --ignore-not-found >/dev/null""",
+ "checks": [
+  C("completions %{c}, parallelism %{p}, backoffLimit 2", 2, r"""eq "$(j -n %{ns} job %{job} '{.spec.completions}')" %{c} && eq "$(j -n %{ns} job %{job} '{.spec.parallelism}')" %{p} && eq "$(j -n %{ns} job %{job} '{.spec.backoffLimit}')" 2"""),
+  C("job succeeded %{c} times", 2, r"""for i in $(seq 1 20); do [ "$(j -n %{ns} job %{job} '{.status.succeeded}')" = %{c} ] && exit 0; sleep 3; done; exit 1"""),
+ ],
+},
+{
+ "id": 32, "domain": "WORK", "title": "ResourceQuota and LimitRange",
+ "params": lambda r: {"ns": ns_pick(r, "team"), "pods": r.choice([4, 5, 6]), "dcpu": r.choice(["100m", "150m", "200m"])},
+ "task": """
+In namespace `%{ns}`:
+1. ResourceQuota `%{ns}-quota`: at most %{pods} pods, total CPU requests 1, total memory limits 2Gi.
+2. LimitRange `%{ns}-defaults`: for containers, default request cpu %{dcpu} / memory 64Mi and
+   default limit cpu 500m / memory 256Mi.
+3. Create Deployment `worker` (nginx:1.25, 2 replicas) **without** specifying resources.
+   Its pods must be Running with the defaults applied.
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete quota,limitrange --all >/dev/null 2>&1; $K -n %{ns} delete deploy worker --ignore-not-found >/dev/null""",
+ "checks": [
+  C("ResourceQuota limits pods/%{pods}, requests.cpu 1, limits.memory 2Gi", 2, r"""q="$K -n %{ns} get quota %{ns}-quota"; eq "$($q -o jsonpath='{.spec.hard.pods}' 2>/dev/null)" %{pods} && eq "$($q -o jsonpath='{.spec.hard.requests\.cpu}')" 1 && eq "$($q -o jsonpath='{.spec.hard.limits\.memory}')" 2Gi"""),
+  C("LimitRange defaults set", 1, r"""l="$K -n %{ns} get limitrange %{ns}-defaults"; eq "$($l -o jsonpath='{.spec.limits[0].defaultRequest.cpu}' 2>/dev/null)" %{dcpu} && eq "$($l -o jsonpath='{.spec.limits[0].default.memory}')" 256Mi"""),
+  C("worker pods Running with defaulted requests", 1, r"""eq "$(j -n %{ns} deploy worker '{.status.availableReplicas}')" 2 && eq "$(j -n %{ns} pod -l app=worker '{.items[0].spec.containers[0].resources.requests.cpu}')" %{dcpu}"""),
+ ],
+},
+# ----------------------------------------------------------------- NET ----
+{
+ "id": 22, "domain": "NET", "title": "Gateway API HTTPRoute",
+ "params": lambda r: {"ns": ns_pick(r, "gw"), "gc": name_pick(r, "lab-gc"), "gw": name_pick(r, "edge"), "host": r.choice(["shop", "api", "app"])},
+ "task": """
+1. Install the Gateway API **standard** channel CRDs (v1.2.1 or newer) into the cluster.
+2. Create a GatewayClass `%{gc}` (controllerName `example.com/gateway`).
+3. In namespace `%{ns}` create Gateway `%{gw}` using that class, one HTTP listener on port 80
+   for hostname `%{host}.example.com`.
+4. Create HTTPRoute `%{host}-route` attached to `%{gw}` that sends `/api` (PathPrefix) to
+   Service `api` port 8080 and everything else to Service `web` port 80.
+
+(No controller runs here; you are graded on the objects.)
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n %{ns} create deploy api --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n %{ns} expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n %{ns} expose deploy api --port=8080 --target-port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K delete gatewayclass %{gc} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete gateway %{gw} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete httproute %{host}-route --ignore-not-found >/dev/null 2>&1
+""",
+ "checks": [
+  C("Gateway API CRDs installed (gateways, httproutes)", 1, r"""$K get crd gateways.gateway.networking.k8s.io httproutes.gateway.networking.k8s.io >/dev/null 2>&1"""),
+  C("GatewayClass %{gc} with the controllerName", 1, r"""eq "$(j gatewayclass %{gc} '{.spec.controllerName}')" example.com/gateway"""),
+  C("Gateway %{gw}: class, HTTP listener :80 for %{host}.example.com", 2, r"""g="$K -n %{ns} get gateway %{gw}"; eq "$($g -o jsonpath='{.spec.gatewayClassName}' 2>/dev/null)" %{gc} && eq "$($g -o jsonpath='{.spec.listeners[0].port}')" 80 && eq "$($g -o jsonpath='{.spec.listeners[0].hostname}')" %{host}.example.com"""),
+  C("HTTPRoute: parentRef %{gw}, /api -> api:8080, default -> web:80", 2, r"""
+h="$K -n %{ns} get httproute %{host}-route"; $h >/dev/null 2>&1 || exit 1
+eq "$($h -o jsonpath='{.spec.parentRefs[0].name}')" %{gw} || exit 1
+$h -o json | python3 -c '
+import sys,json; r=json.load(sys.stdin)["spec"]["rules"]
+api=[x for x in r if any(m.get("path",{}).get("value")=="/api" for m in x.get("matches",[]))]
+web=[x for x in r if x is not api and x.get("backendRefs",[{}])[0].get("name")=="web"]
+ok=api and api[0]["backendRefs"][0]["name"]=="api" and api[0]["backendRefs"][0]["port"]==8080 and web and web[0]["backendRefs"][0]["port"]==80
+sys.exit(0 if ok else 1)'"""),
+ ],
+},
+{
+ "id": 30, "domain": "NET", "title": "Named ports",
+ "params": lambda r: {"ns": ns_pick(r, "ports"), "dep": name_pick(r, "catalog"), "pname": r.choice(["http-web", "app-port", "svc-http"])},
+ "task": """
+1. In `%{ns}`, Deployment `%{dep}` (nginx:1.25, 1 replica) must declare its container port 80
+   with the **name** `%{pname}`.
+2. Create Service `%{dep}` (ClusterIP, port 8080) whose `targetPort` references the port
+   **by name**, not by number.
+3. Add a readiness probe on the container that also references the port by name (HTTP GET /).
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete deploy,svc %{dep} --ignore-not-found >/dev/null""",
+ "checks": [
+  C("container port 80 named %{pname}", 1, r"""eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.containers[0].ports[?(@.containerPort==80)].name}')" %{pname}"""),
+  C("Service port 8080 -> targetPort %{pname} (by name)", 2, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].port}')" 8080 && eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].targetPort}')" %{pname}"""),
+  C("readiness probe uses the named port; endpoints ready", 1, r"""eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.containers[0].readinessProbe.httpGet.port}')" %{pname} && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""),
+ ],
+},
+{
+ "id": 42, "domain": "NET", "title": "Egress-only NetworkPolicy",
+ "params": lambda r: {"ns": ns_pick(r, "egress"), "app": name_pick(r, "client"), "cidr": r.choice(["10.96.0.0/12", "172.16.0.0/16", "192.168.100.0/24"])},
+ "task": """
+Namespace `%{ns}` has a Deployment `%{app}` (label `app=%{app}`).
+
+Create NetworkPolicy `%{app}-egress` that applies only to those pods and allows **egress** to:
+- DNS (UDP and TCP 53) to pods in `kube-system` labelled `k8s-app=kube-dns`,
+- TCP 443 to the CIDR `%{cidr}`,
+and denies all other egress. Do not restrict ingress.
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} create deploy %{app} --image=busybox:1.36 --dry-run=client -o yaml -- sleep 3600 | $K apply -f - >/dev/null; $K -n %{ns} delete netpol %{app}-egress --ignore-not-found >/dev/null""",
+ "checks": [
+  C("applies to app=%{app}, Egress only", 1, r"""n="$K -n %{ns} get netpol %{app}-egress"; eq "$($n -o jsonpath='{.spec.podSelector.matchLabels.app}' 2>/dev/null)" %{app} && eq "$($n -o jsonpath='{.spec.policyTypes}')" '["Egress"]'"""),
+  C("DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP", 2, r"""
+$K -n %{ns} get netpol %{app}-egress -o json 2>/dev/null | python3 -c '
+import sys,json; e=json.load(sys.stdin)["spec"].get("egress",[])
+dns=[r for r in e if any((p.get("port")==53 or p.get("port")=="53") for p in r.get("ports",[]))]
+ok=False
+for r in dns:
+    protos={p.get("protocol","TCP") for p in r["ports"] if str(p.get("port"))=="53"}
+    to=r.get("to",[])
+    nsok=any("kube-system" in json.dumps(t.get("namespaceSelector",{})) for t in to)
+    podok=any(t.get("podSelector",{}).get("matchLabels",{}).get("k8s-app")=="kube-dns" for t in to)
+    if protos>={"UDP","TCP"} and nsok and podok: ok=True
+sys.exit(0 if ok else 1)'"""),
+  C("TCP 443 to %{cidr} only", 1, r"""
+$K -n %{ns} get netpol %{app}-egress -o json 2>/dev/null | python3 -c '
+import sys,json; e=json.load(sys.stdin)["spec"].get("egress",[])
+ok=any(any(t.get("ipBlock",{}).get("cidr")=="%{cidr}" for t in r.get("to",[])) and any(str(p.get("port"))=="443" for p in r.get("ports",[])) for r in e)
+sys.exit(0 if ok else 1)'"""),
+ ],
+},
+{
+ "id": 43, "domain": "NET", "title": "Headless and ExternalName services",
+ "params": lambda r: {"ns": ns_pick(r, "svc2"), "sts": name_pick(r, "db"), "ext": r.choice(["api.example.com", "db.corp.example.org", "cache.internal.example.net"])},
+ "task": """
+1. In `%{ns}`, StatefulSet `%{sts}` (2 replicas) exists but has no Service. Create the **headless**
+   Service `%{sts}` it needs (port 5432) so that `%{sts}-0.%{sts}.%{ns}.svc.cluster.local` resolves.
+2. Create Service `legacy` of type ExternalName pointing at `%{ext}`.
+3. From a pod in the namespace, resolve `%{sts}-0.%{sts}` and write the full nslookup output
+   to `{WS}/q43/pod0.txt`.
+""",
+ "setup": NS + r"""
+mkns %{ns}; rm -rf "$WS/q43"; mkdir -p "$WS/q43"
+$K -n %{ns} delete svc %{sts} legacy --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: apps/v1
+kind: StatefulSet
+metadata: {name: %{sts}, namespace: %{ns}}
+spec:
+  serviceName: %{sts}
+  replicas: 2
+  selector: {matchLabels: {app: %{sts}}}
+  template:
+    metadata: {labels: {app: %{sts}}}
+    spec:
+      containers: [{name: db, image: busybox:1.36, command: [sleep, "3600"], ports: [{containerPort: 5432}]}]
+EOF
+""",
+ "checks": [
+  C("headless Service %{sts} (clusterIP None) selecting app=%{sts}, port 5432", 2, r"""eq "$(j -n %{ns} svc %{sts} '{.spec.clusterIP}')" None && eq "$(j -n %{ns} svc %{sts} '{.spec.selector.app}')" %{sts} && eq "$(j -n %{ns} svc %{sts} '{.spec.ports[0].port}')" 5432"""),
+  C("ExternalName Service legacy -> %{ext}", 1, r"""eq "$(j -n %{ns} svc legacy '{.spec.type}')" ExternalName && eq "$(j -n %{ns} svc legacy '{.spec.externalName}')" %{ext}"""),
+  C("pod0.txt shows %{sts}-0 resolving to its pod IP", 1, r"""ip=$(j -n %{ns} pod %{sts}-0 '{.status.podIP}'); [ -n "$ip" ] && grep -q "$ip" "$WS/q43/pod0.txt" 2>/dev/null"""),
+ ],
+},
+# ---------------------------------------------------------------- STOR ----
+{
+ "id": 35, "domain": "STOR", "title": "Re-bind a Released PersistentVolume",
+ "params": lambda r: {"ns": ns_pick(r, "store"), "pv": name_pick(r, "pv-archive"), "pvc": name_pick(r, "archive-claim")},
+ "task": """
+PersistentVolume `%{pv}` (1Gi, hostPath, class `manual`) is in status **Released**: the claim that used
+it was deleted, but its data must be kept and reused.
+
+Make `%{pv}` bindable again without deleting it, then create PVC `%{pvc}` in `%{ns}`
+(class `manual`, 500Mi, RWO) so it binds to **exactly** that PV. Do not create any other PV.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete pvc %{pvc} --ignore-not-found >/dev/null; $K delete pv %{pv} --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: PersistentVolume
+metadata: {name: %{pv}}
+spec:
+  capacity: {storage: 1Gi}
+  accessModes: [ReadWriteOnce]
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: manual
+  hostPath: {path: /mnt/%{pv}}
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: old-claim, namespace: %{ns}}
+spec: {accessModes: [ReadWriteOnce], storageClassName: manual, resources: {requests: {storage: 500Mi}}}
+EOF
+for i in $(seq 1 20); do [ "$(j -n %{ns} pvc old-claim '{.status.phase}')" = Bound ] && break; sleep 1; done
+$K -n %{ns} delete pvc old-claim >/dev/null
+""",
+ "checks": [
+  C("PV %{pv} still exists and was not recreated", 2, r"""[ -n "$(j pv %{pv} '{.metadata.uid}')" ] && eq "$(j pv %{pv} '{.spec.persistentVolumeReclaimPolicy}')" Retain"""),
+  C("PVC %{pvc} Bound to %{pv}", 3, r"""eq "$(j -n %{ns} pvc %{pvc} '{.status.phase}')" Bound && eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}')" %{pv}"""),
+  C("no extra PVs were created", 1, r"""[ "$($K get pv --no-headers 2>/dev/null | grep -c manual)" -le 2 ]"""),
+ ],
+},
+{
+ "id": 36, "domain": "STOR", "title": "Data survives the pod",
+ "params": lambda r: {"ns": ns_pick(r, "persist"), "pvc": name_pick(r, "notes"), "text": r.choice(["order-9981", "ticket-4471", "batch-2026"])},
+ "task": """
+1. In `%{ns}` create PVC `%{pvc}` (200Mi, RWO) using the cluster's default StorageClass.
+2. Run pod `writer` (busybox:1.36) that mounts it at `/data` and writes the text `%{text}`
+   into `/data/marker.txt`, then exits (restartPolicy Never).
+3. After `writer` has completed, run pod `reader` (busybox:1.36, `sleep 3600`) mounting the
+   same claim at `/data`. `cat /data/marker.txt` inside `reader` must show the text.
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod writer reader --ignore-not-found --wait=false >/dev/null; $K -n %{ns} delete pvc %{pvc} --ignore-not-found >/dev/null""",
+ "checks": [
+  C("PVC %{pvc} Bound with the default StorageClass", 1, r"""eq "$(j -n %{ns} pvc %{pvc} '{.status.phase}')" Bound && [ -n "$(j -n %{ns} pvc %{pvc} '{.spec.storageClassName}')" ]"""),
+  C("writer completed (Succeeded)", 1, r"""eq "$(j -n %{ns} pod writer '{.status.phase}')" Succeeded"""),
+  C("reader sees the marker written by writer", 2, r"""eq "$($K -n %{ns} exec reader -- cat /data/marker.txt 2>/dev/null | tr -d '[:space:]')" %{text}"""),
+ ],
+},
+# -------------------------------------------------------------- TSHOOT ----
+{
+ "id": 37, "domain": "TSHOOT", "breaks": "coredns", "title": "Cluster DNS is broken", "disruptive": True,
+ "params": lambda r: {"bad": r.choice(["forward . 10.255.255.1", "forward . 192.0.2.53"])},
+ "task": """
+Pods across the cluster can no longer resolve names (in-cluster or external).
+Find the cause in CoreDNS and fix it. `nslookup kubernetes.default` from any pod must work
+again, and CoreDNS must be healthy.
+""",
+ "setup": r"""
+$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > /tmp/Corefile.orig
+sed -i 's|forward \. /etc/resolv.conf|%{bad}|' /tmp/Corefile.orig
+$K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.orig --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n kube-system rollout restart deploy coredns >/dev/null
+""",
+ "checks": [
+  C("CoreDNS Corefile forwards to the node resolver again", 2, r"""j -n kube-system cm coredns '{.data.Corefile}' | grep -q 'forward \. /etc/resolv.conf'"""),
+  C("CoreDNS pods available", 1, r"""[ "$(j -n kube-system deploy coredns '{.status.availableReplicas}')" -ge 1 ]"""),
+  C("in-cluster and external names resolve from a pod", 3, r"""
+$K -n default delete pod dns-probe --ignore-not-found --wait=true >/dev/null 2>&1
+$K -n default run dns-probe --image=busybox:1.36 --restart=Never -- sh -c 'for i in 1 2 3 4 5 6; do nslookup kubernetes.default >/dev/null 2>&1 && nslookup kubernetes.io >/dev/null 2>&1 && echo OK && exit 0; sleep 5; done; exit 1' >/dev/null 2>&1
+for i in $(seq 1 20); do ph=$(j -n default pod dns-probe '{.status.phase}'); [ "$ph" = Succeeded ] && { $K -n default delete pod dns-probe --wait=false >/dev/null; exit 0; }; [ "$ph" = Failed ] && break; sleep 3; done
+$K -n default delete pod dns-probe --wait=false >/dev/null; exit 1"""),
+ ],
+},
+{
+ "id": 38, "domain": "TSHOOT", "breaks": "controller-manager", "title": "Deployments never create pods", "disruptive": True,
+ "task": """
+Creating a Deployment results in **no** ReplicaSet or pods, and nothing is reported in
+`kubectl describe deploy`. Existing pods still run.
+
+Find the failing control-plane component, repair it on the control-plane node
+(`{SHELL_CP}`) and make sure new Deployments get their pods.
+""",
+ "setup": r"""
+nx "$CP" '
+f=/etc/kubernetes/manifests/kube-controller-manager.yaml
+grep -q "^ *- kube-controller-manager$" $f && sed -i "s|^\( *\)- kube-controller-manager$|\1- kube-controller-manager\n\1- --leader-elect-lease-duration=1s|" $f
+'
+""",
+ "checks": [
+  C("kube-controller-manager manifest repaired and pod Running", 3, r"""! nx "$CP" 'grep -q "leader-elect-lease-duration=1s" /etc/kubernetes/manifests/kube-controller-manager.yaml' && [ "$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].status.phase}')" = Running ]"""),
+  C("a new Deployment gets a ReplicaSet and running pods", 4, r"""
+$K -n default delete deploy cm-probe --ignore-not-found --wait=true >/dev/null 2>&1
+$K -n default create deploy cm-probe --image=busybox:1.36 -- sleep 300 >/dev/null 2>&1
+ok=1; for i in $(seq 1 24); do [ "$(j -n default deploy cm-probe '{.status.availableReplicas}')" = 1 ] && { ok=0; break; }; sleep 5; done
+$K -n default delete deploy cm-probe --wait=false >/dev/null 2>&1; exit $ok"""),
+ ],
+},
+{
+ "id": 39, "domain": "TSHOOT", "breaks": "w1-kubelet", "title": "kubelet misconfiguration", "disruptive": True,
+ "task": """
+Node `{W1}` is `NotReady`. Someone "tuned" the kubelet on it last night.
+Bring the node back to `Ready`; the kubelet must be running and enabled.
+Reach the node with `{SHELL_W1}`. `journalctl -u kubelet` is your friend.
+""",
+ "setup": r"""
+nx "$W1" '
+cp -n /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.orig 2>/dev/null
+sed -i "s|clientCAFile: /etc/kubernetes/pki/ca.crt|clientCAFile: /etc/kubernetes/pki/ca-bundle.crt|" /var/lib/kubelet/config.yaml
+systemctl restart kubelet
+'
+""",
+ "checks": [
+  C("kubelet config points at a CA file that exists", 2, r"""f=$(nx "$W1" 'grep clientCAFile /var/lib/kubelet/config.yaml' | awk '{print $2}'); [ -n "$f" ] && nx "$W1" "test -f $f" """),
+  C("node Ready", 4, r"""for i in $(seq 1 12); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 5; done; exit 1"""),
+  C("kubelet active and enabled", 1, r"""eq "$(nx "$W1" 'systemctl is-active kubelet')" active && eq "$(nx "$W1" 'systemctl is-enabled kubelet')" enabled"""),
+ ],
+},
+]

tracks/cka/bank_hard.py

@@ -0,0 +1,660 @@
+"""Hard-difficulty CKA tasks (ids 101-122): multi-part, no hints, traps, exact output,
+compound failures, collateral-damage checks, exam-style contexts. Modelled on the kind of
+difficulty people report from killer.sh, not on its content."""
+from tracks.cka.common import C, NS, ns_pick, name_pick
+
+CTX = "Use context: `kubectl config use-context %{ctx}`\n\n"
+
+def ctx_pick(r):
+    return r.choice(["k8s-c1-H", "k8s-c2-AC", "k8s-c3-CCC"])
+
+QUESTIONS = [
+# ================================================================ ARCH ====
+{
+ "id": 101, "domain": "ARCH", "difficulty": "hard", "title": "Exact RBAC with an existing over-grant",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "ci"), "sa": name_pick(r, "ci-bot")},
+ "task": CTX + """
+ServiceAccount `%{sa}` in namespace `%{ns}` is used by a CI system. Its permissions must be
+**exactly**:
+- in `%{ns}`: `get`, `list`, `watch`, `patch` on `deployments` and `replicasets` (apps),
+  and `create` on `pods/exec`,
+- cluster-wide: `list` on `nodes`.
+
+Nothing more. Audit what it can do now and correct it. Use Role `%{sa}-role`, RoleBinding
+`%{sa}-rb`, ClusterRole `%{sa}-nodes`, ClusterRoleBinding `%{sa}-crb`.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} create sa %{sa} --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n %{ns} delete role,rolebinding --all >/dev/null 2>&1
+$K delete clusterrole %{sa}-nodes --ignore-not-found >/dev/null; $K delete clusterrolebinding %{sa}-crb %{sa}-legacy-admin --ignore-not-found >/dev/null
+# trap: someone bound cluster-admin to this SA "temporarily"
+$K create clusterrolebinding %{sa}-legacy-admin --clusterrole=cluster-admin --serviceaccount=%{ns}:%{sa} >/dev/null
+""",
+ "checks": [
+  C("can get/list/watch/patch deployments and replicasets in %{ns}", 2, r"""for res in deployments.apps replicasets.apps; do for v in get list watch patch; do eq "$($K auth can-i $v $res -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes || exit 1; done; done"""),
+  C("can create pods/exec in %{ns}; can list nodes", 2, r"""eq "$($K auth can-i create pods/exec -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes && eq "$($K auth can-i list nodes --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes"""),
+  C("no over-grant: cannot delete deployments, cannot get secrets, cannot get nodes", 3, r"""for t in "delete deployments -n %{ns}" "get secrets -n %{ns}" "get nodes" "create pods -n default"; do eq "$($K auth can-i $t --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null || true)" no || exit 1; done"""),
+  C("the cluster-admin binding is gone and named objects exist", 2, r"""! $K get clusterrolebinding %{sa}-legacy-admin >/dev/null 2>&1 && $K -n %{ns} get role %{sa}-role rolebinding %{sa}-rb >/dev/null 2>&1 && $K get clusterrole %{sa}-nodes clusterrolebinding %{sa}-crb >/dev/null 2>&1"""),
+ ],
+},
+{
+ "id": 102, "domain": "ARCH", "breaks": "etcd", "difficulty": "hard", "title": "etcd restore",
+ "params": lambda r: {"ctx": "k8s-c1-H", "cm": name_pick(r, "restore-proof")},
+ "task": CTX + """
+An etcd snapshot taken earlier is at `/opt/cka/backup.db` on the control-plane node.
+A ConfigMap `%{cm}` in namespace `default` existed when the snapshot was taken and has since
+been deleted.
+
+**Restore the cluster's etcd from that snapshot** so the ConfigMap exists again. The cluster
+must be healthy afterwards (API server up, all kube-system pods Running).
+Do not recreate the ConfigMap by hand — its data must come from the snapshot.
+""",
+ "setup": r"""
+nx "$CP" 'mkdir -p /opt/cka; rm -f /opt/cka/backup.db'
+$K -n default delete cm %{cm} --ignore-not-found >/dev/null
+$K -n default create cm %{cm} --from-literal=proof=$(date +%s)-%{cm} >/dev/null
+POD=$($K -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].metadata.name}')
+$K -n kube-system exec "$POD" -- sh -c 'ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/etcd/q102-backup.db' >/dev/null 2>&1
+nx "$CP" 'mv /var/lib/etcd/q102-backup.db /opt/cka/backup.db'
+for bin in etcdctl etcdutl; do $K -n kube-system exec "$POD" -- cat /usr/local/bin/$bin > /tmp/cka-$bin 2>/dev/null && [ -s /tmp/cka-$bin ] && nx "$CP" "cat > /usr/local/bin/$bin && chmod +x /usr/local/bin/$bin" < /tmp/cka-$bin; rm -f /tmp/cka-$bin; done
+sleep 2; $K -n default delete cm %{cm} >/dev/null
+""",
+ "checks": [
+  C("ConfigMap %{cm} exists again with its original data", 6, r"""j -n default cm %{cm} '{.data.proof}' | grep -q -- '-%{cm}$'"""),
+  C("etcd uses a restored data directory (manifest changed)", 2, r"""! nx "$CP" 'grep -q "path: /var/lib/etcd$" /etc/kubernetes/manifests/etcd.yaml' """),
+  C("control plane healthy: etcd, apiserver, scheduler, controller-manager Running", 2, r"""for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done"""),
+ ],
+},
+{
+ "id": 103, "domain": "ARCH", "breaks": "w3-kubelet", "difficulty": "hard", "title": "Static pod on a relocated manifest path",
+ "params": lambda r: {"ctx": "k8s-c1-H", "ns": ns_pick(r, "edge"), "pod": name_pick(r, "edge-proxy")},
+ "task": CTX + """
+Exactly one worker node is labelled `tier=edge`. On that node create a **static pod**:
+- name `%{pod}`, namespace `%{ns}` (create the namespace),
+- image `nginx:1.27`, requests cpu 50m / memory 32Mi, label `role=edge`.
+
+The mirror pod must be Running. Note that the kubelet on that node may not be configured the
+way you expect.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+for n in "$W1" "$W2" "$W3"; do $K label node $n tier- >/dev/null 2>&1 || true; done
+$K label node "$W3" tier=edge --overwrite >/dev/null
+nx "$W3" '
+rm -f /etc/kubernetes/manifests/%{pod}*.yaml /etc/kubernetes/static/%{pod}*.yaml 2>/dev/null; mkdir -p /etc/kubernetes/static
+sed -i "s|^staticPodPath: .*|staticPodPath: /etc/kubernetes/static|" /var/lib/kubelet/config.yaml
+systemctl restart kubelet'
+""",
+ "checks": [
+  C("mirror pod %{pod}-<edge node> Running in %{ns}", 4, r"""eq "$(j -n %{ns} pod "%{pod}-$W3" '{.status.phase}')" Running"""),
+  C("image, requests and label correct", 2, r"""p="$K -n %{ns} get pod %{pod}-$W3"; eq "$($p -o jsonpath='{.spec.containers[0].image}' 2>/dev/null)" nginx:1.27 && eq "$($p -o jsonpath='{.spec.containers[0].resources.requests.cpu}')" 50m && eq "$($p -o jsonpath='{.metadata.labels.role}')" edge"""),
+  C("manifest lives in the kubelet's actual staticPodPath", 2, r"""d=$(nx "$W3" 'grep staticPodPath /var/lib/kubelet/config.yaml' | awk '{print $2}'); nx "$W3" "grep -ls %{pod} $d/*.yaml" >/dev/null 2>&1"""),
+ ],
+},
+{
+ "id": 104, "domain": "ARCH", "difficulty": "hard", "title": "User onboarding to a working kubeconfig",
+ "params": lambda r: {"ctx": "k8s-c1-H", "user": r.choice(["maya", "theo", "iris"]), "ns": ns_pick(r, "dev")},
+ "task": CTX + """
+Onboard user `%{user}` (group `developers`) end to end:
+1. Key + CSR with CN=%{user}, O=developers; Kubernetes CSR object `%{user}`, approved, certificate issued.
+2. A **self-contained** kubeconfig at `{WS}/q104/%{user}.kubeconfig` (certs embedded, not file paths)
+   with a context `%{user}@lab` whose default namespace is `%{ns}`.
+3. `%{user}` may do everything to pods, deployments and services in `%{ns}` (edit-level), nothing in other namespaces.
+
+`kubectl --kubeconfig {WS}/q104/%{user}.kubeconfig get pods` must work with no extra flags.
+""",
+ "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q104"; mkdir -p "$WS/q104"; $K delete csr %{user} --ignore-not-found >/dev/null; $K -n %{ns} delete rolebinding --all >/dev/null 2>&1""",
+ "checks": [
+  C("CSR %{user} approved with certificate, O=developers", 2, r"""[ -n "$(j csr %{user} '{.status.certificate}')" ] && j csr %{user} '{.status.certificate}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'O *= *developers'"""),
+  C("kubeconfig is self-contained and works without flags", 3, r"""f="$WS/q104/%{user}.kubeconfig"; [ -f "$f" ] || exit 1; grep -q 'client-certificate-data' "$f" && grep -q 'certificate-authority-data' "$f" && ! grep -q 'client-certificate:' "$f" && kubectl --kubeconfig "$f" get pods >/dev/null 2>&1"""),
+  C("context %{user}@lab defaults to %{ns}", 1, r"""eq "$(kubectl --kubeconfig "$WS/q104/%{user}.kubeconfig" config view -o jsonpath='{.contexts[?(@.name=="%{user}@lab")].context.namespace}')" %{ns}"""),
+  C("edit rights in %{ns}, none elsewhere", 2, r"""eq "$($K auth can-i create deployments -n %{ns} --as=%{user} 2>/dev/null)" yes && eq "$($K auth can-i delete pods -n %{ns} --as=%{user} 2>/dev/null)" yes && eq "$($K auth can-i get pods -n default --as=%{user} 2>/dev/null || true)" no && eq "$($K auth can-i get secrets -n kube-system --as=%{user} 2>/dev/null || true)" no"""),
+ ],
+},
+{
+ "id": 105, "domain": "ARCH", "difficulty": "hard", "title": "Kustomize with generators and patches",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "kz"), "rep": r.choice([3, 4]), "lvl": r.choice(["debug", "trace", "warn"])},
+ "task": CTX + """
+Base at `{WS}/q105/base` (Deployment `web`, Service `web`). Build overlay `{WS}/q105/overlays/prod` that:
+- targets namespace `%{ns}` and adds label `tier=prod` to all resources (labels only, not selectors),
+- sets replicas to %{rep} via a **strategic merge patch** file `replicas.yaml`,
+- adds a `configMapGenerator` named `web-config` with literal `LOG_LEVEL=%{lvl}` (hash suffix enabled),
+- injects that ConfigMap into the container via `envFrom` (JSON 6902 patch file `envfrom.yaml`).
+
+Apply with `kubectl apply -k`. The Deployment must reference the hashed ConfigMap name.
+""",
+ "setup": r"""
+rm -rf "$WS/q105"; mkdir -p "$WS/q105/base"
+cat > "$WS/q105/base/kustomization.yaml" <<'EOF'
+resources: [deployment.yaml, service.yaml]
+EOF
+cat > "$WS/q105/base/deployment.yaml" <<'EOF'
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: web}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: web}}
+  template:
+    metadata: {labels: {app: web}}
+    spec:
+      containers: [{name: web, image: nginx:1.25, ports: [{containerPort: 80}]}]
+EOF
+cat > "$WS/q105/base/service.yaml" <<'EOF'
+apiVersion: v1
+kind: Service
+metadata: {name: web}
+spec: {selector: {app: web}, ports: [{port: 80}]}
+EOF
+$K delete ns %{ns} --ignore-not-found --wait=false >/dev/null
+""",
+ "checks": [
+  C("overlay has replicas.yaml (SMP) and envfrom.yaml (JSON6902)", 1, r"""[ -f "$WS/q105/overlays/prod/replicas.yaml" ] && [ -f "$WS/q105/overlays/prod/envfrom.yaml" ] && grep -q 'op:' "$WS/q105/overlays/prod/envfrom.yaml" """),
+  C("deployed to %{ns}: %{rep} replicas, tier=prod label, selector untouched", 2, r"""d="$K -n %{ns} get deploy web"; eq "$($d -o jsonpath='{.spec.replicas}' 2>/dev/null)" %{rep} && eq "$($d -o jsonpath='{.metadata.labels.tier}')" prod && [ -z "$($d -o jsonpath='{.spec.selector.matchLabels.tier}')" ]"""),
+  C("hashed ConfigMap with LOG_LEVEL=%{lvl}, referenced via envFrom", 3, r"""cm=$(j -n %{ns} deploy web '{.spec.template.spec.containers[0].envFrom[0].configMapRef.name}'); [ -n "$cm" ] && echo "$cm" | grep -Eq '^web-config-[a-z0-9]{6,}$' && eq "$(j -n %{ns} cm "$cm" '{.data.LOG_LEVEL}')" %{lvl}"""),
+  C("pods available and env visible", 1, r"""eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" %{rep} && $K -n %{ns} exec deploy/web -- env 2>/dev/null | grep -q '^LOG_LEVEL=%{lvl}$'"""),
+ ],
+},
+{
+ "id": 106, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults",
+ "params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])},
+ "task": CTX + """
+Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural lowercase+s, shortName `%{kind}`-initials) with schema:
+- `spec.replicas` integer, **required**, minimum 1, maximum 10,
+- `spec.environment` string, enum `prod|stage|dev`, **default** `dev`,
+- `spec.image` string, required.
+Then:
+1. Create `%{kind}` `good` in `default` with replicas 3, image `nginx:1.27`, environment `%{env}`.
+2. Save the API server's rejection message for a `%{kind}` with `replicas: 0` to `{WS}/q106/rejected.txt`.
+3. Create `%{kind}` `defaulted` with only replicas 2 and image set; its environment must be `dev` after creation.
+""",
+ "setup": r"""rm -rf "$WS/q106"; mkdir -p "$WS/q106"; k=$(echo %{kind} | tr A-Z a-z); $K delete crd ${k}s.deploy.example.com --ignore-not-found >/dev/null""",
+ "checks": [
+  C("CRD exists, Namespaced, replicas has min/max and is required", 2, r"""k=$(echo %{kind} | tr A-Z a-z); c="$K get crd ${k}s.deploy.example.com"; eq "$($c -o jsonpath='{.spec.scope}' 2>/dev/null)" Namespaced && s=$($c -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec}'); echo "$s" | grep -q '"minimum":1' && echo "$s" | grep -q '"maximum":10' && echo "$s" | grep -q '"required":\[' """),
+  C("good created with environment %{env}", 1, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com good '{.spec.environment}')" %{env}"""),
+  C("replicas 0 is rejected and the message was saved", 2, r"""grep -qiE 'minimum|greater than|>= 1|invalid' "$WS/q106/rejected.txt" 2>/dev/null"""),
+  C("defaulting works: defaulted.spec.environment == dev", 2, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com defaulted '{.spec.environment}')" dev"""),
+ ],
+},
+{
+ "id": 122, "domain": "ARCH", "breaks": "certs", "difficulty": "hard", "title": "Renew control-plane certificates",
+ "params": lambda r: {"ctx": "k8s-c1-H"},
+ "task": CTX + """
+Security policy requires the control-plane certificates to be renewed now.
+On the control-plane node, renew **all** kubeadm-managed certificates, make every control-plane
+component pick up the new certificates, and update the admin kubeconfig on the node
+(`/etc/kubernetes/admin.conf`) so it uses the renewed client certificate.
+Write the new `apiserver` certificate's expiry (as printed by `kubeadm certs check-expiration`) to `{WS}/q122/expiry.txt`.
+""",
+ "setup": r"""rm -rf "$WS/q122"; mkdir -p "$WS/q122"; nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' > "$WS/.q122-before" """,
+ "checks": [
+  C("apiserver certificate was renewed (newer notBefore)", 3, r"""b=$(sed 's/notBefore=//' "$WS/.q122-before"); a=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' | sed 's/notBefore=//'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""),
+  C("all components restarted and Running with the new certs (apiserver serving cert matches file)", 3, r"""
+for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done
+srv=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial'); f=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial'); [ -n "$srv" ] && eq "$srv" "$f" """),
+  C("admin.conf uses the renewed client cert", 2, r"""nx "$CP" 'grep client-certificate-data /etc/kubernetes/admin.conf | awk "{print \$2}" | base64 -d | openssl x509 -noout -startdate' | sed 's/notBefore=//' > /tmp/q122a; b=$(sed 's/notBefore=//' "$WS/.q122-before"); [ "$(date -d "$(cat /tmp/q122a)" +%s)" -gt "$(date -d "$b" +%s)" ]"""),
+  C("expiry.txt matches kubeadm's report for apiserver", 1, r"""exp=$(nx "$CP" 'kubeadm certs check-expiration 2>/dev/null' | awk '$1=="apiserver"{print $2,$3,$4,$5,$6}'); [ -n "$exp" ] && grep -qF "$exp" "$WS/q122/expiry.txt" 2>/dev/null"""),
+ ],
+},
+# ================================================================ WORK ====
+{
+ "id": 107, "domain": "WORK", "breaks": "w1-taint", "difficulty": "hard", "title": "Spread, tolerate, prioritise — under a quota",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "spread"), "dep": name_pick(r, "gateway"), "key": r.choice(["pool", "lane"]), "val": r.choice(["ingress", "edge"])},
+ "task": CTX + """
+Node `{W1}` is tainted `%{key}=%{val}:NoSchedule`. In namespace `%{ns}` create Deployment `%{dep}`
+(nginx:1.27, **3 replicas**) such that:
+- exactly one pod runs on each worker node (required pod anti-affinity on `kubernetes.io/hostname`),
+- it tolerates the taint on `{W1}`,
+- it uses PriorityClass `%{dep}-prio` (value 50000, create it),
+- requests cpu 100m / memory 64Mi.
+
+All 3 pods must be Running. The namespace has a ResourceQuota — respect it or adjust it, but the
+quota must still cap pods in the namespace.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K taint node "$W1" %{key}- >/dev/null 2>&1 || true; $K taint node "$W1" %{key}=%{val}:NoSchedule >/dev/null
+$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null; $K delete priorityclass %{dep}-prio --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: ResourceQuota
+metadata: {name: %{ns}-quota, namespace: %{ns}}
+spec: {hard: {pods: "2", requests.cpu: "250m"}}
+EOF
+""",
+ "checks": [
+  C("3 pods available, one per worker", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 && [ "$(j -n %{ns} pod -l app=%{dep} '{.items[*].spec.nodeName}' | tr ' ' '\n' | sort -u | wc -l)" = 3 ]"""),
+  C("required anti-affinity on hostname + toleration for %{key}", 2, r"""t=$(j -n %{ns} deploy %{dep} '{.spec.template.spec}'); echo "$t" | grep -q 'requiredDuringSchedulingIgnoredDuringExecution' && echo "$t" | grep -q '"topologyKey":"kubernetes.io/hostname"' && echo "$t" | grep -q '"key":"%{key}"' """),
+  C("PriorityClass %{dep}-prio (50000) in use", 1, r"""eq "$(j priorityclass %{dep}-prio '{.value}')" 50000 && eq "$(j -n %{ns} pod -l app=%{dep} '{.items[0].spec.priority}')" 50000"""),
+  C("quota still exists and caps pods (>=3)", 2, r"""p=$(j -n %{ns} quota %{ns}-quota '{.spec.hard.pods}'); [ -n "$p" ] && [ "$p" -ge 3 ]"""),
+ ],
+},
+{
+ "id": 108, "domain": "WORK", "difficulty": "hard", "title": "Rollback to a specific revision, then pause",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "rel"), "dep": name_pick(r, "portal")},
+ "task": CTX + """
+Deployment `%{dep}` in `%{ns}` has gone through several releases; the current one is broken.
+1. Roll it back to the revision that used image `nginx:1.25` (not simply the previous revision).
+2. Scale it to 3 replicas and wait until available.
+3. **Pause** the rollout, then change the image to `nginx:1.27` and add env `RELEASE=next`.
+   Because it is paused, no new ReplicaSet may be created and the running pods must stay on `nginx:1.25`.
+Leave it paused.
+""",
+ "setup": NS + r"""
+mkns %{ns}; $K -n %{ns} delete deploy %{dep} --ignore-not-found --wait=true >/dev/null
+$K -n %{ns} create deploy %{dep} --image=nginx:1.25 >/dev/null
+$K -n %{ns} rollout status deploy %{dep} --timeout=90s >/dev/null 2>&1
+$K -n %{ns} set image deploy %{dep} nginx=nginx:1.26 >/dev/null; $K -n %{ns} rollout status deploy %{dep} --timeout=90s >/dev/null 2>&1
+$K -n %{ns} set image deploy %{dep} nginx=nginx:1.27 >/dev/null; $K -n %{ns} rollout status deploy %{dep} --timeout=90s >/dev/null 2>&1
+$K -n %{ns} set image deploy %{dep} nginx=nginx:9.99-broken >/dev/null
+""",
+ "checks": [
+  C("running pods use nginx:1.25 and 3 are available", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 && for i in $(j -n %{ns} pod -l app=%{dep} '{.items[*].spec.containers[0].image}'); do eq "$i" nginx:1.25 || exit 1; done"""),
+  C("deployment is paused with template image nginx:1.27 and RELEASE=next", 2, r"""d="$K -n %{ns} get deploy %{dep}"; eq "$($d -o jsonpath='{.spec.paused}' 2>/dev/null)" true && eq "$($d -o jsonpath='{.spec.template.spec.containers[0].image}')" nginx:1.27 && eq "$($d -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="RELEASE")].value}')" next"""),
+  C("no ReplicaSet exists for the paused template (1.27 + RELEASE)", 2, r"""! $K -n %{ns} get rs -o json 2>/dev/null | python3 -c '
+import sys,json
+for rs in json.load(sys.stdin)["items"]:
+    c=rs["spec"]["template"]["spec"]["containers"][0]
+    if c["image"]=="nginx:1.27" and any(e.get("name")=="RELEASE" for e in c.get("env",[])): sys.exit(0)
+sys.exit(1)'"""),
+ ],
+},
+{
+ "id": 109, "domain": "WORK", "difficulty": "hard", "title": "Native sidecar with probes and an exact report",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "side"), "pod": name_pick(r, "collector")},
+ "task": CTX + """
+Create pod `%{pod}` in `%{ns}`:
+- a **native sidecar** (an init container with `restartPolicy: Always`) named `agent` (busybox:1.36)
+  that appends `$(date)` to `/var/log/agent.log` every second,
+- main container `app` (nginx:1.27) with a readiness probe (HTTP GET / on 80, period 5s)
+  and a liveness probe (TCP 80),
+- shared `emptyDir` `logs` mounted at `/var/log` in both,
+- `terminationGracePeriodSeconds: 5`.
+When it is Running, write the names of **all** containers of the pod (init and regular), sorted,
+one per line, to `{WS}/q109/containers.txt`, and the pod's `.status.podIP` to `{WS}/q109/ip.txt`.
+""",
+ "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q109"; mkdir -p "$WS/q109"; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""",
+ "checks": [
+  C("native sidecar: initContainer agent with restartPolicy Always", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.spec.initContainers[?(@.name=="agent")].restartPolicy}')" Always"""),
+  C("probes on app as specified", 2, r"""c='{.spec.containers[?(@.name=="app")]'; eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.httpGet.path}")" / && eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.periodSeconds}")" 5 && eq "$(j -n %{ns} pod %{pod} "$c.livenessProbe.tcpSocket.port}")" 80"""),
+  C("Running, shared log written by agent, grace period 5", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.terminationGracePeriodSeconds}')" 5 && [ -n "$($K -n %{ns} exec %{pod} -c app -- sh -c 'tail -1 /var/log/agent.log' 2>/dev/null)" ]"""),
+  C("report files exact", 2, r"""want=$( (j -n %{ns} pod %{pod} '{.spec.initContainers[*].name}'; echo; j -n %{ns} pod %{pod} '{.spec.containers[*].name}') | tr ' ' '\n' | grep . | sort); [ "$(grep . "$WS/q109/containers.txt" 2>/dev/null | sort)" = "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q109/ip.txt" 2>/dev/null)" "$(j -n %{ns} pod %{pod} '{.status.podIP}')" """),
+ ],
+},
+{
+ "id": 110, "domain": "WORK", "difficulty": "hard", "title": "Cluster audit to exact files",
+ "params": lambda r: {"ctx": "k8s-c1-H"},
+ "task": CTX + """
+Produce three files, contents **exactly** as specified (no headers, sorted with `sort`, one per line):
+1. `{WS}/q110/no-readiness.txt` — `<namespace>/<pod>` for every pod in the cluster whose first
+   container has **no** readiness probe. Exclude namespace `kube-system`.
+2. `{WS}/q110/per-node.txt` — `<node> <count>` of Running pods per node, all namespaces.
+3. `{WS}/q110/images.txt` — every distinct container image running in namespace `kube-system`.
+""",
+ "setup": r"""rm -rf "$WS/q110"; mkdir -p "$WS/q110" """,
+ "checks": [
+  C("no-readiness.txt exact", 3, r"""want=$($K get pods -A -o json 2>/dev/null | python3 -c '
+import sys,json
+for p in json.load(sys.stdin)["items"]:
+    if p["metadata"]["namespace"]=="kube-system": continue
+    if not p["spec"]["containers"][0].get("readinessProbe"): print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort); [ "$(grep . "$WS/q110/no-readiness.txt" 2>/dev/null | sort)" = "$want" ]"""),
+  C("per-node.txt exact", 2, r"""want=$($K get pods -A -o json 2>/dev/null | python3 -c '
+import sys,json,collections
+c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"] if p["status"]["phase"]=="Running" and p["spec"].get("nodeName"))
+for n in sorted(c): print(n, c[n])'); [ "$(grep . "$WS/q110/per-node.txt" 2>/dev/null | sort)" = "$(echo "$want" | sort)" ]"""),
+  C("images.txt exact", 2, r"""want=$(j -n kube-system pods '{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u); [ "$(grep . "$WS/q110/images.txt" 2>/dev/null | sort -u)" = "$want" ]"""),
+ ],
+},
+{
+ "id": 121, "domain": "WORK", "difficulty": "hard", "title": "The context's namespace",
+ "params": lambda r: {"ctx": "k8s-c2-AC", "ns": ns_pick(r, "ac"), "dep": name_pick(r, "ledger")},
+ "task": CTX + """
+In the **current namespace of that context** (do not pass `-n`), create Deployment `%{dep}`
+(busybox:1.36, `sleep 3600`, 2 replicas, label `owner=ac`) and a ClusterIP Service `%{dep}`
+on port 9000 targeting port 9000. Then write the namespace name you deployed into, as reported
+by `kubectl config view --minify`, to `{WS}/q121/ns.txt`.
+""",
+ "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q121"; mkdir -p "$WS/q121"; $K -n %{ns} delete deploy,svc %{dep} --ignore-not-found >/dev/null; $K -n default delete deploy,svc %{dep} --ignore-not-found >/dev/null""",
+ "checks": [
+  C("Deployment %{dep} in %{ns} (the context's namespace), 2 available, owner=ac", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && eq "$(j -n %{ns} deploy %{dep} '{.spec.template.metadata.labels.owner}')" ac"""),
+  C("Service %{dep} port 9000 in the same namespace", 1, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].port}')" 9000"""),
+  C("nothing was created in namespace default by mistake", 1, r"""! $K -n default get deploy %{dep} >/dev/null 2>&1"""),
+  C("ns.txt has the namespace", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q121/ns.txt" 2>/dev/null)" %{ns}"""),
+ ],
+},
+# ================================================================= NET ====
+{
+ "id": 111, "domain": "NET", "difficulty": "hard", "title": "Zero-trust namespace",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "zt"), "from": ns_pick(r, "front")},
+ "task": CTX + """
+Namespace `%{ns}` runs `api` (label `app=api`, port 8080) and `db` (label `app=db`, port 5432).
+Implement zero trust with NetworkPolicies:
+1. `default-deny`: deny all ingress **and** egress for every pod in `%{ns}`.
+2. `allow-dns`: every pod may egress to DNS (UDP/TCP 53) in `kube-system` only.
+3. `api-ingress`: `api` accepts ingress on 8080 only from pods in namespaces labelled `team=%{from}`.
+4. `api-to-db`: `api` may egress to `db` on 5432; `db` accepts ingress on 5432 from `api` only.
+Remove anything that contradicts this. Label namespace `%{from}` appropriately.
+""",
+ "setup": NS + r"""
+mkns %{ns}; mkns %{from}
+$K -n %{ns} create deploy api --image=nginx:1.25 --port=8080 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n %{ns} create deploy db --image=nginx:1.25 --port=5432 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n %{ns} delete netpol --all >/dev/null 2>&1
+# trap: an allow-all left behind
+$K apply -f - >/dev/null <<EOF
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: temp-allow-all, namespace: %{ns}}
+spec: {podSelector: {}, policyTypes: [Ingress, Egress], ingress: [{}], egress: [{}]}
+EOF
+$K label ns %{from} team- >/dev/null 2>&1 || true
+""",
+ "checks": [
+  C("default-deny covers Ingress and Egress for all pods; allow-all removed", 2, r"""n="$K -n %{ns} get netpol default-deny"; eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' && $n -o jsonpath='{.spec.policyTypes}' | grep -q Ingress && $n -o jsonpath='{.spec.policyTypes}' | grep -q Egress && [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ] && ! $K -n %{ns} get netpol temp-allow-all >/dev/null 2>&1"""),
+  C("allow-dns: egress 53 UDP+TCP to kube-system only", 2, r"""$K -n %{ns} get netpol allow-dns -o json 2>/dev/null | python3 -c '
+import sys,json; s=json.load(sys.stdin)["spec"]; e=s.get("egress",[])
+ok=s.get("podSelector")=={} and any({p.get("protocol") for p in r.get("ports",[]) if str(p.get("port"))=="53"}>={"UDP","TCP"} and all("kube-system" in json.dumps(t.get("namespaceSelector",{})) for t in r.get("to",[])) and r.get("to") for r in e)
+sys.exit(0 if ok else 1)'"""),
+  C("api-ingress: from namespaces team=%{from} on 8080 only; namespace labelled", 2, r"""eq "$(j ns %{from} '{.metadata.labels.team}')" %{from} && $K -n %{ns} get netpol api-ingress -o json 2>/dev/null | python3 -c '
+import sys,json; s=json.load(sys.stdin)["spec"]
+ok=s["podSelector"].get("matchLabels",{}).get("app")=="api" and len(s.get("ingress",[]))==1 and all(str(p.get("port"))=="8080" for p in s["ingress"][0].get("ports",[])) and s["ingress"][0].get("ports") and all(t.get("namespaceSelector",{}).get("matchLabels",{}).get("team")=="%{from}" for t in s["ingress"][0].get("from",[])) and s["ingress"][0].get("from")
+sys.exit(0 if ok else 1)'"""),
+  C("api-to-db: api egress to db:5432 and db ingress from api:5432", 2, r"""$K -n %{ns} get netpol -o json 2>/dev/null | python3 -c '
+import sys,json; items=json.load(sys.stdin)["items"]; eg=ing=False
+for n in items:
+    s=n["spec"]; app=s["podSelector"].get("matchLabels",{}).get("app")
+    for r in s.get("egress",[]):
+        if app=="api" and any(t.get("podSelector",{}).get("matchLabels",{}).get("app")=="db" for t in r.get("to",[])) and any(str(p.get("port"))=="5432" for p in r.get("ports",[])): eg=True
+    for r in s.get("ingress",[]):
+        if app=="db" and any(t.get("podSelector",{}).get("matchLabels",{}).get("app")=="api" for t in r.get("from",[])) and any(str(p.get("port"))=="5432" for p in r.get("ports",[])): ing=True
+sys.exit(0 if eg and ing else 1)'"""),
+ ],
+},
+{
+ "id": 112, "domain": "NET", "difficulty": "hard", "title": "TLS Ingress with a generated certificate",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "tls"), "host": r.choice(["secure.lab.local", "portal.lab.local", "admin.lab.local"])},
+ "task": CTX + """
+Namespace `%{ns}` has Deployment+Service `web` (port 80).
+1. Generate a self-signed certificate (RSA 2048, 365 days) with CN `%{host}` and store it as a
+   Secret `%{host}-tls` of type `kubernetes.io/tls`.
+2. Create Ingress `web` (class `nginx`) serving host `%{host}` with that TLS secret, path `/` Prefix → `web:80`,
+   and the annotation `nginx.ingress.kubernetes.io/ssl-redirect: "true"`.
+3. Write the certificate's SHA-256 fingerprint (`openssl x509 -fingerprint -sha256`, the value only) to `{WS}/q112/fp.txt`.
+""",
+ "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q112"; mkdir -p "$WS/q112"; $K -n %{ns} create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} delete secret %{host}-tls --ignore-not-found >/dev/null; $K -n %{ns} delete ingress web --ignore-not-found >/dev/null""",
+ "checks": [
+  C("Secret %{host}-tls is kubernetes.io/tls with CN %{host}", 2, r"""eq "$(j -n %{ns} secret %{host}-tls '{.type}')" kubernetes.io/tls && j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'CN *= *%{host}'"""),
+  C("Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", 3, r"""i="$K -n %{ns} get ingress web"; eq "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" nginx && eq "$($i -o jsonpath='{.spec.tls[0].secretName}')" %{host}-tls && eq "$($i -o jsonpath='{.spec.tls[0].hosts[0]}')" %{host} && eq "$($i -o jsonpath='{.spec.rules[0].host}')" %{host} && eq "$($i -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}')" web && eq "$($i -o jsonpath='{.metadata.annotations.nginx\.ingress\.kubernetes\.io/ssl-redirect}')" true"""),
+  C("fp.txt matches the secret's certificate fingerprint", 2, r"""fp=$(j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//'); [ -n "$fp" ] && grep -qiF "$fp" "$WS/q112/fp.txt" 2>/dev/null"""),
+ ],
+},
+{
+ "id": 113, "domain": "NET", "difficulty": "hard", "title": "Gateway API canary with header match",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "canary"), "host": r.choice(["shop", "pay", "cart"]), "w": r.choice([10, 20, 25])},
+ "task": CTX + """
+Gateway API CRDs are installed. Namespace `%{ns}` has Services `v1` and `v2` (port 80).
+Create in `%{ns}`:
+- Gateway `%{host}-gw` (class `%{host}-gc`, create the GatewayClass, controllerName `example.com/gw`)
+  with two HTTP listeners: `web` for `%{host}.example.com` and `api` for `api.%{host}.example.com`, both port 80.
+- HTTPRoute `%{host}-route` attached to listener `web` only (`sectionName`), with:
+  - a rule matching header `X-Canary: always` → `v2` (100%),
+  - a default rule splitting `v1` %{w2}% / `v2` %{w}% by weight.
+""",
+ "params_post": None,
+ "setup": NS + r"""
+mkns %{ns}
+$K get crd gateways.gateway.networking.k8s.io >/dev/null 2>&1 || { curl -sfL --max-time 60 https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml -o /tmp/gwapi.yaml && $K apply -f /tmp/gwapi.yaml >/dev/null 2>&1; } || echo "warn: could not install Gateway API CRDs"
+for v in v1 v2; do $K -n %{ns} create deploy $v --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} expose deploy $v --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; done
+$K delete gatewayclass %{host}-gc --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete gateway %{host}-gw --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete httproute %{host}-route --ignore-not-found >/dev/null 2>&1
+""",
+ "checks": [
+  C("Gateway with listeners web and api on the right hostnames", 2, r"""g="$K -n %{ns} get gateway %{host}-gw"; eq "$($g -o jsonpath='{.spec.gatewayClassName}' 2>/dev/null)" %{host}-gc && eq "$($g -o jsonpath='{.spec.listeners[?(@.name=="web")].hostname}')" %{host}.example.com && eq "$($g -o jsonpath='{.spec.listeners[?(@.name=="api")].hostname}')" api.%{host}.example.com"""),
+  C("route attaches to listener web only (sectionName)", 1, r"""eq "$(j -n %{ns} httproute %{host}-route '{.spec.parentRefs[0].sectionName}')" web && [ -z "$(j -n %{ns} httproute %{host}-route '{.spec.parentRefs[1]}')" ]"""),
+  C("header rule X-Canary: always -> v2", 2, r"""$K -n %{ns} get httproute %{host}-route -o json 2>/dev/null | python3 -c '
+import sys,json; r=json.load(sys.stdin)["spec"]["rules"]
+ok=any(any(any(h.get("name","").lower()=="x-canary" and h.get("value")=="always" for h in m.get("headers",[])) for m in x.get("matches",[])) and x["backendRefs"][0]["name"]=="v2" and len(x["backendRefs"])==1 for x in r)
+sys.exit(0 if ok else 1)'"""),
+  C("default rule weights v1 %{w2} / v2 %{w}", 2, r"""$K -n %{ns} get httproute %{host}-route -o json 2>/dev/null | python3 -c '
+import sys,json; r=json.load(sys.stdin)["spec"]["rules"]
+ok=any(not x.get("matches") or all(not m.get("headers") for m in x["matches"]) and {b["name"]:b.get("weight") for b in x["backendRefs"]}=={"v1":%{w2},"v2":%{w}} for x in r if len(x.get("backendRefs",[]))==2)
+sys.exit(0 if ok else 1)'"""),
+ ],
+},
+{
+ "id": 114, "domain": "NET", "breaks": "coredns", "difficulty": "hard", "title": "DNS down: three faults", "disruptive": True,
+ "params": lambda r: {"ctx": "k8s-c1-H"},
+ "task": CTX + """
+Nothing in the cluster resolves DNS. Restore it. There is more than one cause.
+When done, `nslookup kubernetes.default` and `nslookup kubernetes.io` must succeed from a pod
+in `default`, CoreDNS must have 2 available replicas, and no NetworkPolicy may remain in `kube-system`.
+""",
+ "setup": r"""
+$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > /tmp/Corefile.q114
+sed -i 's|forward \. /etc/resolv.conf|forward . 10.255.255.254|' /tmp/Corefile.q114
+$K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.q114 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n kube-system scale deploy coredns --replicas=0 >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: maintenance-lockdown, namespace: kube-system}
+spec: {podSelector: {matchLabels: {k8s-app: kube-dns}}, policyTypes: [Ingress], ingress: []}
+EOF
+""",
+ "checks": [
+  C("Corefile forwards to /etc/resolv.conf", 2, r"""j -n kube-system cm coredns '{.data.Corefile}' | grep -q 'forward \. /etc/resolv.conf'"""),
+  C("CoreDNS has 2 available replicas", 2, r"""for i in $(seq 1 12); do [ "$(j -n kube-system deploy coredns '{.status.availableReplicas}')" = 2 ] && exit 0; sleep 5; done; exit 1"""),
+  C("no NetworkPolicy in kube-system", 1, r"""[ "$($K -n kube-system get netpol --no-headers 2>/dev/null | wc -l)" = 0 ]"""),
+  C("resolution works from a pod", 3, r"""
+$K -n default delete pod dns-probe --ignore-not-found --wait=true >/dev/null 2>&1
+$K -n default run dns-probe --image=busybox:1.36 --restart=Never -- sh -c 'for i in 1 2 3 4 5 6; do nslookup kubernetes.default >/dev/null 2>&1 && nslookup kubernetes.io >/dev/null 2>&1 && exit 0; sleep 5; done; exit 1' >/dev/null 2>&1
+for i in $(seq 1 20); do ph=$(j -n default pod dns-probe '{.status.phase}'); [ "$ph" = Succeeded ] && { $K -n default delete pod dns-probe --wait=false >/dev/null; exit 0; }; [ "$ph" = Failed ] && break; sleep 3; done
+$K -n default delete pod dns-probe --wait=false >/dev/null; exit 1"""),
+ ],
+},
+# ================================================================ STOR ====
+{
+ "id": 115, "domain": "STOR", "difficulty": "hard", "title": "Bind the right local volume",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "lv"), "pvc": name_pick(r, "fast-data")},
+ "task": CTX + """
+Two PersistentVolumes of class `local-fast` exist. Only one of them lives on the node labelled
+`disk=nvme`. Create PVC `%{pvc}` in `%{ns}` (RWO, 1Gi, class `local-fast`) that binds to **that**
+PV — use a label selector, do not edit the PVs — and pod `consumer` (busybox:1.36, `sleep 3600`)
+mounting it at `/data`. The pod must run on the nvme node. Do not delete either PV.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+for n in "$W1" "$W2" "$W3"; do $K label node $n disk- >/dev/null 2>&1 || true; done
+$K label node "$W2" disk=nvme --overwrite >/dev/null
+$K -n %{ns} delete pod consumer --ignore-not-found --wait=false >/dev/null; $K -n %{ns} delete pvc %{pvc} --ignore-not-found >/dev/null
+$K delete pv lv-hdd lv-nvme --ignore-not-found >/dev/null
+for pair in "lv-hdd:$W1:hdd:2Gi" "lv-nvme:$W2:nvme:1Gi"; do IFS=: read name node kind size <<<"$pair"
+nx "$node" "mkdir -p /mnt/$name"
+$K apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: PersistentVolume
+metadata: {name: $name, labels: {media: $kind}}
+spec:
+  capacity: {storage: $size}
+  accessModes: [ReadWriteOnce]
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: local-fast
+  local: {path: /mnt/$name}
+  nodeAffinity: {required: {nodeSelectorTerms: [{matchExpressions: [{key: kubernetes.io/hostname, operator: In, values: [$node]}]}]}}
+EOF
+done
+""",
+ "checks": [
+  C("PVC uses a selector and is Bound to lv-nvme", 3, r"""[ -n "$(j -n %{ns} pvc %{pvc} '{.spec.selector}')" ] && eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}')" lv-nvme && eq "$(j -n %{ns} pvc %{pvc} '{.status.phase}')" Bound"""),
+  C("consumer Running on the nvme node with /data mounted", 2, r"""eq "$(j -n %{ns} pod consumer '{.status.phase}')" Running && eq "$(j -n %{ns} pod consumer '{.spec.nodeName}')" "$W2" && j -n %{ns} pod consumer '{.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /data"""),
+  C("both PVs still exist, unmodified affinity", 1, r"""eq "$(j pv lv-hdd '{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}')" "$W1" && eq "$(j pv lv-nvme '{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}')" "$W2" """),
+ ],
+},
+{
+ "id": 116, "domain": "STOR", "difficulty": "hard", "title": "Migrate data between claims",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "mig"), "n": r.choice([7, 11, 13])},
+ "task": CTX + """
+In `%{ns}`, PVC `old-data` holds files a running pod `legacy` wrote (`/data`). Storage is being
+replaced:
+1. Create PVC `new-data` (default StorageClass, 1Gi, RWO).
+2. Copy **everything** from `old-data` to `new-data` using a Job named `migrate` (busybox:1.36).
+   The Job must complete; use whatever mounts you need.
+3. Re-point pod `legacy` to `new-data` (same name, same namespace, mount at `/data`) and get it Running.
+4. Delete PVC `old-data` afterwards. Its PV must be released, not deleted (check its reclaim policy first).
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete pod legacy --ignore-not-found --wait=true >/dev/null; $K -n %{ns} delete job migrate --ignore-not-found >/dev/null; $K -n %{ns} delete pvc old-data new-data --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: old-data, namespace: %{ns}}
+spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 1Gi}}}
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: legacy, namespace: %{ns}}
+spec:
+  containers: [{name: app, image: busybox:1.36, command: [sh, -c, "for i in \$(seq 1 %{n}); do echo record-\$i > /data/file-\$i.txt; done; sleep 3600"], volumeMounts: [{name: d, mountPath: /data}]}]
+  volumes: [{name: d, persistentVolumeClaim: {claimName: old-data}}]
+EOF
+for i in $(seq 1 30); do [ "$(j -n %{ns} pod legacy '{.status.phase}')" = Running ] && break; sleep 2; done
+pv=$(j -n %{ns} pvc old-data '{.spec.volumeName}'); [ -n "$pv" ] && $K patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' >/dev/null
+echo "$pv" > "$WS/.q116-pv"
+""",
+ "checks": [
+  C("Job migrate completed", 2, r"""eq "$(j -n %{ns} job migrate '{.status.succeeded}')" 1"""),
+  C("legacy Running on new-data with all %{n} files intact", 3, r"""eq "$(j -n %{ns} pod legacy '{.status.phase}')" Running && eq "$(j -n %{ns} pod legacy '{.spec.volumes[?(@.name!="kube-api-access")].persistentVolumeClaim.claimName}' | tr ' ' '\n' | grep -v '^$' | head -1)" new-data && eq "$($K -n %{ns} exec legacy -- sh -c 'ls /data/file-*.txt | wc -l' 2>/dev/null | tr -d ' ')" %{n} && eq "$($K -n %{ns} exec legacy -- cat /data/file-%{n}.txt 2>/dev/null)" record-%{n}"""),
+  C("old-data deleted; its PV Released (not deleted)", 2, r"""! $K -n %{ns} get pvc old-data >/dev/null 2>&1 && pv=$(cat "$WS/.q116-pv" 2>/dev/null) && eq "$(j pv $pv '{.status.phase}')" Released"""),
+ ],
+},
+# ============================================================== TSHOOT ====
+{
+ "id": 117, "domain": "TSHOOT", "breaks": "w2-kubelet", "difficulty": "hard", "title": "Worker node: three things wrong", "disruptive": True,
+ "params": lambda r: {"ctx": "k8s-c1-H"},
+ "task": CTX + """
+Node `{W2}` is unusable. Make it fully usable again: `Ready`, schedulable, kubelet running and
+enabled, and a new pod pinned to it must reach Running. Nothing else on the cluster may be changed.
+""",
+ "setup": r"""
+$K cordon "$W2" >/dev/null 2>&1 || true
+nx "$W2" '
+sed -i "s|clientCAFile: /etc/kubernetes/pki/ca.crt|clientCAFile: /etc/kubernetes/pki/ca-wrong.crt|" /var/lib/kubelet/config.yaml
+systemctl disable kubelet >/dev/null 2>&1; systemctl stop kubelet'
+""",
+ "checks": [
+  C("node Ready", 3, r"""for i in $(seq 1 12); do [ "$(j node "$W2" '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 5; done; exit 1"""),
+  C("node uncordoned (collateral check)", 2, r"""[ "$(j node "$W2" '{.spec.unschedulable}')" != true ]"""),
+  C("kubelet active and enabled, config points at a real CA", 2, r"""eq "$(nx "$W2" 'systemctl is-active kubelet')" active && eq "$(nx "$W2" 'systemctl is-enabled kubelet')" enabled && f=$(nx "$W2" 'grep clientCAFile /var/lib/kubelet/config.yaml' | awk '{print $2}') && nx "$W2" "test -f $f" """),
+  C("a pod pinned to the node runs", 2, r"""
+$K -n default delete pod node-probe --ignore-not-found --wait=true >/dev/null 2>&1
+$K -n default run node-probe --image=busybox:1.36 --restart=Never --overrides="{\"spec\":{\"nodeName\":\"$W2\"}}" -- sleep 120 >/dev/null 2>&1
+ok=1; for i in $(seq 1 20); do [ "$(j -n default pod node-probe '{.status.phase}')" = Running ] && { ok=0; break; }; sleep 3; done
+$K -n default delete pod node-probe --wait=false >/dev/null 2>&1; exit $ok"""),
+ ],
+},
+{
+ "id": 118, "domain": "TSHOOT", "breaks": "scheduler", "difficulty": "hard", "title": "Two control-plane components down", "disruptive": True,
+ "params": lambda r: {"ctx": "k8s-c1-H"},
+ "task": CTX + """
+New Deployments create no pods, and pods that do get created stay `Pending`. Two different
+control-plane components are failing for two different reasons. Repair both on the control-plane node.
+Afterwards a new Deployment must reach Available within a minute.
+""",
+ "setup": r"""
+nx "$CP" '
+f=/etc/kubernetes/manifests/kube-scheduler.yaml; grep -q "^ *- kube-scheduler$" $f && sed -i "s|^\( *\)- kube-scheduler$|\1- kube-schedulerr|" $f
+g=/etc/kubernetes/manifests/kube-controller-manager.yaml; grep -q "image: registry.k8s.io/kube-controller-manager:v" $g && sed -i "s|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)|\1-broken|" $g
+'
+""",
+ "checks": [
+  C("kube-scheduler manifest repaired and Running", 3, r"""nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml' && eq "$(j -n kube-system pod -l component=kube-scheduler '{.items[0].status.phase}')" Running"""),
+  C("kube-controller-manager image repaired and Running", 3, r"""! nx "$CP" 'grep -q "\-broken" /etc/kubernetes/manifests/kube-controller-manager.yaml' && eq "$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].status.phase}')" Running"""),
+  C("a new Deployment becomes Available", 3, r"""
+$K -n default delete deploy cp-probe --ignore-not-found --wait=true >/dev/null 2>&1
+$K -n default create deploy cp-probe --image=busybox:1.36 -- sleep 300 >/dev/null 2>&1
+ok=1; for i in $(seq 1 20); do [ "$(j -n default deploy cp-probe '{.status.availableReplicas}')" = 1 ] && { ok=0; break; }; sleep 3; done
+$K -n default delete deploy cp-probe --wait=false >/dev/null 2>&1; exit $ok"""),
+ ],
+},
+{
+ "id": 119, "domain": "TSHOOT", "difficulty": "hard", "title": "Application down for four reasons",
+ "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "checkout"), "dep": name_pick(r, "checkout")},
+ "task": CTX + """
+Deployment `%{dep}` in `%{ns}` should have 3 Available replicas and Service `%{dep}` should route
+to them on port 80. Right now nothing works. Fix it **in place** — the Deployment must keep its
+UID (do not delete and recreate it) and the Service must keep its ClusterIP.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete deploy,svc,quota,secret --all >/dev/null 2>&1
+$K apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: ResourceQuota
+metadata: {name: tight, namespace: %{ns}}
+spec: {hard: {pods: "1"}}
+---
+apiVersion: v1
+kind: Secret
+metadata: {name: %{dep}-env, namespace: %{ns}}
+stringData: {API_KEY: abc123}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: %{dep}, namespace: %{ns}}
+spec:
+  replicas: 3
+  selector: {matchLabels: {app: %{dep}}}
+  template:
+    metadata: {labels: {app: %{dep}}}
+    spec:
+      containers:
+        - name: web
+          image: nginx:1.27-nope
+          ports: [{containerPort: 80}]
+          env: [{name: DB_URL, valueFrom: {secretKeyRef: {name: %{dep}-env, key: DATABASE_URL}}}]
+          resources: {requests: {cpu: 10m, memory: 16Mi}}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: %{dep}, namespace: %{ns}}
+spec:
+  selector: {app: %{dep}-web}
+  ports: [{port: 80, targetPort: 8080}]
+EOF
+j -n %{ns} deploy %{dep} '{.metadata.uid}' > "$WS/.q119-uid"; j -n %{ns} svc %{dep} '{.spec.clusterIP}' > "$WS/.q119-ip"
+""",
+ "checks": [
+  C("3 replicas Available", 3, r"""for i in $(seq 1 6); do [ "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" = 3 ] && exit 0; sleep 5; done; exit 1"""),
+  C("Service routes: endpoints ready, targetPort 80", 2, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].targetPort}')" 80 && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""),
+  C("Deployment UID and Service ClusterIP unchanged (fixed in place)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.metadata.uid}')" "$(cat "$WS/.q119-uid")" && eq "$(j -n %{ns} svc %{dep} '{.spec.clusterIP}')" "$(cat "$WS/.q119-ip")" """),
+  C("quota still present and permits the 3 pods", 1, r"""p=$(j -n %{ns} quota tight '{.spec.hard.pods}'); [ -n "$p" ] && [ "$p" -ge 3 ]"""),
+ ],
+},
+{
+ "id": 120, "domain": "TSHOOT", "difficulty": "hard", "title": "Cluster facts, exactly",
+ "params": lambda r: {"ctx": "k8s-c1-H", "node": r.choice(["{W1}", "{W2}", "{W3}"])},
+ "task": CTX + """
+Write `{WS}/q120/facts.txt` with these lines, values taken from the running cluster:
+```
+pod-cidr=<cluster pod CIDR as configured on kube-controller-manager>
+service-cidr=<service CIDR as configured on kube-apiserver>
+dns-ip=<ClusterIP of the kube-dns Service>
+proxy-mode=<kube-proxy mode from its ConfigMap (empty means iptables — write iptables)>
+%{node}-pod-cidr=<.spec.podCIDR of that node>
+```
+""",
+ "setup": r"""rm -rf "$WS/q120"; mkdir -p "$WS/q120" """,
+ "checks": [
+  C("pod-cidr and service-cidr", 2, r"""f="$WS/q120/facts.txt"; pc=$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'cluster-cidr=[^"]*' | cut -d= -f2); sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2); grep -qx "pod-cidr=$pc" "$f" 2>/dev/null && grep -qx "service-cidr=$sc" "$f" """),
+  C("dns-ip and proxy-mode", 2, r"""f="$WS/q120/facts.txt"; d=$(j -n kube-system svc kube-dns '{.spec.clusterIP}'); m=$(j -n kube-system cm kube-proxy '{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"'); [ -n "$m" ] || m=iptables; grep -qx "dns-ip=$d" "$f" 2>/dev/null && grep -qx "proxy-mode=$m" "$f" """),
+  C("%{node} pod CIDR", 1, r"""grep -qx "%{node}-pod-cidr=$(j node %{node} '{.spec.podCIDR}')" "$WS/q120/facts.txt" 2>/dev/null"""),
+ ],
+},
+]
+
+# derived params: w2 = 100 - w for the canary split
+for _q in QUESTIONS:
+    if _q["id"] == 113:
+        _orig = _q["params"]
+        _q["params"] = lambda r, _o=_orig: (lambda p: {**p, "w2": 100 - p["w"]})(_o(r))

tracks/cka/bank_sim_b.py

@@ -0,0 +1,973 @@
+"""Simulator Set B - 17 tasks, fixed set, killer.sh-grade difficulty.
+
+Original scenarios. Weighted toward node-level and control-plane work: etcd flags, kube-proxy
+rules, ServiceCIDR, bound tokens, PDB-blocked drains, PV reclaim, upgrade planning, CSR signing.
+
+Run with:  ./exam sim b
+"""
+from tracks.cka.common import C, NS
+
+Q = []
+
+
+def task(**kw):
+    kw.setdefault("difficulty", "hard")
+    kw["set"] = "b"
+    Q.append(kw)
+
+
+# ------------------------------------------------------------------ 301 ----
+task(
+    id=301, domain="ARCH", title="etcd configuration facts",
+    task="""
+The security team wants the following about the etcd running on the control-plane node.
+Write `{WS}/q301/etcd-info.txt` with exactly these five lines (values after the colon):
+
+```
+data-dir: <etcd data directory>
+server-key: <path to the server private key>
+trusted-ca: <path to the CA used for client certificate verification>
+client-cert-auth: <yes|no>
+server-cert-expiry: <notAfter value of the etcd server certificate>
+```
+
+Take every value from how etcd is actually started, not from memory.
+""",
+    setup=r"""rm -rf "$WS/q301"; mkdir -p "$WS/q301" """,
+    checks=[
+        C("data-dir and server-key are correct", 3, r"""
+f="$WS/q301/etcd-info.txt"; [ -f "$f" ] || exit 1
+cmd=$(j -n kube-system pod -l component=etcd '{.items[0].spec.containers[0].command}')
+dd=$(echo "$cmd" | tr ',' '\n' | grep -o 'data-dir=[^"]*' | cut -d= -f2)
+sk=$(echo "$cmd" | tr ',' '\n' | grep -o '\-\-key-file=[^"]*' | cut -d= -f2)
+grep -q "data-dir: *$dd" "$f" && grep -q "server-key: *$sk" "$f" """),
+        C("trusted-ca and client-cert-auth are correct", 3, r"""
+f="$WS/q301/etcd-info.txt"
+cmd=$(j -n kube-system pod -l component=etcd '{.items[0].spec.containers[0].command}')
+ca=$(echo "$cmd" | tr ',' '\n' | grep -o '\-\-trusted-ca-file=[^"]*' | cut -d= -f2)
+grep -q "trusted-ca: *$ca" "$f" && grep -Eqi 'client-cert-auth: *(yes|true)' "$f" """),
+        C("server-cert-expiry matches the certificate on disk", 3, r"""
+exp=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -enddate' | sed 's/notAfter=//')
+yr=$(echo "$exp" | awk '{print $4}'); mon=$(echo "$exp" | awk '{print $1}'); day=$(echo "$exp" | awk '{print $2}')
+f="$WS/q301/etcd-info.txt"; grep -q "$yr" "$f" 2>/dev/null && grep -qi "$mon" "$f" && grep -q "$day" "$f" """),
+    ],
+    solution="""
+{SHELL_CP}
+cat /etc/kubernetes/manifests/etcd.yaml        # every flag etcd runs with
+#   --data-dir=/var/lib/etcd
+#   --key-file=/etc/kubernetes/pki/etcd/server.key
+#   --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
+#   --client-cert-auth=true
+openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -enddate
+# etcd runs as a static pod, so the manifest in /etc/kubernetes/manifests is the source of truth.
+# k -n kube-system get pod -l component=etcd -o yaml would show the same command.
+""",
+)
+
+# ------------------------------------------------------------------ 302 ----
+task(
+    id=302, domain="NET", title="Service rules on the node", breaks="proxy-rules",
+    task="""
+Confirm that kube-proxy is programming this cluster.
+
+1. In Namespace `project-proxy` create Pod `proxy-pod` (image `nginx:1-alpine`) and expose it
+   internally as Service `proxy-svc` on port **4000** targeting container port 80.
+2. From the node where the Pod runs, capture the proxy rules that mention `proxy-svc` into
+   `{WS}/q302/rules.txt` (the node uses iptables; `iptables-save` is available as root).
+3. Write the Service's ClusterIP to `{WS}/q302/clusterip.txt`.
+4. Write the proxy mode kube-proxy is running in to `{WS}/q302/mode.txt`
+   (an empty `mode` in its ConfigMap means `iptables`).
+""",
+    setup=NS + r"""
+mkns project-proxy
+rm -rf "$WS/q302"; mkdir -p "$WS/q302"
+$K -n project-proxy delete pod proxy-pod --ignore-not-found --wait=false >/dev/null
+$K -n project-proxy delete svc proxy-svc --ignore-not-found >/dev/null
+""",
+    checks=[
+        C("Pod and Service exist, 4000 -> 80, with endpoints", 3, r"""
+eq "$(j -n project-proxy pod proxy-pod '{.status.phase}')" Running \
+ && eq "$(j -n project-proxy svc proxy-svc '{.spec.ports[0].port}')" 4000 && eq "$(j -n project-proxy svc proxy-svc '{.spec.ports[0].targetPort}')" 80 \
+ && [ -n "$(j -n project-proxy endpointslices -l kubernetes.io/service-name=proxy-svc '{.items[*].endpoints[*].addresses[*]}')" ]"""),
+        C("clusterip.txt matches the Service", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q302/clusterip.txt" 2>/dev/null)" "$(j -n project-proxy svc proxy-svc '{.spec.clusterIP}')" """),
+        C("rules.txt contains real proxy rules naming the Service and its ClusterIP", 4, r"""
+f="$WS/q302/rules.txt"; grep -q 'proxy-svc' "$f" 2>/dev/null && grep -q "$(j -n project-proxy svc proxy-svc '{.spec.clusterIP}')" "$f" && grep -Eq 'KUBE-(SVC|SEP|SERVICES)' "$f" """),
+        C("mode.txt says iptables", 2, r"""grep -qi 'iptables' "$WS/q302/mode.txt" 2>/dev/null"""),
+    ],
+    solution="""
+k -n project-proxy run proxy-pod --image=nginx:1-alpine
+k -n project-proxy expose pod proxy-pod --name proxy-svc --port 4000 --target-port 80
+k -n project-proxy get svc proxy-svc -o jsonpath='{.spec.clusterIP}' > {WS}/q302/clusterip.txt
+k -n project-proxy get pod proxy-pod -o wide          # note the node
+{SHELL_W1}                                            # the node it landed on
+iptables-save | grep proxy-svc                        # -> copy into {WS}/q302/rules.txt
+exit
+k -n kube-system get cm kube-proxy -o yaml | grep -E '^ *mode:'   # empty -> iptables
+echo iptables > {WS}/q302/mode.txt
+# Deleting the Service makes the rules disappear within a second - kube-proxy watches the API
+# and rewrites the rules on every Service/EndpointSlice change.
+""",
+)
+
+# ------------------------------------------------------------------ 303 ----
+task(
+    id=303, domain="NET", title="A second Service CIDR",
+    task="""
+The cluster is running out of Service IPs.
+
+1. Write the CIDR the cluster currently allocates Service IPs from to `{WS}/q303/current-cidr.txt`.
+2. Add a **second** range `10.180.0.0/16` to the cluster by creating a `ServiceCIDR` object named
+   `extra-range` (do not remove the existing one).
+3. In Namespace `default` create Deployment `cidr-demo` (image `nginx:1-alpine`) and a ClusterIP
+   Service `cidr-demo` on port 80 whose ClusterIP is taken **from the new range**
+   (request the address explicitly if you need to).
+4. Write that Service's ClusterIP to `{WS}/q303/new-ip.txt`.
+""",
+    setup=r"""
+rm -rf "$WS/q303"; mkdir -p "$WS/q303"
+$K delete svc cidr-demo --ignore-not-found >/dev/null 2>&1
+$K delete deploy cidr-demo --ignore-not-found >/dev/null 2>&1
+$K delete servicecidr extra-range --ignore-not-found >/dev/null 2>&1
+$K get servicecidr >/dev/null 2>&1 || echo "warn: this cluster has no ServiceCIDR API (needs MultiCIDRServiceAllocator)"
+""",
+    checks=[
+        C("current-cidr.txt matches the apiserver's service range", 2, r"""
+sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2)
+grep -qF "$sc" "$WS/q303/current-cidr.txt" 2>/dev/null"""),
+        C("ServiceCIDR extra-range covers 10.180.0.0/16 and is Ready", 4, r"""
+j servicecidr extra-range '{.spec.cidrs[*]}' | grep -q '10.180.0.0/16' || exit 1
+for i in $(seq 1 10); do [ "$(j servicecidr extra-range '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 2; done; exit 1"""),
+        C("Service cidr-demo has an IP from the new range and ready endpoints", 4, r"""
+ip=$(j -n default svc cidr-demo '{.spec.clusterIP}'); case "$ip" in 10.180.*) ;; *) exit 1;; esac
+[ -n "$(j -n default endpointslices -l kubernetes.io/service-name=cidr-demo '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""),
+        C("new-ip.txt records that ClusterIP", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q303/new-ip.txt" 2>/dev/null)" "$(j -n default svc cidr-demo '{.spec.clusterIP}')" """),
+    ],
+    solution="""
+k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
+#   -> e.g. 10.96.0.0/16 ; write it into {WS}/q303/current-cidr.txt
+k get servicecidr                        # the built-in "kubernetes" ServiceCIDR
+cat <<EOF | k apply -f -
+apiVersion: networking.k8s.io/v1
+kind: ServiceCIDR
+metadata: {name: extra-range}
+spec: {cidrs: ["10.180.0.0/16"]}
+EOF
+k get servicecidr extra-range -o yaml    # wait for Ready=True
+k create deploy cidr-demo --image=nginx:1-alpine
+k expose deploy cidr-demo --port=80 --cluster-ip=10.180.0.50
+k get svc cidr-demo -o jsonpath='{.spec.clusterIP}' > {WS}/q303/new-ip.txt
+# New ranges are additive: existing Services keep their IPs, and allocation from a new range
+# happens once the ServiceCIDR is Ready. Requesting --cluster-ip makes it deterministic.
+""",
+)
+
+# ------------------------------------------------------------------ 304 ----
+task(
+    id=304, domain="ARCH", title="Bound tokens versus legacy tokens",
+    task="""
+In Namespace `project-token` there is ServiceAccount `reporter`.
+
+1. Issue a **bound token** for it that is valid for exactly 2 hours and save it to
+   `{WS}/q304/token.jwt`.
+2. Decode the token's payload and write its `exp` and `iat` claims, as
+   `exp=<value>` and `iat=<value>` on two lines, to `{WS}/q304/claims.txt`.
+3. Also create a **long-lived** token for the same ServiceAccount, as a Secret named
+   `reporter-legacy` of the correct type, and confirm the controller populated it.
+""",
+    setup=NS + r"""
+mkns project-token
+rm -rf "$WS/q304"; mkdir -p "$WS/q304"
+$K -n project-token create sa reporter --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n project-token delete secret reporter-legacy --ignore-not-found >/dev/null
+""",
+    checks=[
+        C("token.jwt is a JWT issued for the reporter ServiceAccount", 3, r"""
+f="$WS/q304/token.jwt"; [ -s "$f" ] || exit 1
+python3 - "$f" <<'PY'
+import base64,json,sys
+t=open(sys.argv[1]).read().strip().split(".")
+if len(t)!=3: sys.exit(1)
+p=json.loads(base64.urlsafe_b64decode(t[1]+"=="*3))
+sub=p.get("sub",""); ko=p.get("kubernetes.io",{}).get("serviceaccount",{}).get("name","")
+sys.exit(0 if "reporter" in sub or ko=="reporter" else 1)
+PY"""),
+        C("the token's lifetime is 2 hours", 3, r"""
+python3 - "$WS/q304/token.jwt" <<'PY'
+import base64,json,sys
+p=json.loads(base64.urlsafe_b64decode(open(sys.argv[1]).read().strip().split(".")[1]+"=="*3))
+d=p["exp"]-p["iat"]
+sys.exit(0 if 7000 <= d <= 7400 else 1)
+PY"""),
+        C("claims.txt records exp and iat matching the token", 2, r"""
+python3 - "$WS/q304/token.jwt" "$WS/q304/claims.txt" <<'PY'
+import base64,json,sys,re
+p=json.loads(base64.urlsafe_b64decode(open(sys.argv[1]).read().strip().split(".")[1]+"=="*3))
+try: t=open(sys.argv[2]).read()
+except OSError: sys.exit(1)
+ok=re.search(r"exp\s*=\s*%d"%p["exp"],t) and re.search(r"iat\s*=\s*%d"%p["iat"],t)
+sys.exit(0 if ok else 1)
+PY"""),
+        C("Secret reporter-legacy is a populated service-account-token for reporter", 3, r"""
+eq "$(j -n project-token secret reporter-legacy '{.type}')" kubernetes.io/service-account-token \
+ && eq "$(j -n project-token secret reporter-legacy '{.metadata.annotations.kubernetes\.io/service-account\.name}')" reporter \
+ && [ -n "$(j -n project-token secret reporter-legacy '{.data.token}')" ]"""),
+    ],
+    solution="""
+k -n project-token create token reporter --duration=2h > {WS}/q304/token.jwt
+# decode the middle segment (JWTs are base64url, pad it out):
+cut -d. -f2 {WS}/q304/token.jwt | base64 -d 2>/dev/null | python3 -m json.tool
+#   -> copy exp= and iat= into {WS}/q304/claims.txt
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Secret
+metadata:
+  name: reporter-legacy
+  namespace: project-token
+  annotations: {kubernetes.io/service-account.name: reporter}
+type: kubernetes.io/service-account-token
+EOF
+k -n project-token get secret reporter-legacy -o yaml     # the controller fills in .data.token
+# Bound tokens expire and are tied to an audience; the Secret form never expires - avoid it
+# unless something really needs a static token.
+""",
+)
+
+# ------------------------------------------------------------------ 305 ----
+task(
+    id=305, domain="WORK", title="Schedule without the scheduler",
+    task="""
+In Namespace `project-orion`:
+
+1. Create Pod `direct-assign` (image `busybox:1.36`, command `sleep 86400`) that runs on node
+   `{W3}` **without the scheduler being involved at all**.
+2. Create Pod `scheduled` (same image and command) that lands on `{W3}` **through** the scheduler.
+3. Write to `{WS}/q305/proof.txt` the value of the field you used for Pod 1, in the form
+   `field=<name>`, and on a second line `scheduler=<the schedulerName of the second Pod>`.
+""",
+    setup=NS + r"""
+mkns project-orion
+rm -rf "$WS/q305"; mkdir -p "$WS/q305"
+$K -n project-orion delete pod direct-assign scheduled --ignore-not-found --wait=false >/dev/null
+$K label node "$W3" tier- >/dev/null 2>&1 || true
+""",
+    checks=[
+        C("direct-assign Running on the right node with spec.nodeName set", 3, r"""
+eq "$(j -n project-orion pod direct-assign '{.status.phase}')" Running && eq "$(j -n project-orion pod direct-assign '{.spec.nodeName}')" "$W3" """),
+        C("direct-assign was never scheduled (no Scheduled event from default-scheduler)", 3, r"""
+! $K -n project-orion get events --field-selector involvedObject.name=direct-assign 2>/dev/null | grep -q 'default-scheduler'"""),
+        C("scheduled Running on the same node via the default scheduler", 3, r"""
+eq "$(j -n project-orion pod scheduled '{.status.phase}')" Running && eq "$(j -n project-orion pod scheduled '{.spec.nodeName}')" "$W3" \
+ && $K -n project-orion get events --field-selector involvedObject.name=scheduled 2>/dev/null | grep -q 'default-scheduler'"""),
+        C("proof.txt names nodeName and default-scheduler", 2, r"""
+grep -q 'field= *nodeName' "$WS/q305/proof.txt" 2>/dev/null && grep -q 'scheduler= *default-scheduler' "$WS/q305/proof.txt" """),
+    ],
+    solution="""
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: direct-assign, namespace: project-orion}
+spec:
+  nodeName: {W3}            # bypasses the scheduler entirely - the kubelet just picks it up
+  containers: [{name: c, image: busybox:1.36, command: [sleep, "86400"]}]
+EOF
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: scheduled, namespace: project-orion}
+spec:
+  nodeSelector: {kubernetes.io/hostname: {W3}}     # the scheduler still makes the decision
+  containers: [{name: c, image: busybox:1.36, command: [sleep, "86400"]}]
+EOF
+k -n project-orion get events --field-selector involvedObject.name=direct-assign   # no Scheduled event
+k -n project-orion get events --field-selector involvedObject.name=scheduled       # Scheduled by default-scheduler
+printf 'field=nodeName\\nscheduler=default-scheduler\\n' > {WS}/q305/proof.txt
+""",
+)
+
+# ------------------------------------------------------------------ 306 ----
+task(
+    id=306, domain="STOR", title="Grow a volume in place",
+    task="""
+In Namespace `project-orbit`:
+
+1. Create StorageClass `expandable` using provisioner `rancher.io/local-path`,
+   `volumeBindingMode: WaitForFirstConsumer`, and allowing volume expansion.
+2. Create PVC `growing` (1Gi, RWO) using that class and Pod `grower` (busybox:1.36, `sleep 86400`)
+   mounting it at `/data`, so the claim binds.
+3. Then grow the claim to **2Gi** without deleting it, and write the claim's
+   `.status.capacity.storage` after the resize to `{WS}/q306/capacity.txt`.
+""",
+    setup=NS + r"""
+mkns project-orbit
+rm -rf "$WS/q306"; mkdir -p "$WS/q306"
+$K -n project-orbit delete pod grower --ignore-not-found --wait=false >/dev/null
+$K -n project-orbit delete pvc growing --ignore-not-found >/dev/null
+$K delete sc expandable --ignore-not-found >/dev/null
+""",
+    checks=[
+        C("StorageClass expandable: local-path, WaitForFirstConsumer, expansion allowed", 3, r"""
+s="$K get sc expandable"
+eq "$($s -o jsonpath='{.provisioner}' 2>/dev/null)" rancher.io/local-path && eq "$($s -o jsonpath='{.volumeBindingMode}')" WaitForFirstConsumer \
+ && eq "$($s -o jsonpath='{.allowVolumeExpansion}')" true"""),
+        C("PVC growing Bound on that class with grower Running", 3, r"""
+eq "$(j -n project-orbit pvc growing '{.spec.storageClassName}')" expandable && eq "$(j -n project-orbit pvc growing '{.status.phase}')" Bound \
+ && eq "$(j -n project-orbit pod grower '{.status.phase}')" Running"""),
+        C("claim requests 2Gi and reports at least 2Gi capacity", 4, r"""
+eq "$(j -n project-orbit pvc growing '{.spec.resources.requests.storage}')" 2Gi || exit 1
+for i in $(seq 1 12); do c=$(j -n project-orbit pvc growing '{.status.capacity.storage}'); [ "$c" = 2Gi ] && exit 0; sleep 5; done; exit 1"""),
+        C("capacity.txt records the new capacity", 1, r"""grep -q '2Gi' "$WS/q306/capacity.txt" 2>/dev/null"""),
+    ],
+    solution="""
+cat <<EOF | k apply -f -
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata: {name: expandable}
+provisioner: rancher.io/local-path
+volumeBindingMode: WaitForFirstConsumer
+allowVolumeExpansion: true
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: growing, namespace: project-orbit}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: expandable
+  resources: {requests: {storage: 1Gi}}
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: grower, namespace: project-orbit}
+spec:
+  volumes: [{name: d, persistentVolumeClaim: {claimName: growing}}]
+  containers: [{name: c, image: busybox:1.36, command: [sleep, "86400"], volumeMounts: [{name: d, mountPath: /data}]}]
+EOF
+k -n project-orbit patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
+k -n project-orbit get pvc growing -w        # spec 2Gi first, then status.capacity follows
+k -n project-orbit get pvc growing -o jsonpath='{.status.capacity.storage}' > {WS}/q306/capacity.txt
+# Shrinking is never allowed. Without allowVolumeExpansion the patch is rejected outright.
+""",
+)
+
+# ------------------------------------------------------------------ 307 ----
+task(
+    id=307, domain="TSHOOT", title="Node accepts nothing new", breaks="w1-block",
+    task="""
+Node `{W1}` is `Ready` and its kubelet is healthy, but no new Pod will schedule onto it.
+
+Find every reason and remove them, so that a Pod with a `nodeSelector` pinning it to `{W1}`
+starts successfully. Do not delete or recreate the node, and do not use `nodeName` to work around it.
+""",
+    setup=r"""
+$K cordon "$W1" >/dev/null 2>&1 || true
+$K taint node "$W1" maintenance=inprogress:NoSchedule --overwrite >/dev/null 2>&1 || true
+$K label node "$W1" kubernetes.io/os- >/dev/null 2>&1 || true
+$K label node "$W1" kubernetes.io/os=linux --overwrite >/dev/null 2>&1 || true
+""",
+    checks=[
+        C("node is schedulable (uncordoned)", 3, r"""[ "$(j node "$W1" '{.spec.unschedulable}')" != true ]"""),
+        C("the maintenance taint is gone", 3, r"""! j node "$W1" '{.spec.taints}' | grep -q 'maintenance'"""),
+        C("a pinned Pod actually runs there", 4, r"""
+$K -n default delete pod pin-probe --ignore-not-found --wait=true >/dev/null 2>&1
+cat <<EOF | $K apply -f - >/dev/null 2>&1
+apiVersion: v1
+kind: Pod
+metadata: {name: pin-probe, namespace: default}
+spec:
+  nodeSelector: {kubernetes.io/hostname: $W1}
+  containers: [{name: c, image: busybox:1.36, command: [sleep, "120"]}]
+EOF
+ok=1; for i in $(seq 1 20); do [ "$(j -n default pod pin-probe '{.status.phase}')" = Running ] && { ok=0; break; }; sleep 3; done
+$K -n default delete pod pin-probe --wait=false >/dev/null 2>&1; exit $ok"""),
+    ],
+    solution="""
+k describe node {W1} | head -20
+#   Taints:             maintenance=inprogress:NoSchedule
+#   Unschedulable:      true
+k uncordon {W1}
+k taint node {W1} maintenance-           # trailing dash removes the taint
+k get node {W1} -o jsonpath='{.spec.taints}{"\\n"}{.spec.unschedulable}{"\\n"}'
+# Cordon and taints are two independent mechanisms - `kubectl drain` sets the cordon,
+# and a taint may have been left by an operator or added by hand. Both must go.
+""",
+)
+
+# ------------------------------------------------------------------ 308 ----
+task(
+    id=308, domain="ARCH", title="API resource reference card",
+    task="""
+Write `{WS}/q308/resources.txt` with one line per resource below, in this exact format:
+
+```
+<resource>|<shortname or ->|<apiVersion as served>|<namespaced true|false>
+```
+
+for these four: `networkpolicies`, `persistentvolumes`, `storageclasses`, `certificatesigningrequests`.
+
+Take the values from the cluster (`kubectl api-resources`), not from memory. Use `-` when a
+resource has no shortname. Lines in the order given above.
+""",
+    setup=r"""rm -rf "$WS/q308"; mkdir -p "$WS/q308" """,
+    checks=[
+        C("all four lines present and correctly formatted", 3, r"""
+f="$WS/q308/resources.txt"; [ -f "$f" ] || exit 1
+for r in networkpolicies persistentvolumes storageclasses certificatesigningrequests; do grep -q "^$r|" "$f" || exit 1; done
+[ "$(grep -c '|' "$f")" -ge 4 ]"""),
+        C("shortnames and namespaced flags correct", 4, r"""
+f="$WS/q308/resources.txt"
+grep -Eq '^networkpolicies\|netpol\|networking.k8s.io/v1\|true$' "$f" || exit 1
+grep -Eq '^persistentvolumes\|pv\|v1\|false$' "$f" || exit 1
+grep -Eq '^storageclasses\|sc\|storage.k8s.io/v1\|false$' "$f" || exit 1
+grep -Eq '^certificatesigningrequests\|csr\|certificates.k8s.io/v1\|false$' "$f" || exit 1"""),
+        C("order matches the task", 2, r"""
+f="$WS/q308/resources.txt"
+[ "$(grep -o '^[a-z]*' "$f" | grep . | head -4 | tr '\n' ' ')" = "networkpolicies persistentvolumes storageclasses certificatesigningrequests " ]"""),
+    ],
+    solution="""
+k api-resources | grep -E '^(networkpolicies|persistentvolumes|storageclasses|certificatesigningrequests) '
+# NAME  SHORTNAMES  APIVERSION  NAMESPACED  KIND
+cat > {WS}/q308/resources.txt <<'EOF'
+networkpolicies|netpol|networking.k8s.io/v1|true
+persistentvolumes|pv|v1|false
+storageclasses|sc|storage.k8s.io/v1|false
+certificatesigningrequests|csr|certificates.k8s.io/v1|false
+EOF
+# k api-resources --namespaced=false narrows to cluster-scoped resources;
+# k explain <resource> shows the same apiVersion when you're unsure.
+""",
+)
+
+# ------------------------------------------------------------------ 309 ----
+task(
+    id=309, domain="WORK", title="A PodDisruptionBudget that blocks maintenance", breaks="w3-drain",
+    task="""
+Namespace `project-atlas` runs Deployment `atlas-api` with 2 replicas, and a PodDisruptionBudget
+`atlas-api` that currently makes voluntary eviction impossible.
+
+Node `{W3}` must be drained for maintenance, and `atlas-api` must keep at least one Pod serving
+at all times.
+
+1. Adjust the PodDisruptionBudget so the drain can complete while still guaranteeing one
+   available Pod. Keep the PDB — do not delete it.
+2. Drain `{W3}` (DaemonSet Pods aside), then make it schedulable again.
+3. Write the value of the PDB's `.status.disruptionsAllowed` after your change to
+   `{WS}/q309/allowed.txt`.
+""",
+    setup=NS + r"""
+mkns project-atlas
+rm -rf "$WS/q309"; mkdir -p "$WS/q309"
+$K uncordon "$W3" >/dev/null 2>&1 || true
+$K -n project-atlas delete deploy atlas-api --ignore-not-found >/dev/null
+$K -n project-atlas delete pdb atlas-api --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<'EOF'
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: atlas-api, namespace: project-atlas}
+spec:
+  replicas: 2
+  selector: {matchLabels: {app: atlas-api}}
+  template:
+    metadata: {labels: {app: atlas-api}}
+    spec:
+      containers: [{name: api, image: nginx:1.27, resources: {requests: {cpu: 10m, memory: 16Mi}}}]
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata: {name: atlas-api, namespace: project-atlas}
+spec:
+  minAvailable: 2
+  selector: {matchLabels: {app: atlas-api}}
+EOF
+$K -n project-atlas rollout status deploy atlas-api --timeout=120s >/dev/null 2>&1
+""",
+    checks=[
+        C("PDB still exists and now permits exactly one disruption", 3, r"""
+p="$K -n project-atlas get pdb atlas-api"
+$p >/dev/null 2>&1 || exit 1
+ma=$($p -o jsonpath='{.spec.minAvailable}'); mu=$($p -o jsonpath='{.spec.maxUnavailable}')
+{ [ "$ma" = 1 ] || [ "$mu" = 1 ]; }"""),
+        C("no atlas-api Pod left on the drained node", 3, r"""
+n=$($K -n project-atlas get pod -l app=atlas-api -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' 2>/dev/null | grep -c "^$W3$")
+eq "$n" 0"""),
+        C("both replicas Available and node schedulable again", 3, r"""
+eq "$(j -n project-atlas deploy atlas-api '{.status.availableReplicas}')" 2 && [ "$(j node "$W3" '{.spec.unschedulable}')" != true ]"""),
+        C("allowed.txt records disruptionsAllowed", 1, r"""grep -Eq '^[0-9]+$' <(tr -dc '0-9\n' < "$WS/q309/allowed.txt" 2>/dev/null | grep .) """),
+    ],
+    solution="""
+k -n project-atlas get pdb atlas-api
+#   ALLOWED DISRUPTIONS: 0  -> minAvailable 2 with 2 replicas means nothing may be evicted;
+#   a drain would hang forever waiting for the eviction API.
+k -n project-atlas patch pdb atlas-api -p '{"spec":{"minAvailable":1}}'
+k -n project-atlas get pdb atlas-api        # ALLOWED DISRUPTIONS: 1
+k drain {W3} --ignore-daemonsets --delete-emptydir-data
+k -n project-atlas rollout status deploy atlas-api
+k uncordon {W3}
+k -n project-atlas get pdb atlas-api -o jsonpath='{.status.disruptionsAllowed}' > {WS}/q309/allowed.txt
+""",
+)
+
+# ------------------------------------------------------------------ 310 ----
+task(
+    id=310, domain="NET", title="Route to something outside the cluster",
+    task="""
+An external database lives at `10.190.7.25` port 5432. Applications should reach it through a
+normal in-cluster Service name.
+
+In Namespace `project-orbit`:
+1. Create a Service `external-db` (ClusterIP, port 5432) that has **no selector**.
+2. Make it resolve to that external address by creating the matching `EndpointSlice`
+   named `external-db-1` (port 5432, TCP, addressType IPv4) with the right labels so the
+   Service adopts it.
+3. Write the ClusterIP of the Service to `{WS}/q310/ip.txt`.
+
+Do not use an ExternalName Service.
+""",
+    setup=NS + r"""
+mkns project-orbit
+rm -rf "$WS/q310"; mkdir -p "$WS/q310"
+$K -n project-orbit delete svc external-db --ignore-not-found >/dev/null
+$K -n project-orbit delete endpointslice external-db-1 --ignore-not-found >/dev/null 2>&1
+""",
+    checks=[
+        C("Service external-db is ClusterIP on 5432 with no selector", 3, r"""
+s="$K -n project-orbit get svc external-db"
+eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" ClusterIP && eq "$($s -o jsonpath='{.spec.ports[0].port}')" 5432 && [ -z "$($s -o jsonpath='{.spec.selector}')" ]"""),
+        C("EndpointSlice external-db-1 points at 10.190.7.25:5432", 4, r"""
+e="$K -n project-orbit get endpointslice external-db-1"
+eq "$($e -o jsonpath='{.addressType}' 2>/dev/null)" IPv4 && $e -o jsonpath='{.endpoints[*].addresses[*]}' | grep -q '10.190.7.25' \
+ && eq "$($e -o jsonpath='{.ports[0].port}')" 5432"""),
+        C("the slice is linked to the Service by label", 2, r"""eq "$(j -n project-orbit endpointslice external-db-1 '{.metadata.labels.kubernetes\.io/service-name}')" external-db"""),
+        C("ip.txt records the Service ClusterIP", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q310/ip.txt" 2>/dev/null)" "$(j -n project-orbit svc external-db '{.spec.clusterIP}')" """),
+    ],
+    solution="""
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Service
+metadata: {name: external-db, namespace: project-orbit}
+spec:
+  ports: [{name: pg, port: 5432, protocol: TCP, targetPort: 5432}]
+  # no selector: nothing populates the endpoints automatically, so we do it ourselves
+---
+apiVersion: discovery.k8s.io/v1
+kind: EndpointSlice
+metadata:
+  name: external-db-1
+  namespace: project-orbit
+  labels: {kubernetes.io/service-name: external-db}    # this label is the link to the Service
+addressType: IPv4
+ports: [{name: pg, port: 5432, protocol: TCP}]
+endpoints:
+  - addresses: ["10.190.7.25"]
+    conditions: {ready: true}
+EOF
+k -n project-orbit get svc external-db -o jsonpath='{.spec.clusterIP}' > {WS}/q310/ip.txt
+k -n project-orbit get endpointslice
+# ExternalName would return a CNAME instead and gives you no ClusterIP or port mapping.
+""",
+)
+
+# ------------------------------------------------------------------ 311 ----
+task(
+    id=311, domain="ARCH", title="Image inventory across the cluster",
+    task="""
+Produce `{WS}/q311/images.txt`: for every Deployment in every Namespace, one line
+
+```
+<namespace>/<deployment> <image>[,<image>...]
+```
+
+with the images of its Pod template (init containers included), comma-separated in the order they
+appear, and the file sorted by the first column.
+
+Then write the number of **distinct** images in use across those Deployments to
+`{WS}/q311/distinct.txt`.
+""",
+    setup=r"""rm -rf "$WS/q311"; mkdir -p "$WS/q311" """,
+    checks=[
+        C("images.txt matches the cluster exactly", 5, r"""
+want=$($K get deploy -A -o json 2>/dev/null | python3 -c '
+import sys,json
+out=[]
+for d in json.load(sys.stdin)["items"]:
+    s=d["spec"]["template"]["spec"]
+    imgs=[c["image"] for c in s.get("initContainers",[])]+[c["image"] for c in s["containers"]]
+    out.append("%s/%s %s"%(d["metadata"]["namespace"],d["metadata"]["name"],",".join(imgs)))
+print("\n".join(sorted(out)))')
+[ "$(grep . "$WS/q311/images.txt" 2>/dev/null | sort)" = "$want" ]"""),
+        C("distinct.txt has the right count", 3, r"""
+want=$($K get deploy -A -o json 2>/dev/null | python3 -c '
+import sys,json
+s=set()
+for d in json.load(sys.stdin)["items"]:
+    sp=d["spec"]["template"]["spec"]
+    for c in sp.get("initContainers",[])+sp["containers"]: s.add(c["image"])
+print(len(s))')
+eq "$(tr -dc '0-9' < "$WS/q311/distinct.txt" 2>/dev/null)" "$want" """),
+    ],
+    solution="""
+k get deploy -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" "}{range .spec.template.spec.initContainers[*]}{.image}{","}{end}{range .spec.template.spec.containers[*]}{.image}{","}{end}{"\\n"}{end}' \\
+  | sed 's/,$//' | sort > {WS}/q311/images.txt
+k get deploy -A -o jsonpath='{range .items[*]}{range .spec.template.spec.initContainers[*]}{.image}{"\\n"}{end}{range .spec.template.spec.containers[*]}{.image}{"\\n"}{end}{end}' \\
+  | grep . | sort -u | wc -l > {WS}/q311/distinct.txt
+# Two `range` blocks over the same pod spec keeps init and normal containers in order.
+# A trailing comma is easiest to strip afterwards with sed.
+""",
+)
+
+# ------------------------------------------------------------------ 312 ----
+task(
+    id=312, domain="WORK", title="A Job that fails, then one that doesn't",
+    task="""
+In Namespace `project-orion`:
+
+1. Job `flaky` exists and keeps failing. Let it reach its terminal state, then write the name of
+   the condition that marks it failed and its reason, as `<Type>/<Reason>`, to `{WS}/q312/why.txt`.
+2. Create a **new** Job `steady` that does the same work correctly: image `busybox:1.36`,
+   6 completions, 3 in parallel, `backoffLimit: 2`, restartPolicy Never, each Pod printing
+   `ok` and exiting 0.
+3. `steady` must complete all 6.
+""",
+    setup=NS + r"""
+mkns project-orion
+rm -rf "$WS/q312"; mkdir -p "$WS/q312"
+$K -n project-orion delete job flaky steady --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<'EOF'
+apiVersion: batch/v1
+kind: Job
+metadata: {name: flaky, namespace: project-orion}
+spec:
+  completions: 2
+  parallelism: 1
+  backoffLimit: 1
+  template:
+    spec:
+      restartPolicy: Never
+      containers: [{name: c, image: busybox:1.36, command: [sh, -c, "echo starting; exit 7"]}]
+EOF
+""",
+    checks=[
+        C("why.txt names the Failed condition and BackoffLimitExceeded", 3, r"""
+f="$WS/q312/why.txt"; grep -qi 'Failed' "$f" 2>/dev/null && grep -qi 'BackoffLimitExceeded' "$f" """),
+        C("Job steady: 6 completions, parallelism 3, backoffLimit 2, restartPolicy Never", 3, r"""
+j1="$K -n project-orion get job steady"
+eq "$($j1 -o jsonpath='{.spec.completions}' 2>/dev/null)" 6 && eq "$($j1 -o jsonpath='{.spec.parallelism}')" 3 \
+ && eq "$($j1 -o jsonpath='{.spec.backoffLimit}')" 2 && eq "$($j1 -o jsonpath='{.spec.template.spec.restartPolicy}')" Never"""),
+        C("steady completed all 6", 4, r"""for i in $(seq 1 20); do [ "$(j -n project-orion job steady '{.status.succeeded}')" = 6 ] && exit 0; sleep 3; done; exit 1"""),
+    ],
+    solution="""
+k -n project-orion get job flaky -o yaml | grep -A6 conditions
+#   type: Failed / reason: BackoffLimitExceeded
+k -n project-orion describe job flaky
+k -n project-orion logs job/flaky        # exit 7
+echo Failed/BackoffLimitExceeded > {WS}/q312/why.txt
+k -n project-orion create job steady --image=busybox:1.36 $do -- sh -c "echo ok" > steady.yaml
+# set spec.completions: 6, spec.parallelism: 3, spec.backoffLimit: 2
+k apply -f steady.yaml
+k -n project-orion wait --for=condition=complete job/steady --timeout=120s
+""",
+)
+
+# ------------------------------------------------------------------ 313 ----
+task(
+    id=313, domain="NET", title="Two hosts, TLS, and a default backend",
+    task="""
+Namespace `project-r700` has Services `web-classic` and `web-next` (port 80).
+
+Create Ingress `multi` in that Namespace with class `nginx`:
+
+- host `classic.r700.io`, path `/` (Prefix) → `web-classic:80`, TLS with secret `classic-tls`
+- host `next.r700.io`, path `/` (Prefix) → `web-next:80`, TLS with the same secret
+- a **default backend** (requests matching no rule) → `web-classic:80`
+
+Also create the TLS secret `classic-tls` from a self-signed certificate whose CN is `classic.r700.io`.
+""",
+    setup=NS + r"""
+mkns project-r700
+for s in web-classic web-next; do
+  $K -n project-r700 create deploy $s --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+  $K -n project-r700 expose deploy $s --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+done
+$K -n project-r700 delete ingress multi --ignore-not-found >/dev/null
+$K -n project-r700 delete secret classic-tls --ignore-not-found >/dev/null
+""",
+    checks=[
+        C("Secret classic-tls is a TLS secret with CN classic.r700.io", 3, r"""
+eq "$(j -n project-r700 secret classic-tls '{.type}')" kubernetes.io/tls \
+ && j -n project-r700 secret classic-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'CN *= *classic.r700.io'"""),
+        C("both host rules route to the right Services on /", 3, r"""
+$K -n project-r700 get ingress multi -o json 2>/dev/null | python3 -c '
+import sys,json
+rs=json.load(sys.stdin)["spec"]["rules"]
+m={r["host"]:r["http"]["paths"][0] for r in rs if r.get("host")}
+ok=all(h in m and m[h]["pathType"]=="Prefix" and m[h]["path"]=="/" for h in ("classic.r700.io","next.r700.io"))
+ok=ok and m["classic.r700.io"]["backend"]["service"]["name"]=="web-classic" and m["next.r700.io"]["backend"]["service"]["name"]=="web-next"
+sys.exit(0 if ok else 1)'"""),
+        C("TLS covers both hosts with classic-tls", 2, r"""
+$K -n project-r700 get ingress multi -o json 2>/dev/null | python3 -c '
+import sys,json
+t=json.load(sys.stdin)["spec"].get("tls",[])
+hosts={h for e in t for h in e.get("hosts",[])}; secs={e.get("secretName") for e in t}
+sys.exit(0 if {"classic.r700.io","next.r700.io"}<=hosts and secs=={"classic-tls"} else 1)'"""),
+        C("defaultBackend points at web-classic:80, class is nginx", 3, r"""
+i="$K -n project-r700 get ingress multi"
+eq "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" nginx \
+ && eq "$($i -o jsonpath='{.spec.defaultBackend.service.name}')" web-classic && eq "$($i -o jsonpath='{.spec.defaultBackend.service.port.number}')" 80"""),
+    ],
+    solution="""
+openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=classic.r700.io" -keyout t.key -out t.crt
+k -n project-r700 create secret tls classic-tls --cert=t.crt --key=t.key
+cat <<EOF | k apply -f -
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata: {name: multi, namespace: project-r700}
+spec:
+  ingressClassName: nginx
+  defaultBackend: {service: {name: web-classic, port: {number: 80}}}
+  tls:
+    - hosts: [classic.r700.io, next.r700.io]
+      secretName: classic-tls
+  rules:
+    - host: classic.r700.io
+      http: {paths: [{path: /, pathType: Prefix, backend: {service: {name: web-classic, port: {number: 80}}}}]}
+    - host: next.r700.io
+      http: {paths: [{path: /, pathType: Prefix, backend: {service: {name: web-next, port: {number: 80}}}}]}
+EOF
+# `kubectl create ingress` can do most of this:
+#   k -n project-r700 create ingress multi --class=nginx \\
+#     --rule="classic.r700.io/*=web-classic:80,tls=classic-tls" \\
+#     --rule="next.r700.io/*=web-next:80,tls=classic-tls" \\
+#     --default-backend=web-classic:80
+""",
+)
+
+# ------------------------------------------------------------------ 314 ----
+task(
+    id=314, domain="TSHOOT", title="A static pod that never starts", breaks="w3-kubelet",
+    task="""
+Someone added a static Pod manifest on node `{W3}` for a Pod called `edge-cache`, but no such Pod
+ever appears in the cluster and the node is otherwise healthy.
+
+Find out why, fix it on the node, and get the mirror Pod `edge-cache-{W3}` Running in Namespace
+`default` with image `nginx:1.27`.
+
+Write the one-line reason it wasn't starting to `{WS}/q314/reason.txt`.
+""",
+    setup=r"""
+rm -rf "$WS/q314"; mkdir -p "$WS/q314"
+nx "$W3" 'rm -f /etc/kubernetes/manifests/edge-cache.yaml
+d=$(grep staticPodPath /var/lib/kubelet/config.yaml | awk "{print \$2}")
+mkdir -p "$d"
+cat > "$d/edge-cache.yaml" <<EOF
+apiVersion: v1
+kind: Pod
+metadata:
+  name: edge-cache
+  namespace: default
+spec:
+  containers:
+    - name: cache
+      image: nginx:1.27
+      volumeMounts:
+        - name: conf
+          mountPath: /etc/nginx/conf.d
+  volumes:
+    - name: conf
+      hostPath:
+        path: /opt/edge-cache/conf
+        type: Directory
+EOF
+rm -rf /opt/edge-cache
+systemctl restart kubelet'
+""",
+    checks=[
+        C("mirror pod edge-cache-<node> is Running with nginx:1.27", 5, r"""
+for i in $(seq 1 12); do
+  [ "$(j -n default pod "edge-cache-$W3" '{.status.phase}')" = Running ] && [ "$(j -n default pod "edge-cache-$W3" '{.spec.containers[0].image}')" = nginx:1.27 ] && exit 0
+  sleep 5
+done; exit 1"""),
+        C("the hostPath the manifest references now exists on the node", 3, r"""
+p=$(nx "$W3" 'grep -A2 hostPath /etc/kubernetes/manifests/edge-cache.yaml /etc/kubernetes/static/edge-cache.yaml 2>/dev/null | grep "path:" | head -1' | awk '{print $2}')
+[ -n "$p" ] && nx "$W3" "test -d $p" """),
+        C("reason.txt mentions the missing host directory", 2, r"""grep -Eqi 'director|hostpath|/opt/edge-cache|not exist|missing' "$WS/q314/reason.txt" 2>/dev/null"""),
+    ],
+    solution="""
+k get pod -A | grep edge-cache          # nothing
+{SHELL_W3}
+journalctl -u kubelet -n 40 --no-pager | grep -i edge-cache
+#   ... failed to start ... hostPath type check failed: /opt/edge-cache/conf is not a directory
+grep staticPodPath /var/lib/kubelet/config.yaml
+cat <staticPodPath>/edge-cache.yaml     # hostPath /opt/edge-cache/conf with type: Directory
+mkdir -p /opt/edge-cache/conf           # type: Directory means the kubelet will NOT create it
+exit
+k get pod edge-cache-{W3}
+echo 'hostPath /opt/edge-cache/conf did not exist and type was Directory' > {WS}/q314/reason.txt
+# The alternative fix is type: DirectoryOrCreate, but the task asks for the pod to run as written.
+""",
+)
+
+# ------------------------------------------------------------------ 315 ----
+task(
+    id=315, domain="STOR", title="Keep the data when the claim goes",
+    task="""
+In Namespace `project-orbit`, Pod `keeper` writes to PVC `keeper-claim`, which was provisioned
+dynamically by the default StorageClass — whose reclaim policy is `Delete`.
+
+The data must survive the claim being deleted:
+
+1. Without recreating anything, make sure the bound PersistentVolume is **retained** when its claim
+   goes away.
+2. Write the PV's name to `{WS}/q315/pv.txt`.
+3. Delete Pod `keeper` and PVC `keeper-claim`.
+4. The PV must still exist afterwards, in phase `Released`.
+""",
+    setup=NS + r"""
+mkns project-orbit
+rm -rf "$WS/q315"; mkdir -p "$WS/q315"
+$K -n project-orbit delete pod keeper --ignore-not-found --wait=true >/dev/null
+$K -n project-orbit delete pvc keeper-claim --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<'EOF'
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: keeper-claim, namespace: project-orbit}
+spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 200Mi}}}
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: keeper, namespace: project-orbit}
+spec:
+  volumes: [{name: d, persistentVolumeClaim: {claimName: keeper-claim}}]
+  containers: [{name: c, image: busybox:1.36, command: [sh, -c, "echo important > /data/keep.txt; sleep 86400"], volumeMounts: [{name: d, mountPath: /data}]}]
+EOF
+for i in $(seq 1 30); do [ "$(j -n project-orbit pod keeper '{.status.phase}')" = Running ] && break; sleep 2; done
+""",
+    checks=[
+        C("pv.txt names the PV that was bound to keeper-claim", 3, r"""
+p=$(tr -d '[:space:]' < "$WS/q315/pv.txt" 2>/dev/null); [ -n "$p" ] || exit 1
+eq "$(j pv "$p" '{.spec.claimRef.name}')" keeper-claim"""),
+        C("that PV has reclaim policy Retain", 4, r"""eq "$(j pv "$(tr -d '[:space:]' < "$WS/q315/pv.txt")" '{.spec.persistentVolumeReclaimPolicy}')" Retain"""),
+        C("Pod and PVC are gone", 2, r"""! $K -n project-orbit get pod keeper >/dev/null 2>&1 && ! $K -n project-orbit get pvc keeper-claim >/dev/null 2>&1"""),
+        C("the PV still exists and is Released", 3, r"""
+p=$(tr -d '[:space:]' < "$WS/q315/pv.txt"); for i in $(seq 1 10); do [ "$(j pv "$p" '{.status.phase}')" = Released ] && exit 0; sleep 3; done; exit 1"""),
+    ],
+    solution="""
+k -n project-orbit get pvc keeper-claim -o jsonpath='{.spec.volumeName}' | tee {WS}/q315/pv.txt
+PV=$(cat {WS}/q315/pv.txt)
+k patch pv $PV -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
+k get pv $PV                       # RECLAIM POLICY: Retain
+k -n project-orbit delete pod keeper --now
+k -n project-orbit delete pvc keeper-claim
+k get pv $PV                       # STATUS: Released - data still on disk
+# Patch the PV, not the StorageClass: the class only sets the policy for volumes created later.
+# To reuse it, clear .spec.claimRef and it becomes Available again.
+""",
+)
+
+# ------------------------------------------------------------------ 316 ----
+task(
+    id=316, domain="ARCH", title="Plan a control-plane upgrade",
+    task="""
+The control-plane node must be upgraded to the next patch release.
+
+1. Write the cluster's current control-plane version to `{WS}/q316/current.txt` (as `kubectl` reports it).
+2. Run the kubeadm command that shows what an upgrade would do, and save its output to
+   `{WS}/q316/plan.txt`.
+3. Write to `{WS}/q316/steps.sh` the ordered commands you would run to upgrade the control-plane
+   node — package install of kubeadm, the kubeadm upgrade step, draining, upgrading kubelet and
+   kubectl, restarting the kubelet, and uncordoning — one command per line, in the order they must
+   be executed. Do **not** execute them.
+""",
+    setup=r"""rm -rf "$WS/q316"; mkdir -p "$WS/q316" """,
+    checks=[
+        C("current.txt matches the server version", 2, r"""
+v=$($K version -o json 2>/dev/null | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])')
+grep -qF "$v" "$WS/q316/current.txt" 2>/dev/null"""),
+        C("plan.txt is real kubeadm upgrade plan output", 3, r"""grep -Eqi 'upgrade|component|version' "$WS/q316/plan.txt" 2>/dev/null && [ "$(wc -l < "$WS/q316/plan.txt")" -ge 5 ]"""),
+        C("steps.sh has the commands in the correct order", 5, r"""
+f="$WS/q316/steps.sh"; [ -f "$f" ] || exit 1
+grep -q 'kubeadm upgrade apply' "$f" || exit 1
+grep -Eq '(apt|yum|dnf).*(install).*kubeadm' "$f" || exit 1
+grep -q 'drain' "$f" && grep -q 'uncordon' "$f" || exit 1
+grep -Eq '(apt|yum|dnf).*(install).*kubelet' "$f" || exit 1
+grep -Eq 'systemctl (restart|daemon-reload)' "$f" || exit 1
+n=$(grep -n 'kubeadm upgrade apply' "$f" | head -1 | cut -d: -f1)
+ka=$(grep -nE '(apt|yum|dnf).*install.*kubeadm' "$f" | head -1 | cut -d: -f1)
+kl=$(grep -nE '(apt|yum|dnf).*install.*kubelet' "$f" | head -1 | cut -d: -f1)
+dr=$(grep -n 'drain' "$f" | head -1 | cut -d: -f1)
+un=$(grep -n 'uncordon' "$f" | head -1 | cut -d: -f1)
+[ "$ka" -lt "$n" ] && [ "$n" -lt "$kl" ] && [ "$dr" -lt "$kl" ] && [ "$kl" -lt "$un" ]"""),
+    ],
+    solution="""
+k version -o json | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])' > {WS}/q316/current.txt
+{SHELL_CP}
+kubeadm upgrade plan | tee /tmp/plan.txt      # copy into {WS}/q316/plan.txt
+cat > {WS}/q316/steps.sh <<'EOF'
+apt-get update && apt-get install -y kubeadm=1.34.1-1.1
+kubeadm upgrade plan
+kubeadm upgrade apply v1.34.1
+kubectl drain <control-plane> --ignore-daemonsets
+apt-get install -y kubelet=1.34.1-1.1 kubectl=1.34.1-1.1
+systemctl daemon-reload
+systemctl restart kubelet
+kubectl uncordon <control-plane>
+EOF
+# Order matters and is the point of this task: kubeadm binary first, then `kubeadm upgrade apply`
+# (it upgrades the static pod control plane), then drain, then kubelet/kubectl packages, restart,
+# uncordon. Workers use `kubeadm upgrade node` instead of `upgrade apply`.
+""",
+)
+
+# ------------------------------------------------------------------ 317 ----
+task(
+    id=317, domain="TSHOOT", title="Certificates stop being issued", breaks="controller-manager",
+    task="""
+Certificate signing requests are approved but never get a certificate: `kubectl get csr` shows
+them `Approved` with an empty `CONDITION`-issued certificate forever.
+
+Find the responsible control-plane component, repair it on the control-plane node, and prove it
+works: a CertificateSigningRequest named `issue-probe` (CN `probe`, signer
+`kubernetes.io/kube-apiserver-client`, usage `client auth`) that you create and approve must
+receive a certificate.
+""",
+    setup=r"""
+$K delete csr issue-probe --ignore-not-found >/dev/null 2>&1
+nx "$CP" '
+f=/etc/kubernetes/manifests/kube-controller-manager.yaml
+grep -q "cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt" $f && sed -i "s|--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt|--cluster-signing-cert-file=/etc/kubernetes/pki/ca-signing.crt|" $f
+'
+""",
+    checks=[
+        C("kube-controller-manager points at a signing certificate that exists", 4, r"""
+p=$(nx "$CP" 'grep -o "\-\-cluster-signing-cert-file=[^ ]*" /etc/kubernetes/manifests/kube-controller-manager.yaml' | cut -d= -f2)
+[ -n "$p" ] && nx "$CP" "test -f $p" """),
+        C("kube-controller-manager pod is Running", 3, r"""
+for i in $(seq 1 12); do [ "$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].status.phase}')" = Running ] && exit 0; sleep 5; done; exit 1"""),
+        C("an approved CSR now receives a certificate", 5, r"""
+$K get csr issue-probe >/dev/null 2>&1 || exit 1
+$K get csr issue-probe -o jsonpath='{.status.conditions[*].type}' | grep -q Approved || exit 1
+for i in $(seq 1 15); do [ -n "$(j csr issue-probe '{.status.certificate}')" ] && exit 0; sleep 4; done; exit 1"""),
+    ],
+    solution="""
+k get csr                                     # Approved but never Issued
+k -n kube-system logs kube-controller-manager-{CP} | tail -20
+#   ... unable to load client CA file .../ca-signing.crt: no such file or directory
+{SHELL_CP}
+grep signing /etc/kubernetes/manifests/kube-controller-manager.yaml
+ls /etc/kubernetes/pki/ca.crt                 # the real file
+vi /etc/kubernetes/manifests/kube-controller-manager.yaml
+#   --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
+exit
+openssl genrsa -out probe.key 2048
+openssl req -new -key probe.key -subj "/CN=probe" -out probe.csr
+cat <<EOF | k apply -f -
+apiVersion: certificates.k8s.io/v1
+kind: CertificateSigningRequest
+metadata: {name: issue-probe}
+spec: {request: $(base64 -w0 probe.csr), signerName: kubernetes.io/kube-apiserver-client, usages: [client auth]}
+EOF
+k certificate approve issue-probe
+k get csr issue-probe                          # certificate now populated
+# The csrsigning controller lives in kube-controller-manager: approval is the API's job,
+# but issuing the certificate is the controller's.
+""",
+)
+
+QUESTIONS = Q

tracks/cka/common.py

@@ -0,0 +1,15 @@
+"""Shared helpers for CKA task banks."""
+
+NS = r"""
+mkns() { $K create ns "$1" --dry-run=client -o yaml | $K apply -f - >/dev/null; }
+"""
+
+def C(name, pts, script):
+    return {"name": name, "pts": pts, "script": script}
+
+def ns_pick(r, base):
+    """Namespace name with a random suffix so the same task never lands on the same namespace twice."""
+    return f"{base}-{r.choice(['alpha', 'bravo', 'delta', 'echo', 'kilo', 'nova', 'omega', 'zulu'])}"
+
+def name_pick(r, base):
+    return f"{base}-{r.randint(10, 99)}"

tracks/cka/questions.py

@@ -0,0 +1,652 @@
+"""
+CKA track for the exam simulator.
+
+TRACK  - metadata (name, duration, pass mark, domain table)
+QUESTIONS - list of tasks. Each task:
+  id, domain, title, task (markdown shown to candidate),
+  setup   : bash that seeds the scenario
+  checks  : list of (name, points, bash) - each check is graded independently
+            (exit 0 = pass). Points earned = sum of passing checks, so partial
+            credit works the way it does on the real exam.
+  disruptive: True => setup runs last, after everything else is scheduled.
+
+Scripts get a prelude: $K (kubectl), $WS (workspace), $CP/$W1/$W2/$W3 (node
+names), nx NODE 'cmd' (run a command on a node). {WS}, {CP}, {W2}, {W3},
+{SHELL_CP} etc. are substituted in task text.
+"""
+
+TRACK = {
+    "id": "cka",
+    "name": "Certified Kubernetes Administrator",
+    "minutes": 120,
+    "pass_pct": 66,
+    # tasks per domain drawn into a session (weights follow the exam blueprint)
+    "quota":      {"ARCH": 5, "WORK": 4, "NET": 4, "STOR": 2, "TSHOOT": 5},      # 20 tasks
+    "quota_hard": {"ARCH": 6, "WORK": 5, "NET": 5, "STOR": 3, "TSHOOT": 6},      # 25 tasks, hard ones preferred
+    # fixed simulator sets (./exam sim a|b) - excluded from random sampling
+    "sets": {"a": "Simulator Set A", "b": "Simulator Set B"},
+    "domains": {
+        "ARCH":   "Cluster Architecture, Installation & Configuration (25%)",
+        "WORK":   "Workloads & Scheduling (15%)",
+        "NET":    "Services & Networking (20%)",
+        "STOR":   "Storage (10%)",
+        "TSHOOT": "Troubleshooting (30%)",
+    },
+}
+
+from tracks.cka.common import C, NS
+
+QUESTIONS = [
+# ================================================================ ARCH ====
+{
+ "id": 1, "domain": "ARCH", "title": "RBAC for a deployment bot",
+ "task": """
+Namespace `rbac-app` already exists.
+
+1. Create a ServiceAccount `deploy-bot` in namespace `rbac-app`.
+2. Create a Role `deploy-manager` in `rbac-app` that grants
+   get, list, watch, create, update, patch, delete on **deployments** (apps API group) only.
+3. Bind it to the ServiceAccount with a RoleBinding named `deploy-bot-binding`.
+
+The ServiceAccount must NOT be able to delete pods, and must have no access
+outside the `rbac-app` namespace. Verify with `kubectl auth can-i`.
+""",
+ "setup": NS + r"""
+mkns rbac-app
+$K -n rbac-app delete sa deploy-bot --ignore-not-found >/dev/null
+$K -n rbac-app delete role deploy-manager --ignore-not-found >/dev/null
+$K -n rbac-app delete rolebinding deploy-bot-binding --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("ServiceAccount rbac-app/deploy-bot exists", 1, r"""$K -n rbac-app get sa deploy-bot >/dev/null 2>&1"""),
+  C("Role deploy-manager and RoleBinding deploy-bot-binding exist", 1,
+    r"""$K -n rbac-app get role deploy-manager >/dev/null 2>&1 && $K -n rbac-app get rolebinding deploy-bot-binding >/dev/null 2>&1"""),
+  C("deploy-bot can get/list/watch/create/update/patch/delete deployments", 2, r"""
+for v in get list watch create update patch delete; do
+  [ "$($K auth can-i $v deployments -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot 2>/dev/null)" = yes ] || exit 1
+done"""),
+  C("deploy-bot cannot delete pods", 1, r"""[ "$($K auth can-i delete pods -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot 2>/dev/null || true)" = no ]"""),
+  C("deploy-bot has no access in namespace default", 1, r"""[ "$($K auth can-i create deployments -n default --as=system:serviceaccount:rbac-app:deploy-bot 2>/dev/null || true)" = no ]"""),
+ ],
+},
+{
+ "id": 2, "domain": "ARCH", "title": "etcd snapshot",
+ "task": """
+Create a snapshot of the cluster's etcd database and save it on the
+**control-plane node** at `/opt/cka/etcd-snapshot.db`.
+
+Details you need:
+- etcd listens on `https://127.0.0.1:2379`
+- certificates are under `/etc/kubernetes/pki/etcd/` (`ca.crt`, `server.crt`, `server.key`)
+- "ssh to the control-plane node" is: `{SHELL_CP}`
+- Setup tried to place `etcdctl` at `/usr/local/bin/etcdctl` on that node.
+  If it is missing, run etcdctl inside the etcd pod
+  (`kubectl -n kube-system exec etcd-{CP} -- etcdctl ...`) and copy the file to the node afterwards.
+
+Verify the snapshot with `etcdutl snapshot status` (or `etcdctl snapshot status`).
+""",
+ "setup": r"""
+nx "$CP" 'mkdir -p /opt/cka; rm -f /opt/cka/etcd-snapshot.db'
+POD=$($K -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].metadata.name}')
+for bin in etcdctl etcdutl; do
+  if $K -n kube-system exec "$POD" -- cat /usr/local/bin/$bin > /tmp/cka-$bin 2>/dev/null && [ -s /tmp/cka-$bin ]; then
+    nx "$CP" "cat > /usr/local/bin/$bin && chmod +x /usr/local/bin/$bin" < /tmp/cka-$bin
+  fi
+  rm -f /tmp/cka-$bin
+done
+""",
+ "checks": [
+  C("/opt/cka/etcd-snapshot.db exists on the control-plane node", 3, r"""nx "$CP" 'test -f /opt/cka/etcd-snapshot.db'"""),
+  C("snapshot is a real etcd backup (> 100 KB)", 4, r"""[ "$(nx "$CP" 'stat -c %s /opt/cka/etcd-snapshot.db' 2>/dev/null || echo 0)" -gt 100000 ]"""),
+ ],
+},
+{
+ "id": 3, "domain": "ARCH", "title": "Kustomize overlay",
+ "task": """
+A Kustomize base lives in `{WS}/q03/base` (Deployment `store-api` + Service `store-api`).
+
+Create a production overlay at `{WS}/q03/overlays/prod` that:
+- deploys everything into namespace `store-prod` (create the Namespace as part of the overlay),
+- adds the common label `env=prod` to all resources,
+- sets the `store-api` Deployment to **3** replicas.
+
+Apply it with `kubectl apply -k`.
+""",
+ "setup": r"""
+rm -rf "$WS/q03"; mkdir -p "$WS/q03/base"
+cat > "$WS/q03/base/kustomization.yaml" <<'EOF'
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - deployment.yaml
+  - service.yaml
+EOF
+cat > "$WS/q03/base/deployment.yaml" <<'EOF'
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: store-api
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: store-api
+  template:
+    metadata:
+      labels:
+        app: store-api
+    spec:
+      containers:
+        - name: api
+          image: nginx:1.25
+          ports:
+            - containerPort: 80
+EOF
+cat > "$WS/q03/base/service.yaml" <<'EOF'
+apiVersion: v1
+kind: Service
+metadata:
+  name: store-api
+spec:
+  selector:
+    app: store-api
+  ports:
+    - port: 80
+      targetPort: 80
+EOF
+$K delete ns store-prod --ignore-not-found --wait=false >/dev/null
+""",
+ "checks": [
+  C("overlay kustomization.yaml exists", 1, r"""[ -f "$WS/q03/overlays/prod/kustomization.yaml" ]"""),
+  C("resources deployed into namespace store-prod", 1, r"""$K -n store-prod get deploy store-api >/dev/null 2>&1 && $K -n store-prod get svc store-api >/dev/null 2>&1"""),
+  C("store-api has 3 replicas", 1, r"""[ "$($K -n store-prod get deploy store-api -o jsonpath='{.spec.replicas}')" = 3 ]"""),
+  C("Deployment and Service carry label env=prod", 2, r"""[ "$($K -n store-prod get deploy store-api -o jsonpath='{.metadata.labels.env}')" = prod ] && [ "$($K -n store-prod get svc store-api -o jsonpath='{.metadata.labels.env}')" = prod ]"""),
+ ],
+},
+{
+ "id": 4, "domain": "ARCH", "title": "CustomResourceDefinition",
+ "task": """
+1. Create a CRD with:
+   - group `data.example.com`, version `v1` (served + storage)
+   - kind `Backup`, plural `backups`, singular `backup`, shortName `bk`
+   - scope `Namespaced`
+   - an OpenAPI v3 schema where `spec` has string fields `schedule` and `target`
+2. Create a `Backup` named `nightly` in namespace `default` with
+   `spec.schedule: "0 2 * * *"` and `spec.target: "s3://backups/prod"`.
+
+`kubectl get bk` must list it.
+""",
+ "setup": r"""$K delete crd backups.data.example.com --ignore-not-found >/dev/null""",
+ "checks": [
+  C("CRD backups.data.example.com exists and is Namespaced", 1, r"""[ "$($K get crd backups.data.example.com -o jsonpath='{.spec.scope}' 2>/dev/null)" = Namespaced ]"""),
+  C("shortName bk works", 1, r"""$K get crd backups.data.example.com -o jsonpath='{.spec.names.shortNames}' | grep -q '"bk"'"""),
+  C("Backup nightly has the right schedule", 1, r"""[ "$($K -n default get bk nightly -o jsonpath='{.spec.schedule}' 2>/dev/null)" = "0 2 * * *" ]"""),
+  C("Backup nightly has the right target", 1, r"""[ "$($K -n default get bk nightly -o jsonpath='{.spec.target}' 2>/dev/null)" = "s3://backups/prod" ]"""),
+ ],
+},
+{
+ "id": 5, "domain": "ARCH", "title": "Static pod",
+ "task": """
+Create a **static pod** named `static-web` running image `nginx:1.27`
+on node `{W3}` (namespace `default`).
+
+You must place the manifest in the kubelet's static pod path on that node.
+Reach the node with: `{SHELL_W3}`
+(check `/var/lib/kubelet/config.yaml` for `staticPodPath`).
+
+The mirror pod `static-web-{W3}` must reach Running.
+""",
+ "setup": r"""nx "$W3" 'rm -f /etc/kubernetes/manifests/static-web*.yaml' || true""",
+ "checks": [
+  C("manifest containing static-web is in /etc/kubernetes/manifests on the node", 2, r"""nx "$W3" 'grep -ls static-web /etc/kubernetes/manifests/*.yaml' >/dev/null 2>&1"""),
+  C("mirror pod static-web-<node> is Running", 2, r"""[ "$($K -n default get pod "static-web-$W3" -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ]"""),
+  C("image is nginx:1.27", 1, r"""[ "$($K -n default get pod "static-web-$W3" -o jsonpath='{.spec.containers[0].image}' 2>/dev/null)" = nginx:1.27 ]"""),
+ ],
+},
+{
+ "id": 6, "domain": "ARCH", "title": "Certificates: expiry check and user CSR",
+ "task": """
+Part A — On the control-plane node (`{SHELL_CP}`) determine the expiry date of the
+**kube-apiserver** serving certificate (use `kubeadm certs check-expiration` or `openssl`).
+Write the expiry line to `{WS}/q06/apiserver-expiry.txt` on your machine.
+
+Part B — Onboard a new user `jane`:
+1. Generate a private key and a CSR with CN=jane (openssl).
+2. Create a Kubernetes `CertificateSigningRequest` named `jane`
+   (signer `kubernetes.io/kube-apiserver-client`, usage `client auth`).
+3. Approve it so a certificate is issued.
+4. Create a RoleBinding `jane-view` in namespace `default` giving user `jane`
+   the built-in ClusterRole `view`.
+""",
+ "setup": r"""
+rm -rf "$WS/q06"; mkdir -p "$WS/q06"
+$K delete csr jane --ignore-not-found >/dev/null
+$K -n default delete rolebinding jane-view --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("apiserver-expiry.txt records the real expiry month and year", 2, r"""
+[ -s "$WS/q06/apiserver-expiry.txt" ] || exit 1
+exp=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate' | sed 's/notAfter=//')
+yr=$(echo "$exp" | awk '{print $4}'); mon=$(echo "$exp" | awk '{print $1}')
+grep -q "$yr" "$WS/q06/apiserver-expiry.txt" && grep -qi "$mon" "$WS/q06/apiserver-expiry.txt" """),
+  C("CSR jane exists with signer kubernetes.io/kube-apiserver-client", 1, r"""[ "$($K get csr jane -o jsonpath='{.spec.signerName}' 2>/dev/null)" = kubernetes.io/kube-apiserver-client ]"""),
+  C("CSR jane approved and certificate issued", 2, r"""$K get csr jane -o jsonpath='{.status.conditions[*].type}' 2>/dev/null | grep -q Approved && [ -n "$($K get csr jane -o jsonpath='{.status.certificate}')" ]"""),
+  C("jane can list pods in default (view) but not delete them", 1, r"""[ "$($K auth can-i list pods -n default --as=jane 2>/dev/null || true)" = yes ] && [ "$($K auth can-i delete pods -n default --as=jane 2>/dev/null || true)" = no ]"""),
+ ],
+},
+# ================================================================ WORK ====
+{
+ "id": 7, "domain": "WORK", "title": "Rolling update & scaling",
+ "task": """
+Deployment `web-frontend` in namespace `apps` runs `nginx:1.25` with 2 replicas.
+
+- Update the image to `nginx:1.27`.
+- Scale to **4** replicas.
+- Configure the rolling update strategy: `maxSurge: 1`, `maxUnavailable: 0`.
+- All 4 replicas must be available when you are done.
+""",
+ "setup": NS + r"""
+mkns apps
+$K -n apps delete deploy web-frontend --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<'EOF'
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: web-frontend
+  namespace: apps
+spec:
+  replicas: 2
+  selector:
+    matchLabels: {app: web-frontend}
+  template:
+    metadata:
+      labels: {app: web-frontend}
+    spec:
+      containers:
+        - name: nginx
+          image: nginx:1.25
+          resources:
+            requests: {cpu: 50m, memory: 32Mi}
+EOF
+""",
+ "checks": [
+  C("image is nginx:1.27", 1, r"""[ "$($K -n apps get deploy web-frontend -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null)" = nginx:1.27 ]"""),
+  C("replicas = 4", 1, r"""[ "$($K -n apps get deploy web-frontend -o jsonpath='{.spec.replicas}' 2>/dev/null)" = 4 ]"""),
+  C("strategy maxSurge=1 maxUnavailable=0", 1, r"""[ "$($K -n apps get deploy web-frontend -o jsonpath='{.spec.strategy.rollingUpdate.maxSurge}')" = 1 ] && [ "$($K -n apps get deploy web-frontend -o jsonpath='{.spec.strategy.rollingUpdate.maxUnavailable}')" = 0 ]"""),
+  C("all 4 replicas available", 1, r"""[ "$($K -n apps get deploy web-frontend -o jsonpath='{.status.availableReplicas}' 2>/dev/null)" = 4 ]"""),
+ ],
+},
+{
+ "id": 8, "domain": "WORK", "title": "Multi-container pod with shared volume",
+ "task": """
+Create a pod `log-shipper` in namespace `apps` with two containers:
+
+- `app` (image `busybox:1.36`): appends the current date to `/var/log/app.log` every 2 seconds.
+- `shipper` (image `busybox:1.36`): runs `tail -f /var/log/app.log`.
+
+Both containers share an `emptyDir` volume named `logs` mounted at `/var/log`.
+`kubectl logs log-shipper -c shipper` must show the dates written by `app`.
+""",
+ "setup": NS + r"""
+mkns apps
+$K -n apps delete pod log-shipper --ignore-not-found --wait=false >/dev/null
+""",
+ "checks": [
+  C("pod log-shipper Running with containers app and shipper", 1, r"""
+p="$K -n apps get pod log-shipper"
+[ "$($p -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] || exit 1
+n=$($p -o jsonpath='{.spec.containers[*].name}'); echo "$n" | grep -qw app && echo "$n" | grep -qw shipper"""),
+  C("volume logs is an emptyDir mounted at /var/log in both containers", 2, r"""
+p="$K -n apps get pod log-shipper"
+[ -n "$($p -o jsonpath='{.spec.volumes[?(@.name=="logs")].emptyDir}' 2>/dev/null)" ] || exit 1
+[ "$($p -o jsonpath='{.spec.containers[?(@.name=="app")].volumeMounts[?(@.name=="logs")].mountPath}')" = /var/log ] || exit 1
+[ "$($p -o jsonpath='{.spec.containers[?(@.name=="shipper")].volumeMounts[?(@.name=="logs")].mountPath}')" = /var/log ]"""),
+  C("shipper's logs show lines written by app", 1, r"""sleep 3; [ -n "$($K -n apps logs log-shipper -c shipper --tail=2 2>/dev/null)" ]"""),
+ ],
+},
+{
+ "id": 9, "domain": "WORK", "title": "Scheduling with node affinity and resources",
+ "task": """
+One node in the cluster is labelled `disktype=ssd`.
+
+Create a Deployment `cache` in namespace `apps`, 2 replicas, image `redis:7-alpine`, that:
+- uses **required** node affinity (`requiredDuringSchedulingIgnoredDuringExecution`)
+  so pods only run on nodes with `disktype=ssd` (do not use nodeSelector),
+- sets resource requests `cpu: 100m`, `memory: 64Mi` and limits `cpu: 200m`, `memory: 128Mi`.
+
+Both pods must be Running.
+""",
+ "setup": NS + r"""
+mkns apps
+$K label node "$W3" disktype=ssd --overwrite >/dev/null
+$K label node "$W1" disktype- >/dev/null 2>&1 || true
+$K label node "$W2" disktype- >/dev/null 2>&1 || true
+$K -n apps delete deploy cache --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("deployment cache with 2 replicas", 1, r"""[ "$($K -n apps get deploy cache -o jsonpath='{.spec.replicas}' 2>/dev/null)" = 2 ]"""),
+  C("required node affinity set, no nodeSelector", 1, r"""d="$K -n apps get deploy cache"; [ -n "$($d -o jsonpath='{.spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution}' 2>/dev/null)" ] && [ -z "$($d -o jsonpath='{.spec.template.spec.nodeSelector}')" ]"""),
+  C("requests 100m/64Mi and limits 200m/128Mi", 1, r"""
+c='{.spec.template.spec.containers[0].resources'; d="$K -n apps get deploy cache"
+[ "$($d -o jsonpath="$c.requests.cpu}")" = 100m ] && [ "$($d -o jsonpath="$c.requests.memory}")" = 64Mi ] && [ "$($d -o jsonpath="$c.limits.cpu}")" = 200m ] && [ "$($d -o jsonpath="$c.limits.memory}")" = 128Mi ]"""),
+  C("both pods available and on the ssd node", 1, r"""
+[ "$($K -n apps get deploy cache -o jsonpath='{.status.availableReplicas}')" = 2 ] || exit 1
+for n in $($K -n apps get pod -l app=cache -o jsonpath='{.items[*].spec.nodeName}'); do [ "$n" = "$W3" ] || exit 1; done"""),
+ ],
+},
+{
+ "id": 10, "domain": "WORK", "title": "ConfigMap and Secret consumption",
+ "task": """
+In namespace `apps`:
+1. Create ConfigMap `app-config` with keys `APP_MODE=production` and `LOG_LEVEL=debug`.
+2. Create Secret `db-creds` with keys `username=admin` and `password=S3cr3t!`.
+3. Create pod `config-pod` (image `busybox:1.36`, command `sleep 3600`) that:
+   - exposes `APP_MODE` as an env var sourced from the ConfigMap key,
+   - exposes **all** keys of `db-creds` as env vars (envFrom),
+   - mounts the ConfigMap as a volume at `/etc/config`.
+""",
+ "setup": NS + r"""
+mkns apps
+$K -n apps delete pod config-pod --ignore-not-found --wait=false >/dev/null
+$K -n apps delete cm app-config --ignore-not-found >/dev/null
+$K -n apps delete secret db-creds --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("ConfigMap app-config and Secret db-creds with correct values", 1, r"""[ "$($K -n apps get cm app-config -o jsonpath='{.data.LOG_LEVEL}' 2>/dev/null)" = debug ] && [ "$($K -n apps get secret db-creds -o jsonpath='{.data.password}' 2>/dev/null | base64 -d)" = 'S3cr3t!' ]"""),
+  C("APP_MODE injected via configMapKeyRef", 1, r"""[ "$($K -n apps get pod config-pod -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && [ -n "$($K -n apps get pod config-pod -o jsonpath='{.spec.containers[0].env[?(@.name=="APP_MODE")].valueFrom.configMapKeyRef}')" ] && $K -n apps exec config-pod -- env 2>/dev/null | grep -q '^APP_MODE=production$'"""),
+  C("all db-creds keys injected via envFrom", 1, r"""[ -n "$($K -n apps get pod config-pod -o jsonpath='{.spec.containers[0].envFrom[*].secretRef}' 2>/dev/null)" ] && $K -n apps exec config-pod -- env 2>/dev/null | grep -q '^password=S3cr3t!$'"""),
+  C("ConfigMap mounted at /etc/config", 1, r"""[ "$($K -n apps exec config-pod -- cat /etc/config/LOG_LEVEL 2>/dev/null)" = debug ]"""),
+ ],
+},
+# ================================================================= NET ====
+{
+ "id": 11, "domain": "NET", "title": "NetworkPolicy",
+ "task": """
+Namespace `netpol` has deployments `frontend`, `backend` and `db`
+(labels `app=frontend|backend|db`).
+
+Create a NetworkPolicy `backend-policy` in `netpol` so that pods with `app=backend`
+- accept ingress **only** from pods labelled `app=frontend` in the same namespace,
+  on TCP port **8080**,
+- reject all other ingress.
+
+Do not restrict egress. (The default kind CNI does not enforce policies;
+you are graded on the policy spec.)
+""",
+ "setup": NS + r"""
+mkns netpol
+for a in frontend backend db; do
+  $K -n netpol create deploy $a --image=nginx:1.25 --dry-run=client -o yaml | $K apply -f - >/dev/null
+done
+$K -n netpol delete netpol backend-policy --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("policy selects app=backend, policyTypes Ingress only", 2, r"""n="$K -n netpol get netpol backend-policy"; [ "$($n -o jsonpath='{.spec.podSelector.matchLabels.app}' 2>/dev/null)" = backend ] && $n -o jsonpath='{.spec.policyTypes}' | grep -q Ingress && ! $n -o jsonpath='{.spec.policyTypes}' | grep -q Egress"""),
+  C("single ingress rule from podSelector app=frontend only", 2, r"""n="$K -n netpol get netpol backend-policy"; [ "$($n -o jsonpath='{.spec.ingress[0].from[0].podSelector.matchLabels.app}' 2>/dev/null)" = frontend ] && [ -z "$($n -o jsonpath='{.spec.ingress[1]}')" ] && [ -z "$($n -o jsonpath='{.spec.ingress[0].from[1]}')" ]"""),
+  C("allowed port is TCP 8080", 2, r"""n="$K -n netpol get netpol backend-policy"; [ "$($n -o jsonpath='{.spec.ingress[0].ports[0].port}' 2>/dev/null)" = 8080 ] && { p=$($n -o jsonpath='{.spec.ingress[0].ports[0].protocol}'); [ -z "$p" ] || [ "$p" = TCP ]; }"""),
+ ],
+},
+{
+ "id": 12, "domain": "NET", "title": "NodePort Service",
+ "task": """
+Deployment `shop` (namespace `svc`) serves HTTP on container port 80.
+
+Expose it with a Service `shop-svc` of type **NodePort**:
+port 80, targetPort 80, nodePort **30080**.
+The Service must have ready endpoints.
+""",
+ "setup": NS + r"""
+mkns svc
+$K -n svc create deploy shop --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n svc delete svc shop-svc --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("Service shop-svc is NodePort, port 80 -> targetPort 80", 1, r"""s="$K -n svc get svc shop-svc"; [ "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" = NodePort ] && [ "$($s -o jsonpath='{.spec.ports[0].port}')" = 80 ] && [ "$($s -o jsonpath='{.spec.ports[0].targetPort}')" = 80 ]"""),
+  C("nodePort is 30080", 1, r"""[ "$($K -n svc get svc shop-svc -o jsonpath='{.spec.ports[0].nodePort}' 2>/dev/null)" = 30080 ]"""),
+  C("Service has endpoints", 2, r"""[ -n "$($K -n svc get endpointslices -l kubernetes.io/service-name=shop-svc -o jsonpath='{.items[*].endpoints[*].addresses[*]}' 2>/dev/null)" ]"""),
+ ],
+},
+{
+ "id": 13, "domain": "NET", "title": "Ingress routing",
+ "task": """
+Namespace `ingress-app` has Services `web` (port 80) and `api` (port 8080).
+
+Create an Ingress `app-ingress` in `ingress-app`:
+- `ingressClassName: nginx`
+- host `shop.example.com`
+- path `/api` (Prefix) -> service `api` port 8080
+- path `/`    (Prefix) -> service `web` port 80
+
+(No ingress controller is installed; you are graded on the object.)
+""",
+ "setup": NS + r"""
+mkns ingress-app
+$K -n ingress-app create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n ingress-app create deploy api --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n ingress-app expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n ingress-app expose deploy api --port=8080 --target-port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n ingress-app delete ingress app-ingress --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("ingressClassName nginx and host shop.example.com", 1, r"""i="$K -n ingress-app get ingress app-ingress"; [ "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" = nginx ] && [ "$($i -o jsonpath='{.spec.rules[0].host}')" = shop.example.com ]"""),
+  C("/api (Prefix) -> api:8080", 2, r"""i="$K -n ingress-app get ingress app-ingress"; P='{.spec.rules[0].http.paths[?(@.path=="/api")]'; [ "$($i -o jsonpath="$P.backend.service.name}" 2>/dev/null)" = api ] && [ "$($i -o jsonpath="$P.backend.service.port.number}")" = 8080 ] && [ "$($i -o jsonpath="$P.pathType}")" = Prefix ]"""),
+  C("/ (Prefix) -> web:80", 2, r"""i="$K -n ingress-app get ingress app-ingress"; P='{.spec.rules[0].http.paths[?(@.path=="/")]'; [ "$($i -o jsonpath="$P.backend.service.name}" 2>/dev/null)" = web ] && [ "$($i -o jsonpath="$P.backend.service.port.number}")" = 80 ] && [ "$($i -o jsonpath="$P.pathType}")" = Prefix ]"""),
+ ],
+},
+{
+ "id": 14, "domain": "NET", "title": "Cluster DNS",
+ "task": """
+Service `hello` exists in namespace `dns-check`.
+
+1. Run a pod `dns-test` (image `busybox:1.36`) in namespace `dns-check`.
+2. From inside it, resolve the Service using its **fully-qualified** DNS name.
+3. Save the FQDN you used (one line) to `{WS}/q14/svc-fqdn.txt`.
+4. Save the full `nslookup` output to `{WS}/q14/nslookup.txt`.
+""",
+ "setup": NS + r"""
+mkns dns-check
+rm -rf "$WS/q14"; mkdir -p "$WS/q14"
+$K -n dns-check create deploy hello --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n dns-check expose deploy hello --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K -n dns-check delete pod dns-test --ignore-not-found --wait=false >/dev/null
+""",
+ "checks": [
+  C("pod dns-test exists in dns-check", 1, r"""$K -n dns-check get pod dns-test >/dev/null 2>&1"""),
+  C("svc-fqdn.txt contains hello.dns-check.svc.cluster.local", 2, r"""grep -qx 'hello.dns-check.svc.cluster.local\.\?' "$WS/q14/svc-fqdn.txt" 2>/dev/null"""),
+  C("nslookup.txt contains the service ClusterIP", 1, r"""ip=$($K -n dns-check get svc hello -o jsonpath='{.spec.clusterIP}'); grep -q "$ip" "$WS/q14/nslookup.txt" 2>/dev/null"""),
+ ],
+},
+# ================================================================ STOR ====
+{
+ "id": 15, "domain": "STOR", "title": "PersistentVolume, claim and pod",
+ "task": """
+1. Create a PersistentVolume `pv-data`: 1Gi, `hostPath` `/mnt/data`,
+   accessMode `ReadWriteOnce`, `storageClassName: manual`, reclaim policy `Retain`.
+2. Create a PersistentVolumeClaim `pvc-data` in namespace `storage` requesting 500Mi
+   with `storageClassName: manual` so it binds to `pv-data`.
+3. Create pod `data-pod` (image `nginx:1.25`) in `storage` mounting the claim at `/usr/share/data`.
+""",
+ "setup": NS + r"""
+mkns storage
+$K -n storage delete pod data-pod --ignore-not-found --wait=false >/dev/null
+$K -n storage delete pvc pvc-data --ignore-not-found >/dev/null
+$K delete pv pv-data --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", 2, r"""
+pv="$K get pv pv-data"
+[ "$($pv -o jsonpath='{.spec.capacity.storage}' 2>/dev/null)" = 1Gi ] && [ "$($pv -o jsonpath='{.spec.hostPath.path}')" = /mnt/data ] && [ "$($pv -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')" = Retain ] && [ "$($pv -o jsonpath='{.spec.storageClassName}')" = manual ] && $pv -o jsonpath='{.spec.accessModes}' | grep -q ReadWriteOnce"""),
+  C("PVC pvc-data Bound to pv-data", 2, r"""[ "$($K -n storage get pvc pvc-data -o jsonpath='{.status.phase}' 2>/dev/null)" = Bound ] && [ "$($K -n storage get pvc pvc-data -o jsonpath='{.spec.volumeName}')" = pv-data ]"""),
+  C("data-pod Running with claim mounted at /usr/share/data", 2, r"""[ "$($K -n storage get pod data-pod -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && [ -n "$($K -n storage get pod data-pod -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim.claimName=="pvc-data")].name}')" ] && $K -n storage get pod data-pod -o jsonpath='{.spec.containers[0].volumeMounts[*].mountPath}' | grep -q /usr/share/data"""),
+ ],
+},
+{
+ "id": 16, "domain": "STOR", "title": "Default StorageClass and dynamic provisioning",
+ "task": """
+1. Create a StorageClass `fast-local`:
+   provisioner `rancher.io/local-path`, `volumeBindingMode: WaitForFirstConsumer`,
+   `reclaimPolicy: Delete`.
+2. Make `fast-local` the **default** StorageClass and ensure the existing
+   `standard` class is no longer default.
+3. Create PVC `fast-claim` (200Mi, RWO) in namespace `storage` **without** specifying a storageClassName.
+4. Create pod `fast-pod` (image `nginx:1.25`) in `storage` that mounts `fast-claim` at `/data`.
+   The claim must end up Bound.
+""",
+ "setup": NS + r"""
+mkns storage
+$K -n storage delete pod fast-pod --ignore-not-found --wait=false >/dev/null
+$K -n storage delete pvc fast-claim --ignore-not-found >/dev/null
+$K delete sc fast-local --ignore-not-found >/dev/null
+$K annotate sc standard storageclass.kubernetes.io/is-default-class=true --overwrite >/dev/null
+""",
+ "checks": [
+  C("StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", 1, r"""sc="$K get sc fast-local"; [ "$($sc -o jsonpath='{.provisioner}' 2>/dev/null)" = rancher.io/local-path ] && [ "$($sc -o jsonpath='{.volumeBindingMode}')" = WaitForFirstConsumer ] && [ "$($sc -o jsonpath='{.reclaimPolicy}')" = Delete ]"""),
+  C("fast-local is default and standard is not", 2, r"""[ "$($K get sc fast-local -o jsonpath='{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}' 2>/dev/null)" = true ] && [ "$($K get sc standard -o jsonpath='{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]"""),
+  C("fast-claim picked up fast-local and is Bound", 1, r"""[ "$($K -n storage get pvc fast-claim -o jsonpath='{.spec.storageClassName}' 2>/dev/null)" = fast-local ] && [ "$($K -n storage get pvc fast-claim -o jsonpath='{.status.phase}')" = Bound ]"""),
+  C("fast-pod Running with claim at /data", 1, r"""[ "$($K -n storage get pod fast-pod -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && $K -n storage get pod fast-pod -o jsonpath='{.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /data"""),
+ ],
+},
+# ============================================================== TSHOOT ====
+{
+ "id": 17, "domain": "TSHOOT", "breaks": "w2-kubelet", "title": "Node NotReady", "disruptive": True,
+ "task": """
+Node `{W2}` is `NotReady`.
+
+Investigate and fix it so the node returns to `Ready`. The fix must survive a
+reboot of the node (i.e. the responsible service must be enabled).
+
+Reach the node with: `{SHELL_W2}`
+""",
+ "setup": r"""nx "$W2" 'systemctl disable kubelet >/dev/null 2>&1; systemctl stop kubelet'""",
+ "checks": [
+  C("node is Ready", 4, r"""[ "$($K get node "$W2" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null)" = True ]"""),
+  C("kubelet is active", 2, r"""[ "$(nx "$W2" 'systemctl is-active kubelet')" = active ]"""),
+  C("kubelet is enabled (survives reboot)", 2, r"""[ "$(nx "$W2" 'systemctl is-enabled kubelet')" = enabled ]"""),
+ ],
+},
+{
+ "id": 18, "domain": "TSHOOT", "title": "Broken deployment and service",
+ "task": """
+The `payments` application in namespace `troubleshoot` is down.
+Users report that the Service `payments` returns no backends and the pods are not healthy.
+
+Fix everything necessary so that:
+- the `payments` Deployment has all replicas Available (the app is a plain nginx web server), and
+- the `payments` Service has endpoints.
+
+Do not delete the Deployment or the Service; fix them in place.
+""",
+ "setup": NS + r"""
+mkns troubleshoot
+$K apply -f - >/dev/null <<'EOF'
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: payments
+  namespace: troubleshoot
+spec:
+  replicas: 2
+  selector:
+    matchLabels: {app: payments}
+  template:
+    metadata:
+      labels: {app: payments}
+    spec:
+      containers:
+        - name: payments
+          image: nginx:1.99-nope
+          ports: [{containerPort: 80}]
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: payments
+  namespace: troubleshoot
+spec:
+  selector:
+    app: payment
+  ports:
+    - port: 80
+      targetPort: 8080
+EOF
+""",
+ "checks": [
+  C("all payments replicas available", 3, r"""d="$K -n troubleshoot get deploy payments"; r=$($d -o jsonpath='{.spec.replicas}' 2>/dev/null); a=$($d -o jsonpath='{.status.availableReplicas}'); [ -n "$a" ] && [ "$a" = "$r" ]"""),
+  C("Service payments has endpoints (selector fixed)", 2, r"""[ -n "$($K -n troubleshoot get endpointslices -l kubernetes.io/service-name=payments -o jsonpath='{.items[*].endpoints[*].addresses[*]}' 2>/dev/null)" ]"""),
+  C("Service targetPort matches container port 80", 2, r"""[ "$($K -n troubleshoot get svc payments -o jsonpath='{.spec.ports[0].targetPort}' 2>/dev/null)" = 80 ]"""),
+ ],
+},
+{
+ "id": 19, "domain": "TSHOOT", "title": "CrashLoopBackOff diagnosis",
+ "task": """
+Pod `crasher` in namespace `troubleshoot` keeps restarting.
+
+1. Find out why. Save the pod's (current or previous) container logs to
+   `{WS}/q19/crasher.log`.
+2. Fix the pod so it stays Running. The application must keep the same name,
+   namespace, image and command; it only needs the configuration it is complaining about
+   (use the value `db.internal` if a value is required).
+""",
+ "setup": NS + r"""
+mkns troubleshoot
+rm -rf "$WS/q19"; mkdir -p "$WS/q19"
+$K -n troubleshoot delete pod crasher --ignore-not-found --wait=true >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: Pod
+metadata:
+  name: crasher
+  namespace: troubleshoot
+spec:
+  nodeSelector:
+    kubernetes.io/hostname: $W3
+  containers:
+    - name: app
+      image: busybox:1.36
+      command: ["sh", "-c", "if [ -z \"\$DB_HOST\" ]; then echo 'FATAL: required environment variable DB_HOST is not set'; exit 1; fi; echo \"connected to \$DB_HOST\"; while true; do sleep 5; done"]
+EOF
+""",
+ "checks": [
+  C("crasher.log captured and contains the DB_HOST failure", 2, r"""grep -q DB_HOST "$WS/q19/crasher.log" 2>/dev/null"""),
+  C("pod crasher Running with DB_HOST set", 3, r"""[ "$($K -n troubleshoot get pod crasher -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && [ -n "$($K -n troubleshoot exec crasher -- sh -c 'echo $DB_HOST' 2>/dev/null)" ]"""),
+  C("image unchanged (busybox:1.36)", 1, r"""[ "$($K -n troubleshoot get pod crasher -o jsonpath='{.spec.containers[0].image}' 2>/dev/null)" = busybox:1.36 ]"""),
+ ],
+},
+{
+ "id": 20, "domain": "TSHOOT", "breaks": "scheduler", "title": "Control plane component failure", "disruptive": True,
+ "task": """
+New pods in the cluster are stuck in `Pending` and are never assigned to a node.
+
+Find the failing control-plane component and repair it on the control-plane node
+(`{SHELL_CP}`). Afterwards, newly created pods must be scheduled normally.
+""",
+ "setup": r"""
+nx "$CP" '
+f=/etc/kubernetes/manifests/kube-scheduler.yaml
+grep -q "^ *- kube-scheduler$" $f && sed -i "s|^\( *\)- kube-scheduler$|\1- kube-schedulerr|" $f
+' || true
+""",
+ "checks": [
+  C("kube-scheduler static pod manifest repaired", 3, r"""nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml'"""),
+  C("new pods get scheduled", 5, r"""
+$K -n default delete pod sched-probe --ignore-not-found --wait=true >/dev/null 2>&1
+$K -n default run sched-probe --image=busybox:1.36 --restart=Never -- sleep 300 >/dev/null 2>&1 || exit 1
+n=""; for i in $(seq 1 30); do n=$($K -n default get pod sched-probe -o jsonpath='{.spec.nodeName}' 2>/dev/null); [ -n "$n" ] && break; sleep 2; done
+$K -n default delete pod sched-probe --wait=false >/dev/null 2>&1
+[ -n "$n" ]"""),
+ ],
+},
+]
+
+# ---- aggregate the banks: 1-20 here, 21-43 in bank_extra, 101-122 (hard) in bank_hard
+from tracks.cka import bank_extra, bank_hard, bank_sim_a, bank_sim_b
+QUESTIONS = QUESTIONS + bank_extra.QUESTIONS + bank_hard.QUESTIONS + bank_sim_a.QUESTIONS + bank_sim_b.QUESTIONS
+QUESTIONS.sort(key=lambda q: q["id"])
+_ids = [q["id"] for q in QUESTIONS]
+assert len(_ids) == len(set(_ids)), "duplicate question ids"
+from tracks.cka.solutions import S as _SOL
+for _q in QUESTIONS:
+    _q["solution"] = (_q.get("solution") or _SOL.get(_q["id"], "")).strip()
+    _q.setdefault("set", None)
+_missing = [q["id"] for q in QUESTIONS if not q["solution"]]
+assert not _missing, f"tasks without a solution: {_missing}"

tracks/cka/solutions.py

@@ -0,0 +1,906 @@
+"""Reference solutions, keyed by task id. Rendered with the same %{param} / {WS} / {W1} substitution
+as the task text. Shown only after a task has been graded (drills) or the session submitted (exams).
+These are one correct way, written the way you'd type them under time pressure: imperative first,
+YAML only where imperative can't express it."""
+
+S = {}
+
+# ================================================================= ARCH ====
+S[1] = """
+k -n rbac-app create sa deploy-bot
+k -n rbac-app create role deploy-manager --verb=get,list,watch,create,update,patch,delete --resource=deployments.apps
+k -n rbac-app create rolebinding deploy-bot-binding --role=deploy-manager --serviceaccount=rbac-app:deploy-bot
+k auth can-i create deployments -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot   # yes
+k auth can-i delete pods -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot          # no
+"""
+S[2] = """
+{SHELL_CP}
+ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \\
+  --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \\
+  snapshot save /opt/cka/etcd-snapshot.db
+etcdutl snapshot status /opt/cka/etcd-snapshot.db -w table
+# If etcdctl is only inside the pod: run the same command via
+#   kubectl -n kube-system exec etcd-{CP} -- etcdctl ... snapshot save /var/lib/etcd/snap.db
+# then on the node: mv /var/lib/etcd/snap.db /opt/cka/etcd-snapshot.db  (the pod's /var/lib/etcd is the node's /var/lib/etcd)
+"""
+S[3] = """
+mkdir -p {WS}/q03/overlays/prod && cd {WS}/q03/overlays/prod
+cat > namespace.yaml <<EOF
+apiVersion: v1
+kind: Namespace
+metadata: {name: store-prod}
+EOF
+cat > kustomization.yaml <<EOF
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: store-prod
+resources: [../../base, namespace.yaml]
+labels:
+  - pairs: {env: prod}
+    includeSelectors: false
+replicas:
+  - name: store-api
+    count: 3
+EOF
+kubectl apply -k .
+"""
+S[4] = """
+cat <<EOF | k apply -f -
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata: {name: backups.data.example.com}
+spec:
+  group: data.example.com
+  scope: Namespaced
+  names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]}
+  versions:
+    - name: v1
+      served: true
+      storage: true
+      schema:
+        openAPIV3Schema:
+          type: object
+          properties:
+            spec:
+              type: object
+              properties:
+                schedule: {type: string}
+                target: {type: string}
+EOF
+cat <<EOF | k apply -f -
+apiVersion: data.example.com/v1
+kind: Backup
+metadata: {name: nightly, namespace: default}
+spec: {schedule: "0 2 * * *", target: "s3://backups/prod"}
+EOF
+k get bk
+"""
+S[5] = """
+{SHELL_W3}
+grep staticPodPath /var/lib/kubelet/config.yaml        # /etc/kubernetes/manifests
+cat > /etc/kubernetes/manifests/static-web.yaml <<EOF
+apiVersion: v1
+kind: Pod
+metadata: {name: static-web, namespace: default}
+spec:
+  containers: [{name: web, image: nginx:1.27}]
+EOF
+# kubelet picks it up within seconds; mirror pod is static-web-{W3}
+"""
+S[6] = """
+# Part A
+{SHELL_CP}
+kubeadm certs check-expiration | grep '^apiserver '   >  (copy the line into {WS}/q06/apiserver-expiry.txt)
+# Part B
+openssl genrsa -out jane.key 2048
+openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr
+cat <<EOF | k apply -f -
+apiVersion: certificates.k8s.io/v1
+kind: CertificateSigningRequest
+metadata: {name: jane}
+spec:
+  request: $(base64 -w0 jane.csr)
+  signerName: kubernetes.io/kube-apiserver-client
+  usages: [client auth]
+EOF
+k certificate approve jane
+k get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt
+k -n default create rolebinding jane-view --clusterrole=view --user=jane
+"""
+S[21] = """
+k create ns %{ns}
+helm install %{rel} {WS}/q21/chart -n %{ns} --set replicaCount=%{n}
+helm upgrade %{rel} {WS}/q21/chart -n %{ns} --reuse-values --set service.type=NodePort
+helm -n %{ns} history %{rel}     # revision 2
+"""
+S[33] = """
+cat <<EOF | k apply -f -
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata: {name: %{cr}}
+aggregationRule:
+  clusterRoleSelectors:
+    - matchLabels: {rbac.example.com/aggregate-to-%{cr}: "true"}
+rules: []
+EOF
+k create clusterrole %{cr}-pods --verb=get,list,watch --resource=pods
+k create clusterrole %{cr}-services --verb=get,list,watch --resource=services
+k label clusterrole %{cr}-pods %{cr}-services rbac.example.com/aggregate-to-%{cr}=true
+k create clusterrolebinding %{cr}-binding --clusterrole=%{cr} --user=%{user}
+k get clusterrole %{cr} -o yaml      # rules now populated by the controller
+"""
+S[34] = """
+k explain %{kind}.spec --recursive        # find the field, then:
+k explain %{kind}.spec.tuning.%{field}    # confirm the description
+echo spec.tuning.%{field} > {WS}/q34/path.txt
+cat <<EOF | k apply -f -
+apiVersion: demo.io/v1
+kind: %{kind}
+metadata: {name: sample, namespace: default}
+spec:
+  tuning: {%{field}: %{n}}
+EOF
+"""
+S[40] = """
+{SHELL_CP}
+kubeadm token create --ttl %{ttl} --print-join-command
+# copy the printed line into {WS}/q40/join.sh
+kubeadm token list
+"""
+S[41] = """
+k version                                            # Server Version -> apiserver=
+k get node %{node} -o jsonpath='{.status.nodeInfo.kubeletVersion}{"\\n"}{.status.nodeInfo.containerRuntimeVersion}{"\\n"}'
+k -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].spec.containers[0].image}'   # tag after the colon
+"""
+# --------------------------------------------------------------- hard ARCH
+S[101] = """
+k get clusterrolebinding -o wide | grep %{sa}          # find the cluster-admin binding -> delete it
+k delete clusterrolebinding %{sa}-legacy-admin
+k -n %{ns} create role %{sa}-role --verb=get,list,watch,patch --resource=deployments.apps,replicasets.apps
+k -n %{ns} patch role %{sa}-role --type=json -p='[{"op":"add","path":"/rules/-","value":{"apiGroups":[""],"resources":["pods/exec"],"verbs":["create"]}}]'
+k -n %{ns} create rolebinding %{sa}-rb --role=%{sa}-role --serviceaccount=%{ns}:%{sa}
+k create clusterrole %{sa}-nodes --verb=list --resource=nodes
+k create clusterrolebinding %{sa}-crb --clusterrole=%{sa}-nodes --serviceaccount=%{ns}:%{sa}
+k auth can-i --list -n %{ns} --as=system:serviceaccount:%{ns}:%{sa}
+"""
+S[102] = """
+{SHELL_CP}
+etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore
+# point the etcd static pod at the new dir: the hostPath volume, NOT just --data-dir
+vi /etc/kubernetes/manifests/etcd.yaml      # volumes: - hostPath: path: /var/lib/etcd  ->  /var/lib/etcd-restore
+# kubelet restarts etcd; apiserver reconnects in ~30-60s
+watch crictl ps                              # etcd + kube-apiserver come back
+kubectl -n default get cm %{cm}              # restored
+"""
+S[103] = """
+k get nodes -l tier=edge                                    # -> the node
+{SHELL_W3}
+grep staticPodPath /var/lib/kubelet/config.yaml             # NOT /etc/kubernetes/manifests here
+cat > <staticPodPath>/%{pod}.yaml <<EOF
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}, labels: {role: edge}}
+spec:
+  containers:
+    - name: web
+      image: nginx:1.27
+      resources: {requests: {cpu: 50m, memory: 32Mi}}
+EOF
+"""
+S[104] = """
+openssl genrsa -out %{user}.key 2048
+openssl req -new -key %{user}.key -subj "/CN=%{user}/O=developers" -out %{user}.csr
+k apply -f - <<EOF
+apiVersion: certificates.k8s.io/v1
+kind: CertificateSigningRequest
+metadata: {name: %{user}}
+spec: {request: $(base64 -w0 %{user}.csr), signerName: kubernetes.io/kube-apiserver-client, usages: [client auth]}
+EOF
+k certificate approve %{user}
+k get csr %{user} -o jsonpath='{.status.certificate}' | base64 -d > %{user}.crt
+k -n %{ns} create rolebinding %{user}-edit --clusterrole=edit --user=%{user}
+# self-contained kubeconfig (--embed-certs)
+KC={WS}/q104/%{user}.kubeconfig
+k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
+SERVER=$(k config view --minify -o jsonpath='{.clusters[0].cluster.server}')
+kubectl --kubeconfig $KC config set-cluster lab --server=$SERVER --certificate-authority=ca.crt --embed-certs
+kubectl --kubeconfig $KC config set-credentials %{user} --client-certificate=%{user}.crt --client-key=%{user}.key --embed-certs
+kubectl --kubeconfig $KC config set-context %{user}@lab --cluster=lab --user=%{user} --namespace=%{ns}
+kubectl --kubeconfig $KC config use-context %{user}@lab
+kubectl --kubeconfig $KC get pods
+"""
+S[105] = """
+mkdir -p {WS}/q105/overlays/prod && cd {WS}/q105/overlays/prod
+cat > replicas.yaml <<EOF
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: web}
+spec: {replicas: %{rep}}
+EOF
+cat > envfrom.yaml <<EOF
+- op: add
+  path: /spec/template/spec/containers/0/envFrom
+  value: [{configMapRef: {name: web-config}}]
+EOF
+cat > kustomization.yaml <<EOF
+namespace: %{ns}
+resources: [../../base]
+labels:
+  - pairs: {tier: prod}
+    includeSelectors: false
+configMapGenerator:
+  - name: web-config
+    literals: [LOG_LEVEL=%{lvl}]
+patches:
+  - path: replicas.yaml
+  - path: envfrom.yaml
+    target: {kind: Deployment, name: web}
+EOF
+k create ns %{ns}; k apply -k .        # kustomize rewrites web-config -> web-config-<hash> in the envFrom too
+"""
+S[106] = """
+cat <<EOF | k apply -f -
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata: {name: $(echo %{kind} | tr A-Z a-z)s.deploy.example.com}
+spec:
+  group: deploy.example.com
+  scope: Namespaced
+  names: {kind: %{kind}, plural: $(echo %{kind} | tr A-Z a-z)s, singular: $(echo %{kind} | tr A-Z a-z)}
+  versions:
+    - name: v1
+      served: true
+      storage: true
+      schema:
+        openAPIV3Schema:
+          type: object
+          properties:
+            spec:
+              type: object
+              required: [replicas, image]
+              properties:
+                replicas: {type: integer, minimum: 1, maximum: 10}
+                environment: {type: string, enum: [prod, stage, dev], default: dev}
+                image: {type: string}
+EOF
+k apply -f - <<EOF
+apiVersion: deploy.example.com/v1
+kind: %{kind}
+metadata: {name: good}
+spec: {replicas: 3, image: nginx:1.27, environment: %{env}}
+EOF
+printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: bad}\\nspec: {replicas: 0, image: x}\\n' | k apply -f - 2> {WS}/q106/rejected.txt
+printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: defaulted}\\nspec: {replicas: 2, image: nginx:1.27}\\n' | k apply -f -
+"""
+S[122] = """
+{SHELL_CP}
+kubeadm certs renew all
+# static pods do NOT reload certs by themselves: bounce them
+mv /etc/kubernetes/manifests/*.yaml /tmp/ && sleep 20 && mv /tmp/*.yaml /etc/kubernetes/manifests/
+# admin.conf was regenerated by `renew all`; use it:
+cp /etc/kubernetes/admin.conf ~/.kube/config
+kubeadm certs check-expiration | grep '^apiserver '    # -> {WS}/q122/expiry.txt
+"""
+
+# ================================================================= WORK ====
+S[7] = """
+k -n apps set image deploy web-frontend nginx=nginx:1.27
+k -n apps scale deploy web-frontend --replicas=4
+k -n apps patch deploy web-frontend -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
+k -n apps rollout status deploy web-frontend
+"""
+S[8] = """
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: log-shipper, namespace: apps}
+spec:
+  volumes: [{name: logs, emptyDir: {}}]
+  containers:
+    - name: app
+      image: busybox:1.36
+      command: [sh, -c, "while true; do date >> /var/log/app.log; sleep 2; done"]
+      volumeMounts: [{name: logs, mountPath: /var/log}]
+    - name: shipper
+      image: busybox:1.36
+      command: [sh, -c, "tail -f /var/log/app.log"]
+      volumeMounts: [{name: logs, mountPath: /var/log}]
+EOF
+"""
+S[9] = """
+k -n apps create deploy cache --image=redis:7-alpine --replicas=2 $do > cache.yaml
+# edit: add under spec.template.spec
+#   affinity:
+#     nodeAffinity:
+#       requiredDuringSchedulingIgnoredDuringExecution:
+#         nodeSelectorTerms:
+#           - matchExpressions: [{key: disktype, operator: In, values: [ssd]}]
+# and under the container: resources: {requests: {cpu: 100m, memory: 64Mi}, limits: {cpu: 200m, memory: 128Mi}}
+k apply -f cache.yaml
+"""
+S[10] = """
+k -n apps create cm app-config --from-literal=APP_MODE=production --from-literal=LOG_LEVEL=debug
+k -n apps create secret generic db-creds --from-literal=username=admin --from-literal='password=S3cr3t!'
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: config-pod, namespace: apps}
+spec:
+  volumes: [{name: cfg, configMap: {name: app-config}}]
+  containers:
+    - name: app
+      image: busybox:1.36
+      command: [sleep, "3600"]
+      env: [{name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: APP_MODE}}}]
+      envFrom: [{secretRef: {name: db-creds}}]
+      volumeMounts: [{name: cfg, mountPath: /etc/config}]
+EOF
+"""
+S[23] = """
+k -n kube-system edit deploy metrics-server      # args: add  - --kubelet-insecure-tls
+k top nodes                                      # ~30s later
+k -n %{ns} autoscale deploy %{dep} --min=2 --max=%{max} --cpu-percent=%{cpu} --name=%{dep}
+"""
+S[24] = """
+cat <<EOF | k apply -f -
+apiVersion: scheduling.k8s.io/v1
+kind: PriorityClass
+metadata: {name: %{pc}}
+value: %{val}
+globalDefault: false
+description: "%{pc} workloads"
+EOF
+k -n %{ns} patch deploy payments -p '{"spec":{"template":{"spec":{"priorityClassName":"%{pc}"}}}}'
+k -n %{ns} rollout status deploy payments
+"""
+S[25] = """
+k taint node {W1} %{key}=%{val}:NoSchedule
+k -n %{ns} create deploy %{dep} --image=busybox:1.36 --replicas=2 $do -- sleep 3600 > d.yaml
+# add under spec.template.spec:
+#   nodeSelector: {kubernetes.io/hostname: {W1}}
+#   tolerations: [{key: %{key}, operator: Equal, value: %{val}, effect: NoSchedule}]
+k apply -f d.yaml
+"""
+S[26] = """
+cat <<EOF | k apply -f -
+apiVersion: apps/v1
+kind: DaemonSet
+metadata: {name: %{ds}, namespace: %{ns}}
+spec:
+  selector: {matchLabels: {app: %{ds}}}
+  template:
+    metadata: {labels: {app: %{ds}}}
+    spec:
+      tolerations: [{key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule}]
+      containers:
+        - name: agent
+          image: busybox:1.36
+          command: [sleep, "3600"]
+          resources: {requests: {cpu: 10m, memory: 16Mi}}
+EOF
+"""
+S[27] = """
+k -n %{ns} create cj %{cj} --image=busybox:1.36 --schedule='*/%{every} * * * *' $do -- date > cj.yaml
+# set: spec.concurrencyPolicy: Forbid, successfulJobsHistoryLimit: 3, failedJobsHistoryLimit: 1,
+#      spec.jobTemplate.spec.activeDeadlineSeconds: 30, ...template.spec.restartPolicy: Never
+k apply -f cj.yaml
+k -n %{ns} create job %{cj}-manual --from=cronjob/%{cj}
+"""
+S[28] = """
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  volumes: [{name: content, emptyDir: {}}]
+  initContainers:
+    - name: setup
+      image: busybox:1.36
+      command: [sh, -c, "echo '%{text}' > /work/index.html"]
+      volumeMounts: [{name: content, mountPath: /work}]
+  containers:
+    - name: web
+      image: nginx:1.25
+      volumeMounts: [{name: content, mountPath: /usr/share/nginx/html}]
+EOF
+"""
+S[29] = """
+k -n %{ns} create cm %{cm}-v2 --from-literal=mode=new
+k -n %{ns} patch cm %{cm}-v2 -p '{"immutable":true}'          # optional, keeps the pattern
+k -n %{ns} set volume deploy %{dep} --add=false 2>/dev/null; \\
+k -n %{ns} patch deploy %{dep} --type=json -p='[{"op":"replace","path":"/spec/template/spec/volumes/0/configMap/name","value":"%{cm}-v2"}]'
+k -n %{ns} rollout status deploy %{dep}
+"""
+S[31] = """
+k -n %{ns} create job %{job} --image=busybox:1.36 $do -- sh -c "echo done; sleep 2" > job.yaml
+# spec: completions: %{c}, parallelism: %{p}, backoffLimit: 2  (restartPolicy Never is the default from create job)
+k apply -f job.yaml && k -n %{ns} wait --for=condition=complete job/%{job} --timeout=120s
+"""
+S[32] = """
+k -n %{ns} create quota %{ns}-quota --hard=pods=%{pods},requests.cpu=1,limits.memory=2Gi
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: LimitRange
+metadata: {name: %{ns}-defaults, namespace: %{ns}}
+spec:
+  limits:
+    - type: Container
+      defaultRequest: {cpu: %{dcpu}, memory: 64Mi}
+      default: {cpu: 500m, memory: 256Mi}
+EOF
+k -n %{ns} create deploy worker --image=nginx:1.25 --replicas=2
+"""
+# --------------------------------------------------------------- hard WORK
+S[107] = """
+k -n %{ns} patch quota %{ns}-quota -p '{"spec":{"hard":{"pods":"3","requests.cpu":"400m"}}}'   # keep a cap, make room
+cat <<EOF | k apply -f -
+apiVersion: scheduling.k8s.io/v1
+kind: PriorityClass
+metadata: {name: %{dep}-prio}
+value: 50000
+EOF
+k -n %{ns} create deploy %{dep} --image=nginx:1.27 --replicas=3 $do > d.yaml
+# spec.template.spec:
+#   priorityClassName: %{dep}-prio
+#   tolerations: [{key: %{key}, operator: Equal, value: %{val}, effect: NoSchedule}]
+#   affinity:
+#     podAntiAffinity:
+#       requiredDuringSchedulingIgnoredDuringExecution:
+#         - labelSelector: {matchLabels: {app: %{dep}}}
+#           topologyKey: kubernetes.io/hostname
+# container resources: {requests: {cpu: 100m, memory: 64Mi}}
+k apply -f d.yaml
+"""
+S[108] = """
+k -n %{ns} rollout history deploy %{dep}                     # find the revision whose image is nginx:1.25
+k -n %{ns} rollout history deploy %{dep} --revision=1        # confirm
+k -n %{ns} rollout undo deploy %{dep} --to-revision=1
+k -n %{ns} scale deploy %{dep} --replicas=3 && k -n %{ns} rollout status deploy %{dep}
+k -n %{ns} rollout pause deploy %{dep}
+k -n %{ns} set image deploy %{dep} nginx=nginx:1.27
+k -n %{ns} set env deploy %{dep} RELEASE=next
+k -n %{ns} get rs                                             # no new RS while paused
+"""
+S[109] = """
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  terminationGracePeriodSeconds: 5
+  volumes: [{name: logs, emptyDir: {}}]
+  initContainers:
+    - name: agent
+      image: busybox:1.36
+      restartPolicy: Always                      # <- native sidecar
+      command: [sh, -c, "while true; do date >> /var/log/agent.log; sleep 1; done"]
+      volumeMounts: [{name: logs, mountPath: /var/log}]
+  containers:
+    - name: app
+      image: nginx:1.27
+      volumeMounts: [{name: logs, mountPath: /var/log}]
+      readinessProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5}
+      livenessProbe: {tcpSocket: {port: 80}}
+EOF
+k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name} {.spec.containers[*].name}' | tr ' ' '\\n' | sort > {WS}/q109/containers.txt
+k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' > {WS}/q109/ip.txt
+"""
+S[110] = """
+k get pods -A -o json | jq -r '.items[] | select(.metadata.namespace!="kube-system") | select(.spec.containers[0].readinessProbe==null) | "\\(.metadata.namespace)/\\(.metadata.name)"' | sort > {WS}/q110/no-readiness.txt
+k get pods -A --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\\n"}{end}' | sort | uniq -c | awk '{print $2, $1}' > {WS}/q110/per-node.txt
+k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\\n' | sort -u > {WS}/q110/images.txt
+# no jq? use: k get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,RP:.spec.containers[0].readinessProbe --no-headers | awk '$3=="<none>" && $1!="kube-system"{print $1"/"$2}' | sort
+"""
+S[121] = """
+k config use-context k8s-c2-AC
+k config view --minify -o jsonpath='{..namespace}'         # -> %{ns}, write it to {WS}/q121/ns.txt
+k create deploy %{dep} --image=busybox:1.36 --replicas=2 -- sleep 3600
+k label deploy %{dep} owner=ac && k patch deploy %{dep} -p '{"spec":{"template":{"metadata":{"labels":{"owner":"ac"}}}}}'
+k expose deploy %{dep} --port=9000 --target-port=9000 --name=%{dep}
+"""
+
+# ================================================================== NET ====
+S[11] = """
+cat <<EOF | k apply -f -
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: backend-policy, namespace: netpol}
+spec:
+  podSelector: {matchLabels: {app: backend}}
+  policyTypes: [Ingress]
+  ingress:
+    - from: [{podSelector: {matchLabels: {app: frontend}}}]
+      ports: [{protocol: TCP, port: 8080}]
+EOF
+"""
+S[12] = """
+k -n svc expose deploy shop --name=shop-svc --type=NodePort --port=80 --target-port=80 $do > svc.yaml
+# set spec.ports[0].nodePort: 30080
+k apply -f svc.yaml && k -n svc get endpointslices
+"""
+S[13] = """
+k -n ingress-app create ingress app-ingress --class=nginx \\
+  --rule="shop.example.com/api*=api:8080" --rule="shop.example.com/*=web:80"
+# `*` suffix -> pathType Prefix. Verify: k -n ingress-app get ingress app-ingress -o yaml
+"""
+S[14] = """
+k -n dns-check run dns-test --image=busybox:1.36 -- sleep 3600
+k -n dns-check exec dns-test -- nslookup hello.dns-check.svc.cluster.local | tee {WS}/q14/nslookup.txt
+echo hello.dns-check.svc.cluster.local > {WS}/q14/svc-fqdn.txt
+"""
+S[22] = """
+k apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
+cat <<EOF | k apply -f -
+apiVersion: gateway.networking.k8s.io/v1
+kind: GatewayClass
+metadata: {name: %{gc}}
+spec: {controllerName: example.com/gateway}
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata: {name: %{gw}, namespace: %{ns}}
+spec:
+  gatewayClassName: %{gc}
+  listeners: [{name: http, protocol: HTTP, port: 80, hostname: %{host}.example.com}]
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata: {name: %{host}-route, namespace: %{ns}}
+spec:
+  parentRefs: [{name: %{gw}}]
+  rules:
+    - matches: [{path: {type: PathPrefix, value: /api}}]
+      backendRefs: [{name: api, port: 8080}]
+    - backendRefs: [{name: web, port: 80}]
+EOF
+"""
+S[30] = """
+k -n %{ns} create deploy %{dep} --image=nginx:1.25 $do > d.yaml
+# container: ports: [{name: %{pname}, containerPort: 80}]
+#            readinessProbe: {httpGet: {path: /, port: %{pname}}}
+k apply -f d.yaml
+k -n %{ns} expose deploy %{dep} --port=8080 --target-port=%{pname}
+"""
+S[42] = """
+cat <<EOF | k apply -f -
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: %{app}-egress, namespace: %{ns}}
+spec:
+  podSelector: {matchLabels: {app: %{app}}}
+  policyTypes: [Egress]
+  egress:
+    - to:
+        - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}
+          podSelector: {matchLabels: {k8s-app: kube-dns}}
+      ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
+    - to: [{ipBlock: {cidr: %{cidr}}}]
+      ports: [{protocol: TCP, port: 443}]
+EOF
+"""
+S[43] = """
+k -n %{ns} create svc clusterip %{sts} --clusterip=None --tcp=5432:5432 $do > h.yaml   # fix selector: app: %{sts}
+k apply -f h.yaml
+k -n %{ns} create svc externalname legacy --external-name=%{ext}
+k -n %{ns} exec %{sts}-0 -- nslookup %{sts}-0.%{sts} > {WS}/q43/pod0.txt
+"""
+# --------------------------------------------------------------- hard NET
+S[111] = """
+k -n %{ns} delete netpol temp-allow-all
+k label ns %{from} team=%{from}
+cat <<EOF | k apply -f -
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: default-deny, namespace: %{ns}}
+spec: {podSelector: {}, policyTypes: [Ingress, Egress]}
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: allow-dns, namespace: %{ns}}
+spec:
+  podSelector: {}
+  policyTypes: [Egress]
+  egress:
+    - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}}]
+      ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: api-ingress, namespace: %{ns}}
+spec:
+  podSelector: {matchLabels: {app: api}}
+  policyTypes: [Ingress]
+  ingress:
+    - from: [{namespaceSelector: {matchLabels: {team: %{from}}}}]
+      ports: [{port: 8080}]
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: api-to-db, namespace: %{ns}}
+spec:
+  podSelector: {matchLabels: {app: api}}
+  policyTypes: [Egress]
+  egress:
+    - to: [{podSelector: {matchLabels: {app: db}}}]
+      ports: [{port: 5432}]
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: db-ingress, namespace: %{ns}}
+spec:
+  podSelector: {matchLabels: {app: db}}
+  policyTypes: [Ingress]
+  ingress:
+    - from: [{podSelector: {matchLabels: {app: api}}}]
+      ports: [{port: 5432}]
+EOF
+"""
+S[112] = """
+openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=%{host}" -keyout tls.key -out tls.crt
+k -n %{ns} create secret tls %{host}-tls --cert=tls.crt --key=tls.key
+k -n %{ns} create ingress web --class=nginx --rule="%{host}/*=web:80,tls=%{host}-tls" \\
+  --annotation nginx.ingress.kubernetes.io/ssl-redirect=true
+openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > {WS}/q112/fp.txt
+"""
+S[113] = """
+cat <<EOF | k apply -f -
+apiVersion: gateway.networking.k8s.io/v1
+kind: GatewayClass
+metadata: {name: %{host}-gc}
+spec: {controllerName: example.com/gw}
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata: {name: %{host}-gw, namespace: %{ns}}
+spec:
+  gatewayClassName: %{host}-gc
+  listeners:
+    - {name: web, protocol: HTTP, port: 80, hostname: %{host}.example.com}
+    - {name: api, protocol: HTTP, port: 80, hostname: api.%{host}.example.com}
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata: {name: %{host}-route, namespace: %{ns}}
+spec:
+  parentRefs: [{name: %{host}-gw, sectionName: web}]
+  rules:
+    - matches: [{headers: [{name: X-Canary, value: always}]}]
+      backendRefs: [{name: v2, port: 80}]
+    - backendRefs:
+        - {name: v1, port: 80, weight: %{w2}}
+        - {name: v2, port: 80, weight: %{w}}
+EOF
+"""
+S[114] = """
+k -n kube-system get netpol && k -n kube-system delete netpol maintenance-lockdown
+k -n kube-system get deploy coredns                         # 0/0 -> scale it
+k -n kube-system scale deploy coredns --replicas=2
+k -n kube-system edit cm coredns                            # forward . 10.255.255.254 -> forward . /etc/resolv.conf
+k -n kube-system rollout restart deploy coredns
+k run t --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
+"""
+
+# ================================================================= STOR ====
+S[15] = """
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: PersistentVolume
+metadata: {name: pv-data}
+spec:
+  capacity: {storage: 1Gi}
+  accessModes: [ReadWriteOnce]
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: manual
+  hostPath: {path: /mnt/data}
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: pvc-data, namespace: storage}
+spec: {accessModes: [ReadWriteOnce], storageClassName: manual, resources: {requests: {storage: 500Mi}}}
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: data-pod, namespace: storage}
+spec:
+  volumes: [{name: d, persistentVolumeClaim: {claimName: pvc-data}}]
+  containers: [{name: web, image: nginx:1.25, volumeMounts: [{name: d, mountPath: /usr/share/data}]}]
+EOF
+"""
+S[16] = """
+cat <<EOF | k apply -f -
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: fast-local
+  annotations: {storageclass.kubernetes.io/is-default-class: "true"}
+provisioner: rancher.io/local-path
+volumeBindingMode: WaitForFirstConsumer
+reclaimPolicy: Delete
+EOF
+k annotate sc standard storageclass.kubernetes.io/is-default-class-
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: fast-claim, namespace: storage}
+spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 200Mi}}}
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: fast-pod, namespace: storage}
+spec:
+  volumes: [{name: d, persistentVolumeClaim: {claimName: fast-claim}}]
+  containers: [{name: web, image: nginx:1.25, volumeMounts: [{name: d, mountPath: /data}]}]
+EOF
+"""
+S[35] = """
+k get pv %{pv} -o yaml | grep -A5 claimRef        # points at the deleted PVC -> remove it
+k patch pv %{pv} --type=json -p='[{"op":"remove","path":"/spec/claimRef"}]'
+k get pv %{pv}                                    # Available
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: %{pvc}, namespace: %{ns}}
+spec: {accessModes: [ReadWriteOnce], storageClassName: manual, volumeName: %{pv}, resources: {requests: {storage: 500Mi}}}
+EOF
+"""
+S[36] = """
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: %{pvc}, namespace: %{ns}}
+spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 200Mi}}}
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: writer, namespace: %{ns}}
+spec:
+  restartPolicy: Never
+  volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
+  containers: [{name: w, image: busybox:1.36, command: [sh, -c, "echo %{text} > /data/marker.txt"], volumeMounts: [{name: d, mountPath: /data}]}]
+EOF
+k -n %{ns} wait --for=jsonpath='{.status.phase}'=Succeeded pod/writer
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: reader, namespace: %{ns}}
+spec:
+  volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
+  containers: [{name: r, image: busybox:1.36, command: [sleep, "3600"], volumeMounts: [{name: d, mountPath: /data}]}]
+EOF
+"""
+# --------------------------------------------------------------- hard STOR
+S[115] = """
+k get pv -l media --show-labels; k get node -l disk=nvme        # lv-nvme is on {W2}
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: %{pvc}, namespace: %{ns}}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: local-fast
+  selector: {matchLabels: {media: nvme}}
+  resources: {requests: {storage: 1Gi}}
+---
+apiVersion: v1
+kind: Pod
+metadata: {name: consumer, namespace: %{ns}}
+spec:
+  volumes: [{name: d, persistentVolumeClaim: {claimName: %{pvc}}}]
+  containers: [{name: c, image: busybox:1.36, command: [sleep, "3600"], volumeMounts: [{name: d, mountPath: /data}]}]
+EOF
+# the PV's nodeAffinity pulls the pod to {W2} - no nodeSelector needed
+"""
+S[116] = """
+k -n %{ns} apply -f - <<EOF
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: new-data, namespace: %{ns}}
+spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 1Gi}}}
+---
+apiVersion: batch/v1
+kind: Job
+metadata: {name: migrate, namespace: %{ns}}
+spec:
+  template:
+    spec:
+      restartPolicy: Never
+      volumes:
+        - {name: src, persistentVolumeClaim: {claimName: old-data}}
+        - {name: dst, persistentVolumeClaim: {claimName: new-data}}
+      containers:
+        - name: cp
+          image: busybox:1.36
+          command: [sh, -c, "cp -a /src/. /dst/ && ls /dst"]
+          volumeMounts: [{name: src, mountPath: /src}, {name: dst, mountPath: /dst}]
+EOF
+# RWO: the Job must land on the same node as `legacy` if the PV is node-bound; local-path is, so if it Pending -> add nodeName
+k -n %{ns} wait --for=condition=complete job/migrate --timeout=120s
+k -n %{ns} get pod legacy -o yaml > legacy.yaml   # change claimName old-data -> new-data, strip status/uid/resourceVersion
+k -n %{ns} delete pod legacy --now && k apply -f legacy.yaml
+k get pv $(k -n %{ns} get pvc old-data -o jsonpath='{.spec.volumeName}') -o jsonpath='{.spec.persistentVolumeReclaimPolicy}'   # Retain
+k -n %{ns} delete pvc old-data
+"""
+
+# =============================================================== TSHOOT ====
+S[17] = """
+k get node {W2}                       # NotReady
+{SHELL_W2}
+systemctl status kubelet              # inactive (dead), disabled
+systemctl enable --now kubelet
+journalctl -u kubelet -f              # until "Successfully registered node"
+"""
+S[18] = """
+k -n troubleshoot describe deploy payments | grep -i image        # nginx:1.99-nope -> ErrImagePull
+k -n troubleshoot set image deploy payments payments=nginx:1.27
+k -n troubleshoot get svc payments -o yaml                       # selector app: payment (typo), targetPort 8080 (container is 80)
+k -n troubleshoot patch svc payments -p '{"spec":{"selector":{"app":"payments"},"ports":[{"port":80,"targetPort":80}]}}'
+k -n troubleshoot get endpointslices
+"""
+S[19] = """
+k -n troubleshoot logs crasher --previous | tee {WS}/q19/crasher.log     # FATAL: ... DB_HOST is not set
+k -n troubleshoot get pod crasher -o yaml > crasher.yaml
+# add under the container:  env: [{name: DB_HOST, value: db.internal}]   (strip status/uid/resourceVersion)
+k -n troubleshoot delete pod crasher --now && k apply -f crasher.yaml
+"""
+S[20] = """
+k -n kube-system get pods                                  # kube-scheduler-... CrashLoopBackOff / Error
+k -n kube-system logs kube-scheduler-{CP}                   # exec: "kube-schedulerr": not found
+{SHELL_CP}
+vi /etc/kubernetes/manifests/kube-scheduler.yaml           # command: - kube-schedulerr -> - kube-scheduler
+# kubelet restarts it; k -n kube-system get pods -w
+"""
+S[37] = """
+k -n kube-system get pods -l k8s-app=kube-dns; k -n kube-system logs -l k8s-app=kube-dns --tail=20
+k -n kube-system get cm coredns -o yaml            # forward . 10.255.255.1 -> unreachable upstream
+k -n kube-system edit cm coredns                   # forward . /etc/resolv.conf
+k -n kube-system rollout restart deploy coredns
+k run t --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
+"""
+S[38] = """
+k -n kube-system get pods                                  # kube-controller-manager Error/CrashLoop
+k -n kube-system logs kube-controller-manager-{CP}         # flag error: lease duration ... invalid
+{SHELL_CP}
+vi /etc/kubernetes/manifests/kube-controller-manager.yaml  # remove the bogus --leader-elect-lease-duration=1s line
+k create deploy probe --image=busybox:1.36 -- sleep 60 && k get rs
+"""
+S[39] = """
+{SHELL_W1}
+systemctl status kubelet; journalctl -u kubelet -n 30 --no-pager   # "open /etc/kubernetes/pki/ca-bundle.crt: no such file"
+grep -n clientCAFile /var/lib/kubelet/config.yaml                   # -> /etc/kubernetes/pki/ca.crt
+systemctl restart kubelet && systemctl enable kubelet
+"""
+# -------------------------------------------------------------- hard TSHOOT
+S[117] = """
+k get node {W2}                                   # NotReady,SchedulingDisabled
+{SHELL_W2}
+systemctl status kubelet                          # disabled + dead
+journalctl -u kubelet -n 30 --no-pager            # after enable: "ca-wrong.crt: no such file"
+sed -i 's|ca-wrong.crt|ca.crt|' /var/lib/kubelet/config.yaml
+systemctl enable --now kubelet
+exit
+k uncordon {W2}                                   # the third thing
+k run t --image=busybox:1.36 --restart=Never --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 60
+"""
+S[118] = """
+k -n kube-system get pods                                  # scheduler CrashLoop, controller-manager ImagePullBackOff
+{SHELL_CP}
+vi /etc/kubernetes/manifests/kube-scheduler.yaml           # kube-schedulerr -> kube-scheduler
+vi /etc/kubernetes/manifests/kube-controller-manager.yaml  # image tag ...-broken -> remove suffix (match the others: crictl images)
+k -n kube-system get pods -w
+k create deploy probe --image=busybox:1.36 -- sleep 60 && k get deploy probe -w
+"""
+S[119] = """
+k -n %{ns} get all; k -n %{ns} describe deploy %{dep}; k -n %{ns} describe rs -l app=%{dep}   # quota: pods 1
+k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"3"}}}'
+k -n %{ns} set image deploy %{dep} web=nginx:1.27                                        # image tag typo
+k -n %{ns} patch secret %{dep}-env -p '{"stringData":{"DATABASE_URL":"postgres://db"}}'   # missing key referenced by env
+k -n %{ns} rollout restart deploy %{dep}                                                  # picks up the secret key
+k -n %{ns} patch svc %{dep} -p '{"spec":{"selector":{"app":"%{dep}"},"ports":[{"port":80,"targetPort":80}]}}'
+"""
+S[120] = """
+k -n kube-system get pod -l component=kube-controller-manager -o yaml | grep cluster-cidr
+k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
+k -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}'
+k -n kube-system get cm kube-proxy -o yaml | grep '^ *mode:'      # "" -> iptables
+k get node %{node} -o jsonpath='{.spec.podCIDR}'
+"""

tracks/ckad/questions.py

@@ -0,0 +1,563 @@
+"""CKAD track. Starter bank - same format as the CKA banks, extend it the same way.
+
+CKAD is a build-things exam: less cluster surgery, more "produce the right object, fast".
+The clock is the adversary, so these are mostly single-object tasks graded exactingly.
+"""
+from tracks.cka.common import C, NS, ns_pick, name_pick
+
+TRACK = {
+    "id": "ckad",
+    "name": "Certified Kubernetes Application Developer",
+    "minutes": 120,
+    "pass_pct": 66,
+    "domains": {
+        "DESIGN": "Application Design and Build (20%)",
+        "DEPLOY": "Application Deployment (20%)",
+        "OBS":    "Application Observability and Maintenance (15%)",
+        "ENV":    "Application Environment, Configuration and Security (25%)",
+        "NET":    "Services and Networking (20%)",
+    },
+    "quota":      {"DESIGN": 3, "DEPLOY": 3, "OBS": 2, "ENV": 2, "NET": 2},
+    "quota_hard": {"DESIGN": 3, "DEPLOY": 3, "OBS": 2, "ENV": 2, "NET": 2},
+}
+
+QUESTIONS = [
+# ---------------------------------------------------------------- DESIGN ----
+{
+ "id": 1, "domain": "DESIGN", "title": "Adapter sidecar",
+ "params": lambda r: {"ns": ns_pick(r, "app"), "pod": name_pick(r, "adapter")},
+ "task": """
+Create Pod `%{pod}` in Namespace `%{ns}` with two containers sharing an `emptyDir` named `spool`
+mounted at `/spool` in both:
+
+- `writer` (busybox:1.36): every 2 seconds appends a line `metric 1` to `/spool/raw.txt`
+- `adapter` (busybox:1.36): runs `tail -f /spool/raw.txt`
+
+`kubectl logs %{pod} -c adapter` must show the writer's lines.
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""",
+ "checks": [
+  C("pod Running with containers writer and adapter", 2, r"""
+p="$K -n %{ns} get pod %{pod}"
+eq "$($p -o jsonpath='{.status.phase}' 2>/dev/null)" Running && $p -o jsonpath='{.spec.containers[*].name}' | grep -qw writer && $p -o jsonpath='{.spec.containers[*].name}' | grep -qw adapter"""),
+  C("emptyDir 'spool' mounted at /spool in both containers", 2, r"""
+p="$K -n %{ns} get pod %{pod}"
+[ -n "$($p -o jsonpath='{.spec.volumes[?(@.name=="spool")].emptyDir}')" ] \
+ && eq "$($p -o jsonpath='{.spec.containers[?(@.name=="writer")].volumeMounts[?(@.name=="spool")].mountPath}')" /spool \
+ && eq "$($p -o jsonpath='{.spec.containers[?(@.name=="adapter")].volumeMounts[?(@.name=="spool")].mountPath}')" /spool"""),
+  C("adapter logs show the writer's output", 2, r"""sleep 3; $K -n %{ns} logs %{pod} -c adapter --tail=3 2>/dev/null | grep -q 'metric 1'"""),
+ ],
+ "solution": """
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  volumes: [{name: spool, emptyDir: {}}]
+  containers:
+    - name: writer
+      image: busybox:1.36
+      command: [sh, -c, "while true; do echo 'metric 1' >> /spool/raw.txt; sleep 2; done"]
+      volumeMounts: [{name: spool, mountPath: /spool}]
+    - name: adapter
+      image: busybox:1.36
+      command: [sh, -c, "tail -f /spool/raw.txt"]
+      volumeMounts: [{name: spool, mountPath: /spool}]
+EOF
+k -n %{ns} logs %{pod} -c adapter
+""",
+},
+{
+ "id": 2, "domain": "DESIGN", "title": "Init container waits for a dependency",
+ "params": lambda r: {"ns": ns_pick(r, "boot"), "pod": name_pick(r, "consumer")},
+ "task": """
+Namespace `%{ns}` has a Service `backend` with no endpoints yet.
+
+1. Create Pod `%{pod}` with an init container `wait-for-backend` (busybox:1.36) that loops until
+   the DNS name `backend` resolves, and a main container `app` (nginx:1.27).
+   It will sit in `Init:0/1`.
+2. Then create Deployment `backend` (nginx:1.27, 1 replica, Pod label `app=backend`) so the
+   Service gets an endpoint and your Pod starts.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null
+$K -n %{ns} delete deploy backend --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: Service
+metadata: {name: backend, namespace: %{ns}}
+spec: {selector: {app: backend}, ports: [{port: 80}]}
+EOF
+""",
+ "checks": [
+  C("pod has an init container named wait-for-backend", 2, r"""[ -n "$(j -n %{ns} pod %{pod} '{.spec.initContainers[?(@.name=="wait-for-backend")].name}')" ]"""),
+  C("backend Deployment available with an endpoint on the Service", 3, r"""
+eq "$(j -n %{ns} deploy backend '{.status.availableReplicas}')" 1 \
+ && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=backend '{.items[*].endpoints[*].addresses[*]}')" ]"""),
+  C("pod is now Running", 3, r"""for i in $(seq 1 12); do [ "$(j -n %{ns} pod %{pod} '{.status.phase}')" = Running ] && exit 0; sleep 5; done; exit 1"""),
+ ],
+ "solution": """
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  initContainers:
+    - name: wait-for-backend
+      image: busybox:1.36
+      command: [sh, -c, "until nslookup backend; do echo waiting; sleep 2; done"]
+  containers: [{name: app, image: nginx:1.27}]
+EOF
+k -n %{ns} get pod %{pod}            # Init:0/1
+k -n %{ns} create deploy backend --image=nginx:1.27
+k -n %{ns} patch deploy backend -p '{"spec":{"template":{"metadata":{"labels":{"app":"backend"}}}}}'
+# A ClusterIP Service only resolves once it has endpoints - that is what unblocks the init container.
+""",
+},
+{
+ "id": 3, "domain": "DESIGN", "title": "CronJob with deadlines",
+ "params": lambda r: {"ns": ns_pick(r, "batch"), "cj": name_pick(r, "digest"), "m": r.choice([3, 5, 7])},
+ "task": """
+In `%{ns}` create CronJob `%{cj}`:
+- schedule: every %{m} minutes
+- image `busybox:1.36`, command prints `digest`
+- a run that cannot start within 10 seconds of its schedule is skipped
+- a run that takes longer than 20 seconds is killed
+- keep 2 successful and 2 failed Jobs
+- restartPolicy `Never`
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete cj %{cj} --ignore-not-found >/dev/null""",
+ "checks": [
+  C("schedule and history limits", 2, r"""
+c="$K -n %{ns} get cj %{cj}"
+eq "$($c -o jsonpath='{.spec.schedule}' 2>/dev/null)" "*/%{m} * * * *" && eq "$($c -o jsonpath='{.spec.successfulJobsHistoryLimit}')" 2 && eq "$($c -o jsonpath='{.spec.failedJobsHistoryLimit}')" 2"""),
+  C("startingDeadlineSeconds 10 and activeDeadlineSeconds 20", 3, r"""
+c="$K -n %{ns} get cj %{cj}"
+eq "$($c -o jsonpath='{.spec.startingDeadlineSeconds}' 2>/dev/null)" 10 && eq "$($c -o jsonpath='{.spec.jobTemplate.spec.activeDeadlineSeconds}')" 20"""),
+  C("restartPolicy Never", 1, r"""eq "$(j -n %{ns} cj %{cj} '{.spec.jobTemplate.spec.template.spec.restartPolicy}')" Never"""),
+ ],
+ "solution": """
+k -n %{ns} create cj %{cj} --image=busybox:1.36 --schedule='*/%{m} * * * *' $do -- sh -c "echo digest" > cj.yaml
+#   spec.startingDeadlineSeconds: 10                       (skip a late start)
+#   spec.successfulJobsHistoryLimit: 2 / failedJobsHistoryLimit: 2
+#   spec.jobTemplate.spec.activeDeadlineSeconds: 20        (kill an overrunning run)
+#   spec.jobTemplate.spec.template.spec.restartPolicy: Never
+k apply -f cj.yaml
+# startingDeadlineSeconds lives on the CronJob, activeDeadlineSeconds on the Job template.
+""",
+},
+# ---------------------------------------------------------------- DEPLOY ----
+{
+ "id": 4, "domain": "DEPLOY", "title": "Roll forward, then back",
+ "params": lambda r: {"ns": ns_pick(r, "rel"), "dep": name_pick(r, "frontend")},
+ "task": """
+Deployment `%{dep}` in `%{ns}` runs `nginx:1.25` with 3 replicas.
+
+1. Update the image to `nginx:1.27` and record a change-cause annotation of `pick 1.27`.
+2. Then update it to `nginx:9.99-nope`, which will never become ready.
+3. Roll back so all 3 replicas are Available on `nginx:1.27` again.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete deploy %{dep} --ignore-not-found --wait=true >/dev/null
+$K -n %{ns} create deploy %{dep} --image=nginx:1.25 --replicas=3 >/dev/null
+$K -n %{ns} rollout status deploy %{dep} --timeout=120s >/dev/null 2>&1
+""",
+ "checks": [
+  C("running image is nginx:1.27 with 3 available", 4, r"""
+eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.containers[0].image}')" nginx:1.27 && eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3"""),
+  C("rollout history has at least three revisions", 2, r"""[ "$($K -n %{ns} rollout history deploy %{dep} 2>/dev/null | grep -c '^[0-9]')" -ge 3 ]"""),
+  C("a change-cause annotation is recorded", 2, r"""j -n %{ns} deploy %{dep} '{.metadata.annotations.kubernetes\.io/change-cause}' | grep -q ."""),
+ ],
+ "solution": """
+k -n %{ns} set image deploy %{dep} nginx=nginx:1.27
+k -n %{ns} annotate deploy %{dep} kubernetes.io/change-cause='pick 1.27' --overwrite
+k -n %{ns} rollout status deploy %{dep}
+k -n %{ns} set image deploy %{dep} nginx=nginx:9.99-nope
+k -n %{ns} rollout status deploy %{dep} --timeout=30s     # never completes
+k -n %{ns} rollout history deploy %{dep}
+k -n %{ns} rollout undo deploy %{dep}
+k -n %{ns} rollout status deploy %{dep}
+""",
+},
+{
+ "id": 5, "domain": "DEPLOY", "title": "Blue/green cutover",
+ "params": lambda r: {"ns": ns_pick(r, "bg"), "svc": name_pick(r, "shop")},
+ "task": """
+Namespace `%{ns}` has Deployment `%{svc}-blue` (Pod labels `app=%{svc}`, `version=blue`) behind
+Service `%{svc}`.
+
+1. Create Deployment `%{svc}-green`: 2 replicas, image `nginx:1.27`, Pod labels `app=%{svc}` and
+   `version=green`.
+2. When it is fully Available, switch Service `%{svc}` to serve green only.
+3. Leave blue running untouched so the switch can be reversed.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete deploy %{svc}-blue %{svc}-green --ignore-not-found >/dev/null
+$K -n %{ns} delete svc %{svc} --ignore-not-found >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: %{svc}-blue, namespace: %{ns}}
+spec:
+  replicas: 2
+  selector: {matchLabels: {app: %{svc}, version: blue}}
+  template:
+    metadata: {labels: {app: %{svc}, version: blue}}
+    spec: {containers: [{name: web, image: nginx:1.25, ports: [{containerPort: 80}]}]}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: %{svc}, namespace: %{ns}}
+spec: {selector: {app: %{svc}, version: blue}, ports: [{port: 80, targetPort: 80}]}
+EOF
+""",
+ "checks": [
+  C("green Deployment available with the right labels", 3, r"""
+eq "$(j -n %{ns} deploy %{svc}-green '{.status.availableReplicas}')" 2 \
+ && eq "$(j -n %{ns} deploy %{svc}-green '{.spec.template.metadata.labels.version}')" green \
+ && eq "$(j -n %{ns} deploy %{svc}-green '{.spec.template.metadata.labels.app}')" %{svc}"""),
+  C("Service selector now targets version=green", 3, r"""eq "$(j -n %{ns} svc %{svc} '{.spec.selector.version}')" green"""),
+  C("every Service endpoint belongs to a green Pod", 2, r"""
+ips=$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{svc} '{.items[*].endpoints[*].addresses[*]}')
+[ -n "$ips" ] || exit 1
+for ip in $ips; do j -n %{ns} pod -l version=green '{.items[*].status.podIP}' | grep -qw "$ip" || exit 1; done"""),
+  C("blue is still running", 2, r"""eq "$(j -n %{ns} deploy %{svc}-blue '{.status.availableReplicas}')" 2"""),
+ ],
+ "solution": """
+k -n %{ns} get deploy %{svc}-blue -o yaml > green.yaml
+# rename to %{svc}-green, change every  version: blue  ->  green, image -> nginx:1.27,
+# strip status/uid/resourceVersion/creationTimestamp
+k apply -f green.yaml
+k -n %{ns} rollout status deploy %{svc}-green
+k -n %{ns} patch svc %{svc} -p '{"spec":{"selector":{"app":"%{svc}","version":"green"}}}'
+k -n %{ns} get endpointslices -l kubernetes.io/service-name=%{svc}
+# Rolling back is the same patch with version: blue - which is why blue stays up.
+""",
+},
+{
+ "id": 6, "domain": "DEPLOY", "title": "Helm release lifecycle",
+ "params": lambda r: {"ns": ns_pick(r, "helm"), "rel": name_pick(r, "site")},
+ "task": """
+A chart is at `{WS}/q6/chart`.
+
+1. Install it into Namespace `%{ns}` (create it) as release `%{rel}` with `replicaCount=2`.
+2. Upgrade the release to `replicaCount=4`.
+3. Roll the release back so the Deployment is at 2 replicas again — without uninstalling it.
+4. Write the release's current revision number to `{WS}/q6/revision.txt`.
+""",
+ "setup": r"""
+rm -rf "$WS/q6"; mkdir -p "$WS/q6/chart/templates"
+cat > "$WS/q6/chart/Chart.yaml" <<'EOF'
+apiVersion: v2
+name: site
+version: 0.1.0
+EOF
+cat > "$WS/q6/chart/values.yaml" <<'EOF'
+replicaCount: 1
+image: nginx:1.27
+EOF
+cat > "$WS/q6/chart/templates/deploy.yaml" <<'EOF'
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: {{ .Release.Name }}-site}
+spec:
+  replicas: {{ .Values.replicaCount }}
+  selector: {matchLabels: {app: {{ .Release.Name }}-site}}
+  template:
+    metadata: {labels: {app: {{ .Release.Name }}-site}}
+    spec: {containers: [{name: web, image: "{{ .Values.image }}"}]}
+EOF
+$K create ns %{ns} --dry-run=client -o yaml | $K apply -f - >/dev/null
+helm -n %{ns} uninstall %{rel} >/dev/null 2>&1 || true
+command -v helm >/dev/null || echo "warn: helm not installed"
+""",
+ "checks": [
+  C("release deployed with at least three revisions of history", 3, r"""
+helm -n %{ns} status %{rel} -o json 2>/dev/null | python3 -c 'import sys,json;sys.exit(0 if json.load(sys.stdin)["info"]["status"]=="deployed" else 1)' || exit 1
+[ "$(helm -n %{ns} history %{rel} -o json 2>/dev/null | python3 -c 'import sys,json;print(len(json.load(sys.stdin)))')" -ge 3 ]"""),
+  C("Deployment is back to 2 replicas", 3, r"""eq "$(j -n %{ns} deploy %{rel}-site '{.spec.replicas}')" 2"""),
+  C("revision.txt matches the release's current revision", 2, r"""
+cur=$(helm -n %{ns} history %{rel} -o json 2>/dev/null | python3 -c 'import sys,json;print(max(r["revision"] for r in json.load(sys.stdin)))' 2>/dev/null)
+[ -n "$cur" ] && eq "$(tr -dc '0-9' < "$WS/q6/revision.txt" 2>/dev/null)" "$cur" """),
+ ],
+ "solution": """
+k create ns %{ns}
+helm -n %{ns} install %{rel} {WS}/q6/chart --set replicaCount=2
+helm -n %{ns} upgrade %{rel} {WS}/q6/chart --set replicaCount=4
+helm -n %{ns} history %{rel}
+helm -n %{ns} rollback %{rel} 1          # creates a NEW revision identical to revision 1
+helm -n %{ns} history %{rel} | grep '^[0-9]' | tail -1 | awk '{print $1}' > {WS}/q6/revision.txt
+""",
+},
+# ------------------------------------------------------------------- OBS ----
+{
+ "id": 7, "domain": "OBS", "title": "Three kinds of probe",
+ "params": lambda r: {"ns": ns_pick(r, "probe"), "dep": name_pick(r, "api")},
+ "task": """
+Deployment `%{dep}` in `%{ns}` (nginx:1.27, 2 replicas) needs health checks:
+
+- **startup**: HTTP GET `/` on port 80, `periodSeconds: 5`, allowed a total of 60 seconds to come up
+- **readiness**: HTTP GET `/` on port 80, every 5 seconds, a single failure removes it from endpoints
+- **liveness**: TCP on port 80, every 10 seconds, initial delay 15 seconds
+
+All replicas must end up Ready.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null
+$K -n %{ns} create deploy %{dep} --image=nginx:1.27 --replicas=2 >/dev/null
+""",
+ "checks": [
+  C("startupProbe: httpGet / :80, period 5, failureThreshold 12", 3, r"""
+c='{.spec.template.spec.containers[0].startupProbe'
+eq "$(j -n %{ns} deploy %{dep} "$c.httpGet.path}")" / && eq "$(j -n %{ns} deploy %{dep} "$c.periodSeconds}")" 5 && eq "$(j -n %{ns} deploy %{dep} "$c.failureThreshold}")" 12"""),
+  C("readinessProbe: httpGet, period 5, failureThreshold 1", 3, r"""
+c='{.spec.template.spec.containers[0].readinessProbe'
+eq "$(j -n %{ns} deploy %{dep} "$c.httpGet.path}")" / && eq "$(j -n %{ns} deploy %{dep} "$c.periodSeconds}")" 5 && eq "$(j -n %{ns} deploy %{dep} "$c.failureThreshold}")" 1"""),
+  C("livenessProbe: tcpSocket 80, period 10, initialDelay 15", 3, r"""
+c='{.spec.template.spec.containers[0].livenessProbe'
+eq "$(j -n %{ns} deploy %{dep} "$c.tcpSocket.port}")" 80 && eq "$(j -n %{ns} deploy %{dep} "$c.periodSeconds}")" 10 && eq "$(j -n %{ns} deploy %{dep} "$c.initialDelaySeconds}")" 15"""),
+  C("all replicas Ready", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.readyReplicas}')" 2"""),
+ ],
+ "solution": """
+k -n %{ns} edit deploy %{dep}       # add to the container:
+#   startupProbe:   {httpGet: {path: /, port: 80}, periodSeconds: 5, failureThreshold: 12}
+#   readinessProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5, failureThreshold: 1}
+#   livenessProbe:  {tcpSocket: {port: 80}, periodSeconds: 10, initialDelaySeconds: 15}
+k -n %{ns} rollout status deploy %{dep}
+# 12 x 5s = 60s. While a startupProbe runs, liveness and readiness are suspended, which is why
+# it beats a long initialDelaySeconds on liveness for slow starters.
+""",
+},
+{
+ "id": 8, "domain": "OBS", "title": "Diagnose and fix a broken Pod",
+ "params": lambda r: {"ns": ns_pick(r, "debug"), "pod": name_pick(r, "worker")},
+ "task": """
+Pod `%{pod}` in `%{ns}` is not running.
+
+1. Write the container's waiting **reason**, as reported in the Pod's status, to `{WS}/q8/reason.txt`.
+2. Fix the Pod so it runs. Keep the name, Namespace and command — correct only what is wrong.
+""",
+ "setup": NS + r"""
+mkns %{ns}; rm -rf "$WS/q8"; mkdir -p "$WS/q8"
+$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  containers:
+    - name: worker
+      image: busybox:1.99-doesnotexist
+      command: [sh, -c, "echo working; sleep 86400"]
+EOF
+""",
+ "checks": [
+  C("reason.txt records the image pull failure", 3, r"""grep -Eqi 'ImagePull|ErrImagePull|ImagePullBackOff' "$WS/q8/reason.txt" 2>/dev/null"""),
+  C("pod Running with its original command", 4, r"""
+eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && j -n %{ns} pod %{pod} '{.spec.containers[0].command}' | grep -q sleep"""),
+ ],
+ "solution": """
+k -n %{ns} get pod %{pod}
+k -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].state.waiting.reason}'
+#   ImagePullBackOff  -> {WS}/q8/reason.txt
+k -n %{ns} describe pod %{pod} | tail -15      # "Failed to pull image ... not found"
+k -n %{ns} get pod %{pod} -o yaml > fix.yaml   # set image: busybox:1.36, strip status/uid/resourceVersion
+k -n %{ns} delete pod %{pod} --now && k apply -f fix.yaml
+# A Pod's image can't be changed in place, so delete + recreate is the fix (a Deployment could
+# have been patched with `set image`).
+""",
+},
+# ------------------------------------------------------------------- ENV ----
+{
+ "id": 9, "domain": "ENV", "title": "SecurityContext and capabilities",
+ "params": lambda r: {"ns": ns_pick(r, "sec"), "pod": name_pick(r, "hardened"), "uid": r.choice([1001, 2000, 10001])},
+ "task": """
+Create Pod `%{pod}` in `%{ns}` (image `busybox:1.36`, command `sleep 86400`) that:
+
+- runs as user %{uid} and group %{uid}, and must not run as root,
+- has a read-only root filesystem,
+- drops **all** capabilities and adds back only `NET_BIND_SERVICE`,
+- cannot gain new privileges.
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""",
+ "checks": [
+  C("runs as UID %{uid} with runAsNonRoot", 3, r"""
+p="$K -n %{ns} get pod %{pod}"
+u=$($p -o jsonpath='{.spec.securityContext.runAsUser}' 2>/dev/null); [ -z "$u" ] && u=$($p -o jsonpath='{.spec.containers[0].securityContext.runAsUser}')
+eq "$u" %{uid} && $p -o jsonpath='{.spec.securityContext.runAsNonRoot}{.spec.containers[0].securityContext.runAsNonRoot}' | grep -q true"""),
+  C("readOnlyRootFilesystem and allowPrivilegeEscalation=false", 3, r"""
+c='{.spec.containers[0].securityContext'
+eq "$(j -n %{ns} pod %{pod} "$c.readOnlyRootFilesystem}")" true && eq "$(j -n %{ns} pod %{pod} "$c.allowPrivilegeEscalation}")" false"""),
+  C("capabilities drop ALL, add NET_BIND_SERVICE", 3, r"""
+c='{.spec.containers[0].securityContext.capabilities'
+j -n %{ns} pod %{pod} "$c.drop[*]}" | grep -q ALL && j -n %{ns} pod %{pod} "$c.add[*]}" | grep -q NET_BIND_SERVICE"""),
+  C("pod Running as that UID", 2, r"""
+eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && $K -n %{ns} exec %{pod} -- id 2>/dev/null | grep -q 'uid=%{uid}'"""),
+ ],
+ "solution": """
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  securityContext: {runAsUser: %{uid}, runAsGroup: %{uid}, runAsNonRoot: true}
+  containers:
+    - name: c
+      image: busybox:1.36
+      command: [sleep, "86400"]
+      securityContext:
+        readOnlyRootFilesystem: true
+        allowPrivilegeEscalation: false
+        capabilities: {drop: [ALL], add: [NET_BIND_SERVICE]}
+EOF
+k -n %{ns} exec %{pod} -- id
+# runAsUser/runAsGroup/runAsNonRoot exist at pod OR container level;
+# capabilities, readOnlyRootFilesystem and allowPrivilegeEscalation are container-level only.
+""",
+},
+{
+ "id": 10, "domain": "ENV", "title": "Projected volume: config, secret and token",
+ "params": lambda r: {"ns": ns_pick(r, "cfg"), "pod": name_pick(r, "combined")},
+ "task": """
+In `%{ns}`:
+1. Create ConfigMap `app-cfg` with `mode=live` and Secret `app-sec` with `apikey=xyz789`.
+2. Create Pod `%{pod}` (busybox:1.36, `sleep 86400`) with a **single projected volume** named
+   `bundle` mounted at `/etc/bundle`, exposing:
+   - ConfigMap key `mode` as file `mode`,
+   - Secret key `apikey` as file `apikey`,
+   - a ServiceAccount token with audience `vault` and expiry 3600 seconds as file `token`.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null
+$K -n %{ns} delete cm app-cfg --ignore-not-found >/dev/null; $K -n %{ns} delete secret app-sec --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("ConfigMap and Secret hold the right values", 2, r"""
+eq "$(j -n %{ns} cm app-cfg '{.data.mode}')" live && eq "$(j -n %{ns} secret app-sec '{.data.apikey}' | base64 -d)" xyz789"""),
+  C("one projected volume 'bundle' with all three sources", 3, r"""
+s=$(j -n %{ns} pod %{pod} '{.spec.volumes[?(@.name=="bundle")].projected.sources}')
+echo "$s" | grep -q configMap && echo "$s" | grep -q secret && echo "$s" | grep -q serviceAccountToken"""),
+  C("token source: audience vault, expiry 3600", 3, r"""
+s=$(j -n %{ns} pod %{pod} '{.spec.volumes[?(@.name=="bundle")].projected.sources[*].serviceAccountToken}')
+echo "$s" | grep -q '"audience":"vault"' && echo "$s" | grep -q '"expirationSeconds":3600'"""),
+  C("all three files readable at /etc/bundle", 3, r"""
+eq "$($K -n %{ns} exec %{pod} -- cat /etc/bundle/mode 2>/dev/null)" live \
+ && eq "$($K -n %{ns} exec %{pod} -- cat /etc/bundle/apikey 2>/dev/null)" xyz789 \
+ && [ -n "$($K -n %{ns} exec %{pod} -- cat /etc/bundle/token 2>/dev/null)" ]"""),
+ ],
+ "solution": """
+k -n %{ns} create cm app-cfg --from-literal=mode=live
+k -n %{ns} create secret generic app-sec --from-literal=apikey=xyz789
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  volumes:
+    - name: bundle
+      projected:
+        sources:
+          - configMap: {name: app-cfg, items: [{key: mode, path: mode}]}
+          - secret:    {name: app-sec, items: [{key: apikey, path: apikey}]}
+          - serviceAccountToken: {audience: vault, expirationSeconds: 3600, path: token}
+  containers:
+    - name: c
+      image: busybox:1.36
+      command: [sleep, "86400"]
+      volumeMounts: [{name: bundle, mountPath: /etc/bundle}]
+EOF
+k -n %{ns} exec %{pod} -- ls /etc/bundle
+""",
+},
+# ------------------------------------------------------------------- NET ----
+{
+ "id": 11, "domain": "NET", "title": "Expose a Deployment two ways",
+ "params": lambda r: {"ns": ns_pick(r, "expose"), "dep": name_pick(r, "portal"), "np": r.choice([30310, 30320, 30330])},
+ "task": """
+In `%{ns}` Deployment `%{dep}` (nginx:1.27, 2 replicas) serves on container port 80.
+
+1. Expose it internally as ClusterIP Service `%{dep}`, port 8080 → 80.
+2. Also expose it as NodePort Service `%{dep}-ext`, port 80 → 80, nodePort %{np}.
+3. Create Ingress `%{dep}` (class `nginx`, host `%{dep}.example.com`, path `/` Prefix) pointing at
+   the **ClusterIP** Service on port 8080.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null
+$K -n %{ns} delete svc %{dep} %{dep}-ext --ignore-not-found >/dev/null
+$K -n %{ns} delete ingress %{dep} --ignore-not-found >/dev/null
+$K -n %{ns} create deploy %{dep} --image=nginx:1.27 --replicas=2 --port=80 >/dev/null
+""",
+ "checks": [
+  C("ClusterIP Service 8080 -> 80 with endpoints", 3, r"""
+s="$K -n %{ns} get svc %{dep}"
+eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" ClusterIP && eq "$($s -o jsonpath='{.spec.ports[0].port}')" 8080 && eq "$($s -o jsonpath='{.spec.ports[0].targetPort}')" 80 \
+ && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[*].addresses[*]}')" ]"""),
+  C("NodePort Service on %{np}", 2, r"""
+s="$K -n %{ns} get svc %{dep}-ext"
+eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" NodePort && eq "$($s -o jsonpath='{.spec.ports[0].nodePort}')" %{np}"""),
+  C("Ingress routes the host to the ClusterIP Service on 8080", 3, r"""
+i="$K -n %{ns} get ingress %{dep}"
+eq "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" nginx && eq "$($i -o jsonpath='{.spec.rules[0].host}')" %{dep}.example.com \
+ && eq "$($i -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}')" %{dep} \
+ && eq "$($i -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.port.number}')" 8080"""),
+ ],
+ "solution": """
+k -n %{ns} expose deploy %{dep} --port=8080 --target-port=80
+k -n %{ns} expose deploy %{dep} --name=%{dep}-ext --type=NodePort --port=80 --target-port=80 $do > ext.yaml
+# set spec.ports[0].nodePort: %{np}
+k apply -f ext.yaml
+k -n %{ns} create ingress %{dep} --class=nginx --rule="%{dep}.example.com/*=%{dep}:8080"
+""",
+},
+{
+ "id": 12, "domain": "NET", "title": "Restrict a namespace with NetworkPolicy",
+ "params": lambda r: {"ns": ns_pick(r, "np"), "client": r.choice(["frontend", "gateway", "portal"])},
+ "task": """
+Namespace `%{ns}` runs Pods labelled `app=db` and `app=%{client}`.
+
+Create NetworkPolicy `db-allow` so `app=db` Pods accept ingress **only** from `app=%{client}`
+Pods in the same Namespace on TCP 5432, and nothing else. Do not restrict egress.
+
+(kindnet doesn't enforce policies here, so you are graded on the object.)
+""",
+ "setup": NS + r"""
+mkns %{ns}
+for a in db %{client}; do
+  $K -n %{ns} create deploy $a --image=nginx:1.27 --dry-run=client -o yaml | $K apply -f - >/dev/null
+  $K -n %{ns} patch deploy $a -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app\":\"$a\"}}}}}" >/dev/null
+done
+$K -n %{ns} delete netpol db-allow --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("selects app=db, Ingress only", 2, r"""
+n="$K -n %{ns} get netpol db-allow"
+eq "$($n -o jsonpath='{.spec.podSelector.matchLabels.app}' 2>/dev/null)" db && eq "$($n -o jsonpath='{.spec.policyTypes}')" '["Ingress"]'"""),
+  C("exactly one rule: from app=%{client} on TCP 5432", 4, r"""
+$K -n %{ns} get netpol db-allow -o json 2>/dev/null | python3 -c '
+import sys,json
+i=json.load(sys.stdin)["spec"].get("ingress",[])
+ok=len(i)==1 and len(i[0].get("from",[]))==1 and i[0]["from"][0].get("podSelector",{}).get("matchLabels",{}).get("app")=="%{client}" \
+   and len(i[0].get("ports",[]))==1 and str(i[0]["ports"][0]["port"])=="5432"
+sys.exit(0 if ok else 1)'"""),
+ ],
+ "solution": """
+cat <<EOF | k apply -f -
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: db-allow, namespace: %{ns}}
+spec:
+  podSelector: {matchLabels: {app: db}}
+  policyTypes: [Ingress]
+  ingress:
+    - from: [{podSelector: {matchLabels: {app: %{client}}}}]
+      ports: [{protocol: TCP, port: 5432}]
+EOF
+""",
+},
+]

tracks/cks/questions.py

@@ -0,0 +1,557 @@
+"""CKS track. Starter bank.
+
+Caveat worth knowing before you rely on this: several CKS objectives need infrastructure the
+default kind cluster does not have - kindnet does not enforce NetworkPolicy, and there is no
+Falco, AppArmor profile loader, gVisor runtime or image scanner installed. Tasks here either
+stick to what a stock kubeadm/kind cluster can really do (PSA, RBAC, admission policies, audit
+logging, seccomp, token hygiene) or are graded on the object rather than on enforcement, and say so.
+"""
+from tracks.cka.common import C, NS, ns_pick, name_pick
+
+TRACK = {
+    "id": "cks",
+    "name": "Certified Kubernetes Security Specialist",
+    "minutes": 120,
+    "pass_pct": 67,
+    "domains": {
+        "SETUP":   "Cluster Setup (15%)",
+        "HARDEN":  "Cluster Hardening (15%)",
+        "SYS":     "System Hardening (10%)",
+        "MICRO":   "Minimize Microservice Vulnerabilities (20%)",
+        "SUPPLY":  "Supply Chain Security (20%)",
+        "RUNTIME": "Monitoring, Logging and Runtime Security (20%)",
+    },
+    "quota":      {"SETUP": 2, "HARDEN": 2, "SYS": 1, "MICRO": 2, "SUPPLY": 1, "RUNTIME": 1},
+    "quota_hard": {"SETUP": 2, "HARDEN": 2, "SYS": 1, "MICRO": 2, "SUPPLY": 1, "RUNTIME": 1},
+}
+
+QUESTIONS = [
+# ----------------------------------------------------------------- SETUP ----
+{
+ "id": 1, "domain": "SETUP", "title": "Default-deny, then allow what's needed",
+ "params": lambda r: {"ns": ns_pick(r, "zone")},
+ "task": """
+Namespace `%{ns}` runs `app=web` (port 8080) and `app=db` (port 5432).
+
+Create three NetworkPolicies:
+1. `default-deny` — denies all ingress **and** egress for every Pod in the Namespace.
+2. `allow-dns` — lets every Pod reach DNS (UDP and TCP 53) in `kube-system`.
+3. `web-to-db` — lets `app=web` reach `app=db` on 5432, and lets `app=db` accept that ingress.
+
+(kindnet does not enforce policies; you are graded on the objects.)
+""",
+ "setup": NS + r"""
+mkns %{ns}
+for a in web db; do
+  $K -n %{ns} create deploy $a --image=nginx:1.27 --dry-run=client -o yaml | $K apply -f - >/dev/null
+  $K -n %{ns} patch deploy $a -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app\":\"$a\"}}}}}" >/dev/null
+done
+$K -n %{ns} delete netpol --all >/dev/null 2>&1
+""",
+ "checks": [
+  C("default-deny covers all pods, both directions, with no allow rules", 3, r"""
+n="$K -n %{ns} get netpol default-deny"
+eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' && $n -o jsonpath='{.spec.policyTypes}' | grep -q Ingress \
+ && $n -o jsonpath='{.spec.policyTypes}' | grep -q Egress && [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ]"""),
+  C("allow-dns permits UDP and TCP 53 to kube-system", 3, r"""
+$K -n %{ns} get netpol allow-dns -o json 2>/dev/null | python3 -c '
+import sys,json
+s=json.load(sys.stdin)["spec"]
+ok=any({p.get("protocol") for p in r.get("ports",[]) if str(p.get("port"))=="53"}>={"UDP","TCP"}
+       and r.get("to") and all("kube-system" in json.dumps(t.get("namespaceSelector",{})) for t in r["to"])
+       for r in s.get("egress",[]))
+sys.exit(0 if ok else 1)'"""),
+  C("web-to-db pairs web egress and db ingress on 5432", 4, r"""
+$K -n %{ns} get netpol -o json 2>/dev/null | python3 -c '
+import sys,json
+eg=ing=False
+for n in json.load(sys.stdin)["items"]:
+    s=n["spec"]; app=s["podSelector"].get("matchLabels",{}).get("app")
+    for r in s.get("egress",[]):
+        if app=="web" and any(t.get("podSelector",{}).get("matchLabels",{}).get("app")=="db" for t in r.get("to",[])) \
+           and any(str(p.get("port"))=="5432" for p in r.get("ports",[])): eg=True
+    for r in s.get("ingress",[]):
+        if app=="db" and any(t.get("podSelector",{}).get("matchLabels",{}).get("app")=="web" for t in r.get("from",[])) \
+           and any(str(p.get("port"))=="5432" for p in r.get("ports",[])): ing=True
+sys.exit(0 if eg and ing else 1)'"""),
+ ],
+ "solution": """
+cat <<EOF | k apply -f -
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: default-deny, namespace: %{ns}}
+spec: {podSelector: {}, policyTypes: [Ingress, Egress]}
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: allow-dns, namespace: %{ns}}
+spec:
+  podSelector: {}
+  policyTypes: [Egress]
+  egress:
+    - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}}]
+      ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: web-to-db, namespace: %{ns}}
+spec:
+  podSelector: {matchLabels: {app: web}}
+  policyTypes: [Egress]
+  egress:
+    - to: [{podSelector: {matchLabels: {app: db}}}]
+      ports: [{port: 5432}]
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata: {name: db-from-web, namespace: %{ns}}
+spec:
+  podSelector: {matchLabels: {app: db}}
+  policyTypes: [Ingress]
+  ingress:
+    - from: [{podSelector: {matchLabels: {app: web}}}]
+      ports: [{port: 5432}]
+EOF
+# With a default deny in place, BOTH sides need a rule: egress on the client and ingress on the
+# server. Forgetting one is the most common CKS mistake.
+""",
+},
+{
+ "id": 2, "domain": "SETUP", "title": "Turn off apiserver profiling", "breaks": "apiserver-flag", "difficulty": "hard",
+ "task": """
+An audit found the kube-apiserver's pprof profiling endpoints exposed.
+
+1. Write the current value of the apiserver's `--profiling` setting (or `absent` if the flag is
+   not set) to `{WS}/q2/before.txt`.
+2. Disable profiling on the kube-apiserver, persistently.
+3. Confirm the API server comes back healthy and write its `--profiling` value afterwards to
+   `{WS}/q2/after.txt`.
+""",
+ "setup": r"""
+rm -rf "$WS/q2"; mkdir -p "$WS/q2"
+nx "$CP" 'sed -i "/--profiling=/d" /etc/kubernetes/manifests/kube-apiserver.yaml' || true
+""",
+ "checks": [
+  C("before.txt records the original state", 2, r"""grep -Eqi 'absent|true|not set|none' "$WS/q2/before.txt" 2>/dev/null"""),
+  C("the manifest now sets --profiling=false", 4, r"""nx "$CP" 'grep -q -- "--profiling=false" /etc/kubernetes/manifests/kube-apiserver.yaml'"""),
+  C("apiserver is healthy again", 4, r"""
+for i in $(seq 1 20); do $K get --raw=/healthz >/dev/null 2>&1 && [ "$(j -n kube-system pod -l component=kube-apiserver '{.items[0].status.phase}')" = Running ] && exit 0; sleep 6; done; exit 1"""),
+  C("after.txt records false", 1, r"""grep -q 'false' "$WS/q2/after.txt" 2>/dev/null"""),
+ ],
+ "solution": """
+{SHELL_CP}
+grep profiling /etc/kubernetes/manifests/kube-apiserver.yaml || echo absent   # -> {WS}/q2/before.txt
+vi /etc/kubernetes/manifests/kube-apiserver.yaml
+#   under command:, add    - --profiling=false
+# the kubelet notices the manifest change and restarts the static pod - watch with:
+crictl ps | grep apiserver
+exit
+k get --raw=/healthz
+echo false > {WS}/q2/after.txt
+# Editing the manifest is what makes it persistent; `kubectl edit` on the mirror pod would be lost.
+# Mind the indentation: the flag goes in the container's command list, not under spec.
+""",
+},
+# ---------------------------------------------------------------- HARDEN ----
+{
+ "id": 3, "domain": "HARDEN", "title": "Replace a wildcard ClusterRole",
+ "params": lambda r: {"ns": ns_pick(r, "team"), "sa": name_pick(r, "runner")},
+ "task": """
+ServiceAccount `%{sa}` in `%{ns}` is bound to a ClusterRole granting `*` on `*` cluster-wide.
+
+Reduce it to exactly what it needs:
+- read-only (`get`, `list`, `watch`) on `pods` and `pods/log` **in `%{ns}` only**.
+
+Use Role `%{sa}-view` and RoleBinding `%{sa}-view`. Remove the over-privileged binding.
+The ServiceAccount must have no other permissions anywhere.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} create sa %{sa} --dry-run=client -o yaml | $K apply -f - >/dev/null
+$K delete clusterrole %{sa}-god --ignore-not-found >/dev/null; $K delete clusterrolebinding %{sa}-god --ignore-not-found >/dev/null
+$K -n %{ns} delete role,rolebinding --all >/dev/null 2>&1
+$K create clusterrole %{sa}-god --verb='*' --resource='*.*' >/dev/null
+$K create clusterrolebinding %{sa}-god --clusterrole=%{sa}-god --serviceaccount=%{ns}:%{sa} >/dev/null
+""",
+ "checks": [
+  C("the wildcard binding is gone", 3, r"""! $K get clusterrolebinding %{sa}-god >/dev/null 2>&1"""),
+  C("can get/list/watch pods and pods/log in %{ns}", 3, r"""
+for res in pods pods/log; do for v in get list watch; do
+  eq "$($K auth can-i $v $res -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes || exit 1
+done; done"""),
+  C("nothing else: no delete pods, no secrets, nothing in other namespaces", 4, r"""
+for t in "delete pods -n %{ns}" "get secrets -n %{ns}" "create pods -n %{ns}" "list pods -n default" "list nodes"; do
+  eq "$($K auth can-i $t --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null || true)" no || exit 1
+done"""),
+ ],
+ "solution": """
+k get clusterrolebinding -o wide | grep %{sa}
+k delete clusterrolebinding %{sa}-god
+k -n %{ns} create role %{sa}-view --verb=get,list,watch --resource=pods,pods/log
+k -n %{ns} create rolebinding %{sa}-view --role=%{sa}-view --serviceaccount=%{ns}:%{sa}
+k auth can-i --list -n %{ns} --as=system:serviceaccount:%{ns}:%{sa}
+# Deleting the ClusterRole itself is optional - the binding is what grants the access.
+""",
+},
+{
+ "id": 4, "domain": "HARDEN", "title": "Stop handing out API tokens",
+ "params": lambda r: {"ns": ns_pick(r, "apps"), "dep": name_pick(r, "worker")},
+ "task": """
+In Namespace `%{ns}`:
+
+1. Make the **default** ServiceAccount stop mounting its API token into Pods automatically.
+2. Deployment `%{dep}` genuinely needs API access: give it a dedicated ServiceAccount `%{dep}-sa`
+   that does mount its token, and make the Deployment use it.
+3. Confirm: a new Pod using the default ServiceAccount has no token mounted, while `%{dep}`'s Pods do.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} patch sa default -p '{"automountServiceAccountToken":null}' >/dev/null 2>&1 || true
+$K -n %{ns} delete sa %{dep}-sa --ignore-not-found >/dev/null
+$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null
+$K -n %{ns} delete pod plain --ignore-not-found --wait=false >/dev/null
+$K -n %{ns} create deploy %{dep} --image=nginx:1.27 >/dev/null
+""",
+ "checks": [
+  C("default ServiceAccount has automountServiceAccountToken: false", 3, r"""eq "$(j -n %{ns} sa default '{.automountServiceAccountToken}')" false"""),
+  C("%{dep} uses %{dep}-sa, which does mount a token", 3, r"""
+eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.serviceAccountName}')" %{dep}-sa \
+ && [ "$(j -n %{ns} sa %{dep}-sa '{.automountServiceAccountToken}')" != false ]"""),
+  C("%{dep} pods have a projected token mounted", 3, r"""
+p=$(j -n %{ns} pod -l app=%{dep} '{.items[0].metadata.name}'); [ -n "$p" ] || exit 1
+$K -n %{ns} exec "$p" -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null"""),
+  C("a pod on the default ServiceAccount has no token", 3, r"""
+$K -n %{ns} delete pod plain --ignore-not-found --wait=true >/dev/null 2>&1
+$K -n %{ns} run plain --image=busybox:1.36 --restart=Never -- sleep 120 >/dev/null 2>&1
+for i in $(seq 1 15); do [ "$(j -n %{ns} pod plain '{.status.phase}')" = Running ] && break; sleep 3; done
+r=1; $K -n %{ns} exec plain -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null || r=0
+$K -n %{ns} delete pod plain --wait=false >/dev/null 2>&1; exit $r"""),
+ ],
+ "solution": """
+k -n %{ns} patch sa default -p '{"automountServiceAccountToken": false}'
+k -n %{ns} create sa %{dep}-sa
+k -n %{ns} set serviceaccount deploy %{dep} %{dep}-sa
+k -n %{ns} rollout status deploy %{dep}
+# The setting can also be put on the Pod spec (automountServiceAccountToken: false), which
+# overrides the ServiceAccount - useful when only one workload should differ.
+""",
+},
+# ------------------------------------------------------------------- SYS ----
+{
+ "id": 5, "domain": "SYS", "title": "seccomp and a minimal container",
+ "params": lambda r: {"ns": ns_pick(r, "sys"), "pod": name_pick(r, "confined")},
+ "task": """
+Create Pod `%{pod}` in `%{ns}` (image `nginx:1.27`) that:
+
+- uses the **RuntimeDefault** seccomp profile at Pod level,
+- runs as a non-root user (UID 101, nginx's own user) with `allowPrivilegeEscalation: false`,
+- drops all capabilities and adds back only `NET_BIND_SERVICE` and `CHOWN`,
+- mounts an `emptyDir` at `/var/cache/nginx` and one at `/var/run` so nginx can still write
+  while the root filesystem is read-only.
+
+The Pod must reach Running.
+""",
+ "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""",
+ "checks": [
+  C("seccompProfile RuntimeDefault at pod level", 3, r"""eq "$(j -n %{ns} pod %{pod} '{.spec.securityContext.seccompProfile.type}')" RuntimeDefault"""),
+  C("non-root UID 101, no privilege escalation, read-only root fs", 3, r"""
+p="$K -n %{ns} get pod %{pod}"
+u=$($p -o jsonpath='{.spec.securityContext.runAsUser}' 2>/dev/null); [ -z "$u" ] && u=$($p -o jsonpath='{.spec.containers[0].securityContext.runAsUser}')
+eq "$u" 101 && eq "$($p -o jsonpath='{.spec.containers[0].securityContext.allowPrivilegeEscalation}')" false \
+ && eq "$($p -o jsonpath='{.spec.containers[0].securityContext.readOnlyRootFilesystem}')" true"""),
+  C("capabilities drop ALL, add NET_BIND_SERVICE and CHOWN", 3, r"""
+c='{.spec.containers[0].securityContext.capabilities'
+j -n %{ns} pod %{pod} "$c.drop[*]}" | grep -q ALL && j -n %{ns} pod %{pod} "$c.add[*]}" | grep -q NET_BIND_SERVICE && j -n %{ns} pod %{pod} "$c.add[*]}" | grep -q CHOWN"""),
+  C("pod Running with both writable mounts", 3, r"""
+eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running \
+ && j -n %{ns} pod %{pod} '{.spec.containers[0].volumeMounts[*].mountPath}' | grep -q '/var/cache/nginx' \
+ && j -n %{ns} pod %{pod} '{.spec.containers[0].volumeMounts[*].mountPath}' | grep -q '/var/run'"""),
+ ],
+ "solution": """
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  securityContext:
+    seccompProfile: {type: RuntimeDefault}
+    runAsUser: 101
+    runAsNonRoot: true
+  volumes:
+    - {name: cache, emptyDir: {}}
+    - {name: run, emptyDir: {}}
+  containers:
+    - name: web
+      image: nginx:1.27
+      securityContext:
+        allowPrivilegeEscalation: false
+        readOnlyRootFilesystem: true
+        capabilities: {drop: [ALL], add: [NET_BIND_SERVICE, CHOWN]}
+      volumeMounts:
+        - {name: cache, mountPath: /var/cache/nginx}
+        - {name: run, mountPath: /var/run}
+EOF
+# readOnlyRootFilesystem breaks most images until you mount emptyDirs over the paths they write to;
+# for nginx those are /var/cache/nginx and /var/run.
+""",
+},
+# ----------------------------------------------------------------- MICRO ----
+{
+ "id": 6, "domain": "MICRO", "title": "Enforce Pod Security restricted",
+ "params": lambda r: {"ns": ns_pick(r, "psa"), "dep": name_pick(r, "legacy")},
+ "task": """
+Namespace `%{ns}` runs Deployment `%{dep}`, whose Pods run as root and request extra privileges.
+
+1. Label the Namespace so the **restricted** Pod Security Standard is **enforced** (latest version),
+   and also **warn** and **audit** at restricted.
+2. Fix `%{dep}` so its Pods are admitted under that standard, without changing its image
+   (`nginx:1.27`) or replica count.
+
+All replicas must be Available afterwards.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K label ns %{ns} pod-security.kubernetes.io/enforce- pod-security.kubernetes.io/warn- pod-security.kubernetes.io/audit- >/dev/null 2>&1 || true
+$K -n %{ns} delete deploy %{dep} --ignore-not-found --wait=true >/dev/null
+$K apply -f - >/dev/null <<EOF
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: %{dep}, namespace: %{ns}}
+spec:
+  replicas: 2
+  selector: {matchLabels: {app: %{dep}}}
+  template:
+    metadata: {labels: {app: %{dep}}}
+    spec:
+      volumes: [{name: cache, emptyDir: {}}, {name: run, emptyDir: {}}]
+      containers:
+        - name: web
+          image: nginx:1.27
+          securityContext: {privileged: true}
+          volumeMounts: [{name: cache, mountPath: /var/cache/nginx}, {name: run, mountPath: /var/run}]
+EOF
+$K -n %{ns} rollout status deploy %{dep} --timeout=90s >/dev/null 2>&1
+""",
+ "checks": [
+  C("namespace enforces, warns and audits restricted", 3, r"""
+l=$(j ns %{ns} '{.metadata.labels}')
+echo "$l" | grep -q '"pod-security.kubernetes.io/enforce":"restricted"' && echo "$l" | grep -q '"pod-security.kubernetes.io/warn":"restricted"' && echo "$l" | grep -q '"pod-security.kubernetes.io/audit":"restricted"'"""),
+  C("deployment no longer asks for privilege", 3, r"""
+t=$(j -n %{ns} deploy %{dep} '{.spec.template.spec}')
+! echo "$t" | grep -q '"privileged":true' && echo "$t" | grep -q '"allowPrivilegeEscalation":false' && echo "$t" | grep -q 'RuntimeDefault'"""),
+  C("it satisfies restricted: non-root, drop ALL", 3, r"""
+t=$(j -n %{ns} deploy %{dep} '{.spec.template.spec}')
+echo "$t" | grep -q '"runAsNonRoot":true' && echo "$t" | grep -q '"drop":\["ALL"\]'"""),
+  C("2 replicas Available under enforcement, image unchanged", 4, r"""
+eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.containers[0].image}')" nginx:1.27
+for i in $(seq 1 12); do [ "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" = 2 ] && exit 0; sleep 5; done; exit 1"""),
+ ],
+ "solution": """
+k label ns %{ns} \\
+  pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/enforce-version=latest \\
+  pod-security.kubernetes.io/warn=restricted pod-security.kubernetes.io/audit=restricted --overwrite
+k -n %{ns} rollout restart deploy %{dep}     # now blocked, see the events for the exact reasons
+k -n %{ns} get events --field-selector reason=FailedCreate
+k -n %{ns} edit deploy %{dep}                # pod template needs, at minimum:
+#   securityContext: {runAsNonRoot: true, runAsUser: 101, seccompProfile: {type: RuntimeDefault}}
+#   container securityContext: {allowPrivilegeEscalation: false, capabilities: {drop: [ALL]}}
+#   and remove privileged: true
+k -n %{ns} rollout status deploy %{dep}
+# restricted requires all four: runAsNonRoot, allowPrivilegeEscalation=false, drop ALL,
+# and a RuntimeDefault (or Localhost) seccomp profile.
+""",
+},
+{
+ "id": 7, "domain": "MICRO", "title": "Isolate a workload's ServiceAccount and namespace",
+ "params": lambda r: {"ns": ns_pick(r, "tenant"), "pod": name_pick(r, "isolated")},
+ "task": """
+Create Pod `%{pod}` in Namespace `%{ns}` (image `busybox:1.36`, `sleep 86400`) that:
+
+- uses ServiceAccount `%{pod}-sa` (create it),
+- does **not** get an API token mounted, set on the **Pod**, not the ServiceAccount,
+- runs with `hostNetwork`, `hostPID` and `hostIPC` all explicitly disabled,
+- has both CPU and memory requests and limits set (any sane values).
+""",
+ "setup": NS + r"""
+mkns %{ns}
+$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null
+$K -n %{ns} delete sa %{pod}-sa --ignore-not-found >/dev/null
+""",
+ "checks": [
+  C("uses its own ServiceAccount", 2, r"""$K -n %{ns} get sa %{pod}-sa >/dev/null 2>&1 && eq "$(j -n %{ns} pod %{pod} '{.spec.serviceAccountName}')" %{pod}-sa"""),
+  C("token not mounted, set on the pod spec", 3, r"""
+eq "$(j -n %{ns} pod %{pod} '{.spec.automountServiceAccountToken}')" false \
+ && ! $K -n %{ns} exec %{pod} -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null"""),
+  C("host namespaces explicitly disabled", 3, r"""
+p="$K -n %{ns} get pod %{pod}"
+eq "$($p -o jsonpath='{.spec.hostNetwork}' 2>/dev/null)" false && eq "$($p -o jsonpath='{.spec.hostPID}')" false && eq "$($p -o jsonpath='{.spec.hostIPC}')" false"""),
+  C("requests and limits set for cpu and memory", 2, r"""
+c='{.spec.containers[0].resources'
+for f in requests.cpu requests.memory limits.cpu limits.memory; do [ -n "$(j -n %{ns} pod %{pod} "$c.$f}")" ] || exit 1; done"""),
+ ],
+ "solution": """
+k -n %{ns} create sa %{pod}-sa
+cat <<EOF | k apply -f -
+apiVersion: v1
+kind: Pod
+metadata: {name: %{pod}, namespace: %{ns}}
+spec:
+  serviceAccountName: %{pod}-sa
+  automountServiceAccountToken: false
+  hostNetwork: false
+  hostPID: false
+  hostIPC: false
+  containers:
+    - name: c
+      image: busybox:1.36
+      command: [sleep, "86400"]
+      resources:
+        requests: {cpu: 50m, memory: 64Mi}
+        limits:   {cpu: 200m, memory: 128Mi}
+EOF
+k -n %{ns} exec %{pod} -- ls /var/run/secrets/kubernetes.io/ 2>&1 || echo "no token, as intended"
+""",
+},
+# ---------------------------------------------------------------- SUPPLY ----
+{
+ "id": 8, "domain": "SUPPLY", "title": "Admission policy for allowed registries", "difficulty": "hard",
+ "params": lambda r: {"ns": ns_pick(r, "trusted"), "reg": r.choice(["registry.internal.io", "harbor.corp.io"])},
+ "task": """
+Only images from `%{reg}/` may run in Namespace `%{ns}`.
+
+Using **ValidatingAdmissionPolicy** (no third-party admission controller):
+1. Create a `ValidatingAdmissionPolicy` named `allowed-registry` that denies any Pod whose
+   containers are not all from `%{reg}/`.
+2. Bind it with a `ValidatingAdmissionPolicyBinding` named `allowed-registry` that enforces
+   (`validationActions: [Deny]`) on Namespace `%{ns}` only.
+3. Prove it: creating a Pod from `nginx:1.27` in `%{ns}` must be rejected. Save the rejection
+   message to `{WS}/q8/denied.txt`.
+""",
+ "setup": NS + r"""
+mkns %{ns}
+rm -rf "$WS/q8"; mkdir -p "$WS/q8"
+$K label ns %{ns} registry-policy=enforced --overwrite >/dev/null
+$K delete validatingadmissionpolicybinding allowed-registry --ignore-not-found >/dev/null 2>&1
+$K delete validatingadmissionpolicy allowed-registry --ignore-not-found >/dev/null 2>&1
+""",
+ "checks": [
+  C("ValidatingAdmissionPolicy allowed-registry exists and checks image prefixes", 3, r"""
+v=$(j validatingadmissionpolicy allowed-registry '{.spec.validations[*].expression}' 2>/dev/null)
+[ -n "$v" ] && echo "$v" | grep -q '%{reg}' && j validatingadmissionpolicy allowed-registry '{.spec.matchConstraints.resourceRules[*].resources[*]}' | grep -q pods"""),
+  C("binding enforces Deny and is scoped to %{ns}", 3, r"""
+b="$K get validatingadmissionpolicybinding allowed-registry"
+eq "$($b -o jsonpath='{.spec.policyName}' 2>/dev/null)" allowed-registry && $b -o jsonpath='{.spec.validationActions[*]}' | grep -q Deny \
+ && $b -o jsonpath='{.spec.matchResources}' | grep -Eq '%{ns}|registry-policy'"""),
+  C("a disallowed image is actually rejected", 4, r"""
+out=$($K -n %{ns} run vap-probe --image=nginx:1.27 --restart=Never 2>&1); rc=$?
+$K -n %{ns} delete pod vap-probe --ignore-not-found --wait=false >/dev/null 2>&1
+[ $rc -ne 0 ] && echo "$out" | grep -qi 'denied\|not allowed\|failed'"""),
+  C("denied.txt captures the rejection", 2, r"""grep -Eqi 'denied|not allowed|admission' "$WS/q8/denied.txt" 2>/dev/null"""),
+ ],
+ "solution": """
+cat <<EOF | k apply -f -
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingAdmissionPolicy
+metadata: {name: allowed-registry}
+spec:
+  failurePolicy: Fail
+  matchConstraints:
+    resourceRules:
+      - apiGroups: [""]
+        apiVersions: ["v1"]
+        operations: [CREATE, UPDATE]
+        resources: ["pods"]
+  validations:
+    - expression: "object.spec.containers.all(c, c.image.startsWith('%{reg}/'))"
+      message: "images must come from %{reg}/"
+---
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingAdmissionPolicyBinding
+metadata: {name: allowed-registry}
+spec:
+  policyName: allowed-registry
+  validationActions: [Deny]
+  matchResources:
+    namespaceSelector:
+      matchLabels: {registry-policy: enforced}
+EOF
+k -n %{ns} run vap-probe --image=nginx:1.27 2>&1 | tee {WS}/q8/denied.txt      # rejected
+# CEL runs in-process in the apiserver - no webhook to deploy, no certificate to manage.
+# Add initContainers to the expression too if the exam asks for every container.
+""",
+},
+# --------------------------------------------------------------- RUNTIME ----
+{
+ "id": 9, "domain": "RUNTIME", "title": "Turn on API audit logging", "breaks": "apiserver-flag", "difficulty": "hard",
+ "task": """
+The cluster has no audit trail.
+
+On the control-plane node:
+1. Write an audit policy to `/etc/kubernetes/audit-policy.yaml` that logs Secret access at
+   `Metadata` level and everything else at `Metadata` too, but drops the noisy
+   `RequestReceived` stage.
+2. Configure the kube-apiserver to use it, writing to `/var/log/kubernetes/audit.log`, keeping
+   7 days of logs and at most 5 rotated files.
+3. Bring the API server back healthy, then generate an event (`kubectl get secrets -A`) and copy
+   the last audit line mentioning `secrets` to `{WS}/q9/audit-line.json`.
+""",
+ "setup": r"""
+rm -rf "$WS/q9"; mkdir -p "$WS/q9"
+nx "$CP" 'sed -i "/--audit-/d" /etc/kubernetes/manifests/kube-apiserver.yaml; rm -f /etc/kubernetes/audit-policy.yaml' || true
+""",
+ "checks": [
+  C("audit policy file exists and omits RequestReceived", 3, r"""
+nx "$CP" 'test -f /etc/kubernetes/audit-policy.yaml' && nx "$CP" 'grep -q "omitStages" /etc/kubernetes/audit-policy.yaml' \
+ && nx "$CP" 'grep -q "RequestReceived" /etc/kubernetes/audit-policy.yaml'"""),
+  C("apiserver flags: policy file, log path, maxage 7, maxbackup 5", 4, r"""
+m=$(nx "$CP" 'cat /etc/kubernetes/manifests/kube-apiserver.yaml')
+echo "$m" | grep -q -- '--audit-policy-file=/etc/kubernetes/audit-policy.yaml' \
+ && echo "$m" | grep -q -- '--audit-log-path=/var/log/kubernetes/audit.log' \
+ && echo "$m" | grep -q -- '--audit-log-maxage=7' && echo "$m" | grep -q -- '--audit-log-maxbackup=5'"""),
+  C("apiserver healthy and writing the log", 4, r"""
+for i in $(seq 1 20); do $K get --raw=/healthz >/dev/null 2>&1 && break; sleep 6; done
+$K get --raw=/healthz >/dev/null 2>&1 || exit 1
+$K get secrets -A >/dev/null 2>&1
+for i in $(seq 1 6); do nx "$CP" 'test -s /var/log/kubernetes/audit.log' && exit 0; sleep 5; done; exit 1"""),
+  C("audit-line.json is a real audit event about secrets", 3, r"""
+python3 -c '
+import json,sys
+d=json.load(open("'"$WS"'/q9/audit-line.json"))
+sys.exit(0 if d.get("kind")=="Event" and "secrets" in json.dumps(d) else 1)' 2>/dev/null"""),
+ ],
+ "solution": """
+{SHELL_CP}
+mkdir -p /var/log/kubernetes
+cat > /etc/kubernetes/audit-policy.yaml <<'EOF'
+apiVersion: audit.k8s.io/v1
+kind: Policy
+omitStages:
+  - RequestReceived
+rules:
+  - level: Metadata
+    resources:
+      - group: ""
+        resources: ["secrets"]
+  - level: Metadata
+EOF
+vi /etc/kubernetes/manifests/kube-apiserver.yaml
+#   command:
+#     - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
+#     - --audit-log-path=/var/log/kubernetes/audit.log
+#     - --audit-log-maxage=7
+#     - --audit-log-maxbackup=5
+#   volumeMounts:
+#     - {name: audit-policy, mountPath: /etc/kubernetes/audit-policy.yaml, readOnly: true}
+#     - {name: audit-log, mountPath: /var/log/kubernetes}
+#   volumes:
+#     - {name: audit-policy, hostPath: {path: /etc/kubernetes/audit-policy.yaml, type: File}}
+#     - {name: audit-log, hostPath: {path: /var/log/kubernetes, type: DirectoryOrCreate}}
+exit
+k get secrets -A
+# grab the last secrets line:
+#   grep secrets /var/log/kubernetes/audit.log | tail -1   -> {WS}/q9/audit-line.json
+# Forgetting the two hostPath mounts is the usual reason the apiserver crashloops here:
+# the flags point at paths that don't exist inside the container.
+""",
+},
+]

web/index.html

@@ -0,0 +1,468 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Exam simulator</title>
+<style>
+  :root {
+    --bg: #eceef1; --pane: #ffffff; --ink: #1d2126; --muted: #5f6b78; --line: #d5dae1;
+    --accent: #1f5fbf; --accent-ink: #ffffff; --warn: #b7791f; --danger: #b42318; --ok: #1e7f4f;
+    --flag: #d9a400; --term: #1a1b1e;
+    --sans: "IBM Plex Sans", "Segoe UI", system-ui, -apple-system, sans-serif;
+    --mono: "IBM Plex Mono", "JetBrains Mono", ui-monospace, Menlo, Consolas, monospace;
+  }
+  * { box-sizing: border-box; }
+  html, body { height: 100%; margin: 0; }
+  body { font: 14px/1.5 var(--sans); color: var(--ink); background: var(--bg); }
+  button { font: inherit; cursor: pointer; }
+  a { color: var(--accent); }
+  code { font-family: var(--mono); font-size: .92em; }
+  :focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
+
+  /* ---- top bar: exam name left, clock right, controls far right ---- */
+  header { height: 48px; display: flex; align-items: center; gap: 16px; padding: 0 14px;
+           background: #252a31; color: #e8ebef; }
+  header .name { font-weight: 600; }
+  header .mode { color: #a9b2bd; font-size: 13px; }
+  header .spacer { flex: 1; }
+  #clock { font-family: var(--mono); font-size: 20px; letter-spacing: .04em; font-variant-numeric: tabular-nums; }
+  #clock.warn { color: #f5c451; } #clock.danger { color: #ff7b72; } #clock.paused { color: #a9b2bd; }
+  .btn { border: 1px solid var(--line); background: var(--pane); color: var(--ink); border-radius: 4px; padding: 6px 12px; }
+  .btn.primary { background: var(--accent); border-color: var(--accent); color: var(--accent-ink); }
+  .btn.danger { background: var(--danger); border-color: var(--danger); color: #fff; }
+  .btn.ghost { background: transparent; color: #e8ebef; border-color: #4a525c; }
+  .btn:disabled { opacity: .5; cursor: default; }
+
+  /* ---- main split ---- */
+  main { height: calc(100% - 48px); display: grid; grid-template-columns: 56px 440px 6px 1fr; }
+  #qnav { background: #f6f7f9; border-right: 1px solid var(--line); overflow-y: auto; padding: 8px 0; }
+  #qnav button { display: block; width: 40px; height: 34px; margin: 4px auto; border: 1px solid var(--line); border-radius: 4px;
+                 background: var(--pane); position: relative; font-variant-numeric: tabular-nums; }
+  #qnav button.current { border-color: var(--accent); box-shadow: inset 0 0 0 1px var(--accent); font-weight: 600; }
+  #qnav button.flagged::after { content: ""; position: absolute; top: -4px; right: -4px; width: 10px; height: 10px;
+                                 background: var(--flag); border-radius: 50%; border: 2px solid #f6f7f9; }
+  #qnav button.pass { background: #e3f3ea; border-color: var(--ok); }
+  #qnav button.partial { background: #fbf1dc; border-color: var(--warn); }
+  #qnav button.fail { background: #f9e3e1; border-color: var(--danger); }
+
+  #task { background: var(--pane); display: flex; flex-direction: column; min-height: 0; }
+  #task .head { padding: 14px 18px 10px; border-bottom: 1px solid var(--line); }
+  #task .head .sub { color: var(--muted); font-size: 13px; }
+  #task h2 { margin: 2px 0 0; font-size: 17px; font-weight: 600; }
+  #task .body { flex: 1; overflow-y: auto; padding: 14px 18px; }
+  .prose p { margin: 0 0 10px; white-space: pre-wrap; }
+  .prose code { background: #eef2f7; padding: 1px 5px; border-radius: 3px; cursor: copy; border: 1px solid #dfe5ed; }
+  .prose code:hover { background: #dce6f5; }
+  .prose code.copied { background: #cfe9da; border-color: var(--ok); }
+  #task .foot { border-top: 1px solid var(--line); padding: 10px 18px; display: flex; gap: 8px; align-items: center; }
+  #task .foot .spacer { flex: 1; }
+  .btn.flag.on { background: #fff4c7; border-color: var(--flag); }
+  #checks { margin-top: 12px; border: 1px solid var(--line); border-radius: 4px; padding: 10px 12px; font-size: 13px; }
+  #checks div { padding: 2px 0; }
+  #solution { margin-top: 12px; border: 1px solid var(--line); border-radius: 4px; padding: 8px 12px; font-size: 13px; }
+  #solution summary { cursor: pointer; font-weight: 600; }
+  #solution pre { margin: 8px 0 0; white-space: pre-wrap; font: 12.5px/1.5 var(--mono); background: #f6f7f9; padding: 10px; border-radius: 4px; }
+  #checks .ok::before { content: "✓ "; color: var(--ok); } #checks .no::before { content: "✗ "; color: var(--danger); }
+  #notes { width: 100%; height: 120px; border: 0; border-top: 1px solid var(--line); resize: none; padding: 10px 18px;
+           font: 13px/1.45 var(--mono); background: #fbfbf6; color: var(--ink); }
+  #notes::placeholder { color: #9aa3ad; }
+
+  #handle { background: var(--line); cursor: col-resize; }
+  #termwrap { background: var(--term); position: relative; min-width: 0; }
+  #term { width: 100%; height: 100%; border: 0; display: block; }
+  #noterm { color: #cbd2da; padding: 32px; font-family: var(--mono); font-size: 13px; line-height: 1.7; max-width: 70ch; }
+  #noterm code { color: #f5c451; }
+
+  /* ---- overlays: start, confirm, results ---- */
+  .overlay { position: fixed; inset: 0; background: rgba(20,24,30,.55); display: flex; align-items: center; justify-content: center; }
+  .card { position: relative; background: var(--pane); border-radius: 6px; padding: 26px 28px; width: min(640px, 92vw); max-height: 88vh; overflow-y: auto; }
+  .card h1 { margin: 0 0 4px; font-size: 20px; }
+  .card .sub { color: var(--muted); margin-bottom: 18px; }
+  .card .row { display: flex; gap: 10px; flex-wrap: wrap; align-items: center; margin-top: 16px; }
+  .opts { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; }
+  .opt { border: 1px solid var(--line); border-radius: 6px; padding: 14px; text-align: left; background: var(--pane); }
+  .opt:hover { border-color: var(--accent); }
+  .opt b { display: block; margin-bottom: 4px; }
+  .opt span { color: var(--muted); font-size: 13px; }
+  .picker { columns: 2; margin: 10px 0 0; font-size: 13px; }
+  .picker label { display: block; padding: 2px 0; }
+  pre.log { background: #f6f7f9; border: 1px solid var(--line); padding: 10px; font: 12px/1.5 var(--mono); max-height: 240px; overflow: auto; }
+  .score { font-size: 34px; font-family: var(--mono); }
+  .score small { font-size: 14px; color: var(--muted); font-family: var(--sans); }
+  .verdict { display: inline-block; padding: 2px 10px; border-radius: 4px; color: #fff; margin-left: 10px; font-size: 14px; vertical-align: middle; }
+  .verdict.pass { background: var(--ok); } .verdict.fail { background: var(--danger); }
+  .bar { display: grid; grid-template-columns: 70px 1fr 60px; gap: 10px; align-items: center; margin: 6px 0; font-size: 13px; }
+  .bar .track { height: 10px; background: #e6e9ee; border-radius: 5px; overflow: hidden; }
+  .bar .fill { height: 100%; background: var(--accent); }
+  .qres { margin-top: 16px; }
+  .qres details { border-top: 1px solid var(--line); padding: 6px 0; }
+  .qres summary { cursor: pointer; }
+  .qres summary .pts { float: right; font-family: var(--mono); }
+  .qres .t-pass summary .pts { color: var(--ok); } .qres .t-fail summary .pts { color: var(--danger); } .qres .t-partial summary .pts { color: var(--warn); }
+  .hidden { display: none !important; }
+  .pill { font-size:12px; padding:2px 8px; border-radius:10px; border:1px solid var(--line); color:var(--muted); }
+  .pill.sel { background:var(--accent); border-color:var(--accent); color:#fff; }
+  .pill.pick { cursor:pointer; }
+  .pill.pick:hover { border-color:var(--accent); color:var(--accent); }
+  .pill.done.pick:hover { color:var(--ok); }
+  .pill.done { background:#e3f3ea; border-color:var(--ok); color:var(--ok); }
+  .hardtag { font-size:11px; background:var(--danger); color:#fff; border-radius:3px; padding:1px 5px; margin-left:6px; vertical-align:middle; }
+  @media (prefers-reduced-motion: no-preference) { .prose code { transition: background .15s; } }
+  @media (max-width: 900px) { main { grid-template-columns: 56px 1fr; } #handle, #termwrap { display: none; } }
+</style>
+</head>
+<body>
+<header>
+  <span class="name" id="trackName">Exam simulator</span>
+  <span class="mode" id="modeLabel"></span>
+  <span class="spacer"></span>
+  <span id="clock">--:--:--</span>
+  <button class="btn ghost" id="btnPause">Pause</button>
+  <button class="btn ghost" id="btnHistory">History</button>
+  <button class="btn danger" id="btnEnd">Submit exam</button>
+</header>
+
+<main>
+  <nav id="qnav" aria-label="Questions"></nav>
+  <section id="task">
+    <div class="head">
+      <div class="sub" id="qMeta"></div>
+      <h2 id="qTitle"></h2>
+    </div>
+    <div class="body">
+      <div class="prose" id="qBody"></div>
+      <div id="checks" class="hidden"></div>
+      <details id="solution" class="hidden"><summary>Reference solution</summary><pre id="solutionBody"></pre></details>
+    </div>
+    <div class="foot">
+      <button class="btn" id="btnPrev">Previous</button>
+      <button class="btn" id="btnNext">Next</button>
+      <span class="spacer"></span>
+      <button class="btn" id="btnCheck">Check this task</button>
+      <button class="btn flag" id="btnFlag">Flag for review</button>
+    </div>
+    <textarea id="notes" placeholder="Notepad — kept with the attempt, not graded."></textarea>
+  </section>
+  <div id="handle" role="separator" aria-orientation="vertical"></div>
+  <section id="termwrap"></section>
+</main>
+
+<!-- start / lobby -->
+<div class="overlay" id="lobby">
+  <div class="card">
+    <h1 id="lobbyTitle">Exam simulator</h1>
+    <div class="sub" id="lobbySub"></div>
+    <div id="ver" style="position:absolute;right:28px;top:30px;color:var(--muted);font-size:12px"></div>
+    <div id="trackPick" style="display:flex;gap:6px;margin:0 0 14px"></div>
+    <div style="color:var(--muted);font-size:12px;margin:-8px 0 4px">Drill a domain (✓ = drilled at least once):</div>
+    <div id="coverage" style="display:flex;gap:6px;margin:0 0 14px;flex-wrap:wrap"></div>
+    <label style="display:block;margin-bottom:14px"><input type="checkbox" id="hardMode"> <b>Hard mode</b>
+      <span style="color:var(--muted)"> — killer.sh-style: more tasks in the same time, no hints, traps, exact-output tasks, compound failures, collateral checks, exam contexts.</span></label>
+    <div class="opts" id="lobbyOpts">
+      <button class="opt" id="optStudy"><b>Study mode</b><span>5 tasks, no clock, reference solutions visible from the start. Nothing scored or recorded.</span></button>
+      <button class="opt" id="optNext"><b>Drill next domain</b><span id="nextDesc"></span></button>
+      <button class="opt" id="optAll"><b>Drill the whole bank</b><span id="allDesc"></span></button>
+      <button class="opt" id="optDrill"><b>Drill: weak / pick</b><span>Tasks you're weak on, or choose your own. Feedback per task while you work.</span></button>
+      <button class="opt" id="optSimA"><b>Simulator Set A</b><span id="simADesc"></span></button>
+      <button class="opt" id="optSimB"><b>Simulator Set B</b><span id="simBDesc"></span></button>
+      <button class="opt" id="optExam" style="grid-column:1/3"><b>Full exam</b><span id="examDesc"></span></button>
+    </div>
+    <div id="gate" class="hidden" style="color:var(--warn);margin-top:10px;font-size:13px"></div>
+    <div id="drillPick" class="hidden">
+      <div class="sub" style="margin-top:14px">Pick tasks to drill. Pre-selected: weak or never attempted.</div>
+      <div class="picker" id="picker"></div>
+      <div class="row"><button class="btn primary" id="btnStartDrill">Start drill</button><button class="btn" id="btnBackDrill">Back</button></div>
+    </div>
+    <div id="seeding" class="hidden">
+      <div class="sub">Seeding scenarios on the cluster…</div>
+      <pre class="log" id="seedLog"></pre>
+    </div>
+    <div id="lobbyErr" class="hidden" style="color:var(--danger);margin-top:12px;white-space:pre-wrap"></div>
+  </div>
+</div>
+
+<!-- confirm submit -->
+<div class="overlay hidden" id="confirm">
+  <div class="card">
+    <h1>Submit and end the session?</h1>
+    <div class="sub">Every task is graded against the cluster as it is right now. You can't return after submitting.</div>
+    <div id="confirmFlags"></div>
+    <div class="row"><button class="btn danger" id="btnConfirmEnd">Submit</button><button class="btn" id="btnCancelEnd">Keep working</button></div>
+  </div>
+</div>
+
+<!-- results -->
+<div class="overlay hidden" id="results">
+  <div class="card">
+    <h1>Results</h1>
+    <div class="sub" id="resSub"></div>
+    <div><span class="score" id="resScore"></span><span class="verdict" id="resVerdict"></span></div>
+    <div id="resDomains" style="margin-top:14px"></div>
+    <div class="qres" id="resQuestions"></div>
+    <div class="row">
+      <button class="btn primary" id="btnDrillMissed">Drill what I missed</button>
+      <button class="btn" id="btnNewExam">New full exam</button>
+      <button class="btn" id="btnCloseRes">Review tasks</button>
+    </div>
+  </div>
+</div>
+
+<!-- history -->
+<div class="overlay hidden" id="histOv">
+  <div class="card">
+    <h1>Attempt history</h1>
+    <div class="sub" id="histSub"></div>
+    <div id="histBody"></div>
+    <div class="row"><button class="btn" id="btnCloseHist">Close</button></div>
+  </div>
+</div>
+
+<script>
+(() => {
+  const $ = s => document.querySelector(s);
+  let S = null, cur = null, skew = 0, clockT = null, autoSubmitted = false;
+  const api = (p, body) => fetch(p, body ? {method:'POST', headers:{'Content-Type':'application/json'}, body: JSON.stringify(body)} : {})
+    .then(async r => { const j = await r.json(); if (!r.ok) throw new Error(j.error || r.statusText); return j; });
+  const fmt = s => { s = Math.max(0, Math.floor(s)); return [s/3600, s%3600/60, s%60].map(n => String(Math.floor(n)).padStart(2,'0')).join(':'); };
+  const esc = s => s.replace(/[&<>]/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;'}[c]));
+
+  // minimal markdown: paragraphs, **bold**, `code` (click to copy). Lists stay as literal lines.
+  function render(md) {
+    return md.split(/\n\s*\n/).map(p => '<p>' + esc(p).replace(/\*\*(.+?)\*\*/g, '<b>$1</b>')
+      .replace(/`([^`]+)`/g, (m, c) => `<code title="click to copy">${c}</code>`) + '</p>').join('');
+  }
+
+  function qResultClass(q) {
+    const r = S.session && S.session.results[q.id];
+    if (!r) return '';
+    return r.pass ? 'pass' : (r.earned ? 'partial' : 'fail');
+  }
+
+  function paint() {
+    const sess = S.session, live = sess && !sess.ended_at;
+    $('#trackName').textContent = S.track.name;
+    $('#modeLabel').textContent = sess ? (sess.untimed ? 'Study' : (sess.mode === 'drill' ? 'Drill' : 'Exam')) + (sess.hard ? ' · HARD' : '') : '';
+    $('#btnEnd').classList.toggle('hidden', !live);
+    if (live && sess.untimed) $('#btnEnd').textContent = 'End study session';
+    $('#btnPause').classList.toggle('hidden', !live || sess.untimed);
+    if (live) $('#btnPause').textContent = sess.paused ? 'Resume' : 'Pause';
+    $('#btnCheck').classList.toggle('hidden', !(live && sess.mode === 'drill'));
+    $('#lobby').classList.toggle('hidden', !!live || (sess && sess.ended_at && !$('#results').classList.contains('hidden')));
+    // question nav
+    const nav = $('#qnav'); nav.innerHTML = '';
+    S.questions.forEach(q => {
+      const b = document.createElement('button'); b.textContent = q.id;
+      b.title = `${S.questions.indexOf(q) + 1} of ${S.questions.length} — Q${q.id}: ${q.title}`;
+      b.className = [q.id === cur ? 'current' : '', sess && sess.flags.includes(q.id) ? 'flagged' : '',
+                     (sess && (sess.mode === 'drill' || sess.ended_at)) ? qResultClass(q) : ''].join(' ');
+      b.onclick = () => { cur = q.id; paint(); };
+      nav.appendChild(b);
+    });
+    if (!S.questions.find(q => q.id === cur)) cur = S.questions[0] && S.questions[0].id;
+    const q = S.questions.find(q => q.id === cur);
+    if (q) {
+      const pos = S.questions.indexOf(q) + 1;
+      $('#qMeta').textContent = `Question ${pos} of ${S.questions.length} · Q${q.id} · ${S.track.domains[q.domain]} · ${q.weight} points`;
+      $('#qTitle').innerHTML = esc(q.title) + (q.difficulty === 'hard' ? '<span class="hardtag">hard</span>' : '');
+      $('#qBody').innerHTML = render(q.task);
+      $('#qBody').querySelectorAll('code').forEach(c => c.onclick = () => {
+        navigator.clipboard.writeText(c.textContent).then(() => { c.classList.add('copied'); setTimeout(() => c.classList.remove('copied'), 700); });
+      });
+      $('#btnFlag').classList.toggle('on', !!(sess && sess.flags.includes(q.id)));
+      $('#btnFlag').textContent = sess && sess.flags.includes(q.id) ? 'Flagged' : 'Flag for review';
+      const r = sess && sess.results[q.id], show = r && (sess.mode === 'drill' || sess.ended_at);
+      $('#checks').classList.toggle('hidden', !show);
+      $('#solution').classList.toggle('hidden', !q.solution);
+      if (q.solution) $('#solutionBody').textContent = q.solution;
+      if (show) $('#checks').innerHTML = `<b>${r.earned}/${r.total} points</b>` +
+        r.checks.map(c => `<div class="${c.pass ? 'ok' : 'no'}">${esc(c.name)} <span style="color:var(--muted)">(${c.pts})</span></div>`).join('');
+      const i = S.questions.indexOf(q);
+      $('#btnPrev').disabled = i === 0; $('#btnNext').disabled = i === S.questions.length - 1;
+    }
+    if (sess && document.activeElement !== $('#notes')) $('#notes').value = sess.notes || '';
+    // terminal
+    const tw = $('#termwrap');
+    if (!tw.dataset.ready) {
+      tw.dataset.ready = 1;
+      tw.innerHTML = S.ttyd ? `<iframe id="term" src="http://127.0.0.1:${S.ttyd_port}/" title="terminal"></iframe>` :
+        `<div id="noterm">The terminal pane needs <code>ttyd</code>, which isn't installed.<br><br>
+         Install it (<code>dnf install ttyd</code>, <code>pacman -S ttyd</code>, <code>brew install ttyd</code>, or a release from github.com/tsl0922/ttyd)
+         and restart <code>./exam web</code>.<br><br>Until then, work in any terminal with kubectl pointed at the cluster.
+         Task files go under <code>${S.workspace}/</code>, relative to the cka-sim directory.</div>`;
+    }
+    tickClock();
+  }
+
+  function tickClock() {
+    clearInterval(clockT);
+    const sess = S.session, el = $('#clock');
+    if (!sess || sess.ended_at) { el.textContent = sess ? 'ended' : '--:--:--'; el.className = ''; return; }
+    if (sess.untimed) { el.textContent = 'study'; el.className = 'paused'; return; }
+    const step = () => {
+      const now = Date.now()/1000 - skew;
+      const pausedTotal = sess.paused_total + (sess.paused ? (now - sess.server_time) : 0);
+      const left = sess.ends_at + pausedTotal - now;
+      el.textContent = (sess.paused ? '⏸ ' : '') + fmt(left);
+      el.className = sess.paused ? 'paused' : (left < 5*60 ? 'danger' : (left < 15*60 ? 'warn' : ''));
+      if (sess.paused) return;
+      if (left <= 0 && !autoSubmitted) { autoSubmitted = true; clearInterval(clockT); submit(); }
+    };
+    step(); clockT = setInterval(step, 1000);
+  }
+
+  async function load() {
+    S = await api('/api/session');
+    if (S.session) skew = Date.now()/1000 - S.session.server_time;
+    const sess = S.session;
+    const hard = $('#hardMode').checked, qn = hard ? (S.track.quota_hard || {}) : (S.track.quota || {});
+    const total = Object.values(qn).reduce((a, b) => a + b, 0) || S.bank.length;
+    $('#examDesc').textContent = `${total} tasks sampled across the 5 domains, ${S.track.minutes} minutes, pass mark ${S.track.pass_pct}%. No feedback until you submit.`;
+    const drilled = S.drilled || {}, undone = Object.keys(S.track.domains).filter(d => !drilled[d]);
+    $('#coverage').innerHTML = Object.keys(S.track.domains).map(d =>
+      `<span class="pill pick ${drilled[d] ? 'done' : ''}" data-domain="${d}" title="Drill ${S.track.domains[d]} — 5 tasks">${d} ${drilled[d] ? '✓' : '·'}</span>`).join('');
+    $('#coverage').querySelectorAll('[data-domain]').forEach(el => el.onclick = () => start('drill', null, false, false, el.dataset.domain));
+    const nxt = Object.keys(S.track.domains).sort((a, b) => (drilled[a] || 0) - (drilled[b] || 0))[0];
+    const eligible = S.bank.filter(q => hard || q.difficulty !== 'hard'), seenN = eligible.filter(q => (S.seen || []).includes(q.id)).length;
+    $('#allDesc').textContent = `${seenN}/${eligible.length} tasks attempted. 5 unseen tasks per session, round-robin across domains${hard ? ', hard first' : ''}; then whatever you failed.`;
+    $('#nextDesc').textContent = `${S.track.domains[nxt]} — 5 tasks, ${hard ? 5 : 6} min each. Drills rotate through all domains before the exam.`;
+    for (const s of ['a', 'b']) {
+      const info = (S.sets || {})[s];
+      const el = $('#sim' + s.toUpperCase() + 'Desc');
+      if (el) el.textContent = info ? `${info.count} fixed tasks, ${S.track.minutes} min, exam rules. Hardest set in the bank — closest to a real sitting.` : '';
+    }
+    $('#gate').classList.toggle('hidden', !undone.length);
+    $('#gate').innerHTML = undone.length ? `Drill every domain once before the full exam. Not yet drilled: ${undone.join(', ')}. <a href="#" id="forceExam">Start the exam anyway</a>` : '';
+    $('#optExam').disabled = !!undone.length; $('#optExam').style.opacity = undone.length ? .5 : 1;
+    const fe = $('#forceExam'); if (fe) fe.onclick = e => { e.preventDefault(); start('exam'); };
+    $('#lobbyTitle').textContent = S.track.name;
+    $('#trackPick').innerHTML = (S.tracks || []).map(t =>
+      `<span class="pill pick ${t === S.track_id ? 'sel' : ''}" data-track="${t}">${t.toUpperCase()}</span>`).join('');
+    $('#trackPick').querySelectorAll('[data-track]').forEach(el => el.onclick = async () => {
+      if (el.dataset.track === S.track_id) return;
+      S = await api('/api/track', {track: el.dataset.track}); cur = null; load();
+    }); $('#ver').textContent = 'exam ' + (S.version || '?');
+    $('#lobbySub').textContent = sess && sess.ended_at ? `Last session ended with ${sess.score.pct}%. Start another, or review it under History.` : 'Choose a session. Seeding the cluster takes about a minute.';
+    if (sess && sess.ended_at && !$('#results').dataset.shown) showResults();
+    paint();
+  }
+
+  async function startSim(name) {
+    $('#lobbyOpts').classList.add('hidden'); $('#lobbyErr').classList.add('hidden');
+    $('#seeding').classList.remove('hidden'); $('#seedLog').textContent = 'starting…';
+    try {
+      await api('/api/start', {mode: 'exam', sim: name, hard: true});
+      for (;;) {
+        await new Promise(r => setTimeout(r, 1200));
+        const s = await api('/api/session');
+        $('#seedLog').textContent = ((s.setup_log || []).join('\n') || 'starting…').replace(/\x1b\[[0-9;]*m/g, '');
+        $('#seedLog').scrollTop = $('#seedLog').scrollHeight;
+        if (!s.seeding) { if (s.seed_error) throw new Error(s.seed_error); S = s; break; }
+      }
+      skew = Date.now()/1000 - S.session.server_time; autoSubmitted = false;
+      $('#results').dataset.shown = ''; $('#results').classList.add('hidden'); cur = S.questions[0].id; paint();
+    } catch (e) {
+      $('#lobbyErr').textContent = e.message; $('#lobbyErr').classList.remove('hidden'); $('#lobbyOpts').classList.remove('hidden');
+    } finally { $('#seeding').classList.add('hidden'); }
+  }
+
+  async function start(mode, ids, next, campaign, domain, study) {
+    $('#lobbyOpts').classList.add('hidden'); $('#drillPick').classList.add('hidden'); $('#lobbyErr').classList.add('hidden');
+    $('#seeding').classList.remove('hidden'); $('#seedLog').textContent = 'starting…';
+    try {
+      await api('/api/start', {mode, ids, next: !!next, campaign: !!campaign, domain: domain || null, study: !!study, hard: $('#hardMode').checked});
+      // seeding runs in the background; stream its log until it finishes
+      for (;;) {
+        await new Promise(r => setTimeout(r, 1200));
+        const s = await api('/api/session');
+        $('#seedLog').textContent = ((s.setup_log || []).join('\n') || 'starting…').replace(/\x1b\[[0-9;]*m/g, '');
+        $('#seedLog').scrollTop = $('#seedLog').scrollHeight;
+        if (!s.seeding) {
+          if (s.seed_error) throw new Error(s.seed_error);
+          S = s; break;
+        }
+      }
+      skew = Date.now()/1000 - S.session.server_time; autoSubmitted = false;
+      $('#results').dataset.shown = ''; $('#results').classList.add('hidden'); cur = S.questions[0].id; paint();
+    } catch (e) {
+      $('#lobbyErr').textContent = e.message === 'Failed to fetch' ? 'Lost contact with ./exam web — is it still running? Restart it and reload this page.' : e.message;
+      $('#lobbyErr').classList.remove('hidden'); $('#lobbyOpts').classList.remove('hidden');
+    } finally { $('#seeding').classList.add('hidden'); }
+  }
+
+  async function submit() {
+    $('#confirm').classList.add('hidden');
+    $('#btnEnd').disabled = true; $('#btnEnd').textContent = 'Grading…';
+    try { S = await api('/api/end', {}); } finally { $('#btnEnd').disabled = false; $('#btnEnd').textContent = 'Submit exam'; }
+    showResults(); paint();
+  }
+
+  function showResults() {
+    const s = S.session, sc = s.score;
+    $('#results').dataset.shown = 1;
+    $('#resSub').textContent = `${s.mode === 'drill' ? 'Drill' : 'Full exam'} · ${fmt(s.ended_at - s.started_at - s.paused_total)} used${s.over_time ? ' (over time)' : ''}${s.pauses ? ` · paused ${s.pauses}× (${fmt(s.paused_total)})` : ''}`;
+    $('#resScore').innerHTML = `${sc.pct}% <small>${sc.earned} / ${sc.total} points</small>`;
+    const v = $('#resVerdict'); v.textContent = sc.pct >= sc.pass_pct ? 'Pass' : 'Fail'; v.className = 'verdict ' + (sc.pct >= sc.pass_pct ? 'pass' : 'fail');
+    $('#resDomains').innerHTML = Object.entries(s.domains).map(([d, x]) =>
+      `<div class="bar"><span>${d}</span><div class="track"><div class="fill" style="width:${100*x.earned/x.total}%"></div></div><span>${x.earned}/${x.total}</span></div>`).join('');
+    $('#resQuestions').innerHTML = S.questions.map(q => { const r = s.results[q.id]; if (!r) return '';
+      const cls = r.pass ? 't-pass' : (r.earned ? 't-partial' : 't-fail');
+      return `<details class="${cls}"><summary>Q${q.id} ${esc(q.title)}<span class="pts">${r.earned}/${r.total}</span></summary>` +
+        r.checks.map(c => `<div class="${c.pass ? 'ok' : 'no'}" style="padding-left:14px">${c.pass ? '✓' : '✗'} ${esc(c.name)} (${c.pts})</div>`).join('') +
+        (q.solution ? `<details style="margin:6px 0 4px 14px"><summary style="font-size:13px">Reference solution</summary><pre style="white-space:pre-wrap;font:12px/1.5 var(--mono);background:#f6f7f9;padding:8px;border-radius:4px">${esc(q.solution)}</pre></details>` : '') + '</details>'; }).join('');
+    const missed = S.questions.filter(q => s.results[q.id] && !s.results[q.id].pass).map(q => q.id);
+    $('#btnDrillMissed').disabled = !missed.length; $('#btnDrillMissed').onclick = () => { $('#results').classList.add('hidden'); start('drill', missed); };
+    $('#results').classList.remove('hidden'); $('#lobby').classList.add('hidden');
+  }
+
+  async function showHistory() {
+    const h = await api('/api/history');
+    $('#histSub').textContent = h.history.length ? `${h.history.length} attempt(s) on this track` : 'No attempts yet.';
+    $('#histBody').innerHTML = h.history.slice().reverse().map(e =>
+      `<div class="bar"><span>${e.mode}${e.hard ? '*' : ''}</span><div><b>${e.score_pct}%</b> · ${e.at.replace('T',' ')} · ${fmt(e.time_used)} · ${e.ids.length === S.questions.length ? 'all tasks' : 'Q' + e.ids.join(', Q')}</div><span></span></div>`).join('') +
+      (h.weak.length ? `<div style="margin-top:12px;color:var(--muted)">Weak or unattempted: Q${h.weak.join(', Q')}</div>` : '');
+    $('#histOv').classList.remove('hidden');
+  }
+
+  // ---- wiring ----
+  $('#optExam').onclick = () => start('exam');
+  $('#optNext').onclick = () => start('drill', null, true);
+  $('#optStudy').onclick = () => start('drill', null, true, false, null, true);
+  $('#optAll').onclick = () => start('drill', null, false, true);
+  $('#optSimA').onclick = () => startSim('a');
+  $('#optSimB').onclick = () => startSim('b');
+  $('#hardMode').onchange = () => load();
+  $('#optDrill').onclick = async () => {
+    const h = await api('/api/history');
+    const hard = $('#hardMode').checked;
+    $('#picker').innerHTML = S.bank.filter(q => hard || q.difficulty !== 'hard').map(q => `<label><input type="checkbox" value="${q.id}" ${h.weak.includes(q.id) ? 'checked' : ''}> Q${q.id} ${q.domain} ${esc(q.title)}${q.difficulty === 'hard' ? '<span class="hardtag">hard</span>' : ''}</label>`).join('');
+    $('#lobbyOpts').classList.add('hidden'); $('#drillPick').classList.remove('hidden');
+  };
+  $('#btnBackDrill').onclick = () => { $('#drillPick').classList.add('hidden'); $('#lobbyOpts').classList.remove('hidden'); };
+  $('#btnStartDrill').onclick = () => { const ids = [...$('#picker').querySelectorAll('input:checked')].map(i => +i.value); if (ids.length) start('drill', ids); };
+  $('#btnPrev').onclick = () => { const i = S.questions.findIndex(q => q.id === cur); cur = S.questions[i-1].id; paint(); };
+  $('#btnNext').onclick = () => { const i = S.questions.findIndex(q => q.id === cur); cur = S.questions[i+1].id; paint(); };
+  $('#btnFlag').onclick = async () => { S = await api('/api/flag', {id: cur}); paint(); };
+  $('#btnCheck').onclick = async () => { $('#btnCheck').disabled = true; try { S = await api('/api/grade', {id: cur}); paint(); } catch (e) { alert(e.message); } finally { $('#btnCheck').disabled = false; } };
+  let nt; $('#notes').oninput = () => { clearTimeout(nt); nt = setTimeout(() => api('/api/notes', {notes: $('#notes').value}), 600); };
+  $('#btnEnd').onclick = () => {
+    const f = S.session.flags, un = S.questions.filter(q => !S.session.results[q.id]).length;
+    $('#confirmFlags').innerHTML = f.length ? `<div>Flagged for review: Q${f.join(', Q')}</div>` : '';
+    $('#confirm').classList.remove('hidden');
+  };
+  $('#btnCancelEnd').onclick = () => $('#confirm').classList.add('hidden');
+  $('#btnConfirmEnd').onclick = submit;
+  $('#btnCloseRes').onclick = () => { $('#results').classList.add('hidden'); $('#lobby').classList.add('hidden'); };
+  $('#btnNewExam').onclick = () => { $('#results').classList.add('hidden'); $('#lobby').classList.remove('hidden'); $('#lobbyOpts').classList.remove('hidden'); };
+  $('#btnHistory').onclick = showHistory;
+  $('#btnPause').onclick = async () => { S = await api('/api/pause', {paused: !S.session.paused}); skew = Date.now()/1000 - S.session.server_time; paint(); };
+  $('#btnCloseHist').onclick = () => $('#histOv').classList.add('hidden');
+  // draggable split
+  const handle = $('#handle'), main = document.querySelector('main');
+  handle.onmousedown = e => { e.preventDefault(); const mv = ev => { main.style.gridTemplateColumns = `56px ${Math.max(320, ev.clientX - 56)}px 6px 1fr`; };
+    const up = () => { removeEventListener('mousemove', mv); removeEventListener('mouseup', up); }; addEventListener('mousemove', mv); addEventListener('mouseup', up); };
+  load();
+})();
+</script>
+</body>
+</html>

workspace/q21/chart/Chart.yaml

@@ -0,0 +1,6 @@
+apiVersion: v2
+name: chart
+description: A small web chart for the exam simulator
+type: application
+version: 0.1.0
+appVersion: "1.27"

workspace/q21/chart/templates/_helpers.tpl

@@ -0,0 +1,3 @@
+{{- define "chart.fullname" -}}
+{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}

workspace/q21/chart/templates/deployment.yaml

@@ -0,0 +1,17 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "chart.fullname" . }}
+  labels: {app: {{ include "chart.fullname" . }}}
+spec:
+  replicas: {{ .Values.replicaCount }}
+  selector:
+    matchLabels: {app: {{ include "chart.fullname" . }}}
+  template:
+    metadata:
+      labels: {app: {{ include "chart.fullname" . }}}
+    spec:
+      containers:
+        - name: web
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+          ports: [{containerPort: 80}]

workspace/q21/chart/templates/service.yaml

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ include "chart.fullname" . }}
+spec:
+  type: {{ .Values.service.type }}
+  selector: {app: {{ include "chart.fullname" . }}}
+  ports: [{port: {{ .Values.service.port }}, targetPort: 80}]

workspace/q21/chart/values.yaml

@@ -0,0 +1,7 @@
+replicaCount: 1
+image:
+  repository: nginx
+  tag: "1.27"
+service:
+  type: ClusterIP
+  port: 80