cka-sim/README.md

# cka-sim — Kubernetes certification exam simulator

Entry point is `./exam` (renamed from `cka-sim` so it can't collide with the directory name).

Hands-on exam tasks graded **per check** against a real cluster, an exam clock, a browser
front end modelled on the real PSI exam, attempt history, and drills that rotate through the
domains before you sit a full exam. Tracks: **CKA** (136 tasks: a random pool of 43 normal +
24 hard, plus four fixed simulator sets A–D of 17–18 tasks each), CKAD and CKS (scaffolded — same
format, add questions when you get there).

## How a session is built

Every session draws **fresh** tasks: `start` samples per domain in the exam's proportions
(ARCH 5, WORK 4, NET 4, STOR 2, TSHOOT 5 = 20), preferring tasks you have never seen, then the
least-recently seen. Most tasks are parameterized — namespaces, names, replica counts, ports,
which node is broken — so even a repeat isn't the same task. `drill` with no arguments picks the
domain you've drilled least and gives you 5 tasks from it; a full exam is gated until every
domain has been drilled at least once (`--force` skips that).

**Hard mode** (`--hard`, or the toggle in the UI lobby) is modelled on what people report about
killer.sh, not on its content: 25 tasks in the same 120 minutes, hard tasks preferred, no hints
about paths or nodes, traps (an over-granting binding you must find, an allow-all NetworkPolicy
left behind, a relocated `staticPodPath`, two PVs where only one is right), exact-output tasks
graded byte-for-byte, compound failures (three faults on one node, two control-plane components
down for different reasons, DNS broken three ways), collateral-damage checks (still cordoned,
recreated instead of fixed in place, extra PVs), and exam-style contexts `k8s-c1-H` /
`k8s-c2-AC` / `k8s-c3-CCC` with different default namespaces — each task says which to use.
Grading always runs against the real cluster context regardless of which one you're on.

## Why kind and not k3s / hosted labs

kind is kubeadm-based: real etcd, static-pod control plane, systemd kubelet on every node,
`kubeadm certs` on the control plane. That is what lets the troubleshooting tasks break real
things (kubelet, scheduler manifest) the way the exam does. Hosted labs and CK-X give you no
node access, which rules out exactly those tasks. `podman exec -it <node> bash` plays the role
of `ssh <node>` here; use the Vagrant/ssh mode below if you want a real ssh session.

## Requirements

podman, kind, kubectl, python3, **ttyd** (for the terminal pane in the browser UI).
~4 GB RAM for the 4-node cluster. Rootless podman needs cgroup v2; otherwise `sudo -E ./setup.sh`.
Docker still works: `CONTAINER_RUNTIME=docker ./setup.sh`.

## Run

```
./setup.sh          # create the kind cluster, preload images (idempotent)
./exam web       # open http://127.0.0.1:8080 — start an exam or drill from there
```

Both the built-in terminal and your own shell can have the exam aliases:

```
source ./examrc          # bash: k=kubectl, $do, $now, kubectl completion (also for k), vim defaults
source ./examrc.fish     # fish equivalent
```

`./exam web` sources the right one automatically for its terminal pane.

That's the whole loop. The UI seeds the scenarios, runs the clock, lets you flag questions and
keep notes, submits (or auto-submits at 00:00), shows the score with every check, and offers
"Drill what I missed". The terminal pane is a real `bash` via ttyd with kubectl already pointed
at the cluster (`--shell fish` if you'd rather practise in fish; the real exam is bash).

The same state is available from the terminal if you prefer:

```
./exam drill                      # next least-drilled domain, 5 tasks (add --hard)
./exam drill weak | drill 2 17 20 | drill TSHOOT NET | drill setd
./exam start [--hard]             # full exam (gated until all domains drilled; --force)
./exam sim a|b|c|d                # a fixed 17/18-task simulator set, exam rules, 120 min
./exam study [DOMAIN|weak|N..]    # no clock, solutions visible, nothing recorded
./exam list | show N | timer | status                       # during
./exam end                                                  # submit
./exam weak | history                                       # afterwards
./exam --track ckad web                                     # other tracks
./teardown.sh                                                  # delete cluster + state (history kept)
```

## Scoring, history, drills

Each task is a list of named, weighted checks (`tracks/cka/questions.py`). You get the points
for every check that passes, so a task can score 4/6 — the same partial-credit model as the
exam. Every submit appends to `history-<track>.jsonl` and regenerates `weak-spots-<track>.md`:
latest domain scores, a "drill next" section listing the *failing checks* per task, and a
✓ / n-of-m matrix across all attempts.

A task counts as weak if it failed on its latest attempt, or has failed at least twice and has
not yet been passed twice in a row since, or was never attempted (fixed-set tasks excepted — note
that this makes an early `drill weak` very large; name tasks explicitly for a short drill). Drills
give 6 minutes per task and — unlike exam mode — let you check a task while you work. History
entries marked `"synthetic": true` (scripted verification runs, not a person) are kept in the file
but left out of every statistic.

## Using your own cluster instead of kind

Any kubeadm-style cluster with 1 control-plane + 3 workers works. Skip `setup.sh`, set (Fish):

```
set -x CKA_CONTEXT     kubernetes-admin@kubernetes
set -x CKA_NODE_ACCESS ssh        # podman | docker | ssh | vagrant
set -x CKA_CP cp1;  set -x CKA_W1 node1;  set -x CKA_W2 node2;  set -x CKA_W3 node3
./exam web
```

`ssh` needs passwordless `ssh <node-name>` (for Vagrant: `vagrant ssh-config >> ~/.ssh/config`);
`vagrant` runs `vagrant ssh` from the Vagrantfile directory. Tasks 2 and 20 write to
`/opt/cka` and `/etc/kubernetes/manifests`, so the login must be able to. Managed clusters
(EKS/AKS/GKE) won't work — no node or control-plane access. Startup preflights all of this.

The etcd tasks (2 and 102) need `etcdctl`/`etcdutl` **on the control-plane node**, as the real
exam has them — they cannot be copied out of the running pod, since the etcd image is distroless.
`setup.sh` installs them on kind; on any cluster, seeding those tasks installs them if they are
missing (from the etcd image in the node's containerd, else the upstream release, else apt) and
fails loudly if it cannot. `./etcd-tools.sh` does the same by hand.

## CKA bank

| Domain | Normal (43) | Hard (24) |
|---|---|---|
| Architecture | RBAC, etcd snapshot, Kustomize, CRD, static pod, certs/CSR, Helm, aggregated ClusterRole, `kubectl explain` on a CRD, kubeadm join token, component versions | exact RBAC with an over-grant to remove, **etcd restore**, static pod on a relocated path, user → self-contained kubeconfig, Kustomize generators + patches, CRD validation/defaults, renew all control-plane certs |
| Workloads | rolling update, sidecar, node affinity + resources, ConfigMap/Secret, metrics-server repair + HPA, PriorityClass, taints, DaemonSet, CronJob, init container, immutable ConfigMap rotation, parallel Job, quota + LimitRange | anti-affinity + toleration + priority under a quota, rollback to a specific revision then pause, native sidecar + probes + exact report, cluster audit to exact files, work in the context's namespace |
| Networking | NetworkPolicy, NodePort, Ingress, cluster DNS, Gateway API, named ports, egress-only policy, headless + ExternalName | zero-trust namespace (4 policies + trap), TLS Ingress with generated cert + fingerprint, Gateway canary with header match and weights, DNS down three ways |
| Storage | PV/PVC/pod, default StorageClass, re-bind a Released PV, data survives the pod | bind the right local volume by selector, migrate data between claims via a Job |
| Troubleshooting | node NotReady, broken deployment + service, CrashLoopBackOff, scheduler down, CoreDNS misconfig, controller-manager down, kubelet config drift | worker with three faults (+ uncordon check), two control-plane components down, app down for four reasons fixed in place (UID check), cluster facts exactly |

Disruptive tasks (anything that breaks a node or control-plane component) are seeded last and
can block other tasks until fixed — read everything first, as in the real exam. Tasks that damage
or depend on the same shared state declare it in `breaks` and never share a session, and the etcd
restore (Q102) is `solo`: restoring rolls the whole cluster back, so it only runs on its own.
Before every session a baseline pass repairs anything a previous, unfinished session left broken.

The kindnet CNI here **enforces** NetworkPolicy (it embeds a policy engine), so a wrong policy
really drops traffic; no ingress or Gateway controller runs, so those are graded on the objects.

**Simulator sets** (`./exam sim a|b|c|d`, never drawn into random sessions): A and B are
killer.sh-grade investigate-and-write-to-file sets, C is modelled on an 18-question commercial
mock sitting, and D is killer.sh-style — control-plane inventory, event forensics with crictl,
a broken kubelet unit, kubelet certificates, manual scheduling with the scheduler stopped, Helm
housekeeping with a stuck release, and an enforced AND-vs-OR NetworkPolicy. All are original
scenarios, not copies of any commercial simulator.

## Adding tasks or a new track

Banks live in `tracks/cka/{questions,bank_extra,bank_hard,bank_sim_*}.py`; a task is:

```python
{"id": 44, "domain": "NET", "difficulty": "normal",   # or "hard"
 "params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])},
 "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
 "setup": "bash that seeds the scenario (may use %{ns})",
 "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
 "disruptive": False,          # True: seeded last, after everything else is scheduled
 "breaks": ["w1-taint"],       # shared state it damages or needs intact; never paired with a task sharing any
 "solo": False}                # True: only ever drilled on its own
```

Scripts get `$K` (pinned to the grading context), `$WS`, `$CP/$W1/$W2/$W3`, `nx NODE 'cmd'`,
and helpers `j <get args> '{jsonpath}'` and `eq a b`. Task text may use `{WS}`, `{CP}`, `{W2}`,
`{W3}`, `{SHELL_CP}` etc. `TRACK["quota"]` / `["quota_hard"]` set how many tasks per domain a
session draws. For CKAD/CKS, fill in `QUESTIONS` in the
existing placeholder track; `TRACK` already carries the domain table and pass mark.