cka-sim/README.md

# cka-sim — Kubernetes certification exam simulator

Entry point is `./exam` (renamed from `cka-sim` so it can't collide with the directory name).

Hands-on exam tasks graded **per check** against a real cluster, an exam clock, a browser
front end modelled on the real PSI exam, attempt history, and drills that rotate through the
domains before you sit a full exam. Tracks: **CKA** (102 tasks: a random pool of 43 normal +
24 hard, plus two fixed simulator sets, C and D, of 18 and 17 tasks), **CKAD** (37 pool tasks, at
least one for every competency in the curriculum, plus a 20-task killer.sh-style Set A) and CKS (scaffolded — same format, add
questions when you get there).

## How a session is built

Every session draws **fresh** tasks: `start` samples per domain in the exam's proportions
(ARCH 5, WORK 4, NET 4, STOR 2, TSHOOT 5 = 20), preferring tasks you have never seen, then the
least-recently seen. Most tasks are parameterized — namespaces, names, replica counts, ports,
which node is broken — so even a repeat isn't the same task. `drill` with no arguments picks the
domain you've drilled least and gives you 5 tasks from it; a full exam is gated until every
domain has been drilled at least once (`--force` skips that).

**Hard mode** (`--hard`, or the toggle in the UI lobby) is modelled on what people report about
killer.sh, not on its content: 25 tasks in the same 120 minutes, hard tasks preferred, no hints
about paths or nodes, traps (an over-granting binding you must find, an allow-all NetworkPolicy
left behind, a relocated `staticPodPath`, two PVs where only one is right), exact-output tasks
graded byte-for-byte, compound failures (three faults on one node, two control-plane components
down for different reasons, DNS broken three ways), collateral-damage checks (still cordoned,
recreated instead of fixed in place, extra PVs), and exam-style contexts `k8s-c1-H` /
`k8s-c2-AC` / `k8s-c3-CCC` with different default namespaces — each task says which to use.
Grading always runs against the real cluster context regardless of which one you're on.

## Why kind and not k3s / hosted labs

kind is kubeadm-based: real etcd, static-pod control plane, systemd kubelet on every node,
`kubeadm certs` on the control plane. That is what lets the troubleshooting tasks break real
things (kubelet, scheduler manifest) the way the exam does. Hosted labs and CK-X give you no
node access, which rules out exactly those tasks. `podman exec -it <node> bash` plays the role
of `ssh <node>` here; use the Vagrant/ssh mode below if you want a real ssh session.

## Requirements

podman, kind, kubectl, python3, **ttyd** (for the terminal pane in the browser UI).
~4 GB RAM for the 4-node cluster. Rootless podman needs cgroup v2; otherwise `sudo -E ./setup.sh`.
Docker still works: `CONTAINER_RUNTIME=docker ./setup.sh`.

## Run

```
./setup.sh          # create the kind cluster, preload images (idempotent)
./exam web       # open http://127.0.0.1:8080 — start an exam or drill from there
```

Both the built-in terminal and your own shell can have the exam aliases:

```
source ./examrc          # bash: k=kubectl, $do, $now, kubectl completion (also for k), vim defaults
source ./examrc.fish     # fish equivalent
```

`./exam web` sources the right one automatically for its terminal pane.

That's the whole loop. The UI seeds the scenarios, runs the clock, lets you flag questions and
keep notes, submits (or auto-submits at 00:00), shows the score with every check, and offers
"Drill what I missed". The terminal pane is a real `bash` via ttyd with kubectl already pointed
at the cluster (`--shell fish` if you'd rather practise in fish; the real exam is bash).

The same state is available from the terminal if you prefer:

```
./exam drill                      # next least-drilled domain, 5 tasks (add --hard)
./exam drill weak | drill 2 17 20 | drill TSHOOT NET | drill setd
./exam start [--hard]             # full exam (gated until all domains drilled; --force)
./exam sim c|d                    # a fixed 18/17-task simulator set, exam rules, 120 min
./exam study [DOMAIN|weak|N..]    # no clock, solutions visible, nothing recorded
./exam list | show N | timer | status                       # during
./exam end                                                  # submit
./exam weak | history                                       # afterwards
./exam --track ckad web                                     # other tracks (or CKA_TRACK=ckad)
./teardown.sh                                                  # delete cluster + state (history kept)
```

## CKAD track

`./exam --track ckad ...` (every command takes it). A mock draws **16** tasks in 120 minutes,
weighted like the exam: Design and Build 3, Deployment 3, Observability 3, Environment and
Security 4, Networking 3, from a pool of 37 (8/7/7/10/5), so no two mocks are the same set. Every
line of the curriculum has at least one task: image builds, Jobs, DaemonSets, native sidecars,
volumes; blue/green, canary, rolling-update limits, Helm, Kustomize; API deprecations, probes,
`kubectl top`, logs, debugging (CrashLoopBackOff, OOMKilled, config errors); CRDs, RBAC, Pod
Security admission, quotas and LimitRanges, QoS, ConfigMaps, Secrets, ServiceAccounts;
NetworkPolicy (tested with real traffic), Services, Ingress with TLS.

A few tasks need more than kubectl on this machine: `podman` (Q13 builds an image), `helm`
(Q6, Q21), and network access the first time Q23 runs on a fresh cluster (it installs
metrics-server). Drill a domain with `./exam --track ckad drill ENV`, as for CKA.

**Simulator Set A** (`./exam --track ckad sim a`): 20 original killer.sh-style tasks, 140 points,
120 minutes, exam rules - a fixed set (ids 38-57) kept out of the random pool, so reruns are
comparable. Riverline, a fictional river-freight company, one Namespace per river. Expect several
steps per task, exact names and file outputs, and a few things the pool never asks: debugging a
container with no shell through an ephemeral container, a Kustomize patch with a hashed
configMapGenerator, a Helm release stuck in `failed`, fsGroup on a shared volume, and the
two-source NetworkPolicy trap.

## Scoring, history, drills

Each task is a list of named, weighted checks (`tracks/cka/questions.py`). You get the points
for every check that passes, so a task can score 4/6 — the same partial-credit model as the
exam. Every submit appends to `history-<track>.jsonl` and regenerates `weak-spots-<track>.md`:
latest domain scores, a "drill next" section listing the *failing checks* per task, and a
✓ / n-of-m matrix across all attempts.

A task counts as weak if it failed on its latest attempt, or has failed at least twice and has
not yet been passed twice in a row since, or was never attempted (fixed-set tasks excepted — note
that this makes an early `drill weak` very large; name tasks explicitly for a short drill). Drills
give 6 minutes per task and — unlike exam mode — let you check a task while you work. History
entries marked `"synthetic": true` (scripted verification runs, not a person) are kept in the file
but left out of every statistic.

## Using your own cluster instead of kind

Any kubeadm-style cluster with 1 control-plane + 3 workers works. Skip `setup.sh`, set (Fish):

```
set -x CKA_CONTEXT     kubernetes-admin@kubernetes
set -x CKA_NODE_ACCESS ssh        # podman | docker | ssh | vagrant
set -x CKA_CP cp1;  set -x CKA_W1 node1;  set -x CKA_W2 node2;  set -x CKA_W3 node3
./exam web
```

`ssh` needs passwordless `ssh <node-name>` (for Vagrant: `vagrant ssh-config >> ~/.ssh/config`);
`vagrant` runs `vagrant ssh` from the Vagrantfile directory. Tasks 2 and 20 write to
`/opt/cka` and `/etc/kubernetes/manifests`, so the login must be able to. Managed clusters
(EKS/AKS/GKE) won't work — no node or control-plane access. Startup preflights all of this.

The etcd tasks (2 and 45) need `etcdctl`/`etcdutl` **on the control-plane node**, as the real
exam has them — they cannot be copied out of the running pod, since the etcd image is distroless.
`setup.sh` installs them on kind; on any cluster, seeding those tasks installs them if they are
missing (from the etcd image in the node's containerd, else the upstream release, else apt) and
fails loudly if it cannot. `./etcd-tools.sh` does the same by hand.

## CKA bank

| Domain | Normal (43) | Hard (24) |
|---|---|---|
| Architecture | RBAC, etcd snapshot, Kustomize, CRD, static pod, certs/CSR, Helm, aggregated ClusterRole, `kubectl explain` on a CRD, kubeadm join token, component versions | exact RBAC with an over-grant to remove, **etcd restore**, static pod on a relocated path, user → self-contained kubeconfig, Kustomize generators + patches, CRD validation/defaults, renew all control-plane certs |
| Workloads | rolling update, sidecar, node affinity + resources, ConfigMap/Secret, metrics-server repair + HPA, PriorityClass, taints, DaemonSet, CronJob, init container, immutable ConfigMap rotation, parallel Job, quota + LimitRange | anti-affinity + toleration + priority under a quota, rollback to a specific revision then pause, native sidecar + probes + exact report, cluster audit to exact files, work in the context's namespace |
| Networking | NetworkPolicy, NodePort, Ingress, cluster DNS, Gateway API, named ports, egress-only policy, headless + ExternalName | zero-trust namespace (4 policies + trap), TLS Ingress with generated cert + fingerprint, Gateway canary with header match and weights, DNS down three ways |
| Storage | PV/PVC/pod, default StorageClass, re-bind a Released PV, data survives the pod | bind the right local volume by selector, migrate data between claims via a Job |
| Troubleshooting | node NotReady, broken deployment + service, CrashLoopBackOff, scheduler down, CoreDNS misconfig, controller-manager down, kubelet config drift | worker with three faults (+ uncordon check), two control-plane components down, app down for four reasons fixed in place (UID check), cluster facts exactly |

Disruptive tasks (anything that breaks a node or control-plane component) are seeded last and
can block other tasks until fixed — read everything first, as in the real exam. Tasks that damage
or depend on the same shared state declare it in `breaks` and never share a session, and the etcd
restore (Q45) is `solo`: restoring rolls the whole cluster back, so it only runs on its own.
Before every session a baseline pass repairs anything a previous, unfinished session left broken.

The kindnet CNI here **enforces** NetworkPolicy (it embeds a policy engine), so a wrong policy
really drops traffic; no ingress or Gateway controller runs, so those are graded on the objects.

**Simulator sets** (`./exam sim c|d`, never drawn into random sessions): C is modelled on an
18-question commercial mock sitting, and D is killer.sh-style — control-plane inventory, event
forensics with crictl, a broken kubelet unit, kubelet certificates, manual scheduling with the
scheduler stopped, Helm housekeeping with a stuck release, and an enforced AND-vs-OR
NetworkPolicy. Both are original scenarios, not copies of any commercial simulator. (Sets A and B
were retired in favour of these two; their last versions are in git history.)

## Adding tasks or a new track

Banks live in `tracks/cka/{questions,bank_extra,bank_hard,bank_sim_*}.py`; a task is:

```python
{"id": 103, "domain": "NET", "difficulty": "normal",   # or "hard" - ids stay contiguous
 "params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])},
 "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
 "setup": "bash that seeds the scenario (may use %{ns})",
 "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
 "disruptive": False,          # True: seeded last, after everything else is scheduled
 "breaks": ["w1-taint"],       # shared state it damages or needs intact; never paired with a task sharing any
 "solo": False}                # True: only ever drilled on its own
```

Scripts get `$K` (pinned to the grading context), `$WS`, `$CP/$W1/$W2/$W3`, `nx NODE 'cmd'`,
and helpers `j <get args> '{jsonpath}'` and `eq a b`. Task text may use `{WS}`, `{CP}`, `{W2}`,
`{W3}`, `{SHELL_CP}` etc. `TRACK["quota"]` / `["quota_hard"]` set how many tasks per domain a
session draws. For CKS, fill in `QUESTIONS` in the
existing placeholder track; `TRACK` already carries the domain table and pass mark.