kubestronaut-sim/README.md

# kubestronaut-sim — CKA / CKAD / CKS exam simulator

A local exam simulator laid out like the real thing: a **remote desktop in the browser** (XFCE,
terminal, Firefox limited to the documentation the exam allows), a task pane beside it, and
**one cluster per task** that you reach with `ssh <host>` — the way the Linux Foundation exams have
run since 2025, and the way killer.sh and sailor.sh present them.

It grew out of `cka-sim`: the same task banks, per-check grading, clock, history, weak-spot log,
drills and a path through each curriculum, on a new foundation. Every session runs against the
clock.

```
 browser ── http://127.0.0.1:8080 ──────────────────────────────────────────────┐
 │ task pane                    │ remote desktop (noVNC)                         │
 │  Solve this question on:     │  candidate@base:~$ ssh cka0037                 │
 │  ssh cka0037                 │  candidate@cka0037:~$ sudo -i                  │
 └──────────────┬───────────────┴───────────────────┬───────────────────────────┘
                │ ./ksim (engine, on this machine)  │ container "ksim-desktop"
                │  seeds and grades each task       │  ssh
                ▼                                   ▼
        cka0037  = kind cluster   cka0037-control-plane (+ workers)   ← its own etcd, kubelet,
        cka0012  = kind cluster   cka0012-control-plane (+ workers)     static pods, /opt/course
        ...        one per task in the session, all on the podman network "ksim"
```

## Why one cluster per task

On a shared cluster a task that breaks the kubelet, restores etcd or changes API server flags
damages every other task. `cka-sim` worked around that with `breaks`/`solo`/`disruptive` flags, a
baseline-restore pass before each session, and tasks that could never share a session (the etcd
restore could only be drilled alone). All of that is gone:

- any task can sit next to any other, including the etcd restore in a full mock;
- `./ksim reset N` deletes the task's cluster and builds it again — a real reset;
- tasks are provisioned, seeded and graded in parallel;
- you work as `candidate` on the task's host and use `sudo -i` for node work, as on the day.

Each host is a kind node built from `images/node/` on **Ubuntu 24.04**, as the exam's hosts are —
a kubeadm cluster (real etcd, static-pod control plane, systemd kubelet). kubelet, kubeadm and
kubectl are apt packages from pkgs.k8s.io, held, with the repository line of the cluster's minor
(the CNI plugins come with them, as `kubernetes-cni`): a node is upgraded with `apt-get install`
and `kubeadm upgrade`, for real, and a task can hold one back on an older release. containerd,
runc, crictl, kind's entrypoint and the preloaded control-plane images are copied from the pinned
kind node image, which kind builds on Debian. On top: sshd, `candidate` with passwordless sudo,
`k` and kubectl completion preset, vim (2-space tabs), tmux, jq, yq, helm, man pages, podman (also
as `docker`) for the image tasks, and `etcdctl`/`etcdutl` at the version of the cluster's etcd.

**CKS hosts** add what that exam's hosts carry (`images/node-cks/`, built on the common image):
kube-bench, trivy, kubesec, kube-linter, bom, cosign, istioctl and gVisor (`runsc`, as a containerd
runtime handler).

**VM hosts.** A container shares this machine's kernel, and under rootless podman it can neither
load eBPF programs nor use a kernel feature this machine lacks: no Falco, no AppArmor. Tasks that
need them (`"backend": "vm"`) get an Ubuntu 24.04 virtual machine instead (`images/vm/`): QEMU with
KVM, run in a container that carries the host's name, so `ssh cks0105` from the desktop reaches
the VM exactly as it reaches a container host. Each VM makes its own single-node kubeadm cluster
when it first boots, and is ready in about half a minute.

A fresh cluster has no images, so every host pulls Docker Hub images through a cache on this
machine (the container `ksim-registry`, a pull-through mirror with its own volume). Without it a
full mock would fetch the same nginx and busybox dozens of times and hit Docker Hub's limit of 100
anonymous pulls an hour; with it, each image is fetched from Docker Hub once.

## Requirements

podman (rootless is fine, cgroup v2), kind, kubectl, python3, ssh-keygen; helm, openssl and curl
for seeding some tasks. Nothing else: no ttyd, no kubectl context of yours is ever touched (every
host has its own kubeconfig under `.state/`). VM hosts additionally need KVM (`/dev/kvm` usable
by you); QEMU itself comes in a container image.

Memory and start-up time are the price of a cluster per task. Measured on a 32-core machine:

| Session | Clusters | Ready in | Memory |
|---|---|---|---|
| CKAD drill, 3 tasks | 3 × 1 node | about 1 min | about 1 GB |
| CKA mock, 20 tasks | 20 clusters, 50 nodes | just under 6 min, 6 at a time | about 8 GB |
| CKS drill, 1 VM task + 1 container task | 1 VM, 1 × 2 nodes | about 1 min | about 2 GB |

A control-plane node costs roughly 0.3 GB and a worker under 0.1 GB once seeded; a session is
refused before anything is created if a (more cautious) estimate does not fit (`KSIM_FORCE=1`
overrides). Clusters are provisioned one per four cores at a time, at most 8 (`KSIM_PARALLEL=N`).
A VM host is given 2 GB (it used about 1.6 GB right after starting) and is up in about 30 s.

Disk: the images take 1.5 GB per Kubernetes minor (v1.37, and v1.36 for the tasks that upgrade
their cluster), 2.7 GB for the CKS host, 1.2 GB for the desktop; the VM host adds a 3.5 GB base
disk and the 0.6 GB Ubuntu cloud image under `.state/vm/`.

## Run

```
./ksim build all        # once: the exam host image per Kubernetes minor, and the desktop image
./ksim build cks --vm   # once, for CKS: also the VM host its Falco and AppArmor tasks need (~10 min)
./ksim web              # http://127.0.0.1:8080 — pick a track and a session on the landing page
```

The browser UI follows the layout of [CK-X](https://github.com/sailor-sh/CK-X) (itself modelled on
the Linux Foundation's exam UI), written anew for this simulator:

- **Landing page** — the track (CKA, CKAD, CKS) in the top bar; **Start exam** opens a dialog to
  pick the session: the next step on the path, an objective, a domain, the next domain, the whole
  bank, tasks of your choice, a simulator set or a full exam, with or without hard mode. Below it:
  your path, a card per domain, and every objective with its notes and the "I know this — skip it"
  links. While the clusters are built, an overlay shows their progress and the log.
- **Exam screen** — a dark top bar with the clock and three menus: *Exam Interface* (fullscreen,
  panel sizes, reconnect the desktop), *Exam Controls* (submit, pause, **Restart clock**,
  **Terminate session**) and *Exam Information* (exam instructions, Kubernetes docs, your history).
  On the left, `< Question N >` and each task headed by its host: click `ssh ckad0037` (or any
  highlighted value) to put it on the desktop's clipboard and paste in the terminal with
  **Ctrl+Shift+V**. **Flag for review** and **Satisfied with answer** sit under the task; a drill
  adds **Check this task**, and a task's reference solution opens once it is graded. On the right,
  the remote desktop. The clock starts when you click **Start** in the dialog that opens the session.
- **Results page** — points and pass or fail, every task with its checks, the answers, what to drill
  next, and a way back to the tasks to review them on their hosts.
- **Theme** — Auto (your system's light or dark setting, the default), Light or Dark, from the
  landing page's top bar, the *Exam Interface* menu or the results page; the browser remembers it.
  The remote desktop keeps its own look.

Firefox in the desktop opens only the allowed documentation for the track (`KSIM_OPEN_WEB=1`
lifts that). Submitting grades every task on its own host; the hosts stay up for review until the
next session starts, or until **Delete the hosts** on the results page. **Restart clock**
(`./ksim restart`) puts the session's full time back on the clock without touching its hosts or
your work; like Pause, it is recorded nowhere. **Terminate session** ends a session without grading
it and deletes its hosts, as `./ksim abort` does.

Everything is also available from a terminal. A command is for the track named with `--track` or
`KSIM_TRACK`; with neither, for the track whose session is under way, and for CKA when there is
none. Task numbers are per track — CKA's task 13 and CKAD's task 13 are different tasks — and a
host name such as `ckad0013` only resolves in the desktop's terminal: from your own, it is
`./ksim ssh 13`.

```
./ksim --track ckad drill            # next least-drilled domain, 5 tasks      (--track or KSIM_TRACK)
./ksim drill weak | drill 2 17 20 | drill TSHOOT NET | drill 3.2
./ksim start [--hard]                # full mock, sampled by domain weight
./ksim sim c|d                       # a fixed simulator set (CKA: c, d  ·  CKAD: a)
./ksim objectives | next | focus     # the curriculum as a path, one objective at a time against the clock
./ksim notes 3.2                     # an objective's notes: what to know, fastest commands, traps, docs
./ksim lesson [OBJ]                  # the track's lessons, or start one: a guided lab, no clock (CKA: all 27 · CKAD: 2.1-2.4)
./ksim lesson ARCH [--all]           # teach a domain: its lessons not done yet (--all: every one), back to back
./ksim lesson next | lesson 4.3      # in a domain's run: the next lesson, or any lesson of the run
./ksim lesson step | check | hint | show [N]   # inside a lesson; ./ksim end finishes it
./ksim skip DESIGN | unskip DESIGN   # "I am good with this domain (or objective 2.3)": off the path, and back
./ksim drill OBS:3 | next OBS:3      # three tasks of a domain, each from another objective - alone, or added
./ksim list | show N | grade N | solution N | timer | end
./ksim pause | resume | restart      # stop the clock, or put the session's full time back (hosts and work kept)
./ksim hosts [start]                 # the session's hosts; `start` brings them back after a reboot
./ksim ssh N                         # a shell on the host of task N from your own terminal
./ksim reset N                       # rebuild the host of task N from scratch
./ksim down                          # delete every host and the desktop (history and images stay)
./ksim coverage                      # tasks per domain and objective against the 17–20 target
./ksim audit [all|DOMAIN|N..]        # bank check: seed each task on its own cluster, grade it untouched,
                                     # then solve it with its script and grade again (--settle N, --keep)
./ksim lesson audit [OBJ..]          # lesson check: every step fails, is done by its own commands, then passes
```

**Lessons.** A lesson teaches one objective as a guided lab on a host of its own: an introduction,
then steps you do in the desktop's terminal, each with a **Check step** button, a hint and **Show me
how** (the exam way: what you type, what you edit in vim), and a wrap-up that sends you to
the objective's timed drill. Lessons are the one
untimed part of the simulator — they teach, the drills test — and they write nothing to the
history, the statistics or the weak list; a lesson whose every step was checked is marked done on
its objective (`lesson ✓` in the objective list). Steps count in order. Start one from an
objective's **Lesson** button, **Lesson first** on the path card, the *Take a lesson* session in the
Start exam dialog, or `./ksim lesson 2.1`.

**Teach a domain** runs a domain's lessons back to back: the ones you have not done, or every
one again once they all are (`--all` forces that). It is one session without a clock. Every
lesson's host is made before it starts, as many at a time as for any session (`KSIM_PARALLEL`),
so you never wait between lessons. Each lesson's wrap-up leads to the next, the list at the top
of a lesson goes to any lesson of the run, and a lesson counts as done the moment its last step
is checked, whether or not the run is finished. Finishing the run sends you to the domain's
timed drill. Start one from the **Teach a domain** card in the lobby, the *Teach a domain*
session in the Start exam dialog, or `./ksim lesson ARCH`.

**CKA has a lesson for each of its 27 objectives** (`cka/lessons_*.py`, 135 steps), which also
cover what a full CKA course teaches under them — checked against the outline of KubeDojo's CKA
course and written from scratch: projected volumes, volume expansion, snapshots and restores, eviction thresholds, debug containers, kube-proxy's
rules, encryption at rest, in-place resize, PodDisruptionBudgets, QoS classes, ServiceAccount
tokens, a real upgrade from v1.36 to v1.37, EndpointSlices, an operator. Tasks 132–147 test those
topics against the clock. The CKAD pilot
covers the Application Deployment domain (2.1 blue/green and canary, 2.2 rolling updates, 2.3
Helm, 2.4 Kustomize).

**Skipping what you know.** `./ksim skip DESIGN` (or the "I know this — skip it" link beside a
domain in the browser's objective list) marks a domain or an objective as known: the path, the
no-argument drill and `drill all` leave it out, and it no longer has to be drilled before a full
exam. It is a statement, not a result — nothing is written to the history, the statistics or the
weak list, the objective list shows `»` rather than `✓`, and `./ksim skip` prints what is on record
for it. A mock exam still draws from every domain, as the real one will. To keep such a domain
warm, `DOMAIN:N` takes a few of its tasks, each from another objective: `./ksim drill OBS:3`, or
`./ksim next OBS:3` to add them to the next step on the path.

## Tracks

Each track is a directory: `cka/`, `ckad/`, `cks/`.

| File | What |
|---|---|
| `questions.py` | `TRACK` (name, minutes, pass mark, Kubernetes minor, domains, mock quota, default `workers`, host `image`) and the first tasks; pulls in the other banks |
| `bank_*.py` | more tasks; `bank_sim_*` are the fixed simulator sets |
| `curriculum.py` | the exam's objectives in order, each with study notes and the tasks that practise it |
| `solutions.py` | reference solutions (CKA keeps them apart; the others inline them) |
| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved - for the tasks whose reference solution is not in parts |
| `lessons.py` | guided labs, one per objective (CKA: all, one `lessons_*.py` per domain · CKAD: 2.1–2.4): introduction, steps with their check, hint and commands, wrap-up |

| Track | Kubernetes | Bank today | Target (17–20 pool tasks per domain) |
|---|---|---|---|
| CKA | v1.37 | 147 tasks: 112 pool (18–28 per domain) + sets C, D (35) | reached in all five domains |
| CKAD | v1.37 | 105 tasks: 85 pool (17 per domain) + set A (20) | reached in all five domains |
| CKS | v1.37 | 110 tasks, all pool: 17 per domain on container hosts, plus 5 Falco and 3 AppArmor tasks on VM hosts | reached in all six domains |

`./ksim coverage` prints this per domain and per objective.

**CKA, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 147 tasks seed cleanly on a host of
their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when
their scripted solution is run on the host as `candidate` — for 80 of them that script is the
reference solution exactly as a drill shows it once the task is graded. Q132 really upgrades its
cluster from v1.36.4 to v1.37.1, worker included.

**CKAD, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 105 tasks seed cleanly on a host of
their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when
their scripted solution is run on the host as `candidate`.

**CKS, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 110 tasks seed cleanly on a host of
their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when
their scripted solution is run on the host as `candidate`. Q103, a Falco task on a VM host, failed
to seed once that day — a minute passed without an alert naming the Pod — and passed on
2026-10-20, alone and with the other seven VM tasks six at a time; its setup now tries a new Pod
before it gives up. The 26 objectives are the curriculum's own (`cks/curriculum.py`), each with
notes, commands, traps and the documentation the exam allows — which, per the Linux Foundation's
list read on 2026-10-12, is the Kubernetes docs and blog plus Falco, bom, etcd, the NGINX Ingress
Controller, Cilium and Istio. Trivy's and AppArmor's own documentation are not on it.

**On 2026-10-20**, after `hexdump` and `xxd` were added to the hosts, CKA Q132 (the upgrade from
v1.36), CKAD Q1 and CKS Q60, Q61 and Q64 passed again on the rebuilt images.

### A task

```python
{"id": 114, "domain": "NET", "difficulty": "normal",      # or "hard"
 "params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])},
 "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
 "setup": "bash that seeds the scenario on the task's own cluster",
 "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
 "solution": [cmd("k ... --dry-run=client -o yaml > f.yaml"),   # or one string, a script as it stands
              hand("vim f.yaml", "what to add, Markdown", run="the same, scripted: yq, a heredoc")],
 "workers": 2,                                              # optional: worker nodes of its cluster
 "k8s": "v1.36"}                                            # optional: a minor of its own (an upgrade task)
```

Setup and check scripts run on this machine against the task's own cluster and get:

- `$K` — kubectl for that cluster (`KUBECONFIG` is exported too, so `helm` just works);
- `$CP`, `$W1`, `$W2`, `$W3` — its node names, and `nx NODE 'cmd'` to run a command on one as root;
- `$WS` — the task's `/opt/course` directory as a path here (it is mounted into the host), so a
  check reads a file the candidate wrote with plain `cat "$WS/q14/answer.txt"`. Once seeded, the
  files belong to `candidate`: one that is readable by its owner only — a private key, a kubeconfig
  `kubectl config` wrote — has to be read on the host, `nx "$CP" 'cat /opt/course/q37/tls.key'`,
  and a kubeconfig made there is tested there too (its server address is the host's);
- `$PRIV` — a directory beside it that is *not* mounted into the host: where a setup records what
  its checks compare against later (the UID of an object that must not be re-created, say);
- `podman ...` — root's podman on the task's host, the store the candidate's `podman`/`docker` use;
- `j <get args> '{jsonpath}'`, `eq a b`, `mkns` (from `sim.bank.NS`), `ensure_gateway_crds`,
  `ensure_metrics_server`, and `ensure_csi_hostpath` — the CSI hostpath driver with the snapshot
  controller, and StorageClass `csi-hostpath-sc` (expandable) as the exam's clusters have it, for
  what local-path cannot do: growing a claim, snapshots, clones.

Task text may use `{WS}` (`/opt/course`), `{CP}`, `{W1}`.., and `{SHELL_CP}`, `{SHELL_W1}`..
(`ssh cka0114`, `ssh cka0114-worker`).

**Topology.** A task's cluster has the workers it asks for with `"workers"`. Without it, a task
that names a worker anywhere (`$W1`, `{W2}`, …) gets three — what the CKA bank was written for —
and every other task gets its track's default (`TRACK["workers"]`: CKA 1, CKAD 0, CKS 1). A task
whose `params` draw a node (`r.choice(["{W1}", "{W2}"])`) has to say `"workers"` itself: the
lambda is not searched. A
track's `WORKERS = {id: n}` map sets it without editing the task; `PENDING = {id: why}` holds a
task back from every session. A task with `"backend": "vm"` gets a single-node VM host; such
tasks hold themselves back until `./ksim build cks --vm` has been run. On a VM host, `nx` runs
its command in the VM, and nothing enforces NetworkPolicy (its Pod network is a plain bridge).

**Proving a task.** `./ksim audit N..` builds the task's host, seeds it and grades it untouched:
the seed must be clean and the task must not score full marks before anyone has worked on it. If
`<track>/solve.py` has a script for the task — what a candidate would type on the host — the audit
then runs it there as `candidate` and grades again, and only full marks pass (what the script
created gets up to 90 seconds to come up). A task with no script is proven to seed, not to be
solvable; write the script when you write the task. `--settle 45` waits that long between seeding
and the untouched grade, which catches a check that only passes once the seeded Pods run, or
until a stopped kubelet shows as NotReady; `--keep` leaves the host up to look at
(`KUBECONFIG=.state/envs/<host>/kubeconfig`, `./ksim down` removes it).

A reference solution in parts - `cmd()` and `hand()`, as a lesson step's `show` - is its own
script: the audit runs what is typed as it stands, and for each part done by hand its `run`, so the
solution the candidate reads after a drill is the one that is proven, and one that stops working
with a new Kubernetes release fails the next audit. 105 of the 147 CKA solutions are written so
(the page and `./ksim solution N` show each thing to type as a block of its own, which a click
copies, with the edit to make in `vim` beneath it); the other 42 are commands typed as they stand,
each with its script in `solve.py`. `./ksim solution lint` reads what they have you type, against
the same list of things you would not type in the exam as `./ksim lesson lint`. In `cka/solve.py`,
`AS_WRITTEN` lists the few tasks whose reference solution is still one script, which the audit
runs as it stands.

**Writing a lesson.** A lesson in `<track>/lessons.py` is a dict with `id` (9000 plus the objective,
`9201` for 2.1 — its host is `ckad9201`, which no task's can be), `objective`, `title`, `workers`,
`intro`, `setup` (bash, as a task's), `steps` and `outro` - and, like a task, `k8s` or `image` if
it needs a host of its own kind (lesson 4.4 runs on v1.36 to upgrade it). Each step comes from
`step(title, text, check, hint, show)` in `sim/bank.py`: `check` runs like a task's check, and
`show` is how you would do the step in the exam, as parts. `cmd(text)` is what you type as it
stands. `hand(text, how, run)` is what you do by hand: what you type to begin (`vim role.yaml`,
`k edit cm app-config`, `ssh NODE` and `sudo -i`, `echo NAME > file`), what to do there, and a script
that does the same unattended. `./ksim lesson audit` seeds each lesson on its own host and, step by
step, requires the check to fail, runs the commands (and each hand part's script) there as
`candidate`, and requires the check to pass, so a lesson can always be done as it shows, and no
step is done before you do it. `./ksim lesson lint` reads what every step has you type and refuses
scripts: no `sed`, `awk`, `yq`, heredocs, loops, `kubectl wait`, `--overrides`, JSON patches or
`jsonpath` into a file, and `k` rather than `kubectl`. In a script, a command on another node is
`ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain `ssh NODE` would read the rest of
the script as its own input. **As audited on 2026-10-26** (v1.37, Ubuntu hosts): CKA's 27 lessons,
done the exam way, 135 of 135 steps proven, and none has you type a script. CKAD 2.1–2.4 (24
steps, audited on 2026-10-19) still show their commands as scripts.

## What is different from the real exam

- The desktop, the container hosts and the VM hosts are Ubuntu 24.04 LTS. The Linux Foundation does
  not publish which Ubuntu release the exam runs; 24.04 is the current LTS. The container hosts'
  containerd and runc are kind's builds, not Ubuntu's packages.
- CKA and CKAD run v1.37, as the real exams do. CKS runs v1.37 too, ahead of the real exam: on
  2026-10-19 the Linux Foundation still listed v1.35 for it.
- Container hosts share this machine's kernel. That is invisible for CKA and CKAD, and for most of
  CKS — gVisor, seccomp, Istio's sidecars, kube-bench, real kubeadm upgrades all work in them.
  Falco and AppArmor do not, which is what the VM hosts are for.
- No Cilium runs anywhere (kindnet is the CNI, and it does enforce NetworkPolicy): the one Cilium
  task is graded on the object. The CSI hostpath driver is installed only on the hosts of the tasks
  and lessons that use it. No Ingress controller either: Ingress tasks are graded on the
  objects, with their certificates checked by openssl.
- On a container host `kubeadm upgrade` passes one preflight check it would fail (a rootless
  container has no cpuset cgroup controller); a wrapper adds the flag for that, nothing else.
- Worker nodes are `cka0037-worker`, `cka0037-worker2`, … rather than `node01`.

## Coming from cka-sim

Task ids are unchanged, so your record carries over if you want it:

```
cp ../cka-sim/history-*.jsonl ../cka-sim/study-*.json .
```

What changed for existing tasks: files go to `/opt/course/...` on the task's host instead of
`workspace/...` on this machine; the `kubectl config use-context` line is gone (the host replaces
it); node access is `ssh` + `sudo -i` instead of `podman exec`; and the image-build tasks use the
host's podman (`podman`, `sudo podman` and `docker` are one store there) instead of this machine's.