kubestronaut-sim/README.md

# kubestronaut-sim — CKA / CKAD / CKS exam simulator

A local exam simulator laid out like the real thing: a **remote desktop in the browser** (XFCE,
terminal, Firefox limited to the documentation the exam allows), a task pane beside it, and
**one cluster per task** that you reach with `ssh <host>` — the way the Linux Foundation exams have
run since 2025, and the way killer.sh and sailor.sh present them.

It grew out of `cka-sim`: the same task banks, per-check grading, clock, history, weak-spot log,
drills and a path through each curriculum, on a new foundation. Every session runs against the
clock.

```
 browser ── http://127.0.0.1:8080 ──────────────────────────────────────────────┐
 │ task pane                    │ remote desktop (noVNC)                         │
 │  Solve this question on:     │  candidate@base:~$ ssh cka0037                 │
 │  ssh cka0037                 │  candidate@cka0037:~$ sudo -i                  │
 └──────────────┬───────────────┴───────────────────┬───────────────────────────┘
                │ ./ksim (engine, on this machine)  │ container "ksim-desktop"
                │  seeds and grades each task       │  ssh
                ▼                                   ▼
        cka0037  = kind cluster   cka0037-control-plane (+ workers)   ← its own etcd, kubelet,
        cka0012  = kind cluster   cka0012-control-plane (+ workers)     static pods, /opt/course
        ...        one per task in the session, all on the podman network "ksim"
```

## Why one cluster per task

On a shared cluster a task that breaks the kubelet, restores etcd or changes API server flags
damages every other task. `cka-sim` worked around that with `breaks`/`solo`/`disruptive` flags, a
baseline-restore pass before each session, and tasks that could never share a session (the etcd
restore could only be drilled alone). All of that is gone:

- any task can sit next to any other, including the etcd restore in a full mock;
- `./ksim reset N` deletes the task's cluster and builds it again — a real reset;
- tasks are provisioned, seeded and graded in parallel;
- you work as `candidate` on the task's host and use `sudo -i` for node work, as on the day.

Each host is a kind node built from `images/node/` — a kubeadm cluster (real etcd, static-pod
control plane, systemd kubelet) with sshd, `candidate` with passwordless sudo, `k` and kubectl
completion preset, vim (2-space tabs), tmux, jq, yq, helm, podman (also as `docker`) for the
image tasks, and `etcdctl`/`etcdutl` at the version of the cluster's etcd.

**CKS hosts** add what that exam's hosts carry (`images/node-cks/`, built on the common image):
kube-bench, trivy, kubesec, kube-linter, bom, cosign, istioctl, gVisor (`runsc`, as a containerd
runtime handler) and Kubernetes as apt packages from pkgs.k8s.io — so a node can be held back on
an older patch release and upgraded with `apt-get install` and `kubeadm upgrade`, for real.

**VM hosts.** A container shares this machine's kernel, and under rootless podman it can neither
load eBPF programs nor use a kernel feature this machine lacks: no Falco, no AppArmor. Tasks that
need them (`"backend": "vm"`) get an Ubuntu 24.04 virtual machine instead (`images/vm/`): QEMU with
KVM, run in a container that carries the host's name, so `ssh cks0105` from the desktop reaches
the VM exactly as it reaches a container host. Each VM makes its own single-node kubeadm cluster
when it first boots, and is ready in about half a minute.

A fresh cluster has no images, so every host pulls Docker Hub images through a cache on this
machine (the container `ksim-registry`, a pull-through mirror with its own volume). Without it a
full mock would fetch the same nginx and busybox dozens of times and hit Docker Hub's limit of 100
anonymous pulls an hour; with it, each image is fetched from Docker Hub once.

## Requirements

podman (rootless is fine, cgroup v2), kind, kubectl, python3, ssh-keygen; helm, openssl and curl
for seeding some tasks. Nothing else: no ttyd, no kubectl context of yours is ever touched (every
host has its own kubeconfig under `.state/`). VM hosts additionally need KVM (`/dev/kvm` usable
by you); QEMU itself comes in a container image.

Memory and start-up time are the price of a cluster per task. Measured on a 32-core machine:

| Session | Clusters | Ready in | Memory |
|---|---|---|---|
| CKAD drill, 3 tasks | 3 × 1 node | about 1 min | about 1 GB |
| CKA mock, 20 tasks | 20 clusters, 50 nodes | just under 6 min, 6 at a time | about 8 GB |
| CKS drill, 1 VM task + 1 container task | 1 VM, 1 × 2 nodes | about 1 min | about 2 GB |

A control-plane node costs roughly 0.3 GB and a worker under 0.1 GB once seeded; a session is
refused before anything is created if a (more cautious) estimate does not fit (`KSIM_FORCE=1`
overrides). Clusters are provisioned one per four cores at a time, at most 8 (`KSIM_PARALLEL=N`).
A VM host is given 2 GB (it used about 1.6 GB right after starting) and is up in about 30 s.

Disk: the images take 1.3 GB per Kubernetes minor, 2.8 GB for the CKS host, 1.2 GB for the
desktop; the VM host adds a 3.5 GB base disk and the 0.6 GB Ubuntu cloud image under `.state/vm/`.

## Run

```
./ksim build all        # once: the exam host image per Kubernetes minor, and the desktop image
./ksim build cks --vm   # once, for CKS: also the VM host its Falco and AppArmor tasks need (~10 min)
./ksim web              # http://127.0.0.1:8080 — pick a track and a session on the landing page
```

The browser UI follows the layout of [CK-X](https://github.com/sailor-sh/CK-X) (itself modelled on
the Linux Foundation's exam UI), written anew for this simulator:

- **Landing page** — the track (CKA, CKAD, CKS) in the top bar; **Start exam** opens a dialog to
  pick the session: the next step on the path, an objective, a domain, the next domain, the whole
  bank, tasks of your choice, a simulator set or a full exam, with or without hard mode. Below it:
  your path, a card per domain, and every objective with its notes and the "I know this — skip it"
  links. While the clusters are built, an overlay shows their progress and the log.
- **Exam screen** — a dark top bar with the clock and three menus: *Exam Interface* (fullscreen,
  panel sizes, reconnect the desktop), *Exam Controls* (submit, pause, **Restart clock**,
  **Terminate session**) and *Exam Information* (exam instructions, Kubernetes docs, your history).
  On the left, `< Question N >` and each task headed by its host: click `ssh ckad0037` (or any
  highlighted value) to put it on the desktop's clipboard and paste in the terminal with
  **Ctrl+Shift+V**. **Flag for review** and **Satisfied with answer** sit under the task; a drill
  adds **Check this task**, and a task's reference solution opens once it is graded. On the right,
  the remote desktop. The clock starts when you click **Start** in the dialog that opens the session.
- **Results page** — points and pass or fail, every task with its checks, the answers, what to drill
  next, and a way back to the tasks to review them on their hosts.

Firefox in the desktop opens only the allowed documentation for the track (`KSIM_OPEN_WEB=1`
lifts that). Submitting grades every task on its own host; the hosts stay up for review until the
next session starts, or until **Delete the hosts** on the results page. **Restart clock**
(`./ksim restart`) puts the session's full time back on the clock without touching its hosts or
your work; like Pause, it is recorded nowhere. **Terminate session** ends a session without grading
it and deletes its hosts, as `./ksim abort` does.

Everything is also available from a terminal. A command is for the track named with `--track` or
`KSIM_TRACK`; with neither, for the track whose session is under way, and for CKA when there is
none. Task numbers are per track — CKA's task 13 and CKAD's task 13 are different tasks — and a
host name such as `ckad0013` only resolves in the desktop's terminal: from your own, it is
`./ksim ssh 13`.

```
./ksim --track ckad drill            # next least-drilled domain, 5 tasks      (--track or KSIM_TRACK)
./ksim drill weak | drill 2 17 20 | drill TSHOOT NET | drill 3.2
./ksim start [--hard]                # full mock, sampled by domain weight
./ksim sim c|d                       # a fixed simulator set (CKA: c, d  ·  CKAD: a)
./ksim objectives | next | focus     # the curriculum as a path, one objective at a time against the clock
./ksim notes 3.2                     # an objective's notes: what to know, fastest commands, traps, docs
./ksim skip DESIGN | unskip DESIGN   # "I am good with this domain (or objective 2.3)": off the path, and back
./ksim drill OBS:3 | next OBS:3      # three tasks of a domain, each from another objective - alone, or added
./ksim list | show N | grade N | solution N | timer | end
./ksim pause | resume | restart      # stop the clock, or put the session's full time back (hosts and work kept)
./ksim hosts [start]                 # the session's hosts; `start` brings them back after a reboot
./ksim ssh N                         # a shell on the host of task N from your own terminal
./ksim reset N                       # rebuild the host of task N from scratch
./ksim down                          # delete every host and the desktop (history and images stay)
./ksim coverage                      # tasks per domain and objective against the 17–20 target
./ksim audit [all|DOMAIN|N..]        # bank check: seed each task on its own cluster, grade it untouched,
                                     # then solve it with its script and grade again (--settle N, --keep)
```

**Skipping what you know.** `./ksim skip DESIGN` (or the "I know this — skip it" link beside a
domain in the browser's objective list) marks a domain or an objective as known: the path, the
no-argument drill and `drill all` leave it out, and it no longer has to be drilled before a full
exam. It is a statement, not a result — nothing is written to the history, the statistics or the
weak list, the objective list shows `»` rather than `✓`, and `./ksim skip` prints what is on record
for it. A mock exam still draws from every domain, as the real one will. To keep such a domain
warm, `DOMAIN:N` takes a few of its tasks, each from another objective: `./ksim drill OBS:3`, or
`./ksim next OBS:3` to add them to the next step on the path.

## Tracks

Each track is a directory: `cka/`, `ckad/`, `cks/`.

| File | What |
|---|---|
| `questions.py` | `TRACK` (name, minutes, pass mark, Kubernetes minor, domains, mock quota, default `workers`, host `image`) and the first tasks; pulls in the other banks |
| `bank_*.py` | more tasks; `bank_sim_*` are the fixed simulator sets |
| `curriculum.py` | the exam's objectives in order, each with study notes and the tasks that practise it |
| `solutions.py` | reference solutions (CKA keeps them apart; the others inline them) |
| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved |

| Track | Kubernetes | Bank today | Target (17–20 pool tasks per domain) |
|---|---|---|---|
| CKA | v1.35 | 131 tasks: 96 pool (17–25 per domain) + sets C, D (35) | reached in all five domains |
| CKAD | v1.37 | 105 tasks: 85 pool (17 per domain) + set A (20) | reached in all five domains |
| CKS | v1.35 | 110 tasks, all pool: 17 per domain on container hosts, plus 5 Falco and 3 AppArmor tasks on VM hosts | reached in all six domains |

`./ksim coverage` prints this per domain and per objective.

**CKA, as audited on 2026-10-13:** all 131 tasks seed cleanly on a host of their own, score
nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted
solution is run on the host as `candidate` — for 66 of them that script is the reference solution
exactly as a drill shows it once the task is graded.

**CKAD, as audited on 2026-10-12:** all 105 tasks seed cleanly on a host of their own, score
nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted
solution is run on the host as `candidate`.

**CKS, as audited on 2026-10-12:** all 110 tasks seed cleanly on a host of their own, score nothing
before anyone has worked on them, and reach full marks when their scripted solution is run on the
host as `candidate`. The 26 objectives are the curriculum's own (`cks/curriculum.py`), each with
notes, commands, traps and the documentation the exam allows — which, per the Linux Foundation's
list read that day, is the Kubernetes docs and blog plus Falco, bom, etcd, the NGINX Ingress
Controller, Cilium and Istio. Trivy's and AppArmor's own documentation are not on it.

### A task

```python
{"id": 114, "domain": "NET", "difficulty": "normal",      # or "hard"
 "params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])},
 "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
 "setup": "bash that seeds the scenario on the task's own cluster",
 "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
 "solution": "...",
 "workers": 2}                                              # optional: worker nodes of its cluster
```

Setup and check scripts run on this machine against the task's own cluster and get:

- `$K` — kubectl for that cluster (`KUBECONFIG` is exported too, so `helm` just works);
- `$CP`, `$W1`, `$W2`, `$W3` — its node names, and `nx NODE 'cmd'` to run a command on one as root;
- `$WS` — the task's `/opt/course` directory as a path here (it is mounted into the host), so a
  check reads a file the candidate wrote with plain `cat "$WS/q14/answer.txt"`. Once seeded, the
  files belong to `candidate`: one that is readable by its owner only — a private key, a kubeconfig
  `kubectl config` wrote — has to be read on the host, `nx "$CP" 'cat /opt/course/q37/tls.key'`,
  and a kubeconfig made there is tested there too (its server address is the host's);
- `$PRIV` — a directory beside it that is *not* mounted into the host: where a setup records what
  its checks compare against later (the UID of an object that must not be re-created, say);
- `podman ...` — root's podman on the task's host, the store the candidate's `podman`/`docker` use;
- `j <get args> '{jsonpath}'`, `eq a b`, `mkns` (from `sim.bank.NS`), `ensure_gateway_crds`,
  `ensure_metrics_server`.

Task text may use `{WS}` (`/opt/course`), `{CP}`, `{W1}`.., and `{SHELL_CP}`, `{SHELL_W1}`..
(`ssh cka0114`, `ssh cka0114-worker`).

**Topology.** A task's cluster has the workers it asks for with `"workers"`. Without it, a task
that names a worker anywhere (`$W1`, `{W2}`, …) gets three — what the CKA bank was written for —
and every other task gets its track's default (`TRACK["workers"]`: CKA 1, CKAD 0, CKS 1). A task
whose `params` draw a node (`r.choice(["{W1}", "{W2}"])`) has to say `"workers"` itself: the
lambda is not searched. A
track's `WORKERS = {id: n}` map sets it without editing the task; `PENDING = {id: why}` holds a
task back from every session. A task with `"backend": "vm"` gets a single-node VM host; such
tasks hold themselves back until `./ksim build cks --vm` has been run. On a VM host, `nx` runs
its command in the VM, and nothing enforces NetworkPolicy (its Pod network is a plain bridge).

**Proving a task.** `./ksim audit N..` builds the task's host, seeds it and grades it untouched:
the seed must be clean and the task must not score full marks before anyone has worked on it. If
`<track>/solve.py` has a script for the task — what a candidate would type on the host — the audit
then runs it there as `candidate` and grades again, and only full marks pass (what the script
created gets up to 90 seconds to come up). A task with no script is proven to seed, not to be
solvable; write the script when you write the task. `--settle 45` waits that long between seeding
and the untouched grade, which catches a check that only passes once the seeded Pods run, or
until a stopped kubelet shows as NotReady; `--keep` leaves the host up to look at
(`KUBECONFIG=.state/envs/<host>/kubeconfig`, `./ksim down` removes it).

In `cka/solve.py`, `AS_WRITTEN` lists the tasks whose reference solution is a script as it stands:
for those the audit runs exactly the text the candidate is shown, so a solution that stops working
with a new Kubernetes release fails the next audit instead of misleading you after a drill.

## What is different from the real exam

- The desktop's base machine and the container hosts are Debian, not Ubuntu; `apt`, systemd and
  kubeadm behave the same. VM hosts are Ubuntu 24.04.
- Container hosts share this machine's kernel. That is invisible for CKA and CKAD, and for most of
  CKS — gVisor, seccomp, Istio's sidecars, kube-bench, real kubeadm upgrades all work in them.
  Falco and AppArmor do not, which is what the VM hosts are for.
- No Cilium runs anywhere (kindnet is the CNI, and it does enforce NetworkPolicy): the one Cilium
  task is graded on the object. No Ingress controller either: Ingress tasks are graded on the
  objects, with their certificates checked by openssl.
- On a container host `kubeadm upgrade` passes one preflight check it would fail (a rootless
  container has no cpuset cgroup controller); a wrapper adds the flag for that, nothing else.
- Worker nodes are `cka0037-worker`, `cka0037-worker2`, … rather than `node01`.

## Coming from cka-sim

Task ids are unchanged, so your record carries over if you want it:

```
cp ../cka-sim/history-*.jsonl ../cka-sim/study-*.json .
```

What changed for existing tasks: files go to `/opt/course/...` on the task's host instead of
`workspace/...` on this machine; the `kubectl config use-context` line is gone (the host replaces
it); node access is `ssh` + `sudo -i` instead of `podman exec`; and the image-build tasks use the
host's podman (`podman`, `sudo podman` and `docker` are one store there) instead of this machine's.