kubestronaut-sim/README.md

kubestronaut-sim — CKA / CKAD / CKS exam simulator

A local exam simulator laid out like the real thing: a remote desktop in the browser (XFCE, terminal, Firefox limited to the documentation the exam allows), a task pane beside it, and one cluster per task that you reach with ssh <host> — the way the Linux Foundation exams have run since 2025, and the way killer.sh and sailor.sh present them.

It grew out of cka-sim: the same task banks, per-check grading, clock, history, weak-spot log, drills and a path through each curriculum, on a new foundation. Every session runs against the clock.

 browser ── http://127.0.0.1:8080 ──────────────────────────────────────────────┐
 │ task pane                    │ remote desktop (noVNC)                         │
 │  Solve this question on:     │  candidate@base:~$ ssh cka0037                 │
 │  ssh cka0037                 │  candidate@cka0037:~$ sudo -i                  │
 └──────────────┬───────────────┴───────────────────┬───────────────────────────┘
                │ ./ksim (engine, on this machine)  │ container "ksim-desktop"
                │  seeds and grades each task       │  ssh
                ▼                                   ▼
        cka0037  = kind cluster   cka0037-control-plane (+ workers)   ← its own etcd, kubelet,
        cka0012  = kind cluster   cka0012-control-plane (+ workers)     static pods, /opt/course
        ...        one per task in the session, all on the podman network "ksim"

Why one cluster per task

On a shared cluster a task that breaks the kubelet, restores etcd or changes API server flags damages every other task. cka-sim worked around that with breaks/solo/disruptive flags, a baseline-restore pass before each session, and tasks that could never share a session (the etcd restore could only be drilled alone). All of that is gone:

• any task can sit next to any other, including the etcd restore in a full mock;
• ./ksim reset N deletes the task's cluster and builds it again — a real reset;
• tasks are provisioned, seeded and graded in parallel;
• you work as candidate on the task's host and use sudo -i for node work, as on the day.

Each host is a kind node built from images/node/ on Ubuntu 24.04, as the exam's hosts are — a kubeadm cluster (real etcd, static-pod control plane, systemd kubelet). kubelet, kubeadm and kubectl are apt packages from pkgs.k8s.io, held, with the repository line of the cluster's minor (the CNI plugins come with them, as kubernetes-cni): a node is upgraded with apt-get install and kubeadm upgrade, for real, and a task can hold one back on an older release. containerd, runc, crictl, kind's entrypoint and the preloaded control-plane images are copied from the pinned kind node image, which kind builds on Debian. On top: sshd, candidate with passwordless sudo, k and kubectl completion preset, vim (2-space tabs), tmux, jq, yq, helm, man pages, podman (also as docker) for the image tasks, and etcdctl/etcdutl at the version of the cluster's etcd.

CKS hosts add what that exam's hosts carry (images/node-cks/, built on the common image): kube-bench, trivy, kubesec, kube-linter, bom, cosign, istioctl and gVisor (runsc, as a containerd runtime handler).

VM hosts. A container shares this machine's kernel, and under rootless podman it can neither load eBPF programs nor use a kernel feature this machine lacks: no Falco, no AppArmor. Tasks that need them ("backend": "vm") get an Ubuntu 24.04 virtual machine instead (images/vm/): QEMU with KVM, run in a container that carries the host's name, so ssh cks0105 from the desktop reaches the VM exactly as it reaches a container host. Each VM makes its own single-node kubeadm cluster when it first boots, and is ready in about half a minute.

A fresh cluster has no images, so every host pulls Docker Hub images through a cache on this machine (the container ksim-registry, a pull-through mirror with its own volume). Without it a full mock would fetch the same nginx and busybox dozens of times and hit Docker Hub's limit of 100 anonymous pulls an hour; with it, each image is fetched from Docker Hub once.

Requirements

podman (rootless is fine, cgroup v2), kind, kubectl, python3, ssh-keygen; helm, openssl and curl for seeding some tasks. Nothing else: no ttyd, no kubectl context of yours is ever touched (every host has its own kubeconfig under .state/). VM hosts additionally need KVM (/dev/kvm usable by you); QEMU itself comes in a container image.

Memory and start-up time are the price of a cluster per task. Measured on a 32-core machine:

Session	Clusters	Ready in	Memory
CKAD drill, 3 tasks	3 × 1 node	about 1 min	about 1 GB
CKA mock, 20 tasks	20 clusters, 50 nodes	just under 6 min, 6 at a time	about 8 GB
CKS drill, 1 VM task + 1 container task	1 VM, 1 × 2 nodes	about 1 min	about 2 GB

A control-plane node costs roughly 0.3 GB and a worker under 0.1 GB once seeded; a session is refused before anything is created if a (more cautious) estimate does not fit (KSIM_FORCE=1 overrides). Clusters are provisioned one per four cores at a time, at most 8 (KSIM_PARALLEL=N). A VM host is given 2 GB (it used about 1.6 GB right after starting) and is up in about 30 s.

Disk: the images take 1.5 GB per Kubernetes minor (v1.37, and v1.36 for the tasks that upgrade their cluster), 2.7 GB for the CKS host, 1.2 GB for the desktop; the VM host adds a 3.5 GB base disk and the 0.6 GB Ubuntu cloud image under .state/vm/.

Run

./ksim build all        # once: the exam host image per Kubernetes minor, and the desktop image
./ksim build cks --vm   # once, for CKS: also the VM host its Falco and AppArmor tasks need (~10 min)
./ksim web              # http://127.0.0.1:8080 — pick a track and a session on the landing page

The browser UI follows the layout of CK-X (itself modelled on the Linux Foundation's exam UI), written anew for this simulator:

• Landing page — the track (CKA, CKAD, CKS) in the top bar; Start exam opens a dialog to pick the session: the next step on the path, an objective, a domain, the next domain, the whole bank, tasks of your choice, a simulator set or a full exam, with or without hard mode. Below it: your path, a card per domain, and every objective with its notes and the "I know this — skip it" links. While the clusters are built, an overlay shows their progress and the log.
• Exam screen — a dark top bar with the clock and three menus: Exam Interface (fullscreen, panel sizes, reconnect the desktop), Exam Controls (submit, pause, Restart clock, Terminate session) and Exam Information (exam instructions, Kubernetes docs, your history). On the left, < Question N > and each task headed by its host: click ssh ckad0037 (or any highlighted value) to put it on the desktop's clipboard and paste in the terminal with Ctrl+Shift+V. Flag for review and Satisfied with answer sit under the task; a drill adds Check this task, and a task's reference solution opens once it is graded. On the right, the remote desktop. The clock starts when you click Start in the dialog that opens the session.
• Results page — points and pass or fail, every task with its checks, the answers, what to drill next, and a way back to the tasks to review them on their hosts.
• Theme — Auto (your system's light or dark setting, the default), Light or Dark, from the landing page's top bar, the Exam Interface menu or the results page; the browser remembers it. The remote desktop keeps its own look.

Firefox in the desktop opens only the allowed documentation for the track (KSIM_OPEN_WEB=1 lifts that). Submitting grades every task on its own host; the hosts stay up for review until the next session starts, or until Delete the hosts on the results page. Restart clock (./ksim restart) puts the session's full time back on the clock without touching its hosts or your work; like Pause, it is recorded nowhere. Terminate session ends a session without grading it and deletes its hosts, as ./ksim abort does.

Everything is also available from a terminal. A command is for the track named with --track or KSIM_TRACK; with neither, for the track whose session is under way, and for CKA when there is none. Task numbers are per track — CKA's task 13 and CKAD's task 13 are different tasks — and a host name such as ckad0013 only resolves in the desktop's terminal: from your own, it is ./ksim ssh 13.

./ksim --track ckad drill            # next least-drilled domain, 5 tasks      (--track or KSIM_TRACK)
./ksim drill weak | drill 2 17 20 | drill TSHOOT NET | drill 3.2
./ksim start [--hard]                # full mock, sampled by domain weight
./ksim sim c|d                       # a fixed simulator set (CKA: c, d  ·  CKAD: a)
./ksim objectives | next | focus     # the curriculum as a path, one objective at a time against the clock
./ksim notes 3.2                     # an objective's notes: what to know, fastest commands, traps, docs
./ksim lesson [OBJ]                  # the track's lessons, or start one: a guided lab, no clock (CKA: all 27 · CKAD: 2.1-2.4)
./ksim lesson ARCH [--all]           # teach a domain: its lessons not done yet (--all: every one), back to back
./ksim lesson next | lesson 4.3      # in a domain's run: the next lesson, or any lesson of the run
./ksim lesson step | check | hint | show [N]   # inside a lesson; ./ksim end finishes it
./ksim skip DESIGN | unskip DESIGN   # "I am good with this domain (or objective 2.3)": off the path, and back
./ksim drill OBS:3 | next OBS:3      # three tasks of a domain, each from another objective - alone, or added
./ksim list | show N | grade N | solution N | timer | end
./ksim pause | resume | restart      # stop the clock, or put the session's full time back (hosts and work kept)
./ksim hosts [start]                 # the session's hosts; `start` brings them back after a reboot
./ksim ssh N                         # a shell on the host of task N from your own terminal
./ksim reset N                       # rebuild the host of task N from scratch
./ksim down                          # delete every host and the desktop (history and images stay)
./ksim coverage                      # tasks per domain and objective against the 17–20 target
./ksim audit [all|DOMAIN|N..]        # bank check: seed each task on its own cluster, grade it untouched,
                                     # then solve it with its script and grade again (--settle N, --keep)
./ksim lesson audit [OBJ..]          # lesson check: every step fails, is done by its own commands, then passes

Lessons. A lesson teaches one objective as a guided lab on a host of its own: an introduction, then steps you do in the desktop's terminal, each with a Check step button, a hint and Show me how (the exam way: what you type, what you edit in vim), and a wrap-up that sends you to the objective's timed drill. Lessons are the one untimed part of the simulator — they teach, the drills test — and they write nothing to the history, the statistics or the weak list; a lesson whose every step was checked is marked done on its objective (lesson ✓ in the objective list). Steps count in order. Start one from an objective's Lesson button, Lesson first on the path card, the Take a lesson session in the Start exam dialog, or ./ksim lesson 2.1.

Teach a domain runs a domain's lessons back to back: the ones you have not done, or every one again once they all are (--all forces that). It is one session without a clock. Every lesson's host is made before it starts, as many at a time as for any session (KSIM_PARALLEL), so you never wait between lessons. Each lesson's wrap-up leads to the next, the list at the top of a lesson goes to any lesson of the run, and a lesson counts as done the moment its last step is checked, whether or not the run is finished. Finishing the run sends you to the domain's timed drill. Start one from the Teach a domain card in the lobby, the Teach a domain session in the Start exam dialog, or ./ksim lesson ARCH.

CKA has a lesson for each of its 27 objectives (cka/lessons_*.py, 135 steps), which also cover what a full CKA course teaches under them — checked against the outline of KubeDojo's CKA course and written from scratch: projected volumes, volume expansion, snapshots and restores, eviction thresholds, debug containers, kube-proxy's rules, encryption at rest, in-place resize, PodDisruptionBudgets, QoS classes, ServiceAccount tokens, a real upgrade from v1.36 to v1.37, EndpointSlices, an operator. Tasks 132–147 test those topics against the clock. The CKAD pilot covers the Application Deployment domain (2.1 blue/green and canary, 2.2 rolling updates, 2.3 Helm, 2.4 Kustomize).

Skipping what you know. ./ksim skip DESIGN (or the "I know this — skip it" link beside a domain in the browser's objective list) marks a domain or an objective as known: the path, the no-argument drill and drill all leave it out, and it no longer has to be drilled before a full exam. It is a statement, not a result — nothing is written to the history, the statistics or the weak list, the objective list shows » rather than ✓, and ./ksim skip prints what is on record for it. A mock exam still draws from every domain, as the real one will. To keep such a domain warm, DOMAIN:N takes a few of its tasks, each from another objective: ./ksim drill OBS:3, or ./ksim next OBS:3 to add them to the next step on the path.

Tracks

Each track is a directory: cka/, ckad/, cks/.

File	What
questions.py	TRACK (name, minutes, pass mark, Kubernetes minor, domains, mock quota, default workers, host image) and the first tasks; pulls in the other banks
bank_*.py	more tasks; bank_sim_* are the fixed simulator sets
curriculum.py	the exam's objectives in order, each with study notes and the tasks that practise it
solutions.py	reference solutions (CKA keeps them apart; the others inline them)
solve.py	scripted solutions that ./ksim audit runs on the task's host to prove a task can be solved - for the tasks whose reference solution is not in parts
lessons.py	guided labs, one per objective (CKA: all, one lessons_*.py per domain · CKAD: 2.1–2.4): introduction, steps with their check, hint and commands, wrap-up

Track	Kubernetes	Bank today	Target (17–20 pool tasks per domain)
CKA	v1.37	147 tasks: 112 pool (18–28 per domain) + sets C, D (35)	reached in all five domains
CKAD	v1.37	105 tasks: 85 pool (17 per domain) + set A (20)	reached in all five domains
CKS	v1.37	110 tasks, all pool: 17 per domain on container hosts, plus 5 Falco and 3 AppArmor tasks on VM hosts	reached in all six domains

./ksim coverage prints this per domain and per objective.

CKA, as audited on 2026-10-19 (v1.37, Ubuntu hosts): all 147 tasks seed cleanly on a host of their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted solution is run on the host as candidate — for 80 of them that script is the reference solution exactly as a drill shows it once the task is graded. Q132 really upgrades its cluster from v1.36.4 to v1.37.1, worker included.

CKAD, as audited on 2026-10-19 (v1.37, Ubuntu hosts): all 105 tasks seed cleanly on a host of their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted solution is run on the host as candidate.

CKS, as audited on 2026-10-19 (v1.37, Ubuntu hosts): all 110 tasks seed cleanly on a host of their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted solution is run on the host as candidate. Q103, a Falco task on a VM host, failed to seed once that day — a minute passed without an alert naming the Pod — and passed on 2026-10-20, alone and with the other seven VM tasks six at a time; its setup now tries a new Pod before it gives up. The 26 objectives are the curriculum's own (cks/curriculum.py), each with notes, commands, traps and the documentation the exam allows — which, per the Linux Foundation's list read on 2026-10-12, is the Kubernetes docs and blog plus Falco, bom, etcd, the NGINX Ingress Controller, Cilium and Istio. Trivy's and AppArmor's own documentation are not on it.

On 2026-10-20, after hexdump and xxd were added to the hosts, CKA Q132 (the upgrade from v1.36), CKAD Q1 and CKS Q60, Q61 and Q64 passed again on the rebuilt images.

A task

{"id": 114, "domain": "NET", "difficulty": "normal",      # or "hard"
 "params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])},
 "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
 "setup": "bash that seeds the scenario on the task's own cluster",
 "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
 "solution": [cmd("k ... --dry-run=client -o yaml > f.yaml"),   # or one string, a script as it stands
              hand("vim f.yaml", "what to add, Markdown", run="the same, scripted: yq, a heredoc")],
 "workers": 2,                                              # optional: worker nodes of its cluster
 "k8s": "v1.36"}                                            # optional: a minor of its own (an upgrade task)

Setup and check scripts run on this machine against the task's own cluster and get:

• $K — kubectl for that cluster (KUBECONFIG is exported too, so helm just works);
• $CP, $W1, $W2, $W3 — its node names, and nx NODE 'cmd' to run a command on one as root;
• $WS — the task's /opt/course directory as a path here (it is mounted into the host), so a check reads a file the candidate wrote with plain cat "$WS/q14/answer.txt". Once seeded, the files belong to candidate: one that is readable by its owner only — a private key, a kubeconfig kubectl config wrote — has to be read on the host, nx "$CP" 'cat /opt/course/q37/tls.key', and a kubeconfig made there is tested there too (its server address is the host's);
• $PRIV — a directory beside it that is not mounted into the host: where a setup records what its checks compare against later (the UID of an object that must not be re-created, say);
• podman ... — root's podman on the task's host, the store the candidate's podman/docker use;
• j <get args> '{jsonpath}', eq a b, mkns (from sim.bank.NS), ensure_gateway_crds, ensure_metrics_server, and ensure_csi_hostpath — the CSI hostpath driver with the snapshot controller, and StorageClass csi-hostpath-sc (expandable) as the exam's clusters have it, for what local-path cannot do: growing a claim, snapshots, clones.

Task text may use {WS} (/opt/course), {CP}, {W1}.., and {SHELL_CP}, {SHELL_W1}.. (ssh cka0114, ssh cka0114-worker).

Topology. A task's cluster has the workers it asks for with "workers". Without it, a task that names a worker anywhere ($W1, {W2}, …) gets three — what the CKA bank was written for — and every other task gets its track's default (TRACK["workers"]: CKA 1, CKAD 0, CKS 1). A task whose params draw a node (r.choice(["{W1}", "{W2}"])) has to say "workers" itself: the lambda is not searched. A track's WORKERS = {id: n} map sets it without editing the task; PENDING = {id: why} holds a task back from every session. A task with "backend": "vm" gets a single-node VM host; such tasks hold themselves back until ./ksim build cks --vm has been run. On a VM host, nx runs its command in the VM, and nothing enforces NetworkPolicy (its Pod network is a plain bridge).

Proving a task. ./ksim audit N.. builds the task's host, seeds it and grades it untouched: the seed must be clean and the task must not score full marks before anyone has worked on it. If <track>/solve.py has a script for the task — what a candidate would type on the host — the audit then runs it there as candidate and grades again, and only full marks pass (what the script created gets up to 90 seconds to come up). A task with no script is proven to seed, not to be solvable; write the script when you write the task. --settle 45 waits that long between seeding and the untouched grade, which catches a check that only passes once the seeded Pods run, or until a stopped kubelet shows as NotReady; --keep leaves the host up to look at (KUBECONFIG=.state/envs/<host>/kubeconfig, ./ksim down removes it).

A reference solution in parts - cmd() and hand(), as a lesson step's show - is its own script: the audit runs what is typed as it stands, and for each part done by hand its run, so the solution the candidate reads after a drill is the one that is proven, and one that stops working with a new Kubernetes release fails the next audit. 105 of the 147 CKA solutions are written so (the page and ./ksim solution N show each thing to type as a block of its own, which a click copies, with the edit to make in vim beneath it); the other 42 are commands typed as they stand, each with its script in solve.py. ./ksim solution lint reads what they have you type, against the same list of things you would not type in the exam as ./ksim lesson lint. In cka/solve.py, AS_WRITTEN lists the few tasks whose reference solution is still one script, which the audit runs as it stands.

Writing a lesson. A lesson in <track>/lessons.py is a dict with id (9000 plus the objective, 9201 for 2.1 — its host is ckad9201, which no task's can be), objective, title, workers, intro, setup (bash, as a task's), steps and outro - and, like a task, k8s or image if it needs a host of its own kind (lesson 4.4 runs on v1.36 to upgrade it). Each step comes from step(title, text, check, hint, show) in sim/bank.py: check runs like a task's check, and show is how you would do the step in the exam, as parts. cmd(text) is what you type as it stands. hand(text, how, run) is what you do by hand: what you type to begin (vim role.yaml, k edit cm app-config, ssh NODE and sudo -i, echo NAME > file), what to do there, and a script that does the same unattended. ./ksim lesson audit seeds each lesson on its own host and, step by step, requires the check to fail, runs the commands (and each hand part's script) there as candidate, and requires the check to pass, so a lesson can always be done as it shows, and no step is done before you do it. ./ksim lesson lint reads what every step has you type and refuses scripts: no sed, awk, yq, heredocs, loops, kubectl wait, --overrides, JSON patches or jsonpath into a file, and k rather than kubectl. In a script, a command on another node is ssh NODE sudo bash <<'EOF' or ssh -n NODE 'COMMAND': a plain ssh NODE would read the rest of the script as its own input. As audited on 2026-10-26 (v1.37, Ubuntu hosts): CKA's 27 lessons, done the exam way, 135 of 135 steps proven, and none has you type a script. CKAD 2.1–2.4 (24 steps, audited on 2026-10-19) still show their commands as scripts.

What is different from the real exam

• The desktop, the container hosts and the VM hosts are Ubuntu 24.04 LTS. The Linux Foundation does not publish which Ubuntu release the exam runs; 24.04 is the current LTS. The container hosts' containerd and runc are kind's builds, not Ubuntu's packages.
• CKA and CKAD run v1.37, as the real exams do. CKS runs v1.37 too, ahead of the real exam: on 2026-10-19 the Linux Foundation still listed v1.35 for it.
• Container hosts share this machine's kernel. That is invisible for CKA and CKAD, and for most of CKS — gVisor, seccomp, Istio's sidecars, kube-bench, real kubeadm upgrades all work in them. Falco and AppArmor do not, which is what the VM hosts are for.
• No Cilium runs anywhere (kindnet is the CNI, and it does enforce NetworkPolicy): the one Cilium task is graded on the object. The CSI hostpath driver is installed only on the hosts of the tasks and lessons that use it. No Ingress controller either: Ingress tasks are graded on the objects, with their certificates checked by openssl.
• On a container host kubeadm upgrade passes one preflight check it would fail (a rootless container has no cpuset cgroup controller); a wrapper adds the flag for that, nothing else.
• Worker nodes are cka0037-worker, cka0037-worker2, … rather than node01.

Coming from cka-sim

Task ids are unchanged, so your record carries over if you want it:

cp ../cka-sim/history-*.jsonl ../cka-sim/study-*.json .

What changed for existing tasks: files go to /opt/course/... on the task's host instead of workspace/... on this machine; the kubectl config use-context line is gone (the host replaces it); node access is ssh + sudo -i instead of podman exec; and the image-build tasks use the host's podman (podman, sudo podman and docker are one store there) instead of this machine's.