kubestronaut-sim/README.md

kubestronaut-sim — CKA / CKAD / CKS exam simulator

A local exam simulator laid out like the real thing: a remote desktop in the browser (XFCE, terminal, Firefox limited to the documentation the exam allows), a task pane beside it, and one cluster per task that you reach with ssh <host> — the way the Linux Foundation exams have run since 2025, and the way killer.sh and sailor.sh present them.

It grew out of cka-sim: the same task banks, per-check grading, clock, history, weak-spot log, drills and study path, on a new foundation.

 browser ── http://127.0.0.1:8080 ──────────────────────────────────────────────┐
 │ task pane                    │ remote desktop (noVNC)                         │
 │  Solve this question on:     │  candidate@base:~$ ssh cka0037                 │
 │  ssh cka0037                 │  candidate@cka0037:~$ sudo -i                  │
 └──────────────┬───────────────┴───────────────────┬───────────────────────────┘
                │ ./ksim (engine, on this machine)  │ container "ksim-desktop"
                │  seeds and grades each task       │  ssh
                ▼                                   ▼
        cka0037  = kind cluster   cka0037-control-plane (+ workers)   ← its own etcd, kubelet,
        cka0012  = kind cluster   cka0012-control-plane (+ workers)     static pods, /opt/course
        ...        one per task in the session, all on the podman network "ksim"

Why one cluster per task

On a shared cluster a task that breaks the kubelet, restores etcd or changes API server flags damages every other task. cka-sim worked around that with breaks/solo/disruptive flags, a baseline-restore pass before each session, and tasks that could never share a session (the etcd restore could only be drilled alone). All of that is gone:

• any task can sit next to any other, including the etcd restore in a full mock;
• ./ksim reset N deletes the task's cluster and builds it again — a real reset;
• tasks are provisioned, seeded and graded in parallel;
• you work as candidate on the task's host and use sudo -i for node work, as on the day.

Each host is a kind node built from images/node/ — a kubeadm cluster (real etcd, static-pod control plane, systemd kubelet) with sshd, candidate with passwordless sudo, k and kubectl completion preset, vim (2-space tabs), tmux, jq, yq, helm, podman (also as docker) for the image tasks, and etcdctl/etcdutl at the version of the cluster's etcd.

CKS hosts add what that exam's hosts carry (images/node-cks/, built on the common image): kube-bench, trivy, kubesec, kube-linter, bom, cosign, istioctl, gVisor (runsc, as a containerd runtime handler) and Kubernetes as apt packages from pkgs.k8s.io — so a node can be held back on an older patch release and upgraded with apt-get install and kubeadm upgrade, for real.

VM hosts. A container shares this machine's kernel, and under rootless podman it can neither load eBPF programs nor use a kernel feature this machine lacks: no Falco, no AppArmor. Tasks that need them ("backend": "vm") get an Ubuntu 24.04 virtual machine instead (images/vm/): QEMU with KVM, run in a container that carries the host's name, so ssh cks0105 from the desktop reaches the VM exactly as it reaches a container host. Each VM makes its own single-node kubeadm cluster when it first boots, and is ready in about half a minute.

A fresh cluster has no images, so every host pulls Docker Hub images through a cache on this machine (the container ksim-registry, a pull-through mirror with its own volume). Without it a full mock would fetch the same nginx and busybox dozens of times and hit Docker Hub's limit of 100 anonymous pulls an hour; with it, each image is fetched from Docker Hub once.

Requirements

podman (rootless is fine, cgroup v2), kind, kubectl, python3, ssh-keygen; helm, openssl and curl for seeding some tasks. Nothing else: no ttyd, no kubectl context of yours is ever touched (every host has its own kubeconfig under .state/). VM hosts additionally need KVM (/dev/kvm usable by you); QEMU itself comes in a container image.

Memory and start-up time are the price of a cluster per task. Measured on a 32-core machine:

Session	Clusters	Ready in	Memory
CKAD drill, 3 tasks	3 × 1 node	about 1 min	about 1 GB
CKA mock, 20 tasks	20 clusters, 50 nodes	just under 6 min, 6 at a time	about 8 GB
CKS drill, 1 VM task + 1 container task	1 VM, 1 × 2 nodes	about 1 min	about 2 GB

A control-plane node costs roughly 0.3 GB and a worker under 0.1 GB once seeded; a session is refused before anything is created if a (more cautious) estimate does not fit (KSIM_FORCE=1 overrides). Clusters are provisioned one per four cores at a time, at most 8 (KSIM_PARALLEL=N). A VM host is given 2 GB (it used about 1.6 GB right after starting) and is up in about 30 s.

Disk: the images take 1.3 GB per Kubernetes minor, 2.8 GB for the CKS host, 1.2 GB for the desktop; the VM host adds a 3.5 GB base disk and the 0.6 GB Ubuntu cloud image under .state/vm/.

Run

./ksim build all        # once: the exam host image per Kubernetes minor, and the desktop image
./ksim build cks --vm   # once, for CKS: also the VM host its Falco and AppArmor tasks need (~10 min)
./ksim web              # http://127.0.0.1:8080 — pick a track and a session in the lobby

In the browser: the task on the left names its host; click ssh cka0037 (or any highlighted value) to put it on the desktop's clipboard and paste in the terminal with Ctrl+Shift+V. Firefox in the desktop opens only the allowed documentation for the track (KSIM_OPEN_WEB=1 lifts that). Submit grades every task on its own host; the hosts stay up for review until the next session starts.

Everything is also available from a terminal:

./ksim --track ckad drill            # next least-drilled domain, 5 tasks      (--track or KSIM_TRACK)
./ksim drill weak | drill 2 17 20 | drill TSHOOT NET | drill 3.2
./ksim start [--hard]                # full mock, sampled by domain weight
./ksim sim c|d                       # a fixed simulator set (CKA: c, d  ·  CKAD: a)
./ksim objectives | next | focus     # the curriculum as a study path
./ksim study 3.2 | notes 3.2         # an objective: notes, then its tasks with no clock
./ksim list | show N | grade N | solution N | timer | end
./ksim hosts [start]                 # the session's hosts; `start` brings them back after a reboot
./ksim ssh N                         # a shell on the host of task N from your own terminal
./ksim reset N                       # rebuild the host of task N from scratch
./ksim down                          # delete every host and the desktop (history and images stay)
./ksim coverage                      # tasks per domain and objective against the 17–20 target
./ksim audit [all|DOMAIN|N..]        # bank check: seed each task on its own cluster, grade it untouched,
                                     # then solve it with its script and grade again (--settle N, --keep)

Tracks

Each track is a directory: cka/, ckad/, cks/.

File	What
questions.py	TRACK (name, minutes, pass mark, Kubernetes minor, domains, mock quota, default workers, host image) and the first tasks; pulls in the other banks
bank_*.py	more tasks; bank_sim_* are the fixed simulator sets
curriculum.py	the exam's objectives in order, each with study notes and the tasks that practise it
solutions.py	reference solutions (CKA keeps them apart; the others inline them)
solve.py	scripted solutions that ./ksim audit runs on the task's host to prove a task can be solved

Track	Kubernetes	Bank today	Target (17–20 pool tasks per domain)
CKA	v1.35	131 tasks: 96 pool (17–25 per domain) + sets C, D (35)	reached in all five domains
CKAD	v1.37	105 tasks: 85 pool (17 per domain) + set A (20)	reached in all five domains
CKS	v1.35	110 tasks, all pool: 17 per domain on container hosts, plus 5 Falco and 3 AppArmor tasks on VM hosts	reached in all six domains

./ksim coverage prints this per domain and per objective.

CKA, as audited on 2026-10-13: all 131 tasks seed cleanly on a host of their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted solution is run on the host as candidate — for 66 of them that script is the reference solution exactly as a study session shows it.

CKAD, as audited on 2026-10-12: all 105 tasks seed cleanly on a host of their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted solution is run on the host as candidate.

CKS, as audited on 2026-10-12: all 110 tasks seed cleanly on a host of their own, score nothing before anyone has worked on them, and reach full marks when their scripted solution is run on the host as candidate. The 26 objectives are the curriculum's own (cks/curriculum.py), each with notes, commands, traps and the documentation the exam allows — which, per the Linux Foundation's list read that day, is the Kubernetes docs and blog plus Falco, bom, etcd, the NGINX Ingress Controller, Cilium and Istio. Trivy's and AppArmor's own documentation are not on it.

A task

{"id": 114, "domain": "NET", "difficulty": "normal",      # or "hard"
 "params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])},
 "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
 "setup": "bash that seeds the scenario on the task's own cluster",
 "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
 "solution": "...",
 "workers": 2}                                              # optional: worker nodes of its cluster

Setup and check scripts run on this machine against the task's own cluster and get:

• $K — kubectl for that cluster (KUBECONFIG is exported too, so helm just works);
• $CP, $W1, $W2, $W3 — its node names, and nx NODE 'cmd' to run a command on one as root;
• $WS — the task's /opt/course directory as a path here (it is mounted into the host), so a check reads a file the candidate wrote with plain cat "$WS/q14/answer.txt". Once seeded, the files belong to candidate: one that is readable by its owner only — a private key, a kubeconfig kubectl config wrote — has to be read on the host, nx "$CP" 'cat /opt/course/q37/tls.key', and a kubeconfig made there is tested there too (its server address is the host's);
• $PRIV — a directory beside it that is not mounted into the host: where a setup records what its checks compare against later (the UID of an object that must not be re-created, say);
• podman ... — root's podman on the task's host, the store the candidate's podman/docker use;
• j <get args> '{jsonpath}', eq a b, mkns (from sim.bank.NS), ensure_gateway_crds, ensure_metrics_server.

Task text may use {WS} (/opt/course), {CP}, {W1}.., and {SHELL_CP}, {SHELL_W1}.. (ssh cka0114, ssh cka0114-worker).

Topology. A task's cluster has the workers it asks for with "workers". Without it, a task that names a worker anywhere ($W1, {W2}, …) gets three — what the CKA bank was written for — and every other task gets its track's default (TRACK["workers"]: CKA 1, CKAD 0, CKS 1). A task whose params draw a node (r.choice(["{W1}", "{W2}"])) has to say "workers" itself: the lambda is not searched. A track's WORKERS = {id: n} map sets it without editing the task; PENDING = {id: why} holds a task back from every session. A task with "backend": "vm" gets a single-node VM host; such tasks hold themselves back until ./ksim build cks --vm has been run. On a VM host, nx runs its command in the VM, and nothing enforces NetworkPolicy (its Pod network is a plain bridge).

Proving a task. ./ksim audit N.. builds the task's host, seeds it and grades it untouched: the seed must be clean and the task must not score full marks before anyone has worked on it. If <track>/solve.py has a script for the task — what a candidate would type on the host — the audit then runs it there as candidate and grades again, and only full marks pass (what the script created gets up to 90 seconds to come up). A task with no script is proven to seed, not to be solvable; write the script when you write the task. --settle 45 waits that long between seeding and the untouched grade, which catches a check that only passes once the seeded Pods run, or until a stopped kubelet shows as NotReady; --keep leaves the host up to look at (KUBECONFIG=.state/envs/<host>/kubeconfig, ./ksim down removes it).

In cka/solve.py, AS_WRITTEN lists the tasks whose reference solution is a script as it stands: for those the audit runs exactly the text the candidate is shown, so a solution that stops working with a new Kubernetes release fails the next audit instead of misleading you in a study session.

What is different from the real exam

• The desktop's base machine and the container hosts are Debian, not Ubuntu; apt, systemd and kubeadm behave the same. VM hosts are Ubuntu 24.04.
• Container hosts share this machine's kernel. That is invisible for CKA and CKAD, and for most of CKS — gVisor, seccomp, Istio's sidecars, kube-bench, real kubeadm upgrades all work in them. Falco and AppArmor do not, which is what the VM hosts are for.
• No Cilium runs anywhere (kindnet is the CNI, and it does enforce NetworkPolicy): the one Cilium task is graded on the object. No Ingress controller either: Ingress tasks are graded on the objects, with their certificates checked by openssl.
• On a container host kubeadm upgrade passes one preflight check it would fail (a rootless container has no cpuset cgroup controller); a wrapper adds the flag for that, nothing else.
• Worker nodes are cka0037-worker, cka0037-worker2, … rather than node01.

Coming from cka-sim

Task ids are unchanged, so your record carries over if you want it:

cp ../cka-sim/history-*.jsonl ../cka-sim/study-*.json .

What changed for existing tasks: files go to /opt/course/... on the task's host instead of workspace/... on this machine; the kubectl config use-context line is gone (the host replaces it); node access is ssh + sudo -i instead of podman exec; and the image-build tasks use the host's podman (podman, sudo podman and docker are one store there) instead of this machine's.