vumast3r/kubestronaut-sim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add guided lessons; move CKA and CKS to v1.37 on Ubuntu hosts

Browse Source 
Add lessons: untimed guided labs, one per curriculum objective, on a
host of their own, in the browser and in the terminal. Each step has
a check, a hint and its commands; a lesson writes no history or
scores, only a done mark on its objective. CKAD 2.1-2.4 (24 steps)
and all 27 CKA objectives (135 steps) have one, and `./ksim lesson
audit` proves that every step fails, is done by its own commands,
then passes.

Add CKA tasks 132-147 (cka/bank_course.py) on what a full CKA course
teaches beyond the bank: a real upgrade from v1.36, a drain under a
PodDisruptionBudget, QoS and eviction, in-place resize, kube-proxy's
chains, encryption at rest, an operator, and CSI expansion, snapshots
and clones (ensure_csi_hostpath).

Move CKA and CKS to Kubernetes v1.37 and rebuild the hosts on Ubuntu
24.04, with kubelet, kubeadm and kubectl as held packages from
pkgs.k8s.io; the desktop runs Ubuntu too. Hosts gain man pages,
hexdump and xxd.

Fix CKS Q4, which gave 3 of 12 points untouched, and CKA Q4 and Q49,
whose solutions raced their new CRD.

Audited on v1.37: CKA 147/147, CKAD 105/105, CKS 109/110 (Q103, a
Falco task, failed to seed once and awaits a re-audit); lessons: 135
of 135 CKA steps and 24 of 24 CKAD steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
vumas3r 2026-10-20 20:15:17 -06:00
parent a87668c183
commit 0e5f7686ee
30 changed files with 6248 additions and 211 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

72
CHANGELOG.md
View File

@ -4,6 +4,78 @@ Notable changes to kubestronaut-sim, newest first. A version is the `VERSION` st
(year.month.day.n); `./ksim help` and the top bar of the web page show the one you are running.
Every entry names the commit it came in.
## Unreleased — 2026.10.19.1, on branch `feat/lessons`
### Changed
- **CKA and CKS run Kubernetes v1.37** (CKAD already did). CKA follows the real exam; CKS runs
ahead of it - the Linux Foundation still listed v1.35 for CKS on 2026-10-19. The CKA curriculum
is unchanged (`CKA_Curriculum_v1.37.pdf` is the v1.35 document). Tasks and notes that named
1.35/1.36 now name 1.37 and 1.38, and the etcd ones etcd 3.7.
- CKS Q37 and Q39 hold their worker back on **v1.36** - the v1.37 repository has nothing older
than the cluster's 1.37.0 - so upgrading it is a minor upgrade: the repository line first.
- **Exam hosts and the desktop are Ubuntu 24.04**, not Debian. The host image is kind's node
rebuilt on `ubuntu:24.04`: kind's packages and systemd setup, and its containerd, runc, crictl,
CNI plugins and preloaded images copied from the pinned kind image. The desktop gets Firefox ESR
from Mozilla's APT repository (Ubuntu's own `firefox` is a snap).
- **Every host has Kubernetes as held apt packages** from pkgs.k8s.io, with the repository line of
its minor, as an exam node has - before, only CKS hosts did. The `kubeadm upgrade` wrapper moved
into the common image. Hosts also gained man pages, `hexdump` (bsdextrautils) and `xxd`.
- A task or lesson may name a Kubernetes minor (`"k8s"`) or host flavour (`"image"`) of its own;
`./ksim build` builds what any of them names. v1.36 is pinned for the upgrade task and lesson.
- `./ksim build --vm` builds VM disks only for the tracks that have VM tasks.
### Added
- **CKA lessons for all 27 objectives** (`cka/lessons_storage.py`, `_tshoot`, `_work`, `_arch`,
`_net`; 135 steps). Besides each competency, they cover what a full CKA course teaches under it
- checked against the outline of KubeDojo's CKA course, written from scratch: projected volumes,
volume expansion, snapshots and restores, eviction thresholds, debug containers, kube-proxy's
chains, encryption at rest, in-place resize,
PodDisruptionBudgets, QoS classes, ServiceAccount tokens, a real upgrade from v1.36 to v1.37 after
an etcd backup and restore, an HA join, EndpointSlices, an operator. `./ksim lesson audit`: 135
of 135 steps proven.
- **CKA tasks 132-145** (`cka/bank_course.py`) on the same topics: a real cluster upgrade to
v1.37.1, a drain under a PodDisruptionBudget, a container without a shell, ServiceAccount tokens
and automounting, a projected volume, QoS classes, in-place resize, eviction thresholds,
kube-proxy's chains, a Service backed by a hand-written EndpointSlice, encryption at rest,
required anti-affinity (and the rollout it blocks), a StatefulSet canary by partition, an
operator whose RBAC lacks `backups/status`, and (146-147) growing a claim in use, snapshot,
restore and clone on `csi-hostpath-sc`. CKA now has 147 tasks, 112 of them in the pool.
- `ensure_csi_hostpath` for setups: the CSI hostpath driver (v1.18.0) with external-snapshotter's
CRDs and controller, its `csi-hostpath-snapclass`, and StorageClass `csi-hostpath-sc` with
`allowVolumeExpansion: true` - the class the real CKA's storage tasks use.
- **Lessons**: guided labs, one per curriculum objective, on a host of their own and with no
clock. A lesson is an introduction, steps you do in the desktop's terminal (each with a
**Check step** button, a hint and **Show me the commands**), and a wrap-up that sends you to the
objective's timed drill. Steps count in order. A lesson writes nothing to the history or the
scores; one whose every step was checked is marked done on its objective.
- The pilot covers CKAD's Application Deployment domain: 2.1 blue/green and canary, 2.2
rolling updates and rollbacks, 2.3 Helm, 2.4 Kustomize (`ckad/lessons.py`).
- Browser: a lesson screen in the exam screen's frame (Introduction, Step N of M, Wrap-up), a
**Lesson** button and a `lesson ✓` mark on objectives, **Lesson first** on the path card, a
*Take a lesson* session in the Start exam dialog, and a summary page after a lesson.
- Terminal: `./ksim lesson [OBJ]`, `./ksim lesson step | check | hint | show [N]`, `./ksim end`.
- `./ksim lesson audit [OBJ..]` proves lessons on real clusters: every step must fail its check,
be done by its own commands as `candidate`, then pass. CKAD 2.1–2.4: 24 of 24 steps proven.
- Engine: lessons are sessions of mode `lesson` (hosts `ckad9201`.., ids 9000 + objective),
`POST /api/lesson/check`, done stamps under `lessons` in `study-<track>.json`, and `step()`
in `sim/bank.py` for writing them.
- Tasks and lessons render fenced code blocks (kept as written, a click copies them) and
paragraphs of `- ` lines as lists.
### Fixed
- CKA Q4 and Q49: their solutions wait for the new CRD to be `Established` before creating an
object of its kind (on v1.37 it lost that race: "no matches for kind").
- CKS Q4 gave 3 of its 12 points untouched: its token check took any Pod of the Deployment, and
an untouched Deployment's Pods run as `default`, with a token mounted. It now takes a Pod that
runs as `<deployment>-sa`.
- Building a host image for a kind image not pulled yet: podman's pull progress was read as the
etcd image's name.
- A solution or lesson step that prints binary data (an etcd value) no longer stops the audit.
## 2026.10.15.3 — 2026-10-15
### Added

105
README.md
View File

@ -35,15 +35,19 @@ restore could only be drilled alone). All of that is gone:
- tasks are provisioned, seeded and graded in parallel;
- you work as `candidate` on the task's host and use `sudo -i` for node work, as on the day.
Each host is a kind node built from `images/node/` — a kubeadm cluster (real etcd, static-pod
control plane, systemd kubelet) with sshd, `candidate` with passwordless sudo, `k` and kubectl
completion preset, vim (2-space tabs), tmux, jq, yq, helm, podman (also as `docker`) for the
image tasks, and `etcdctl`/`etcdutl` at the version of the cluster's etcd.
Each host is a kind node built from `images/node/` on **Ubuntu 24.04**, as the exam's hosts are —
a kubeadm cluster (real etcd, static-pod control plane, systemd kubelet). kubelet, kubeadm and
kubectl are apt packages from pkgs.k8s.io, held, with the repository line of the cluster's minor
(the CNI plugins come with them, as `kubernetes-cni`): a node is upgraded with `apt-get install`
and `kubeadm upgrade`, for real, and a task can hold one back on an older release. containerd,
runc, crictl, kind's entrypoint and the preloaded control-plane images are copied from the pinned
kind node image, which kind builds on Debian. On top: sshd, `candidate` with passwordless sudo,
`k` and kubectl completion preset, vim (2-space tabs), tmux, jq, yq, helm, man pages, podman (also
as `docker`) for the image tasks, and `etcdctl`/`etcdutl` at the version of the cluster's etcd.
**CKS hosts** add what that exam's hosts carry (`images/node-cks/`, built on the common image):
kube-bench, trivy, kubesec, kube-linter, bom, cosign, istioctl, gVisor (`runsc`, as a containerd
runtime handler) and Kubernetes as apt packages from pkgs.k8s.io — so a node can be held back on
an older patch release and upgraded with `apt-get install` and `kubeadm upgrade`, for real.
kube-bench, trivy, kubesec, kube-linter, bom, cosign, istioctl and gVisor (`runsc`, as a containerd
runtime handler).
**VM hosts.** A container shares this machine's kernel, and under rootless podman it can neither
load eBPF programs nor use a kernel feature this machine lacks: no Falco, no AppArmor. Tasks that
@ -77,8 +81,9 @@ refused before anything is created if a (more cautious) estimate does not fit (`
overrides). Clusters are provisioned one per four cores at a time, at most 8 (`KSIM_PARALLEL=N`).
A VM host is given 2 GB (it used about 1.6 GB right after starting) and is up in about 30 s.
Disk: the images take 1.3 GB per Kubernetes minor, 2.8 GB for the CKS host, 1.2 GB for the
desktop; the VM host adds a 3.5 GB base disk and the 0.6 GB Ubuntu cloud image under `.state/vm/`.
Disk: the images take 1.5 GB per Kubernetes minor (v1.37, and v1.36 for the tasks that upgrade
their cluster), 2.7 GB for the CKS host, 1.2 GB for the desktop; the VM host adds a 3.5 GB base
disk and the 0.6 GB Ubuntu cloud image under `.state/vm/`.
## Run
@ -130,6 +135,8 @@ host name such as `ckad0013` only resolves in the desktop's terminal: from your
./ksim sim c|d # a fixed simulator set (CKA: c, d · CKAD: a)
./ksim objectives | next | focus # the curriculum as a path, one objective at a time against the clock
./ksim notes 3.2 # an objective's notes: what to know, fastest commands, traps, docs
./ksim lesson [OBJ] # the track's lessons, or start one: a guided lab, no clock (CKA: all 27 · CKAD: 2.1-2.4)
./ksim lesson step | check | hint | show [N] # inside a lesson; ./ksim end finishes it
./ksim skip DESIGN | unskip DESIGN # "I am good with this domain (or objective 2.3)": off the path, and back
./ksim drill OBS:3 | next OBS:3 # three tasks of a domain, each from another objective - alone, or added
./ksim list | show N | grade N | solution N | timer | end
@ -141,8 +148,26 @@ host name such as `ckad0013` only resolves in the desktop's terminal: from your
./ksim coverage # tasks per domain and objective against the 17–20 target
./ksim audit [all|DOMAIN|N..] # bank check: seed each task on its own cluster, grade it untouched,
# then solve it with its script and grade again (--settle N, --keep)
./ksim lesson audit [OBJ..] # lesson check: every step fails, is done by its own commands, then passes
```
**Lessons.** A lesson teaches one objective as a guided lab on a host of its own: an introduction,
then steps you do in the desktop's terminal, each with a **Check step** button, a hint and **Show me
the commands**, and a wrap-up that sends you to the objective's timed drill. Lessons are the one
untimed part of the simulator — they teach, the drills test — and they write nothing to the
history, the statistics or the weak list; a lesson whose every step was checked is marked done on
its objective (`lesson ✓` in the objective list). Steps count in order. Start one from an
objective's **Lesson** button, **Lesson first** on the path card, the *Take a lesson* session in the
Start exam dialog, or `./ksim lesson 2.1`. **CKA has a lesson for each of its 27 objectives**
(`cka/lessons_*.py`, 135 steps), which also cover what a full CKA course teaches under them —
checked against the outline of KubeDojo's CKA course and written from scratch: projected volumes,
volume expansion, snapshots and restores, eviction thresholds, debug containers, kube-proxy's
rules, encryption at rest, in-place resize, PodDisruptionBudgets, QoS classes, ServiceAccount
tokens, a real upgrade from v1.36 to v1.37, EndpointSlices, an operator. Tasks 132–147 test those
topics against the clock. The CKAD pilot
covers the Application Deployment domain (2.1 blue/green and canary, 2.2 rolling updates, 2.3
Helm, 2.4 Kustomize).
**Skipping what you know.** `./ksim skip DESIGN` (or the "I know this — skip it" link beside a
domain in the browser's objective list) marks a domain or an objective as known: the path, the
no-argument drill and `drill all` leave it out, and it no longer has to be drilled before a full
@ -163,29 +188,33 @@ Each track is a directory: `cka/`, `ckad/`, `cks/`.
| `curriculum.py` | the exam's objectives in order, each with study notes and the tasks that practise it |
| `solutions.py` | reference solutions (CKA keeps them apart; the others inline them) |
| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved |
| `lessons.py` | guided labs, one per objective (CKA: all, one `lessons_*.py` per domain · CKAD: 2.1–2.4): introduction, steps with their check, hint and commands, wrap-up |
| Track | Kubernetes | Bank today | Target (17–20 pool tasks per domain) |
|---|---|---|---|
| CKA | v1.35 | 131 tasks: 96 pool (17–25 per domain) + sets C, D (35) | reached in all five domains |
| CKA | v1.37 | 147 tasks: 112 pool (18–28 per domain) + sets C, D (35) | reached in all five domains |
| CKAD | v1.37 | 105 tasks: 85 pool (17 per domain) + set A (20) | reached in all five domains |
| CKS | v1.35 | 110 tasks, all pool: 17 per domain on container hosts, plus 5 Falco and 3 AppArmor tasks on VM hosts | reached in all six domains |
| CKS | v1.37 | 110 tasks, all pool: 17 per domain on container hosts, plus 5 Falco and 3 AppArmor tasks on VM hosts | reached in all six domains |
`./ksim coverage` prints this per domain and per objective.
**CKA, as audited on 2026-10-13:** all 131 tasks seed cleanly on a host of their own, score
nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted
solution is run on the host as `candidate` — for 66 of them that script is the reference solution
exactly as a drill shows it once the task is graded.
**CKA, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 147 tasks seed cleanly on a host of
their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when
their scripted solution is run on the host as `candidate` — for 80 of them that script is the
reference solution exactly as a drill shows it once the task is graded. Q132 really upgrades its
cluster from v1.36.4 to v1.37.1, worker included.
**CKAD, as audited on 2026-10-12:** all 105 tasks seed cleanly on a host of their own, score
nothing when graded untouched 45 seconds after seeding, and reach full marks when their scripted
solution is run on the host as `candidate`.
**CKAD, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 105 tasks seed cleanly on a host of
their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when
their scripted solution is run on the host as `candidate`.
**CKS, as audited on 2026-10-12:** all 110 tasks seed cleanly on a host of their own, score nothing
before anyone has worked on them, and reach full marks when their scripted solution is run on the
host as `candidate`. The 26 objectives are the curriculum's own (`cks/curriculum.py`), each with
notes, commands, traps and the documentation the exam allows — which, per the Linux Foundation's
list read that day, is the Kubernetes docs and blog plus Falco, bom, etcd, the NGINX Ingress
**CKS, as audited on 2026-10-19** (v1.37, Ubuntu hosts): 109 of 110 tasks seed cleanly on a host
of their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks
when their scripted solution is run on the host as `candidate`. Q103, a Falco task on a VM host,
failed to seed once — Falco raised no alert naming the Pod within a minute — and is still to be
audited again. The 26 objectives are the curriculum's own (`cks/curriculum.py`), each with notes,
commands, traps and the documentation the exam allows — which, per the Linux Foundation's list
read on 2026-10-12, is the Kubernetes docs and blog plus Falco, bom, etcd, the NGINX Ingress
Controller, Cilium and Istio. Trivy's and AppArmor's own documentation are not on it.
### A task
@ -197,7 +226,8 @@ Controller, Cilium and Istio. Trivy's and AppArmor's own documentation are not o
"setup": "bash that seeds the scenario on the task's own cluster",
"checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
"solution": "...",
"workers": 2} # optional: worker nodes of its cluster
"workers": 2, # optional: worker nodes of its cluster
"k8s": "v1.36"} # optional: a minor of its own (an upgrade task)
```
Setup and check scripts run on this machine against the task's own cluster and get:
@ -213,7 +243,9 @@ Setup and check scripts run on this machine against the task's own cluster and g
its checks compare against later (the UID of an object that must not be re-created, say);
- `podman ...` — root's podman on the task's host, the store the candidate's `podman`/`docker` use;
- `j <get args> '{jsonpath}'`, `eq a b`, `mkns` (from `sim.bank.NS`), `ensure_gateway_crds`,
`ensure_metrics_server`.
`ensure_metrics_server`, and `ensure_csi_hostpath` — the CSI hostpath driver with the snapshot
controller, and StorageClass `csi-hostpath-sc` (expandable) as the exam's clusters have it, for
what local-path cannot do: growing a claim, snapshots, clones.
Task text may use `{WS}` (`/opt/course`), `{CP}`, `{W1}`.., and `{SHELL_CP}`, `{SHELL_W1}`..
(`ssh cka0114`, `ssh cka0114-worker`).
@ -242,15 +274,32 @@ In `cka/solve.py`, `AS_WRITTEN` lists the tasks whose reference solution is a sc
for those the audit runs exactly the text the candidate is shown, so a solution that stops working
with a new Kubernetes release fails the next audit instead of misleading you after a drill.
**Writing a lesson.** A lesson in `<track>/lessons.py` is a dict with `id` (9000 plus the objective,
`9201` for 2.1 — its host is `ckad9201`, which no task's can be), `objective`, `title`, `workers`,
`intro`, `setup` (bash, as a task's), `steps` and `outro` - and, like a task, `k8s` or `image` if
it needs a host of its own kind (lesson 4.4 runs on v1.36 to upgrade it). Each step comes from
`step(title, text, check, hint, show)` in `sim/bank.py`: `check` runs like a task's check, and
`show` is what the learner would type. `./ksim lesson audit` seeds each lesson on its own host and,
step by step, requires the check to fail, runs `show` there as `candidate`, and requires the check
to pass — so a lesson can always be done exactly as it shows, and no step is done before you do it.
Work on another node is shown as `ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain
`ssh NODE` would read the rest of the commands as its own input. **As audited on 2026-10-19**
(v1.37, Ubuntu hosts): CKA's 27 lessons, 135 of 135 steps, and CKAD 2.1–2.4, 24 of 24 steps,
proven.
## What is different from the real exam
- The desktop's base machine and the container hosts are Debian, not Ubuntu; `apt`, systemd and
kubeadm behave the same. VM hosts are Ubuntu 24.04.
- The desktop, the container hosts and the VM hosts are Ubuntu 24.04 LTS. The Linux Foundation does
not publish which Ubuntu release the exam runs; 24.04 is the current LTS. The container hosts'
containerd and runc are kind's builds, not Ubuntu's packages.
- CKA and CKAD run v1.37, as the real exams do. CKS runs v1.37 too, ahead of the real exam: on
2026-10-19 the Linux Foundation still listed v1.35 for it.
- Container hosts share this machine's kernel. That is invisible for CKA and CKAD, and for most of
CKS — gVisor, seccomp, Istio's sidecars, kube-bench, real kubeadm upgrades all work in them.
Falco and AppArmor do not, which is what the VM hosts are for.
- No Cilium runs anywhere (kindnet is the CNI, and it does enforce NetworkPolicy): the one Cilium
task is graded on the object. No Ingress controller either: Ingress tasks are graded on the
task is graded on the object. The CSI hostpath driver is installed only on the hosts of the tasks
and lessons that use it. No Ingress controller either: Ingress tasks are graded on the
objects, with their certificates checked by openssl.
- On a container host `kubeadm upgrade` passes one preflight check it would fail (a rootless
container has no cpuset cgroup controller); a wrapper adds the flag for that, nothing else.

1033
cka/bank_course.py Normal file
View File

File diff suppressed because it is too large Load Diff

14
cka/bank_sim_c.py
View File

@ -896,19 +896,21 @@ grep -Eq 'kubeadm[[:space:]]+upgrade[[:space:]]+node' "$WS/q80/worker.txt" 2>/de
k version -o json | jq -r .serverVersion.gitVersion > {WS}/q80/current.txt
k get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion --no-headers > {WS}/q80/kubelets.txt
cat > {WS}/q80/steps.sh <<'EOF'
sed -i 's|/v1.35/|/v1.36/|' /etc/apt/sources.list.d/kubernetes.list
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.36.5-1.1 && apt-mark hold kubeadm
sed -i 's|/v1.37/|/v1.38/|' /etc/apt/sources.list.d/kubernetes.list
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.38.0-1.1 && apt-mark hold kubeadm
kubeadm upgrade plan
kubeadm upgrade apply v1.36.5 -y
kubeadm upgrade apply v1.38.0 -y
kubectl drain <control-plane> --ignore-daemonsets
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.36.5-1.1 kubectl=1.36.5-1.1 && apt-mark hold kubelet kubectl
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon <control-plane>
EOF
echo 'kubeadm upgrade node' > {WS}/q80/worker.txt
# The cluster runs v1.35, so the next minor is 1.36 - and pkgs.k8s.io serves one apt repo per
# minor, so the repo line has to move to v1.36 before 1.36 packages exist for apt at all.
# The cluster runs v1.37, so the next minor is 1.38 - and pkgs.k8s.io serves one apt repo per
# minor, so the repo line has to move to v1.38 before 1.38 packages exist for apt at all. (1.38
# is due in December 2026; until then that repo is empty, which is why this is only a plan. Take
# the newest 1.38 patch on offer then: `apt-cache madison kubeadm`.)
# kubeadm goes first because it drives the whole thing; the control plane is upgraded by
# `upgrade apply`, and only then do the node binaries (kubelet/kubectl) follow, ending with
# uncordon. Workers run `kubeadm upgrade node` - they host no control-plane static pods, so

109
cka/common.py
View File

@ -44,3 +44,112 @@ look() { $K -n "$1" exec "$2" -- nslookup "$3" 2>/dev/null | awk '/^Name:/ {n=1}
# configuration by itself, but not at once)
resolves() { local i; for i in $(seq 1 10); do [ "$(look "$1" "$2" "$3")" = "$4" ] && return 0; sleep 3; done; return 1; }
"""
# A small operator for Backup objects (group data.example.com): a controller that gives each
# Backup a ConfigMap and marks it Scheduled. It is kubectl in a loop - the node's own kubectl,
# mounted into a busybox container - which is enough to show the pattern: a CRD, RBAC for its
# controller, and a Deployment that acts on it. With status_rbac=False the ClusterRole lacks
# `backups/status`, and the controller can make the ConfigMaps but not set the phase.
_BACKUP_CRD = r"""apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: backups.data.example.com
spec:
group: data.example.com
scope: Namespaced
names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]}
versions:
- name: v1
served: true
storage: true
subresources:
status: {}
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [schedule]
properties:
schedule: {type: string}
retention: {type: integer, minimum: 1, default: 7}
status:
type: object
properties:
phase: {type: string}
---
"""
def backup_operator(crd=False, status_rbac=True):
"""The operator's manifests, as one YAML document stream (the CRD first, when asked for)."""
status = ", backups/status" if status_rbac else ""
return (_BACKUP_CRD if crd else "") + r"""apiVersion: v1
kind: Namespace
metadata:
name: backup-operator
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: controller
namespace: backup-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: backup-operator
rules:
- apiGroups: [data.example.com]
resources: [backups""" + status + r"""]
verbs: [get, list, watch, update, patch]
- apiGroups: [""]
resources: [configmaps]
verbs: [get, list, create, update, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: backup-operator
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: backup-operator}
subjects:
- {kind: ServiceAccount, name: controller, namespace: backup-operator}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: controller
namespace: backup-operator
spec:
replicas: 1
selector:
matchLabels: {app: backup-operator}
template:
metadata:
labels: {app: backup-operator}
spec:
serviceAccountName: controller
containers:
- name: controller
image: busybox:1.36
command:
- sh
- -c
- |
set -f
while true; do
for b in $(kubectl get backups -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'); do
ns=${b%/*}; name=${b#*/}
sched=$(kubectl -n "$ns" get backup "$name" -o jsonpath='{.spec.schedule}')
kubectl -n "$ns" create configmap "backup-$name" --from-literal=schedule="$sched" --dry-run=client -o yaml | kubectl apply -f - >/dev/null
kubectl -n "$ns" patch backup "$name" --subresource=status --type=merge -p '{"status":{"phase":"Scheduled"}}' >/dev/null
done
sleep 5
done
volumeMounts:
- {name: kubectl, mountPath: /bin/kubectl, readOnly: true}
volumes:
- name: kubectl
hostPath: {path: /usr/bin/kubectl, type: File}
"""

39
cka/curriculum.py
View File

@ -1,5 +1,5 @@
"""CKA curriculum as a path to the exam: the 27 objectives of the five domains, in the curriculum's own
words and order (CKA_Curriculum_v1.35.pdf, cncf/curriculum), each with what to know, the fastest
words and order (CKA_Curriculum_v1.37.pdf, cncf/curriculum), each with what to know, the fastest
commands, the traps, the documentation pages the exam lets you open, and the tasks that practise it.
./ksim objectives the path, with your progress per objective
@ -26,7 +26,7 @@ OBJECTIVES = [
{
"id": "1.1", "key": "storageclasses", "domain": "STOR",
"text": "Implement storage classes and dynamic volume provisioning",
"tasks": [16, 36, 114, 123, 124, 59, 118, 122, 99],
"tasks": [16, 36, 114, 123, 124, 146, 59, 118, 122, 99],
"know": [
"A StorageClass names a provisioner and its settings. A claim that asks for the class gets a PersistentVolume made for it - that is dynamic provisioning.",
"Fields: provisioner, reclaimPolicy (Delete by default, or Retain), volumeBindingMode (Immediate or WaitForFirstConsumer), allowVolumeExpansion, parameters.",
@ -58,7 +58,7 @@ k get pvc,pv; k describe pvc data # the Events say why a cla
{
"id": "1.2", "key": "volumes", "domain": "STOR",
"text": "Configure volume types, access modes and reclaim policies",
"tasks": [15, 35, 114, 115, 124, 58, 119, 120, 68, 72, 102],
"tasks": [15, 35, 114, 115, 124, 136, 58, 119, 120, 68, 72, 102],
"know": [
"Volume types you meet: hostPath (a directory of whatever node the Pod lands on), local (a disk of one node, needs nodeAffinity), emptyDir, nfs, csi, and the configMap / secret / projected volumes.",
"Access modes: ReadWriteOnce (one node), ReadOnlyMany, ReadWriteMany, ReadWriteOncePod (one Pod). A claim binds only to a volume that offers its mode.",
@ -91,7 +91,7 @@ k get pv -o custom-columns=NAME:.metadata.name,MODES:.spec.accessModes,RECLAIM:.
{
"id": "1.3", "key": "claims", "domain": "STOR",
"text": "Manage persistent volumes and persistent volume claims",
"tasks": [15, 36, 35, 116, 121, 123, 58, 59, 117, 118, 119, 120, 68, 72, 81, 99, 102],
"tasks": [15, 36, 35, 116, 121, 123, 146, 147, 58, 59, 117, 118, 119, 120, 68, 72, 81, 99, 102],
"know": [
"A claim binds to one volume, and only if class, access mode and size fit (the volume may be larger); `volumeName` or a selector narrows it to one volume.",
"Phases: a volume goes Available, Bound, Released; a claim Pending, Bound.",
@ -125,7 +125,7 @@ k describe pod POD | tail # unbound claim -> the Pod stays
{
"id": "2.1", "key": "nodes", "domain": "TSHOOT",
"text": "Troubleshoot clusters and nodes",
"tasks": [17, 39, 60, 129, 92, 90, 95],
"tasks": [17, 39, 60, 129, 139, 92, 90, 95],
"know": [
"Start from the cluster: `k get nodes`, then `k describe node` - Conditions, Taints, Unschedulable, Events.",
"A NotReady node is nearly always its kubelet: on the node, `systemctl status kubelet` and `journalctl -u kubelet` name the cause.",
@ -205,7 +205,7 @@ k -n kube-system logs deploy/metrics-server
{
"id": "2.4", "key": "logs", "domain": "TSHOOT",
"text": "Manage and evaluate container output streams",
"tasks": [19, 104, 8, 131, 85],
"tasks": [19, 104, 8, 131, 134, 85],
"know": [
"`kubectl logs` shows what a container wrote to stdout and stderr. The runtime keeps it on the node under /var/log/pods/NAMESPACE_POD_UID/CONTAINER/, and it goes when the Pod is deleted.",
"Pick the container with -c (init containers too) or take --all-containers; --previous is the run before the last restart.",
@ -233,7 +233,7 @@ crictl ps -a; crictl logs --tail=20 CONTAINER-ID; ls /var/log/pods/
{
"id": "2.5", "key": "netdebug", "domain": "TSHOOT",
"text": "Troubleshoot services and networking",
"tasks": [18, 62, 37, 57, 73, 101, 70, 90],
"tasks": [18, 62, 37, 57, 140, 73, 101, 70, 90],
"know": [
"Work outwards from the Pod: is it Ready, does the Service have endpoints (selector against Pod labels), is targetPort the port the container listens on, does the name resolve, does a NetworkPolicy stand in the way.",
"DNS is a Service like any other: the CoreDNS Pods, the kube-dns Service and its endpoints, the Corefile in ConfigMap coredns.",
@ -261,7 +261,7 @@ k get netpol -A
{
"id": "3.1", "key": "rollouts", "domain": "WORK",
"text": "Understand application deployments and how to perform rolling update and rollbacks",
"tasks": [7, 51],
"tasks": [7, 51, 144],
"know": [
"Any change to `spec.template` starts a rollout: a new ReplicaSet scales up while the old one scales down, within maxSurge and maxUnavailable (25% each by default).",
"Every rollout is a revision. `kubectl rollout`: status, history, undo, pause, resume, restart.",
@ -287,7 +287,7 @@ k patch deploy web -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUn
{
"id": "3.2", "key": "config", "domain": "WORK",
"text": "Use ConfigMaps and Secrets to configure applications",
"tasks": [10, 29, 79, 93, 84],
"tasks": [10, 29, 142, 79, 93, 84],
"know": [
"Both hold keys. A key becomes an environment variable (configMapKeyRef / secretKeyRef, or every key at once with envFrom) or a file (a configMap / secret volume; `items` picks and renames keys).",
"Secret data is base64 - an encoding, not encryption. Types: Opaque, kubernetes.io/tls, kubernetes.io/dockerconfigjson.",
@ -318,7 +318,7 @@ k set env deploy/web --from=configmap/app-config; k rollout restart deploy/web
{
"id": "3.3", "key": "autoscaling", "domain": "WORK",
"text": "Configure workload autoscaling",
"tasks": [105, 23, 82],
"tasks": [105, 23, 138, 82],
"know": [
"A HorizontalPodAutoscaler (autoscaling/v2) keeps a workload between minReplicas and maxReplicas by a metric - usually average CPU or memory utilisation, which is usage divided by the container's request.",
"It reads the metrics API, so it needs metrics-server, and the containers need a request for the resource it scales on.",
@ -355,7 +355,7 @@ spec:
{
"id": "3.4", "key": "selfhealing", "domain": "WORK",
"text": "Understand the primitives used to create robust, self-healing, application deployments",
"tasks": [106, 8, 28, 26, 27, 31, 52, 53, 76, 88, 70],
"tasks": [106, 133, 8, 28, 26, 27, 31, 52, 53, 76, 88, 70],
"know": [
"Controllers put back what is missing: a Deployment its replicas, a DaemonSet one Pod per node, a StatefulSet named replicas with their own storage, a Job its completions (backoffLimit retries), a CronJob its Jobs.",
"Probes tell the kubelet about health: liveness restarts the container, readiness takes the Pod out of Service endpoints, startup holds both off until the app is up.",
@ -389,7 +389,7 @@ k describe pod POD | grep -E 'Liveness|Readiness|Startup'; k get events --field
{
"id": "3.5", "key": "scheduling", "domain": "WORK",
"text": "Configure Pod admission and scheduling (limits, node affinity, etc.)",
"tasks": [9, 25, 24, 32, 50, 78, 96, 81, 97],
"tasks": [9, 25, 24, 32, 50, 137, 143, 78, 96, 81, 97],
"know": [
"Requests are what the scheduler reserves, limits what the node enforces. A LimitRange gives containers defaults and bounds; a ResourceQuota caps a Namespace's totals - both act at admission.",
"Where a Pod may go: nodeSelector, node affinity (required or preferred), Pod affinity and anti-affinity with a topologyKey, topologySpreadConstraints, or nodeName, which skips the scheduler.",
@ -426,7 +426,7 @@ k describe pod POD | tail # FailedScheduling: 0/4 nodes are a
{
"id": "4.1", "key": "rbac", "domain": "ARCH",
"text": "Manage role based access control (RBAC)",
"tasks": [1, 33, 6, 44, 47, 75, 91],
"tasks": [1, 33, 6, 44, 47, 135, 75, 91],
"know": [
"A Role (one Namespace) or ClusterRole (everywhere, and cluster-scoped resources) lists verbs on resources; a RoleBinding or ClusterRoleBinding grants it to users, groups or ServiceAccounts.",
"A RoleBinding may point at a ClusterRole: the rules then apply in that one Namespace only.",
@ -514,7 +514,7 @@ grep staticPodPath /var/lib/kubelet/config.yaml; k run web --image=nginx:1.27 $
{
"id": "4.4", "key": "lifecycle", "domain": "ARCH",
"text": "Manage the lifecycle of Kubernetes clusters",
"tasks": [109, 2, 65, 45, 80],
"tasks": [109, 2, 65, 45, 132, 80],
"know": [
"Upgrade one minor version at a time, control plane first: the kubeadm package, `kubeadm upgrade plan` and `apply`, drain, the kubelet and kubectl packages, restart the kubelet, uncordon. Workers: `kubeadm upgrade node` in place of `apply`.",
"Maintenance on a node: `kubectl drain` cordons it and evicts its Pods; `kubectl uncordon` gives it back.",
@ -523,8 +523,8 @@ grep staticPodPath /var/lib/kubelet/config.yaml; k run web --image=nginx:1.27 $
],
"fast": """
k drain NODE --ignore-daemonsets --delete-emptydir-data; k uncordon NODE
apt-get install -y kubeadm='1.36.*'; kubeadm upgrade plan; kubeadm upgrade apply v1.36.x # workers: kubeadm upgrade node
apt-get install -y kubelet='1.36.*' kubectl='1.36.*'; systemctl daemon-reload; systemctl restart kubelet
apt-get install -y kubeadm='1.38.*'; kubeadm upgrade plan; kubeadm upgrade apply v1.38.x # workers: kubeadm upgrade node
apt-get install -y kubelet='1.38.*' kubectl='1.38.*'; systemctl daemon-reload; systemctl restart kubelet
etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \\
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup.db
etcdutl snapshot restore /opt/backup.db --data-dir /var/lib/etcd-restore # then etcd.yaml: the hostPath of volume etcd-data
@ -533,7 +533,7 @@ kubeadm certs renew all; kubeadm certs check-expiration
"traps": [
"drain stops at DaemonSet Pods (--ignore-daemonsets), at emptyDir data (--delete-emptydir-data) and at bare Pods (--force, which deletes them for good).",
"A drained node stays cordoned until you uncordon it.",
"With etcd 3.6, as here, restore exists only in etcdutl - `etcdctl snapshot restore` is gone.",
"Since etcd 3.6 (v1.37 runs 3.7), restore exists only in etcdutl - `etcdctl snapshot restore` is gone.",
"Restoring etcd takes the whole cluster back to the snapshot: everything created since is lost.",
],
"docs": ["https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/",
@ -626,7 +626,7 @@ k get csidrivers; k get csinodes; k get sc # PROVISIONER
{
"id": "4.8", "key": "crds", "domain": "ARCH",
"text": "Understand CRDs, install and configure operators",
"tasks": [4, 34, 49, 91],
"tasks": [4, 34, 49, 145, 91],
"know": [
"A CustomResourceDefinition adds a kind to the API: group, names (kind, plural, singular, shortNames), scope (Namespaced or Cluster) and versions, each with an OpenAPI v3 schema.",
"The schema validates custom resources - types, required fields, enum, minimum, maximum, defaults - and feeds `kubectl explain`.",
@ -658,6 +658,7 @@ k get crd; k api-resources --api-group=data.example.com; k explain backup.spec
""",
"traps": [
"The CRD's name must be PLURAL.GROUP, and exactly one version has `storage: true`.",
"A custom resource applied right after its CRD can fail with \"no matches for kind\": the new type is served a moment later. `k wait --for condition=established crd/NAME` first.",
"A custom resource's apiVersion is GROUP/VERSION of the CRD, not apiextensions.k8s.io/v1.",
"`default` and validation only work inside a structural schema: every level needs its `type`.",
"Deleting a CRD deletes every object of that kind.",
@ -727,7 +728,7 @@ k describe netpol backend-policy -n NS; k exec CLIENT -- wget -qO- -T2 http://b
{
"id": "5.3", "key": "services", "domain": "NET",
"text": "Use ClusterIP, NodePort, LoadBalancer service types and endpoints",
"tasks": [12, 30, 43, 64, 71, 73, 94],
"tasks": [12, 30, 43, 64, 141, 71, 73, 94],
"know": [
"A Service selects Pods by label and forwards `port` to `targetPort` - a number, or the name of a container port.",
"ClusterIP is reachable inside the cluster; NodePort adds a port (30000-32767) on every node; LoadBalancer adds an external address from the cloud; ExternalName is a DNS alias; headless (clusterIP: None) gives every Pod its own DNS record.",

27
cka/lessons.py Normal file
View File

@ -0,0 +1,27 @@
"""CKA lessons: guided labs, one per curriculum objective, on a cluster of your own and without a
clock. A lesson explains, then walks you through steps that you do on its host; each step has a
check, a hint and the commands that do it. It ends by sending you to the objective's timed drill.
Lessons write nothing to the history or the scores - only a "lesson done" mark on the objective.
./ksim lesson the lessons of this track, and which you have done
./ksim lesson 1.1 start the lesson of objective 1.1 (id or key)
./ksim lesson check check the step you are on; the next one follows
./ksim lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after
A lesson's host is named after its id - lesson 9101 runs on `cka9101` - which never clashes with a
task's. Its files go under {WS}/lesson-<objective>/. A step's `show` commands are what the learner
types; the audit runs exactly them, as `candidate` on the host, so what a lesson shows always
works. Work on another node is shown as `ssh NODE sudo bash <<'EOF' ... EOF`: the same commands
you would type after `ssh NODE` and `sudo -i`, in a form that runs unattended. A single command
there is `ssh -n NODE 'COMMAND'`: without -n, ssh would read the rest of the commands as its input.
One file per domain, as with the task banks. Besides the curriculum's competencies, the lessons
cover what a full CKA course teaches under each of them (2026-10-19: the topics of KubeDojo's CKA
course, written here from scratch): projected volumes, eviction thresholds, debug containers,
kube-proxy's rules, encryption at rest, in-place resize, PodDisruptionBudgets, QoS classes,
ServiceAccount tokens, a real kubeadm upgrade, EndpointSlices.
"""
from cka import lessons_arch, lessons_net, lessons_storage, lessons_tshoot, lessons_work
LESSONS = (lessons_storage.LESSONS + lessons_tshoot.LESSONS + lessons_work.LESSONS
+ lessons_arch.LESSONS + lessons_net.LESSONS)

943
cka/lessons_arch.py Normal file
View File

@ -0,0 +1,943 @@
"""CKA lessons of the Cluster Architecture, Installation and Configuration domain (4.1-4.8). See
cka/lessons.py for what a lesson is."""
from cka.common import backup_operator
from sim.bank import CAN, NS, PY, step
_ETCDCTL = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
"--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key")
# ---- 4.1 RBAC and ServiceAccounts -------------------------------------------------------------
L41 = {
"id": 9401, "objective": "4.1", "workers": 0,
"title": "RBAC: Roles, bindings, ServiceAccounts and their tokens",
"intro": r"""Every request to the API server is made by someone - a **user** (a client certificate, or a token from outside), or a **ServiceAccount** (a token the cluster issues) - and RBAC decides what they may do:
```
Role / ClusterRole verbs on resources (in apiGroups) what may be done
RoleBinding subjects -> a Role or a ClusterRole in ONE Namespace
ClusterRoleBinding subjects -> a ClusterRole everywhere
kubectl auth can-i VERB RESOURCE --as=USER | --as=system:serviceaccount:NS:NAME [-n NS] [--list]
```
A RoleBinding may point at a ClusterRole: the ClusterRole's rules then apply only in the binding's Namespace - the usual way to reuse one set of rules in many Namespaces.
Pods authenticate as their ServiceAccount, with a short-lived token the kubelet mounts at `/var/run/secrets/kubernetes.io/serviceaccount/`. `kubectl create token` issues one for you.
Namespace `team` is empty.""",
"setup": NS + r"""
mkns team
mkdir -p "$WS/lesson-4.1"
""",
"steps": [
step("A ServiceAccount that deploys",
r"""In `team`, create ServiceAccount `deployer`, Role `deploy-edit` that allows `get`, `list`, `watch`, `create`, `update`, `patch` and `delete` on **deployments** (API group `apps`) and `get`, `list` on **pods**, and RoleBinding `deployer` that gives the Role to the ServiceAccount.
Prove it with `kubectl auth can-i ... --as=system:serviceaccount:team:deployer`.""",
check=CAN + r"""
S=system:serviceaccount:team:deployer
can --as=$S -- create deployments.apps -n team && can --as=$S -- list pods -n team && cannot --as=$S -- delete pods -n team \
&& cannot --as=$S -- create deployments.apps -n default""",
hint=r"""`kubectl create role deploy-edit --verb=... --resource=deployments.apps` - a Role with two resources and different verbs needs two rules, so write YAML or create it and `kubectl edit`. `kubectl create rolebinding NAME --role=ROLE --serviceaccount=team:deployer`.""",
show=r"""kubectl -n team create serviceaccount deployer
kubectl -n team apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deploy-edit
rules:
- apiGroups: [apps]
resources: [deployments]
verbs: [get, list, watch, create, update, patch, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get, list]
EOF
kubectl -n team create rolebinding deployer --role=deploy-edit --serviceaccount=team:deployer
kubectl auth can-i create deployments -n team --as=system:serviceaccount:team:deployer
kubectl auth can-i delete pods -n team --as=system:serviceaccount:team:deployer"""),
step("One ClusterRole, one Namespace",
r"""Create ClusterRole `pod-reader` (`get`, `list`, `watch` on pods). Give it to the user **`jane`** in Namespace `team` only - with a **RoleBinding** `jane-reads-pods`, not a ClusterRoleBinding.
`jane` can then list Pods in `team`, and nowhere else.""",
check=CAN + r"""
eq "$(j clusterrole pod-reader '{.rules[0].verbs}')" '["get","list","watch"]' || exit 1
eq "$(j -n team rolebinding jane-reads-pods '{.roleRef.kind}/{.roleRef.name}')" ClusterRole/pod-reader || exit 1
can --as=jane -- list pods -n team && cannot --as=jane -- list pods -n default && cannot --as=jane -- list pods -A""",
hint=r"""`kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods`, then `kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane`. Users exist only in certificates and tokens: nothing to create for `jane`.""",
show=r"""kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods
kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane
kubectl auth can-i list pods -n team --as=jane
kubectl auth can-i list pods -n default --as=jane"""),
step("A token for the ServiceAccount",
r"""Issue a token for `deployer` that is valid for **one hour**, and save it in `{WS}/lesson-4.1/deployer.token`. Then use it - as the **only** credential: your kubeconfig carries the admin's client certificate, and the API server checks a certificate before a token, so `kubectl --token=...` alone would still be you. With no kubeconfig, just the server, the CA and the token, `kubectl auth whoami` says `system:serviceaccount:team:deployer`, `get deployments` works in `team` and `get secrets` is forbidden.""",
check=r"""
t=$(tr -d ' \r\n' < "$WS/lesson-4.1/deployer.token" 2>/dev/null); [ -n "$t" ] || exit 1
srv=$($K config view --minify -o jsonpath='{.clusters[0].cluster.server}')
eq "$($K --kubeconfig=/dev/null --server="$srv" --insecure-skip-tls-verify --token="$t" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)" system:serviceaccount:team:deployer || exit 1
echo "$t" | cut -d. -f2 | tr '_-' '/+' | python3 -c '
import base64, json, sys
s = sys.stdin.read().strip(); c = json.loads(base64.b64decode(s + "=" * (-len(s) % 4)))
sys.exit(0 if c["exp"] - c["iat"] == 3600 else 1)'""",
hint=r"""`kubectl -n team create token deployer --duration=1h`. Then `kubectl --kubeconfig=/dev/null --server=URL --certificate-authority=/etc/kubernetes/pki/ca.crt --token=TOKEN ...`; the URL is in your kubeconfig. The token is a JWT: its middle part, base64-decoded, holds `sub`, `iat` and `exp`.""",
show=r"""kubectl -n team create token deployer --duration=1h > {WS}/lesson-4.1/deployer.token
server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
as_deployer() { kubectl --kubeconfig=/dev/null --server=$server --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$(cat {WS}/lesson-4.1/deployer.token) "$@"; }
as_deployer auth whoami
as_deployer -n team get deployments
as_deployer -n team get secrets || true"""),
step("A Pod without a token",
r"""Most Pods never talk to the API server, and do not need a token in their filesystem. Create ServiceAccount `quiet` in `team` with `automountServiceAccountToken: false`, and run Pod `notoken` (`busybox:1.36`, `sleep 86400`) as that ServiceAccount. Inside it, `/var/run/secrets/kubernetes.io/serviceaccount` does not exist.""",
check=r"""
eq "$(j -n team sa quiet '{.automountServiceAccountToken}')" false && eq "$(j -n team pod notoken '{.spec.serviceAccountName} {.status.phase}')" "quiet Running" || exit 1
! $K -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount >/dev/null 2>&1""",
hint=r"""The field sits at the top level of the ServiceAccount (and can be set on a Pod's `spec` too, which wins). The Pod names its ServiceAccount with `spec.serviceAccountName`.""",
show=r"""kubectl -n team apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: quiet
automountServiceAccountToken: false
---
apiVersion: v1
kind: Pod
metadata:
name: notoken
spec:
serviceAccountName: quiet
containers:
- name: app
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n team wait --for=condition=Ready pod/notoken --timeout=120s
kubectl -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount || echo 'no token mounted'"""),
step("Everything one subject may do",
r"""Write the complete list of what `deployer` may do in `team` to `{WS}/lesson-4.1/deployer-can.txt` - `kubectl auth can-i` prints it with `--list`. It includes what every authenticated subject gets from the built-in ClusterRoles, such as creating `selfsubjectreviews`.""",
check=r"""f="$WS/lesson-4.1/deployer-can.txt"; grep -q '^deployments.apps' "$f" 2>/dev/null && grep -q '^pods ' "$f" && grep -qi 'verbs' "$f" """,
hint=r"""`kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer`.""",
show=r"""kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer > {WS}/lesson-4.1/deployer-can.txt
cat {WS}/lesson-4.1/deployer-can.txt"""),
],
"outro": r"""You gave a ServiceAccount a Role, reused a ClusterRole in one Namespace, issued and used a token, ran a Pod without one, and listed a subject's permissions.
**For the exam**
- `kubectl create role|clusterrole NAME --verb=... --resource=...` and `kubectl create rolebinding|clusterrolebinding NAME --role|--clusterrole=... --user|--group|--serviceaccount=NS:NAME`.
- Resources in a group are written `RESOURCE.GROUP` on the command line (`deployments.apps`) and as `apiGroups: [apps]` in YAML; the core group is `""`. Subresources: `pods/log`, `pods/exec`, `deployments/scale`.
- A binding's `roleRef` cannot be changed: delete and re-create.
- `--as=system:serviceaccount:NS:NAME` to test a ServiceAccount; `--as=USER --as-group=GROUP` for users.
- Users are certificates: a CSR with `CN=user, O=group`, signed via a `CertificateSigningRequest` (`kubectl certificate approve`), then a kubeconfig with `kubectl config set-credentials --embed-certs`. Drill 4.1 has that end to end.
Next, prove it against the clock: **drill 4.1**.""",
}
# ---- 4.2 underlying infrastructure ------------------------------------------------------------
L42 = {
"id": 9402, "objective": "4.2", "workers": 1,
"title": "Preparing a machine for kubeadm: modules, sysctls, swap, runtime",
"intro": r"""Before `kubeadm init` or `join`, each machine needs:
```
kernel modules overlay, br_netfilter /etc/modules-load.d/*.conf (modprobe now)
sysctls net.ipv4.ip_forward = 1, net.bridge.bridge-nf-call-iptables = 1 /etc/sysctl.d/*.conf (sysctl --system)
swap off, or the kubelet told to tolerate it (failSwapOn: false)
container runtime containerd, CRI enabled, its cgroup driver matching the kubelet's (systemd)
packages kubeadm, kubelet, kubectl from pkgs.k8s.io for ONE minor, held
ports 6443 (API server), 2379-2380 (etcd), 10250 (kubelet), 30000-32767 (NodePorts)
```
A setting made with `sysctl -w` or `modprobe` alone is gone after the next reboot; the files are what make it stick.
Worker `{W1}` stands in for the new machine. (The nodes of this simulator are containers: a module cannot really be loaded there, but the files and what you read from the node are real.)""",
"setup": r"""
mkdir -p "$WS/lesson-4.2"
nx "$W1" 'rm -f /etc/modules-load.d/k8s.conf /etc/sysctl.d/k8s.conf' </dev/null
""",
"steps": [
step("Modules at every boot",
r"""On `{W1}`, make `overlay` and `br_netfilter` load at every boot: list them, one per line, in `/etc/modules-load.d/k8s.conf`.""",
check=r"""eq "$(nx "$W1" 'grep -v "^#" /etc/modules-load.d/k8s.conf 2>/dev/null | tr -d " " | grep . | sort | tr "\n" " "' </dev/null)" "br_netfilter overlay " """,
hint=r"""`printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/k8s.conf`. On a real machine you would also `modprobe` both now, and `lsmod | grep br_netfilter` would show it.""",
show=r"""ssh {W1} sudo bash <<'EOF'
printf 'overlay\nbr_netfilter\n' > /etc/modules-load.d/k8s.conf
cat /etc/modules-load.d/k8s.conf
EOF"""),
step("Kernel parameters, for good",
r"""On `{W1}`, put the three parameters Kubernetes needs in `/etc/sysctl.d/k8s.conf` - `net.bridge.bridge-nf-call-iptables = 1`, `net.bridge.bridge-nf-call-ip6tables = 1`, `net.ipv4.ip_forward = 1` - and apply every sysctl file now with `sysctl --system`.""",
check=r"""
f=$(nx "$W1" 'cat /etc/sysctl.d/k8s.conf 2>/dev/null' </dev/null | tr -d ' ')
for p in net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1; do echo "$f" | grep -qx "$p" || exit 1; done
eq "$(nx "$W1" 'sysctl -n net.ipv4.ip_forward' </dev/null)" 1""",
hint=r"""A here-document into `sudo tee`, then `sudo sysctl --system` - it prints every file it reads, yours among them. (The bridge parameters only exist once `br_netfilter` is loaded.)""",
show=r"""ssh {W1} sudo bash <<'EOF'
cat > /etc/sysctl.d/k8s.conf <<'CONF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
CONF
sysctl --system 2>/dev/null | grep -A3 k8s.conf
sysctl net.ipv4.ip_forward
EOF"""),
step("Swap and the cgroup driver",
r"""Two more things kubeadm's preflight checks. Write to `{WS}/lesson-4.2/node.txt`, in this format:
```
swap=<on or off: is any swap device active on {W1}>
failSwapOn=<the kubelet's failSwapOn setting>
kubelet-cgroup=<the kubelet's cgroupDriver>
containerd-systemd-cgroup=<SystemdCgroup in containerd's config: true or false>
```
The last two must agree, or Pods fail to start.""",
check=r"""
swap=$(nx "$W1" '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off' </dev/null)
fso=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" | python3 -c 'import json,sys; c=json.load(sys.stdin)["kubeletconfig"]; print(str(c.get("failSwapOn", True)).lower(), c["cgroupDriver"])')
sc=$(nx "$W1" 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml | head -1 | tr -d " " | cut -d= -f2' </dev/null); sc=${sc:-false}
want=$(printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s' "$swap" "${fso% *}" "${fso#* }" "$sc")
eq "$(tr -d ' \r' < "$WS/lesson-4.2/node.txt" 2>/dev/null | grep .)" "$want" """,
hint=r"""`cat /proc/swaps` (or `swapon --show`): any line below the header is swap. `grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml` - a field that is not there has its default (`failSwapOn` is `true`). `grep SystemdCgroup /etc/containerd/config.toml`; no such line means `false`.""",
show=r"""ssh {W1} sudo bash <<'EOF'
cat /proc/swaps
grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml
grep -n SystemdCgroup /etc/containerd/config.toml || echo 'SystemdCgroup not set'
EOF
swap=$(ssh -n {W1} '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off')
fso=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.failSwapOn)
cg=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver)
sc=$(ssh -n {W1} 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml | head -1 | tr -d " " | cut -d= -f2')
printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s\n' "$swap" "$fso" "$cg" "${sc:-false}" > {WS}/lesson-4.2/node.txt
cat {WS}/lesson-4.2/node.txt"""),
step("The packages, as kubeadm wants them",
r"""On an exam node, kubeadm, kubelet and kubectl come from the repository of **one** minor at `pkgs.k8s.io`, and are **held** so that an `apt upgrade` never moves them by accident.
Write to `{WS}/lesson-4.2/packages.txt`: the line of `{W1}`'s Kubernetes repository (from `/etc/apt/sources.list.d/`), then the held packages, one per line, as `apt-mark` prints them.""",
check=r"""
want=$(nx "$W1" 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' </dev/null | grep .)
[ -n "$want" ] && eq "$(grep . "$WS/lesson-4.2/packages.txt" 2>/dev/null | tr -d '\r')" "$want" """,
hint=r"""`cat /etc/apt/sources.list.d/kubernetes.list` and `apt-mark showhold`. `apt-cache madison kubeadm` lists the versions that repository offers; `dpkg -l | grep kube` what is installed.""",
show=r"""ssh -n {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt
cat {WS}/lesson-4.2/packages.txt
ssh -n {W1} 'dpkg -l | grep -E "kube(adm|let|ctl)"'"""),
],
"outro": r"""You set up modules and sysctls that survive a reboot, checked swap and the cgroup drivers, and read how Kubernetes is installed on a node.
**For the exam**
- The docs page "Installing kubeadm" and "Container Runtimes" have every file to copy - module list, sysctl file, the containerd `SystemdCgroup = true` line, the apt repository and `apt-mark hold`.
- `sysctl --system` applies every file under `/etc/sysctl.d`; `sysctl -p FILE` just one.
- containerd: `containerd config default > /etc/containerd/config.toml`, set `SystemdCgroup = true`, `systemctl restart containerd`. `crictl info` shows what it runs with.
- A machine with a CNI already configured has a file in `/etc/cni/net.d/` - the CNI itself is installed after `kubeadm init`, as a manifest.
Next, prove it against the clock: **drill 4.2**.""",
}
# ---- 4.3 kubeadm clusters ---------------------------------------------------------------------
L43 = {
"id": 9403, "objective": "4.3", "workers": 1,
"title": "kubeadm: the cluster's configuration, joining nodes, certificates, static Pods",
"intro": r"""`kubeadm init` builds a control plane from a `ClusterConfiguration` and leaves you everything needed to manage it:
```
/etc/kubernetes/manifests/ static Pods: etcd, kube-apiserver, kube-controller-manager, kube-scheduler
/etc/kubernetes/pki/ the cluster CA, etcd CA, front-proxy CA, and the certificates they signed
/etc/kubernetes/*.conf kubeconfigs: admin.conf, super-admin.conf, controller-manager.conf, scheduler.conf, kubelet.conf
ConfigMap kubeadm-config the ClusterConfiguration, in kube-system
```
Nodes join with a **bootstrap token** and the hash of the CA's public key: `kubeadm token create --print-join-command` prints the whole command. The certificates kubeadm makes last a year - the CA ten years - and `kubeadm certs check-expiration` / `renew` look after them.
This cluster was made by kubeadm; its worker is `{W1}`.""",
"setup": r"""
mkdir -p "$WS/lesson-4.3"
nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' </dev/null | cut -d= -f2 > "$PRIV/lesson-4.3.before"
nx "$W1" 'rm -f /etc/kubernetes/manifests/node-web.yaml' </dev/null
""",
"steps": [
step("Read the cluster's configuration",
r"""Write the cluster's Pod subnet and Service subnet, as kubeadm has them in its `ClusterConfiguration`, to `{WS}/lesson-4.3/networks.txt`, separated by a space.""",
check=r"""
want=$($K -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}')
eq "$(tr -s ' \t' ' ' < "$WS/lesson-4.3/networks.txt" 2>/dev/null | tr -d '\r\n')" "$want" """,
hint=r"""`kubectl -n kube-system get cm kubeadm-config -o yaml` - look under `networking:`. `kubeadm config print init-defaults` shows every field such a file can have.""",
show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep -A4 networking
kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}' > {WS}/lesson-4.3/networks.txt
cat {WS}/lesson-4.3/networks.txt"""),
step("A join command for a new worker",
r"""Create a new bootstrap token that expires in **2 hours**, and save the complete `kubeadm join` command for a worker in `{WS}/lesson-4.3/join.sh`, on one line. Run it on the control-plane node.""",
check=r"""
f="$WS/lesson-4.3/join.sh"; grep -q '^kubeadm join .*--token .*--discovery-token-ca-cert-hash sha256:' "$f" 2>/dev/null || exit 1
id=$(grep -o -- '--token [a-z0-9]*\.[a-z0-9]*' "$f" | awk '{print $2}' | cut -d. -f1); [ -n "$id" ] || exit 1
exp=$(j -n kube-system secret "bootstrap-token-$id" '{.data.expiration}' | base64 -d); [ -n "$exp" ] || exit 1
left=$(( $(date -d "$exp" +%s) - $(date +%s) )); [ "$left" -gt 6600 ] && [ "$left" -le 7300 ]""",
hint=r"""`sudo kubeadm token create --ttl 2h --print-join-command`. `kubeadm token list` shows the tokens and when they expire; each is a Secret `bootstrap-token-ID` in `kube-system`.""",
show=r"""sudo kubeadm token create --ttl 2h --print-join-command > {WS}/lesson-4.3/join.sh
cat {WS}/lesson-4.3/join.sh
sudo kubeadm token list"""),
step("Renew the API server's certificate",
r"""Check the certificates' expiry dates, then **renew** the API server's serving certificate - only that one - and make the API server use it. A running process keeps the certificate it read at start, so the static Pod has to be restarted: move its manifest out of `/etc/kubernetes/manifests`, wait until the container is gone, and move it back.
Afterwards `kubeadm certs check-expiration` shows the new date, and the API server presents the new certificate.""",
check=r"""
b=$(cat "$PRIV/lesson-4.3.before")
nb=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' </dev/null | cut -d= -f2); [ -n "$nb" ] && [ -n "$b" ] || exit 1
[ "$(date -d "$nb" +%s)" -gt "$(date -d "$b" +%s)" ] || exit 1
file=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial' </dev/null)
served=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial' </dev/null)
[ -n "$file" ] && eq "$file" "$served" """,
hint=r"""`sudo kubeadm certs check-expiration`, `sudo kubeadm certs renew apiserver`. Restart: `sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/`, wait for `sudo crictl ps --name kube-apiserver` to be empty, move it back. Kubernetes renews the certificates itself on every `kubeadm upgrade`.""",
show=r"""sudo kubeadm certs check-expiration | head -8
sudo kubeadm certs renew apiserver
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name kube-apiserver -q)" ] && break; sleep 2; done
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
sudo kubeadm certs check-expiration | grep '^apiserver '"""),
step("A static Pod on the worker",
r"""Every kubelet runs static Pods, not only the control plane's. On `{W1}`, find the kubelet's `staticPodPath` and put a manifest there for Pod `node-web` (`nginx:1.27`).
The API server shows it as a **mirror Pod** named after the node: `node-web-{W1}`. Deleting the mirror Pod with kubectl does nothing lasting - the kubelet puts it back; only removing the file does.""",
check=r"""eq "$(j -n default pod "node-web-$W1" '{.status.phase} {.metadata.annotations.kubernetes\.io/config\.source}')" "Running file" """,
hint=r"""`grep staticPodPath /var/lib/kubelet/config.yaml` on the node. `kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml` gives you the manifest; write it with `sudo tee`.""",
show=r"""kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml > {WS}/lesson-4.3/node-web.yaml
ssh -n {W1} 'grep staticPodPath /var/lib/kubelet/config.yaml'
ssh {W1} 'sudo tee /etc/kubernetes/manifests/node-web.yaml >/dev/null' < {WS}/lesson-4.3/node-web.yaml
sleep 15
kubectl get pods -o wide"""),
],
"outro": r"""You read the cluster's kubeadm configuration, made a join command, renewed a certificate and made the API server load it, and ran a static Pod on a worker.
**For the exam**
- `kubeadm init --pod-network-cidr=... [--control-plane-endpoint=...] [--upload-certs]`, then copy `admin.conf` to `~/.kube/config`, then install a CNI. `kubeadm join` on the others. `kubeadm reset` undoes either.
- `kubeadm token create --print-join-command`; for a control-plane node add `--certificate-key` from `kubeadm init phase upload-certs --upload-certs`.
- `kubeadm certs check-expiration | renew all|apiserver|...`; restart the static Pods afterwards. `kubeadm kubeconfig user --client-name=NAME` writes a kubeconfig for a user.
- Static Pods: the kubelet's `staticPodPath` (`/etc/kubernetes/manifests`); mirror Pods are named `POD-NODE`.
Next, prove it against the clock: **drill 4.3**.""",
}
# ---- 4.4 cluster lifecycle: etcd and a real upgrade ----------------------------------------------
_APT = r"""
pkgs() { nx "$1" "dpkg-query -W -f='\${Version} ' kubelet kubeadm kubectl" </dev/null; }
ver() { nx "$1" "dpkg-query -W -f='\${Version}' $2" </dev/null; }
held() { [ "$(nx "$1" 'apt-mark showhold | grep -c -x -e kubelet -e kubeadm -e kubectl' </dev/null)" = 3 ]; }
newest() { nx "$CP" "apt-cache madison kubeadm | awk '\$3 ~ /^1\.37\./ {print \$3; exit}'" </dev/null; }
"""
L44 = {
"id": 9404, "objective": "4.4", "workers": 1, "k8s": "v1.36",
"title": "Lifecycle: back up and restore etcd, then upgrade the cluster to v1.37",
"intro": r"""This cluster runs **Kubernetes v1.36**, and you will take it to **v1.37** the documented way - after a backup.
**etcd** holds the cluster's whole state. A snapshot is taken with `etcdctl snapshot save` against the running server, checked with `etcdutl snapshot status`, and restored with `etcdutl snapshot restore` into a data directory that etcd then starts from.
An **upgrade** goes one minor at a time, control plane first:
```
control plane repo line -> v1.37, apt install kubeadm kubeadm upgrade plan / apply v1.37.x
drain, apt install kubelet kubectl, systemctl daemon-reload, restart kubelet, uncordon
each worker drain (from the control plane); on the node: repo line, kubeadm, kubeadm upgrade node,
kubelet kubectl, restart kubelet; uncordon
```
pkgs.k8s.io has one repository per minor, so the first step on every node is the line in `/etc/apt/sources.list.d/kubernetes.list`. The packages are held: `apt-mark unhold` before, `hold` after.
Namespace `life` has ConfigMap `marker`; etcd's certificates are under `/etc/kubernetes/pki/etcd/`.""",
"setup": NS + r"""
mkns life
$K -n life create configmap marker --from-literal=state=before-backup >/dev/null
nx "$CP" 'mkdir -p /opt/backup; rm -f /opt/backup/*.db' </dev/null
mkdir -p "$WS/lesson-4.4"
""",
"steps": [
step("Back up etcd",
r"""On the control-plane node, save a snapshot of etcd to `/opt/backup/etcd.db`, and check it with `etcdutl snapshot status`. Write that status, as a table, to `{WS}/lesson-4.4/snapshot.txt`.
etcd only accepts clients with a certificate: give `etcdctl` the endpoint and the three files - look them up in `/etc/kubernetes/manifests/etcd.yaml` rather than guessing.""",
check=r"""
nx "$CP" 'etcdutl snapshot status /opt/backup/etcd.db >/dev/null 2>&1' </dev/null && grep -qi 'hash' "$WS/lesson-4.4/snapshot.txt" 2>/dev/null \
&& grep -qi 'total key' "$WS/lesson-4.4/snapshot.txt" """,
hint=r"""`etcdctl --endpoints=https://127.0.0.1:2379 --cacert=.../ca.crt --cert=.../server.crt --key=.../server.key snapshot save FILE`, as root. Since etcd 3.6, `status` and `restore` are `etcdutl` only.""",
show=r"""sudo grep -E -- '--(listen-client-urls|cert-file|key-file|trusted-ca-file)' /etc/kubernetes/manifests/etcd.yaml
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup/etcd.db
sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/snapshot.txt"""),
step("Lose something",
r"""Now the mistake a backup is for: delete ConfigMap `marker` in `life`.""",
check=r"""[ -z "$(j -n life cm marker '{.metadata.name}')" ] && nx "$CP" 'test -s /opt/backup/etcd.db' </dev/null""",
hint=r"""`kubectl -n life delete configmap marker`.""",
show=r"""kubectl -n life delete configmap marker"""),
step("Restore the snapshot",
r"""Restore `/opt/backup/etcd.db` so that `marker` is back. Do it in place, so that etcd keeps its data directory `/var/lib/etcd`:
1. Stop etcd: move `etcd.yaml` out of `/etc/kubernetes/manifests`, and wait until its container is gone. (The API server loses its storage meanwhile: kubectl fails until etcd is back.)
2. Move `/var/lib/etcd` aside, and restore the snapshot into `/var/lib/etcd`.
3. Move the manifest back, and wait for the API server.
Restoring into a **new** directory and pointing the manifest's `hostPath` at it works too, and is what many tasks ask for - but then the `dataDir` in kubeadm's configuration still names the old one, and the next `kubeadm upgrade` would write it back into the manifest.""",
check=r"""eq "$(j -n life cm marker '{.data.state}')" before-backup && nx "$CP" 'test -d /var/lib/etcd/member' </dev/null""",
hint=r"""`sudo etcdutl snapshot restore /opt/backup/etcd.db --data-dir /var/lib/etcd` - the target directory must not exist yet. `sudo crictl ps --name etcd` shows whether etcd is running.""",
show=r"""sudo mv /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml
for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name '^etcd$' -q)" ] && break; sleep 2; done
sudo mv /var/lib/etcd /var/lib/etcd.before-restore
sudo etcdutl snapshot restore /opt/backup/etcd.db --data-dir /var/lib/etcd
sudo mv /root/etcd.yaml /etc/kubernetes/manifests/etcd.yaml
for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
kubectl -n life get configmap marker -o yaml | grep state"""),
step("Plan the upgrade",
r"""On the control-plane node, point the Kubernetes repository at **v1.37**, install the **newest 1.37 kubeadm** the repository offers (unhold it first, hold it again after), and save the output of `kubeadm upgrade plan` in `{WS}/lesson-4.4/plan.txt`.
The plan shows two choices: the newest 1.36 patch, and the 1.37 release your kubeadm can go to.""",
check=_APT + r"""
t=$(newest); [ -n "$t" ] || exit 1
eq "$(ver "$CP" kubeadm)" "$t" && held "$CP" && grep -q "kubeadm upgrade apply v${t%%-*}" "$WS/lesson-4.4/plan.txt" 2>/dev/null""",
hint=r"""`sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list`, `sudo apt-get update`, `apt-cache madison kubeadm` (newest first), `sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.X-1.1 && sudo apt-mark hold kubeadm`.""",
show=r"""sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -qq
apt-cache madison kubeadm | head -3
v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo apt-mark unhold kubeadm && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$v && sudo apt-mark hold kubeadm
kubeadm version -o short
sudo kubeadm upgrade plan | tee {WS}/lesson-4.4/plan.txt | tail -12"""),
step("Upgrade the control plane",
r"""Apply the upgrade to that 1.37 release (`kubeadm upgrade apply`, a few minutes - it upgrades etcd, the static Pods, CoreDNS and kube-proxy). Then drain the node, upgrade `kubelet` and `kubectl` to the same version, reload systemd, restart the kubelet, and uncordon.
`kubectl get nodes` shows the kubelet's version: the control-plane node says v1.37 only after its kubelet has been upgraded and restarted.""",
check=_APT + r"""
t=$(newest); v="v${t%%-*}"
eq "$($K version -o json 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["serverVersion"]["gitVersion"])')" "$v" || exit 1
eq "$(j node "$CP" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$CP")" "$t $t $t " && held "$CP" \
&& [ "$(j node "$CP" '{.spec.unschedulable}')" != true ]""",
hint=r"""`sudo kubeadm upgrade apply v1.37.X -y`; `kubectl drain {CP} --ignore-daemonsets`; `sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=... kubectl=... && sudo apt-mark hold kubelet kubectl`; `sudo systemctl daemon-reload && sudo systemctl restart kubelet`; `kubectl uncordon {CP}`.""",
show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo kubeadm upgrade apply v${v%%-*} -y | tail -3
kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$v kubectl=$v && sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done
kubectl get nodes"""),
step("Upgrade the worker",
r"""Now `{W1}`: drain it from the control-plane node; on `{W1}`, the repository line, kubeadm, **`kubeadm upgrade node`** (a worker has no control-plane Pods to replace - it only refreshes the kubelet's configuration), kubelet and kubectl, daemon-reload, restart; then uncordon it.
Both nodes then report the new version, and every package is held again.""",
check=_APT + r"""
t=$(newest); v="v${t%%-*}"
eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$W1")" "$t $t $t " && held "$W1" \
&& [ "$(j node "$W1" '{.spec.unschedulable}')" != true ] && eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""The same packages, from the same repository line, as on the control plane - only `kubeadm upgrade node` instead of `apply`. Drain and uncordon are kubectl commands: run them on the control-plane node.""",
show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data
ssh {W1} sudo bash -s $v <<'EOF'
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq
apt-mark unhold kubeadm kubelet kubectl >/dev/null
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$1
kubeadm upgrade node | tail -2
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$1 kubectl=$1
apt-mark hold kubeadm kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet
EOF
kubectl uncordon {W1}
kubectl get nodes"""),
],
"outro": r"""You backed etcd up, restored it after a loss, and took a whole cluster from v1.36 to v1.37: control plane first, then the worker, with the packages held at every step.
**For the exam**
- etcd: `etcdctl snapshot save` (with `--endpoints --cacert --cert --key`), `etcdutl snapshot status`, `etcdutl snapshot restore --data-dir NEWDIR`, then the `etcd-data` `hostPath` in `etcd.yaml` - or restore in place, as here.
- Upgrade one minor at a time; kubeadm before `upgrade apply`, kubelet after it. `kubeadm upgrade plan` names the versions; take the exact patch the task asks for, or the newest.
- Workers: `kubeadm upgrade node`, never `apply`. Drain and uncordon from where kubectl works.
- Version skew: kubelets may be up to three minors behind the API server, never ahead.
- `apt-mark unhold` → `apt-get install PKG=VERSION` → `apt-mark hold`; `systemctl daemon-reload` before restarting the kubelet.
Next, prove it against the clock: **drill 4.4**.""",
}
# ---- 4.5 highly available control plane --------------------------------------------------------
L45 = {
"id": 9405, "objective": "4.5", "workers": 0,
"title": "A highly available control plane: endpoint, etcd members, leaders, joining",
"intro": r"""An HA control plane is three (or five) control-plane nodes behind one address:
```
controlPlaneEndpoint one stable address - a load balancer or a VIP - for every kubelet and kubectl
etcd stacked: a member on each control-plane node external: hosts of its own
kube-apiserver active on every node at once, behind the endpoint
kube-scheduler, kube-controller-manager one leader at a time, chosen through a Lease in kube-system
```
etcd needs a majority (quorum): three members survive one failure, five survive two. A second control-plane node joins with the certificates the first uploads (`kubeadm init phase upload-certs --upload-certs`, valid two hours) and `kubeadm join ... --control-plane --certificate-key KEY`.
This cluster has one control-plane node, `{CP}`, made with a `controlPlaneEndpoint` so that more can join.""",
"setup": r"""
mkdir -p "$WS/lesson-4.5"
$K -n kube-system delete secret kubeadm-certs --ignore-not-found >/dev/null 2>&1
""",
"steps": [
step("The one stable address",
r"""Write the cluster's control-plane endpoint, as kubeadm has it, to `{WS}/lesson-4.5/endpoint.txt`. It is also the `server:` of every kubeconfig on the nodes.""",
check=r"""
want=$($K -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}')
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-4.5/endpoint.txt" 2>/dev/null)" "$want" """,
hint=r"""`controlPlaneEndpoint:` in the `ClusterConfiguration` of ConfigMap `kubeadm-config`.""",
show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}' > {WS}/lesson-4.5/endpoint.txt
cat {WS}/lesson-4.5/endpoint.txt
sudo grep server: /etc/kubernetes/kubelet.conf"""),
step("etcd's members and its leader",
r"""Ask etcd itself. Write to `{WS}/lesson-4.5/etcd.txt`:
```
members=<how many members the etcd cluster has>
leader=<the name of the member that leads>
```""",
check=r"""
out=$(nx "$CP" '""" + _ETCDCTL + r""" endpoint status -w json' </dev/null)
n=$(nx "$CP" '""" + _ETCDCTL + r""" member list' </dev/null | grep -c .)
lead=$(nx "$CP" '""" + _ETCDCTL + r""" member list -w json' </dev/null | python3 -c '
import json, sys; m = json.load(sys.stdin)
st = json.loads(sys.argv[1])[0]["Status"]; print(next(x["name"] for x in m["members"] if x["ID"] == st["leader"]))' "$out")
eq "$(tr -d ' \r' < "$WS/lesson-4.5/etcd.txt" 2>/dev/null | grep .)" "$(printf 'members=%s\nleader=%s' "$n" "$lead")" """,
hint=r"""`etcdctl member list -w table` and `etcdctl endpoint status -w table` (the IS LEADER column), with the endpoint and certificates as for a snapshot. `export ETCDCTL_ENDPOINTS=... ETCDCTL_CACERT=...` saves typing.""",
show=r"""sudo bash <<'EOF'
export ETCDCTL_API=3 ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl member list -w table
etcdctl endpoint status -w table
EOF
printf 'members=1\nleader={CP}\n' > {WS}/lesson-4.5/etcd.txt"""),
step("Who leads the scheduler and the controller manager",
r"""Only one kube-scheduler and one kube-controller-manager act at a time; the others wait for the **Lease** in `kube-system` to expire. Write the node of each holder to `{WS}/lesson-4.5/leaders.txt`:
```
scheduler=<node>
controller-manager=<node>
```""",
check=r"""
s=$(j -n kube-system lease kube-scheduler '{.spec.holderIdentity}'); c=$(j -n kube-system lease kube-controller-manager '{.spec.holderIdentity}')
eq "$(tr -d ' \r' < "$WS/lesson-4.5/leaders.txt" 2>/dev/null | grep .)" "$(printf 'scheduler=%s\ncontroller-manager=%s' "${s%%_*}" "${c%%_*}")" """,
hint=r"""`kubectl -n kube-system get lease kube-scheduler kube-controller-manager` - HOLDER is `NODE_UUID`.""",
show=r"""kubectl -n kube-system get lease kube-scheduler kube-controller-manager
s=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}')
c=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}')
printf 'scheduler=%s\ncontroller-manager=%s\n' "${s%%_*}" "${c%%_*}" > {WS}/lesson-4.5/leaders.txt"""),
step("What a second control-plane node needs",
r"""Upload the control-plane certificates for a joining node, and save the complete `kubeadm join` command a new **control-plane** node would run - new token, CA hash, `--control-plane` and the certificate key - to `{WS}/lesson-4.5/join-cp.sh`, on one line. Do not run it anywhere.""",
check=r"""
f="$WS/lesson-4.5/join-cp.sh"; grep -q '^kubeadm join .*--token .*--discovery-token-ca-cert-hash sha256:.*--control-plane' "$f" 2>/dev/null || exit 1
key=$(grep -o -- '--certificate-key [0-9a-f]\{64\}' "$f" | awk '{print $2}'); [ -n "$key" ] && [ -n "$(j -n kube-system secret kubeadm-certs '{.metadata.name}')" ]""",
hint=r"""`sudo kubeadm init phase upload-certs --upload-certs` prints the key as its last line; `sudo kubeadm token create --print-join-command --certificate-key KEY` adds `--control-plane --certificate-key` itself.""",
show=r"""key=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
sudo kubeadm token create --print-join-command --certificate-key $key > {WS}/lesson-4.5/join-cp.sh
cat {WS}/lesson-4.5/join-cp.sh
kubectl -n kube-system get secret kubeadm-certs"""),
],
"outro": r"""You read the endpoint that makes HA possible, asked etcd for its members and leader, found the leaders of the scheduler and controller manager, and prepared a control-plane join.
**For the exam**
- HA starts at `kubeadm init --control-plane-endpoint LB:6443 --upload-certs`; without an endpoint, adding control-plane nodes later means re-issuing certificates.
- Stacked etcd: an etcd static Pod on each control-plane node. External: `etcd.external.endpoints` in the ClusterConfiguration, and the API server's `--etcd-servers` names other hosts.
- Quorum is a majority of members: an even number adds no tolerance.
- The uploaded certificates expire after two hours; re-run `upload-certs` for a fresh key.
Next, prove it against the clock: **drill 4.5**.""",
}
# ---- 4.6 Helm and Kustomize -------------------------------------------------------------------
_CHART = r"""
c="$WS/lesson-4.6/chart"; rm -rf "$WS/lesson-4.6"; mkdir -p "$c/templates" "$WS/lesson-4.6/kustomize/base"
cat > "$c/Chart.yaml" <<'EOF'
apiVersion: v2
name: shop
description: A small web shop
type: application
version: 0.1.0
appVersion: "1.27"
EOF
cat > "$c/values.yaml" <<'EOF'
replicaCount: 1
image:
repository: nginx
tag: "1.27"
service:
type: ClusterIP
port: 80
EOF
cat > "$c/templates/deployment.yaml" <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-shop
labels: {app: {{ .Release.Name }}-shop}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels: {app: {{ .Release.Name }}-shop}
template:
metadata:
labels: {app: {{ .Release.Name }}-shop}
spec:
containers:
- name: shop
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
ports: [{containerPort: 80}]
EOF
cat > "$c/templates/service.yaml" <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: {{ .Release.Name }}-shop
spec:
type: {{ .Values.service.type }}
selector: {app: {{ .Release.Name }}-shop}
ports: [{port: {{ .Values.service.port }}, targetPort: 80}]
EOF
b="$WS/lesson-4.6/kustomize/base"
cat > "$b/deployment.yaml" <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 1
selector:
matchLabels: {app: api}
template:
metadata:
labels: {app: api}
spec:
containers:
- name: api
image: nginx:1.26
EOF
cat > "$b/kustomization.yaml" <<'EOF'
resources:
- deployment.yaml
EOF
helm -n apps uninstall web >/dev/null 2>&1 || true
"""
L46 = {
"id": 9406, "objective": "4.6", "workers": 1,
"title": "Helm releases and Kustomize overlays",
"intro": r"""Two ways to install cluster components from more than a single manifest:
```
Helm a chart = templates + values.yaml helm install | upgrade | rollback | uninstall
a release = one installed copy, with numbered revisions (stored as Secrets in its Namespace)
Kustomize a base + overlays that change it, no templates kubectl apply -k DIR / kubectl kustomize DIR
```
With Helm you change **values** (`--set`, `-f values.yaml`), never the templates; `helm template` renders without installing. With Kustomize you never edit the base; an overlay sets the Namespace, prefixes, images, replicas, labels, and patches the rest.
A chart is at `{WS}/lesson-4.6/chart` (values `replicaCount`, `image.tag`, `service.type`, `service.port`); a Kustomize base is at `{WS}/lesson-4.6/kustomize/base`.""",
"setup": _CHART,
"steps": [
step("Render before you install",
r"""Render the chart as release `web` with `replicaCount=2`, without installing anything, and save the manifests in `{WS}/lesson-4.6/rendered.yaml`. That is how you check what a set of values does.""",
check=r"""f="$WS/lesson-4.6/rendered.yaml"; grep -q 'name: web-shop' "$f" 2>/dev/null && grep -q 'replicas: 2' "$f" && [ -z "$(j -n apps deploy web-shop '{.metadata.name}')" ]""",
hint=r"""`helm template RELEASE CHART --set KEY=VALUE`.""",
show=r"""helm template web {WS}/lesson-4.6/chart --set replicaCount=2 > {WS}/lesson-4.6/rendered.yaml
grep -E 'kind:|replicas:|image:' {WS}/lesson-4.6/rendered.yaml"""),
step("Install a release",
r"""Install the chart as release `web` in Namespace `apps` (create it), with `replicaCount=2`.""",
check=r"""eq "$(helm -n apps status web -o json 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["info"]["status"], d["version"])')" "deployed 1" && eq "$(j -n apps deploy web-shop '{.status.readyReplicas}')" 2""",
hint=r"""`helm install web CHART -n apps --create-namespace --set replicaCount=2`; `helm list -n apps`.""",
show=r"""helm install web {WS}/lesson-4.6/chart -n apps --create-namespace --set replicaCount=2
helm list -n apps
kubectl -n apps rollout status deploy web-shop --timeout=120s"""),
step("Upgrade it, keeping what you set",
r"""Upgrade release `web` so that its Service becomes a `NodePort` - and keep `replicaCount=2`. A plain `helm upgrade --set service.type=NodePort` would quietly put the replica count back to the chart's default.""",
check=r"""eq "$(j -n apps svc web-shop '{.spec.type}')" NodePort && eq "$(j -n apps deploy web-shop '{.spec.replicas}')" 2 \
&& [ "$(helm -n apps history web -o json 2>/dev/null | python3 -c 'import json,sys; print(len(json.load(sys.stdin)))')" -ge 2 ]""",
hint=r"""`--reuse-values` keeps the values of the last release and adds yours; `helm get values web -n apps` shows what is set.""",
show=r"""helm upgrade web {WS}/lesson-4.6/chart -n apps --reuse-values --set service.type=NodePort
helm get values web -n apps
kubectl -n apps get svc web-shop"""),
step("Roll the release back",
r"""The NodePort was a mistake: roll release `web` back to **revision 1**. A rollback is itself a new revision.""",
check=r"""
eq "$(j -n apps svc web-shop '{.spec.type}')" ClusterIP || exit 1
helm -n apps history web -o json 2>/dev/null | python3 -c '
import json, sys; h = json.load(sys.stdin)
sys.exit(0 if len(h) >= 3 and h[-1]["status"] == "deployed" and "Rollback to 1" in h[-1]["description"] else 1)'""",
hint=r"""`helm rollback web 1 -n apps`, then `helm history web -n apps`.""",
show=r"""helm rollback web 1 -n apps
helm history web -n apps
kubectl -n apps get svc web-shop"""),
step("An overlay for production",
r"""Create a Kustomize overlay at `{WS}/lesson-4.6/kustomize/prod` on top of the base, which: puts everything in Namespace `prod`, prefixes names with `prod-`, runs **3** replicas, and uses image `nginx:1.27`. Apply it. Leave the base as it is.""",
check=r"""eq "$(j -n prod deploy prod-api '{.spec.replicas} {.spec.template.spec.containers[0].image} {.status.readyReplicas}')" "3 nginx:1.27 3" \
&& grep -q 'image: nginx:1.26' "$WS/lesson-4.6/kustomize/base/deployment.yaml" """,
hint=r"""A `kustomization.yaml` with `resources: [../base]`, `namespace: prod`, `namePrefix: prod-`, `replicas: [{name: api, count: 3}]`, `images: [{name: nginx, newTag: "1.27"}]`. The Namespace itself must exist (or be one of the resources). `kubectl kustomize DIR` shows the result first.""",
show=r"""mkdir -p {WS}/lesson-4.6/kustomize/prod
cat > {WS}/lesson-4.6/kustomize/prod/kustomization.yaml <<'EOF'
resources:
- ../base
namespace: prod
namePrefix: prod-
replicas:
- name: api
count: 3
images:
- name: nginx
newTag: "1.27"
EOF
kubectl kustomize {WS}/lesson-4.6/kustomize/prod
kubectl create namespace prod
kubectl apply -k {WS}/lesson-4.6/kustomize/prod
kubectl -n prod rollout status deploy prod-api --timeout=120s"""),
],
"outro": r"""You rendered a chart, installed, upgraded and rolled back a release, and built a production overlay with Kustomize without touching its base.
**For the exam**
- `helm repo add NAME URL && helm repo update`, `helm search repo KEYWORD`, `helm show values CHART`, `helm pull CHART --untar`.
- `helm install|upgrade RELEASE CHART -n NS [--create-namespace] [--set k=v] [-f values.yaml] [--version X]`; `helm upgrade --install` for either. `--reuse-values` on an upgrade, or you lose earlier `--set`s.
- `helm list -A`, `helm history`, `helm rollback RELEASE REVISION`, `helm get values|manifest`, `helm uninstall`.
- Kustomize: `resources`, `namespace`, `namePrefix`/`nameSuffix`, `labels`, `images`, `replicas`, `configMapGenerator`/`secretGenerator`, `patches` (strategic merge or JSON 6902). `kubectl apply -k`, `kubectl delete -k`.
Next, prove it against the clock: **drill 4.6**.""",
}
# ---- 4.7 extension interfaces -----------------------------------------------------------------
L47 = {
"id": 9407, "objective": "4.7", "workers": 1,
"title": "CNI, CRI and CSI: what plugs into this cluster",
"intro": r"""Kubernetes leaves three jobs to plugins, each behind an interface:
```
CRI Container Runtime Interface the kubelet asks the runtime (containerd, CRI-O) over a socket crictl
CNI Container Network Interface the runtime calls the plugins in /etc/cni/net.d for each Pod sandbox
CSI Container Storage Interface a driver (Deployment + DaemonSet) makes and mounts volumes CSIDriver, CSINode
```
A **RuntimeClass** names a handler the runtime has configured - `runc`, or a sandbox such as `runsc` (gVisor) - and a Pod picks it with `runtimeClassName`.
This cluster runs containerd, kindnet as its CNI, and the local-path provisioner for storage. Its worker is `{W1}`.""",
"setup": NS + r"""
mkns plug
mkdir -p "$WS/lesson-4.7"
$K -n plug run probe --image=busybox:1.36 -- sleep 86400 >/dev/null
$K -n plug wait --for=condition=Ready pod/probe --timeout=180s >/dev/null
""",
"steps": [
step("The CNI chain on a node",
r"""On `{W1}`, find the CNI configuration (the runtime uses the first file in `/etc/cni/net.d`, by name) and write to `{WS}/lesson-4.7/cni.txt`:
```
config=<full path of that file>
plugins=<the type of each plugin in its chain, in order, comma-separated>
```""",
check=r"""
f=$(nx "$W1" 'ls /etc/cni/net.d/*.conf* 2>/dev/null | sort | head -1' </dev/null)
p=$(nx "$W1" "cat $f" </dev/null | python3 -c 'import json,sys; c=json.load(sys.stdin); print(",".join(x["type"] for x in c.get("plugins", [c])))')
eq "$(tr -d ' \r' < "$WS/lesson-4.7/cni.txt" 2>/dev/null | grep .)" "$(printf 'config=%s\nplugins=%s' "$f" "$p")" """,
hint=r"""`ls /etc/cni/net.d/` and `cat` the file: a `.conflist` has a `"plugins"` list, each with a `"type"` - the binary under `/opt/cni/bin` that is run.""",
show=r"""ssh -n {W1} 'ls /etc/cni/net.d/; cat /etc/cni/net.d/10-kindnet.conflist; ls /opt/cni/bin'
f=$(ssh -n {W1} 'ls /etc/cni/net.d/*.conf* | sort | head -1')
p=$(ssh -n {W1} "cat $f" | jq -r '[.plugins[].type] | join(",")')
printf 'config=%s\nplugins=%s\n' "$f" "$p" > {WS}/lesson-4.7/cni.txt
cat {WS}/lesson-4.7/cni.txt"""),
step("The runtime, through CRI",
r"""Write to `{WS}/lesson-4.7/cri.txt`:
```
endpoint=<the runtime endpoint the kubelet on {W1} uses>
runtime=<the runtime and its version, as the node reports it - for example containerd://1.7.0>
```""",
check=r"""
e=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["containerRuntimeEndpoint"])')
r=$(j node "$W1" '{.status.nodeInfo.containerRuntimeVersion}')
eq "$(tr -d ' \r' < "$WS/lesson-4.7/cri.txt" 2>/dev/null | grep .)" "$(printf 'endpoint=%s\nruntime=%s' "$e" "$r")" """,
hint=r"""`containerRuntimeEndpoint` in the kubelet's configuration (`/var/lib/kubelet/config.yaml`, or `configz`); `kubectl get node -o wide` shows CONTAINER-RUNTIME.""",
show=r"""e=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.containerRuntimeEndpoint)
r=$(kubectl get node {W1} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}')
printf 'endpoint=%s\nruntime=%s\n' "$e" "$r" > {WS}/lesson-4.7/cri.txt
cat {WS}/lesson-4.7/cri.txt"""),
step("A Pod's container, from the runtime's side",
r"""Pod `probe` in `plug` runs on `{W1}`. Ask the runtime about it there, with `crictl`: find its container, and write the container's **process ID on the node** to `{WS}/lesson-4.7/pid.txt`.""",
check=r"""
want=$(nx "$W1" 'crictl inspect $(crictl ps --name probe -q | head -1) 2>/dev/null' </dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["info"]["pid"])')
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-4.7/pid.txt" 2>/dev/null)" "$want" """,
hint=r"""`sudo crictl pods --name probe`, `sudo crictl ps --name probe`, `sudo crictl inspect ID` - the PID is under `info.pid`. `crictl` talks to the same socket as the kubelet.""",
show=r"""ssh -n {W1} 'sudo crictl pods --name probe; sudo crictl ps --name probe'
ssh -n {W1} 'sudo crictl inspect $(sudo crictl ps --name probe -q | head -1)' | jq .info.pid > {WS}/lesson-4.7/pid.txt
cat {WS}/lesson-4.7/pid.txt"""),
step("A RuntimeClass",
r"""Create RuntimeClass `plain` for containerd's handler `runc`, and run Pod `classy` in `plug` (`busybox:1.36`, `sleep 86400`) with that runtime class. A Pod that names a handler the node does not have stays in `ContainerCreating`.""",
check=r"""eq "$(j runtimeclass plain '{.handler}')" runc && eq "$(j -n plug pod classy '{.spec.runtimeClassName} {.status.phase}')" "plain Running" """,
hint=r"""A RuntimeClass is `node.k8s.io/v1` with a top-level `handler:` and no `spec`. The Pod: `spec.runtimeClassName: plain`.""",
show=r"""kubectl apply -f - <<'EOF'
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: plain
handler: runc
EOF
kubectl -n plug run classy --image=busybox:1.36 --overrides='{"spec":{"runtimeClassName":"plain"}}' -- sleep 86400
kubectl -n plug wait --for=condition=Ready pod/classy --timeout=120s
kubectl get runtimeclass"""),
step("Storage drivers",
r"""Write to `{WS}/lesson-4.7/csi.txt`:
```
csidrivers=<number of CSIDriver objects in the cluster>
provisioner=<provisioner of the default StorageClass>
```
A CSI driver registers a `CSIDriver` object and appears in each node's `CSINode`. The local-path provisioner is not one - it is an external provisioner of its own.""",
check=r"""
n=$($K get csidrivers --no-headers 2>/dev/null | grep -c .)
p=$($K get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.provisioner}{end}')
eq "$(tr -d ' \r' < "$WS/lesson-4.7/csi.txt" 2>/dev/null | grep .)" "$(printf 'csidrivers=%s\nprovisioner=%s' "$n" "$p")" """,
hint=r"""`kubectl get csidrivers`, `kubectl get csinodes`, `kubectl get sc`.""",
show=r"""kubectl get csidrivers
kubectl get csinodes
kubectl get sc
printf 'csidrivers=%s\nprovisioner=%s\n' "$(kubectl get csidrivers --no-headers 2>/dev/null | grep -c .)" "$(kubectl get sc standard -o jsonpath='{.provisioner}')" > {WS}/lesson-4.7/csi.txt
cat {WS}/lesson-4.7/csi.txt"""),
],
"outro": r"""You read the CNI chain on a node, the runtime the kubelet talks to, a container as the runtime sees it, ran a Pod with a RuntimeClass, and counted the storage drivers.
**For the exam**
- CNI: `/etc/cni/net.d/` (config), `/opt/cni/bin/` (plugins); the CNI itself (Calico, Cilium, Flannel, ...) is installed as a manifest or a Helm chart after `kubeadm init`, and its Pods run in `kube-system` or a Namespace of its own.
- CRI: `crictl ps|pods|images|inspect|logs|exec`; the endpoint is in `/etc/crictl.yaml` and in the kubelet's `containerRuntimeEndpoint`.
- RuntimeClass `handler` must match a runtime configured in containerd (`[plugins..."cri".containerd.runtimes.NAME]`).
- CSI: `kubectl get csidrivers,csinodes`; a StorageClass names the driver as its `provisioner`.
Next, prove it against the clock: **drill 4.7**.""",
}
# ---- 4.8 CRDs and operators --------------------------------------------------------------------
_OPERATOR = r"""
o="$WS/lesson-4.8/operator"; rm -rf "$WS/lesson-4.8"; mkdir -p "$o"
cat > "$o/operator.yaml" <<'EOF'
""" + backup_operator() + r"""EOF
$K delete crd backups.data.example.com --ignore-not-found >/dev/null 2>&1
"""
L48 = {
"id": 9408, "objective": "4.8", "workers": 1,
"title": "CustomResourceDefinitions, and an operator that acts on them",
"intro": r"""A **CustomResourceDefinition** adds a kind to the API: once it is `Established`, you create, list and watch its objects like any other, and its OpenAPI schema validates them and fills in defaults.
```
CRD name PLURAL.GROUP backups.data.example.com
names kind, plural, singular, shortNames
scope Namespaced | Cluster
versions each served, exactly one storage: true, each with a schema (and subresources: status)
```
An **operator** is a CRD plus a controller - a Deployment with RBAC - that watches those objects and makes the world match them. Installing one is applying its manifests or its Helm chart; using it is creating custom resources.
The manifests of a small backup operator are in `{WS}/lesson-4.8/operator/`. It expects a `Backup` kind that you will define.""",
"setup": NS + _OPERATOR + r"""
mkns data
""",
"steps": [
step("Define the kind",
r"""Create CRD `backups.data.example.com`: group `data.example.com`, kind `Backup` (plural `backups`, singular `backup`, short name `bk`), **Namespaced**, one version `v1` (served, storage) with:
- `spec.schedule`: string, **required**
- `spec.retention`: integer, at least `1`, **default `7`**
- a `status` subresource, with `status.phase` a string""",
check=PY + r"""
c=backups.data.example.com
eq "$(j crd $c '{.spec.scope} {.spec.names.kind} {.spec.names.shortNames[0]}')" "Namespaced Backup bk" || exit 1
py crd $c 's = o["spec"]["versions"][0]; p = s["schema"]["openAPIV3Schema"]["properties"]["spec"]
s["served"] and s["storage"] and "status" in s.get("subresources", {}) and "schedule" in p.get("required", []) and p["properties"]["retention"].get("default") == 7 and p["properties"]["retention"].get("minimum") == 1' \
&& eq "$(j crd $c '{.status.conditions[?(@.type=="Established")].status}')" True""",
hint=r"""The docs page "Extend the Kubernetes API with CustomResourceDefinitions" has the skeleton. `required:` sits beside `properties:`, at the level of `spec`'s schema; `default:` and `minimum:` inside `retention`. `subresources: {status: {}}` goes under the version.""",
show=r"""kubectl apply -f - <<'EOF'
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: backups.data.example.com
spec:
group: data.example.com
scope: Namespaced
names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]}
versions:
- name: v1
served: true
storage: true
subresources:
status: {}
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [schedule]
properties:
schedule: {type: string}
retention: {type: integer, minimum: 1, default: 7}
status:
type: object
properties:
phase: {type: string}
EOF
kubectl wait --for condition=established crd/backups.data.example.com
kubectl api-resources --api-group=data.example.com"""),
step("Defaults and validation",
r"""In Namespace `data`, create Backup `nightly` with `schedule: "0 2 * * *"` and no retention: the API server fills in `7`. Then try a Backup `broken` with `retention: 0` - it is refused. Save the refusal (kubectl writes it to **stderr**) in `{WS}/lesson-4.8/refused.txt`.""",
check=r"""
eq "$(j -n data bk nightly '{.spec.schedule} {.spec.retention}')" "0 2 * * * 7" && [ -z "$(j -n data bk broken '{.metadata.name}')" ] \
&& grep -q 'retention' "$WS/lesson-4.8/refused.txt" 2>/dev/null && grep -qi 'greater than or equal to 1' "$WS/lesson-4.8/refused.txt" """,
hint=r"""`apiVersion: data.example.com/v1`, `kind: Backup`. Redirect with `2>` - with `>` the file stays empty.""",
show=r"""kubectl -n data apply -f - <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: nightly
spec:
schedule: "0 2 * * *"
EOF
kubectl -n data get bk nightly -o jsonpath='{.spec}'; echo
printf 'apiVersion: data.example.com/v1\nkind: Backup\nmetadata: {name: broken}\nspec: {schedule: "@hourly", retention: 0}\n' | kubectl -n data apply -f - 2> {WS}/lesson-4.8/refused.txt
cat {WS}/lesson-4.8/refused.txt"""),
step("What the schema says",
r"""`kubectl explain` reads the CRD's schema like any built-in kind's. Save the output of `kubectl explain backup.spec` in `{WS}/lesson-4.8/explain.txt`.""",
check=r"""f="$WS/lesson-4.8/explain.txt"; grep -q 'retention' "$f" 2>/dev/null && grep -q 'schedule' "$f" && grep -qi 'required' "$f" """,
hint=r"""`kubectl explain backup.spec` - or `--recursive` for the whole tree.""",
show=r"""kubectl explain backup.spec | tee {WS}/lesson-4.8/explain.txt"""),
step("Install the operator",
r"""Install the operator from `{WS}/lesson-4.8/operator/` and wait until its controller runs. Read the manifests first: a Namespace, a ServiceAccount, a ClusterRole for `backups` and ConfigMaps, the binding, and the controller's Deployment.""",
check=r"""eq "$(j -n backup-operator deploy controller '{.status.readyReplicas}')" 1 && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller""",
hint=r"""`kubectl apply -f DIR` applies every file in it; `kubectl -n backup-operator rollout status deploy controller`.""",
show=r"""grep -E '^kind:|^ name:' {WS}/lesson-4.8/operator/operator.yaml
kubectl apply -f {WS}/lesson-4.8/operator/
kubectl -n backup-operator rollout status deploy controller --timeout=120s"""),
step("Let it work",
r"""Create Backup `weekly` in `data` with `schedule: "0 3 * * 0"`. Within a few seconds the controller has acted on it - and on `nightly`: each has `status.phase: Scheduled`, and a ConfigMap `backup-NAME` holds its schedule. `kubectl -n backup-operator logs deploy/controller` is where you would look if it did not.""",
check=r"""
eq "$(j -n data bk weekly '{.status.phase}')" Scheduled && eq "$(j -n data cm backup-weekly '{.data.schedule}')" "0 3 * * 0" && eq "$(j -n data bk nightly '{.status.phase}')" Scheduled""",
hint=r"""A Backup like `nightly`. `kubectl -n data get bk` - add `additionalPrinterColumns` to the CRD and the phase would show there too.""",
show=r"""kubectl -n data apply -f - <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: weekly
spec:
schedule: "0 3 * * 0"
EOF
sleep 10
kubectl -n data get bk -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PHASE:.status.phase
kubectl -n data get cm"""),
],
"outro": r"""You defined a kind with validation and defaults, saw the schema refuse bad objects, installed an operator with its RBAC, and watched it act on your custom resources.
**For the exam**
- `kubectl get crd`, `kubectl api-resources --api-group=GROUP`, `kubectl explain KIND.spec --recursive` - the fastest way to learn an unfamiliar custom resource.
- Wait for `Established` before creating objects of a new kind.
- Installing an operator: its manifests (`kubectl apply -f URL|DIR`) or chart (`helm install`); then check its CRDs, its controller's Pods and logs, and create the custom resource the task asks for.
- Deleting a CRD deletes every object of its kind.
Next, prove it against the clock: **drill 4.8**.""",
}
LESSONS = [L41, L42, L43, L44, L45, L46, L47, L48]

676
cka/lessons_net.py Normal file
View File

@ -0,0 +1,676 @@
"""CKA lessons of the Services and Networking domain (5.1-5.6). See cka/lessons.py for what a lesson is."""
from cka.common import DNS
from sim.bank import NET, NS, PODS, step
# ---- 5.1 connectivity between Pods ------------------------------------------------------------
L51 = {
"id": 9501, "objective": "5.1", "workers": 2,
"title": "The Pod network: addresses, routes between nodes, and the host network",
"intro": r"""Kubernetes asks three things of a network, and leaves the rest to the CNI plugin:
```
every Pod has its own IP from the node's slice of the Pod CIDR (node.spec.podCIDR)
every Pod reaches every other Pod across nodes, without NAT
agents on a node reach its Pods the kubelet, for probes
```
How the packets get from one node to another is the plugin's business: kindnet, here, adds a route on each node for every other node's Pod CIDR, via that node's address. Others use overlays (VXLAN) or BGP.
A Pod with `hostNetwork: true` gets no IP of its own - it uses the node's network namespace and address.
Namespace `net` has Pod `a` on `{W1}` and Pod `b` on `{W2}`; each serves its own name on port 80.""",
"setup": PODS + r"""
mkns net
mkdir -p "$WS/lesson-5.1"
for p in a:$W1 b:$W2; do
$K -n net run "${p%%:*}" --image=busybox:1.36 --overrides="{\"spec\":{\"nodeName\":\"${p#*:}\"}}" --command -- \
sh -c 'mkdir -p /www && echo "$HOSTNAME" > /www/index.html && exec httpd -f -p 80 -h /www' >/dev/null
done
ready net || exit 1
""",
"steps": [
step("Each node's share of the Pod network",
r"""Write each node's Pod CIDR to `{WS}/lesson-5.1/cidrs.txt`, one line per node: `NODE CIDR`. Then compare with the Pod IPs in `kubectl get pods -A -o wide`: every Pod's address is from its node's range.""",
check=r"""
want=$($K get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' | sort)
eq "$(tr -s ' \t' ' ' < "$WS/lesson-5.1/cidrs.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """,
hint=r"""`.spec.podCIDR` of each node: `-o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}'`, or `custom-columns`. The cluster-wide range is the `podSubnet` in kubeadm's configuration.""",
show=r"""kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' > {WS}/lesson-5.1/cidrs.txt
cat {WS}/lesson-5.1/cidrs.txt
kubectl -n net get pods -o wide"""),
step("Across nodes, Pod to Pod",
r"""From Pod `a`, send a request straight to Pod `b`'s **IP address** on port 80 - no Service - and save the answer in `{WS}/lesson-5.1/answer.txt`. It crossed from `{W1}` to `{W2}`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/answer.txt" 2>/dev/null)" b""",
hint=r"""`kubectl -n net get pod b -o jsonpath='{.status.podIP}'`, then `kubectl -n net exec a -- wget -qO- -T3 http://IP`.""",
show=r"""ip=$(kubectl -n net get pod b -o jsonpath='{.status.podIP}')
kubectl -n net exec a -- wget -qO- -T3 http://$ip | tee {WS}/lesson-5.1/answer.txt"""),
step("The route that carries it",
r"""On `{W1}`, look at the routing table: there is a route for `{W2}`'s Pod CIDR, via `{W2}`'s node address. Write that route's destination and gateway to `{WS}/lesson-5.1/route.txt`, as `CIDR via ADDRESS`.""",
check=r"""
c=$(j node "$W2" '{.spec.podCIDR}'); gw=$(j node "$W2" '{.status.addresses[?(@.type=="InternalIP")].address}')
nx "$W1" "ip route | grep -q '^$c via $gw '" </dev/null && eq "$(tr -s ' ' < "$WS/lesson-5.1/route.txt" 2>/dev/null | tr -d '\r\n' | sed 's/ *$//')" "$c via $gw" """,
hint=r"""`ip route` on the node. The node's address is its `InternalIP` (`kubectl get nodes -o wide`).""",
show=r"""ssh -n {W1} ip route
c=$(kubectl get node {W2} -o jsonpath='{.spec.podCIDR}')
ssh -n {W1} ip route | grep "^$c " | awk '{print $1, $2, $3}' > {WS}/lesson-5.1/route.txt
cat {WS}/lesson-5.1/route.txt"""),
step("A Pod on the host's network",
r"""Run Pod `hostnet` in `net` (`busybox:1.36`, `sleep 86400`) with `hostNetwork: true`, on `{W1}`. Its IP is not from the Pod CIDR: it is `{W1}`'s own address, and it sees the node's interfaces. Write its IP to `{WS}/lesson-5.1/hostnet.txt`.""",
check=r"""
want=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}')
eq "$(j -n net pod hostnet '{.spec.hostNetwork} {.status.podIP}')" "true $want" && eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/hostnet.txt" 2>/dev/null)" "$want" """,
hint=r"""`spec.hostNetwork: true` and `spec.nodeName: {W1}` (or a nodeSelector). `kubectl run ... --overrides='{"spec":{...}}'` adds both.""",
show=r"""kubectl -n net run hostnet --image=busybox:1.36 --overrides='{"spec":{"hostNetwork":true,"nodeName":"{W1}"}}' -- sleep 86400
kubectl -n net wait --for=condition=Ready pod/hostnet --timeout=120s
kubectl -n net get pod hostnet -o jsonpath='{.status.podIP}' > {WS}/lesson-5.1/hostnet.txt
kubectl -n net exec hostnet -- ip -4 addr show eth0"""),
],
"outro": r"""You read how the Pod network is divided among nodes, sent a request from one node's Pod to another's, found the route that carried it, and ran a Pod on the host network.
**For the exam**
- Pod IPs: `kubectl get pods -o wide`; the node's range: `.spec.podCIDR`; the cluster's: kubeadm's `podSubnet` (or the CNI's own configuration).
- Pods reach each other by IP across nodes with no NAT; Services and DNS are on top of that.
- Which CNI runs: its Pods in `kube-system` (or its own Namespace) and its file in `/etc/cni/net.d/`. NetworkPolicy works only if the plugin enforces it.
- `hostNetwork: true` Pods share the node's ports: two of them cannot listen on the same port on one node.
Next, prove it against the clock: **drill 5.1**.""",
}
# ---- 5.2 NetworkPolicies ----------------------------------------------------------------------
L52 = {
"id": 9502, "objective": "5.2", "workers": 1,
"title": "NetworkPolicies: deny by default, then allow exactly what is needed",
"intro": r"""Without NetworkPolicies, every Pod accepts traffic from everywhere. A policy **selects** Pods (`podSelector`) and, for those Pods only, allows the traffic its rules list - everything else in that direction is dropped. Policies add up: a packet passes if any policy allows it.
```
podSelector: {} every Pod in the Namespace
from: [{podSelector}] Pods of THIS Namespace
from: [{namespaceSelector}] every Pod of the matching Namespaces
from: [{namespaceSelector, podSelector}] one element: those Pods IN those Namespaces (AND)
from: [{namespaceSelector}, {podSelector}] two elements: either (OR)
```
An egress policy also blocks DNS unless it allows UDP and TCP 53 to `kube-dns`.
Namespace `prod` (label `team=prod`) runs Pods `db`, `api` and `web`, each with a Service of its name on port 80. Namespace `dev` (label `team=dev`) runs `api`, `tester` and `echo`. Nothing is restricted yet.""",
"setup": PODS + r"""
mkns prod; mkns dev
$K label ns prod team=prod --overwrite >/dev/null; $K label ns dev team=dev --overwrite >/dev/null
for p in db api web; do srv prod $p; done
for p in api tester echo; do srv dev $p; done
ready prod && ready dev || exit 1
""",
"steps": [
step("Close the Namespace",
r"""Create NetworkPolicy `default-deny` in `prod`: it selects every Pod and allows **no** ingress. From now on, nothing reaches the Pods of `prod` - not even from inside `prod`.""",
check=NET + r"""
eq "$(j -n prod netpol default-deny '{.spec.policyTypes[0]}')" Ingress && [ -z "$(j -n prod netpol default-deny '{.spec.ingress}')" ] || exit 1
blocked prod api http://db && blocked dev tester http://web.prod""",
hint=r"""`podSelector: {}` and `policyTypes: [Ingress]`, with no `ingress:` list. Test with `kubectl -n prod exec api -- wget -qO- -T3 db`.""",
show=r"""kubectl -n prod apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes: [Ingress]
EOF
kubectl -n prod exec api -- wget -qO- -T3 db || echo blocked"""),
step("Let the API reach the database",
r"""Create NetworkPolicy `db-from-api` in `prod`: Pods `app=db` accept TCP 80 from Pods `app=api` **of `prod`**. `web` still cannot reach `db`, and neither can `dev`'s `api` - a `podSelector` alone means "in this Namespace".""",
check=NET + r"""okr prod api http://db && blocked prod web http://db && blocked dev api http://db.prod""",
hint=r"""`podSelector: {matchLabels: {app: db}}`, `ingress: [{from: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}]`.""",
show=r"""kubectl -n prod apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: db-from-api
spec:
podSelector:
matchLabels: {app: db}
policyTypes: [Ingress]
ingress:
- from:
- podSelector:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
EOF
kubectl -n prod exec api -- wget -qO- -T3 db
kubectl -n prod exec web -- wget -qO- -T3 db || echo blocked"""),
step("Those Pods in that Namespace",
r"""Create NetworkPolicy `web-from-dev-api` in `prod`: Pods `app=web` accept TCP 80 from Pods `app=api` **in** Namespaces labelled `team=dev` - both conditions in **one** element of `from`.
`dev`'s `api` gets in; `dev`'s `tester` does not, and neither does `prod`'s own `api`. Written as two elements, the rule would let every Pod of `dev` in, and every `api` Pod of `prod` too.""",
check=NET + r"""okr dev api http://web.prod && blocked dev tester http://web.prod && blocked prod api http://web""",
hint=r"""`from: [{namespaceSelector: {matchLabels: {team: dev}}, podSelector: {matchLabels: {app: api}}}]` - mind the single `-`.""",
show=r"""kubectl -n prod apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-from-dev-api
spec:
podSelector:
matchLabels: {app: web}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels: {team: dev}
podSelector:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
EOF
kubectl -n dev exec api -- wget -qO- -T3 web.prod
kubectl -n dev exec tester -- wget -qO- -T3 web.prod || echo blocked"""),
step("Egress, and DNS",
r"""Now limit what `tester` in `dev` may **send**: NetworkPolicy `tester-egress` in `dev` selects `app=tester` and allows egress only to `dev`'s `app=api` Pods on TCP 80 - and DNS, to `kube-dns` in `kube-system` on UDP and TCP 53.
`tester` reaches `api` by its name; `echo` is now out of its reach. Without the DNS rule, even `api` would fail - the name would not resolve.""",
check=NET + r"""
eq "$(j -n dev netpol tester-egress '{.spec.policyTypes[0]}')" Egress || exit 1
okr dev tester http://api && blocked dev tester http://echo""",
hint=r"""Two egress rules: `{to: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}` and `{to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}], ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]}`. Every Namespace carries the label `kubernetes.io/metadata.name`.""",
show=r"""kubectl -n dev apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: tester-egress
spec:
podSelector:
matchLabels: {app: tester}
policyTypes: [Egress]
egress:
- to:
- podSelector:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: kube-system}
podSelector:
matchLabels: {k8s-app: kube-dns}
ports:
- {protocol: UDP, port: 53}
- {protocol: TCP, port: 53}
EOF
kubectl -n dev exec tester -- wget -qO- -T3 api
kubectl -n dev exec tester -- wget -qO- -T3 echo || echo blocked"""),
],
"outro": r"""You closed a Namespace, opened single paths into it, used the AND form of a rule, and limited a Pod's egress without breaking its DNS.
**For the exam**
- Test every policy from the Pods in question: `kubectl exec POD -- wget -qO- -T3 TARGET` (or `nc -zv -w3 HOST PORT`). A timeout means dropped.
- `policyTypes` decides which directions the policy restricts; a policy with only `Ingress` leaves egress open.
- Ports in a policy are the Pod's ports (`targetPort`), not the Service's.
- `ipBlock: {cidr: ..., except: [...]}` for addresses outside the cluster.
- `kubectl describe netpol` spells the rules out in words - good for checking the AND/OR.
Next, prove it against the clock: **drill 5.2**.""",
}
# ---- 5.3 Services and endpoints ---------------------------------------------------------------
L53 = {
"id": 9503, "objective": "5.3", "workers": 1,
"title": "Service types, headless Services, ExternalName, and endpoints by hand",
"intro": r"""A Service is a stable name and address in front of a changing set of Pods - its **endpoints**, kept in EndpointSlices by a controller that follows the selector:
```
ClusterIP a virtual IP inside the cluster the default
NodePort ClusterIP + a port (30000-32767) on every node
LoadBalancer NodePort + an external address from a cloud controller (none here: it stays <pending>)
headless clusterIP: None - DNS returns the Pod IPs themselves
ExternalName a DNS alias (CNAME) for a name outside the cluster; no proxying
no selector you write the EndpointSlice yourself - for something outside the cluster
```
Namespace `svc` runs Deployment `web` (nginx, 2 replicas), Pod `client`, and Pod `legacy-box`, which serves on port 80 but carries no label any Service selects.""",
"setup": PODS + r"""
mkns svc
mkdir -p "$WS/lesson-5.3"
$K -n svc create deploy web --image=nginx:1.27 --replicas=2 --port=80 >/dev/null
cli svc client
$K -n svc run legacy-box --image=busybox:1.36 --labels=role=legacy --command -- sh -c 'mkdir -p /www && echo legacy > /www/index.html && exec httpd -f -p 80 -h /www' >/dev/null
$K -n svc rollout status deploy web --timeout=180s >/dev/null && ready svc
""",
"steps": [
step("ClusterIP",
r"""Expose Deployment `web` as Service `web`, port 80. From `client`, `wget -qO- web` returns nginx's page.""",
check=NET + r"""eq "$(j -n svc svc web '{.spec.type} {.spec.ports[0].port}')" "ClusterIP 80" && okr svc client http://web""",
hint=r"""`kubectl expose deploy web --port=80` - it copies the Deployment's selector.""",
show=r"""kubectl -n svc expose deploy web --port=80
kubectl -n svc get svc,endpointslices
kubectl -n svc exec client -- wget -qO- -T3 web | head -4"""),
step("NodePort",
r"""Create Service `web-np` of type **NodePort** for the same Pods, with node port **30080**. Then reach it from the control-plane node: `curl http://{W1}-ADDRESS:30080` - every node answers on that port, whether or not it runs one of the Pods.""",
check=r"""
eq "$(j -n svc svc web-np '{.spec.type} {.spec.ports[0].nodePort}')" "NodePort 30080" || exit 1
ip=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}'); nx "$CP" "curl -s -m 3 http://$ip:30080 | grep -q nginx" </dev/null""",
hint=r"""`kubectl create service nodeport web-np --tcp=80:80 --node-port=30080` makes a selector `app=web-np` - change it to `app=web`; or write the YAML, or `kubectl expose deploy web --name=web-np --type=NodePort` and then set the `nodePort`.""",
show=r"""kubectl -n svc expose deploy web --name=web-np --type=NodePort --port=80 --dry-run=client -o yaml > {WS}/lesson-5.3/web-np.yaml
yq -i '.spec.ports[0].nodePort = 30080' {WS}/lesson-5.3/web-np.yaml
kubectl -n svc apply -f {WS}/lesson-5.3/web-np.yaml
ip=$(kubectl get node {W1} -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
curl -s -m 3 http://$ip:30080 | head -4"""),
step("Headless",
r"""Create Service `web-headless` with `clusterIP: None` for the `web` Pods. Look it up from `client` with `nslookup web-headless`: you get the two Pod IPs, not a Service IP. Write them, one per line, to `{WS}/lesson-5.3/headless.txt`.""",
check=DNS + r"""
eq "$(j -n svc svc web-headless '{.spec.clusterIP}')" None || exit 1
want=$($K -n svc get pods -l app=web -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' | sort)
eq "$(tr -d ' \r' < "$WS/lesson-5.3/headless.txt" 2>/dev/null | grep . | sort)" "$want" """,
hint=r"""`kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80`. In busybox's `nslookup` output the addresses follow the `Name:` line.""",
show=r"""kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80
sleep 5
kubectl -n svc exec client -- nslookup web-headless
kubectl -n svc exec client -- nslookup web-headless | awk '/^Name:/ {n=1} n && /^Address/ {print $NF}' > {WS}/lesson-5.3/headless.txt
cat {WS}/lesson-5.3/headless.txt"""),
step("ExternalName",
r"""Create Service `docs` of type **ExternalName** pointing at `kubernetes.io`. Inside the cluster, `docs.svc.svc.cluster.local` is now a CNAME for that name - DNS only: nothing is proxied, and the port does not matter.""",
check=r"""eq "$(j -n svc svc docs '{.spec.type} {.spec.externalName}')" "ExternalName kubernetes.io" """,
hint=r"""`kubectl -n svc create service externalname docs --external-name=kubernetes.io`.""",
show=r"""kubectl -n svc create service externalname docs --external-name=kubernetes.io
kubectl -n svc get svc docs
kubectl -n svc exec client -- nslookup docs | head -6"""),
step("A Service without a selector",
r"""Create Service `legacy` (port 80) with **no selector**, and an EndpointSlice for it by hand that holds `legacy-box`'s Pod IP on port 80. From `client`, `wget -qO- legacy` then answers `legacy`.
An EndpointSlice belongs to a Service by the label `kubernetes.io/service-name`; its port's name must match the Service port's name (here: none).""",
check=NET + r"""
[ -z "$(j -n svc svc legacy '{.spec.selector}')" ] || exit 1
for i in 1 2 3 4 5; do [ "$($K -n svc exec client -- wget -qO- -T3 legacy 2>/dev/null)" = legacy ] && exit 0; sleep 3; done; exit 1""",
hint=r"""Service: `spec.ports: [{port: 80, targetPort: 80}]` and no `selector`. EndpointSlice (`discovery.k8s.io/v1`): `addressType: IPv4`, `endpoints: [{addresses: [IP]}]`, `ports: [{port: 80, protocol: TCP}]`, and the label.""",
show=r"""ip=$(kubectl -n svc get pod legacy-box -o jsonpath='{.status.podIP}')
kubectl -n svc apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: legacy
spec:
ports:
- {port: 80, targetPort: 80, protocol: TCP}
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: legacy-1
labels:
kubernetes.io/service-name: legacy
addressType: IPv4
endpoints:
- addresses: ["$ip"]
ports:
- {port: 80, protocol: TCP}
EOF
kubectl -n svc exec client -- wget -qO- -T3 legacy"""),
],
"outro": r"""You exposed the same Pods four ways, saw what headless DNS returns, aliased an outside name, and backed a Service with endpoints you wrote yourself.
**For the exam**
- `kubectl expose deploy|pod|svc NAME --port --target-port --type --name` copies the selector; `kubectl create service clusterip|nodeport|loadbalancer|externalname` does not know your Pods.
- `kubectl get endpointslices -l kubernetes.io/service-name=NAME` - empty means the selector matches no ready Pod.
- NodePorts are 30000-32767 by default and open on every node; `externalTrafficPolicy: Local` keeps traffic on nodes that run a Pod.
- A Service's port may be named; then `targetPort` can be the container port's **name**, and EndpointSlice ports must carry the same name.
Next, prove it against the clock: **drill 5.3**.""",
}
# ---- 5.4 Gateway API --------------------------------------------------------------------------
L54 = {
"id": 9504, "objective": "5.4", "workers": 1,
"title": "Gateway API: GatewayClass, Gateway, HTTPRoute - paths, weights, headers, TLS",
"intro": r"""The Gateway API splits what Ingress kept in one object across three roles:
```
GatewayClass which controller implements gateways (cluster-wide; the infrastructure provider)
Gateway an entry point: listeners - protocol, port, hostname, TLS - and which routes may attach
HTTPRoute routing rules: matches (path, header, method, query) -> backendRefs (Services, with weights)
```
A route attaches to a Gateway through `parentRefs`; a listener's `allowedRoutes` says from which Namespaces. Weights split traffic between backends - a canary without a second Deployment of tricks.
The Gateway API's CRDs are installed. No controller runs here, so nothing will actually serve traffic: you are graded, as in many exam tasks, on the objects. Namespace `gw` has Services `web` (port 80), `web-v2` (80) and `api` (8080).""",
"setup": NS + r"""
ensure_gateway_crds || { echo "could not install the Gateway API CRDs" >&2; exit 1; }
mkns gw
for s in web:80 web-v2:80 api:8080; do $K -n gw create service clusterip "${s%%:*}" --tcp="${s#*:}:80" >/dev/null; done
mkdir -p "$WS/lesson-5.4"
""",
"steps": [
step("A class and a Gateway",
r"""Create GatewayClass `example` (controller name `example.com/gateway-controller`) and, in `gw`, Gateway `shop` of that class with one listener: `http`, protocol HTTP, port 80, hostname `shop.example.com`, accepting routes from its own Namespace.""",
check=r"""
eq "$(j gatewayclass example '{.spec.controllerName}')" example.com/gateway-controller || exit 1
eq "$(j -n gw gateway shop '{.spec.gatewayClassName} {.spec.listeners[0].name} {.spec.listeners[0].protocol} {.spec.listeners[0].port} {.spec.listeners[0].hostname} {.spec.listeners[0].allowedRoutes.namespaces.from}')" "example http HTTP 80 shop.example.com Same" """,
hint=r"""Both are `gateway.networking.k8s.io/v1`. `kubectl explain gateway.spec.listeners` once the CRDs are in; the docs at gateway-api.sigs.k8s.io have every example.""",
show=r"""kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: example
spec:
controllerName: example.com/gateway-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: shop
namespace: gw
spec:
gatewayClassName: example
listeners:
- name: http
protocol: HTTP
port: 80
hostname: shop.example.com
allowedRoutes:
namespaces:
from: Same
EOF
kubectl -n gw get gateway shop"""),
step("Route by path",
r"""Create HTTPRoute `shop` in `gw`, attached to Gateway `shop`: requests for paths under `/api` go to Service `api` port 8080, everything else to `web` port 80.""",
check=r"""
py() { $K -n gw get httproute shop -o json 2>/dev/null | python3 -c "import json,sys; r=json.load(sys.stdin)['spec']; $1"; }
py '
p = r["parentRefs"][0]["name"]; rules = r["rules"]
def to(path):
for x in rules:
for m in x.get("matches", []):
if m.get("path", {}).get("value") == path and m["path"].get("type", "PathPrefix") == "PathPrefix":
return [(b["name"], b.get("port")) for b in x["backendRefs"]]
sys.exit(0 if p == "shop" and to("/api") == [("api", 8080)] and to("/") == [("web", 80)] else 1)'""",
hint=r"""`parentRefs: [{name: shop}]`, then two `rules`, each with `matches: [{path: {type: PathPrefix, value: ...}}]` and `backendRefs: [{name: ..., port: ...}]`.""",
show=r"""kubectl -n gw apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop
spec:
parentRefs:
- name: shop
hostnames: [shop.example.com]
rules:
- matches:
- path: {type: PathPrefix, value: /api}
backendRefs:
- {name: api, port: 8080}
- matches:
- path: {type: PathPrefix, value: /}
backendRefs:
- {name: web, port: 80}
EOF
kubectl -n gw describe httproute shop | grep -A3 'Backend Refs' | head -8"""),
step("A canary by weight",
r"""Change the `/` rule of route `shop` so that **90%** of its requests go to `web` and **10%** to `web-v2` (both port 80). The `/api` rule stays as it is.""",
check=r"""
$K -n gw get httproute shop -o json 2>/dev/null | python3 -c '
import json, sys; rules = json.load(sys.stdin)["spec"]["rules"]
def by(path): return next((x for x in rules for m in x.get("matches", []) if m.get("path", {}).get("value") == path), None)
root, api = by("/"), by("/api")
w = sorted((b["name"], b.get("weight", 1)) for b in root["backendRefs"]) if root else []
sys.exit(0 if w == [("web", 90), ("web-v2", 10)] and api and [b["name"] for b in api["backendRefs"]] == ["api"] else 1)'""",
hint=r"""Two `backendRefs` in the same rule, each with a `weight`. The weights are relative: 9 and 1 would do the same.""",
show=r"""kubectl -n gw get httproute shop -o yaml > {WS}/lesson-5.4/shop.yaml
yq -i '(.spec.rules[] | select(.matches[0].path.value == "/")).backendRefs = [{"name": "web", "port": 80, "weight": 90}, {"name": "web-v2", "port": 80, "weight": 10}]' {WS}/lesson-5.4/shop.yaml
kubectl apply -f {WS}/lesson-5.4/shop.yaml
kubectl -n gw get httproute shop -o jsonpath='{.spec.rules[1].backendRefs}'; echo"""),
step("Beta testers by header",
r"""Add HTTPRoute `shop-beta` in `gw`, attached to `shop`: requests carrying the header `x-env: beta` go **all** to `web-v2`. A more specific match wins, so these requests no longer take part in the 90/10 split.""",
check=r"""
$K -n gw get httproute shop-beta -o json 2>/dev/null | python3 -c '
import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0]
h = r["matches"][0].get("headers", [{}])[0]
sys.exit(0 if s["parentRefs"][0]["name"] == "shop" and h.get("name", "").lower() == "x-env" and h.get("value") == "beta" and [b["name"] for b in r["backendRefs"]] == ["web-v2"] else 1)'""",
hint=r"""`matches: [{headers: [{name: x-env, value: beta}]}]` (type `Exact` by default).""",
show=r"""kubectl -n gw apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop-beta
spec:
parentRefs:
- name: shop
hostnames: [shop.example.com]
rules:
- matches:
- headers:
- {name: x-env, value: beta}
backendRefs:
- {name: web-v2, port: 80}
EOF
kubectl -n gw get httproute"""),
step("HTTPS on the Gateway",
r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `gw`, and add a second listener to Gateway `shop`: `https`, protocol HTTPS, port 443, the same hostname, TLS mode `Terminate` with that Secret as its certificate.""",
check=r"""
eq "$(j -n gw secret shop-tls '{.type}')" kubernetes.io/tls || exit 1
eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="https")].protocol} {.spec.listeners[?(@.name=="https")].port} {.spec.listeners[?(@.name=="https")].tls.mode} {.spec.listeners[?(@.name=="https")].tls.certificateRefs[0].name}')" "HTTPS 443 Terminate shop-tls" \
&& eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="http")].port}')" 80""",
hint=r"""`openssl req -x509 ...`, `kubectl create secret tls`, then a listener with `tls: {mode: Terminate, certificateRefs: [{name: shop-tls}]}`. Keep the `http` listener.""",
show=r"""cd {WS}/lesson-5.4
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
kubectl -n gw create secret tls shop-tls --cert=tls.crt --key=tls.key
kubectl -n gw patch gateway shop --type json -p '[{"op":"add","path":"/spec/listeners/-","value":{"name":"https","protocol":"HTTPS","port":443,"hostname":"shop.example.com","tls":{"mode":"Terminate","certificateRefs":[{"name":"shop-tls"}]},"allowedRoutes":{"namespaces":{"from":"Same"}}}}]'
kubectl -n gw get gateway shop -o jsonpath='{range .spec.listeners[*]}{.name} {.port}{"\n"}{end}'
cd"""),
],
"outro": r"""You built a Gateway with HTTP and HTTPS listeners, routed by path, split traffic by weight, and sent one header's requests elsewhere.
**For the exam**
- The CRDs come from the Gateway API release (`standard-install.yaml`); the controller is separate (Envoy Gateway, NGINX Gateway Fabric, Istio, Cilium, ...). A Gateway's `status` stays empty without one.
- `parentRefs` names the Gateway (and `sectionName` a listener); `allowedRoutes.namespaces.from` is `Same`, `All` or `Selector`.
- Matches: `path` (`PathPrefix`, `Exact`, `RegularExpression`), `headers`, `queryParams`, `method`. Filters: `RequestRedirect` (http → https), `URLRewrite`, `RequestHeaderModifier`.
- Moving from Ingress: one Ingress rule ≈ one HTTPRoute rule; TLS moves from the Ingress to the Gateway's listener.
Next, prove it against the clock: **drill 5.4**.""",
}
# ---- 5.5 Ingress ------------------------------------------------------------------------------
L55 = {
"id": 9505, "objective": "5.5", "workers": 1,
"title": "Ingress: classes, host and path rules, TLS, and the default backend",
"intro": r"""An **Ingress** routes HTTP(S) by host and path to Services. It does nothing on its own: an **Ingress controller** (ingress-nginx, Traefik, HAProxy, ...) watches Ingresses of its **IngressClass** and configures a proxy from them.
```
ingressClassName which controller (or the IngressClass marked default)
rules[].host shop.example.com (none = every host)
paths[].pathType Prefix | Exact | ImplementationSpecific
paths[].backend service: {name, port: {number | name}}
tls[] hosts + a kubernetes.io/tls Secret
defaultBackend what matches no rule
```
No controller runs here, so you are graded on the objects. Namespace `ing` has Services `web` (port 80), `api` (8080) and `health` (8081).""",
"setup": NS + r"""
mkns ing
for s in web:80 api:8080 health:8081; do $K -n ing create service clusterip "${s%%:*}" --tcp="${s#*:}:80" >/dev/null; done
$K delete ingressclass nginx --ignore-not-found >/dev/null 2>&1
mkdir -p "$WS/lesson-5.5"
""",
"steps": [
step("An IngressClass, the default",
r"""Create IngressClass `nginx` for controller `k8s.io/ingress-nginx`, and make it the cluster's **default** class: an Ingress without `ingressClassName` then belongs to it.""",
check=r"""eq "$(j ingressclass nginx '{.spec.controller} {.metadata.annotations.ingressclass\.kubernetes\.io/is-default-class}')" "k8s.io/ingress-nginx true" """,
hint=r"""`networking.k8s.io/v1`, kind `IngressClass`, `spec.controller`, and the annotation `ingressclass.kubernetes.io/is-default-class: "true"`. Installing ingress-nginx makes this object for you.""",
show=r"""kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: nginx
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: k8s.io/ingress-nginx
EOF
kubectl get ingressclass"""),
step("Host and paths, from the command line",
r"""Create Ingress `shop` in `ing` with class `nginx` and host `shop.example.com`: paths under `/api` go to `api:8080`, everything else to `web:80`. `kubectl create ingress` can do it in one line.""",
check=r"""
$K -n ing get ingress shop -o json 2>/dev/null | python3 -c '
import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0]
p = {x["path"]: (x["pathType"], x["backend"]["service"]["name"], x["backend"]["service"]["port"]["number"]) for x in r["http"]["paths"]}
sys.exit(0 if s.get("ingressClassName") == "nginx" and r["host"] == "shop.example.com" and p.get("/api") == ("Prefix", "api", 8080) and p.get("/") == ("Prefix", "web", 80) else 1)'""",
hint=r"""`kubectl create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'` - a trailing `*` means `Prefix`, without it `Exact`.""",
show=r"""kubectl -n ing create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'
kubectl -n ing describe ingress shop | sed -n '/Rules/,/Annotations/p'"""),
step("An exact path",
r"""Add a rule to Ingress `shop`: exactly `/healthz` (pathType `Exact`) goes to `health:8081`. `/healthz/deep` would not match it - it would fall through to the `/` prefix.""",
check=r"""
$K -n ing get ingress shop -o json 2>/dev/null | python3 -c '
import json, sys; ps = json.load(sys.stdin)["spec"]["rules"][0]["http"]["paths"]
h = [x for x in ps if x["path"] == "/healthz"]
sys.exit(0 if h and h[0]["pathType"] == "Exact" and h[0]["backend"]["service"]["name"] == "health" and h[0]["backend"]["service"]["port"]["number"] == 8081 and len(ps) == 3 else 1)'""",
hint=r"""`kubectl edit ingress shop` and add a third entry under `paths:`, or a JSON patch that appends to `/spec/rules/0/http/paths`.""",
show=r"""kubectl -n ing patch ingress shop --type json -p '[{"op":"add","path":"/spec/rules/0/http/paths/-","value":{"path":"/healthz","pathType":"Exact","backend":{"service":{"name":"health","port":{"number":8081}}}}}]'
kubectl -n ing get ingress shop -o jsonpath='{range .spec.rules[0].http.paths[*]}{.pathType} {.path} -> {.backend.service.name}{"\n"}{end}'"""),
step("TLS",
r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `ing`, and add a `tls` section to Ingress `shop` for that host with that Secret. The controller would then serve HTTPS for it and, by default, redirect HTTP there.""",
check=r"""
eq "$(j -n ing secret shop-tls '{.type}')" kubernetes.io/tls && eq "$(j -n ing ingress shop '{.spec.tls[0].hosts[0]} {.spec.tls[0].secretName}')" "shop.example.com shop-tls" \
&& j -n ing secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q shop.example.com""",
hint=r"""`spec.tls: [{hosts: [shop.example.com], secretName: shop-tls}]`. The Secret must be in the Ingress's Namespace.""",
show=r"""cd {WS}/lesson-5.5
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
kubectl -n ing create secret tls shop-tls --cert=tls.crt --key=tls.key
kubectl -n ing patch ingress shop --type merge -p '{"spec":{"tls":[{"hosts":["shop.example.com"],"secretName":"shop-tls"}]}}'
kubectl -n ing get ingress shop
cd"""),
step("What matches nothing",
r"""Give Ingress `shop` a **default backend**: `web:80`, for requests whose host or path no rule matches.""",
check=r"""eq "$(j -n ing ingress shop '{.spec.defaultBackend.service.name} {.spec.defaultBackend.service.port.number}')" "web 80" """,
hint=r"""`spec.defaultBackend: {service: {name: web, port: {number: 80}}}`; `kubectl create ingress --default-backend=web:80` does it at creation.""",
show=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"defaultBackend":{"service":{"name":"web","port":{"number":80}}}}}'
kubectl -n ing describe ingress shop | head -12"""),
],
"outro": r"""You made a default IngressClass and an Ingress with host, prefix and exact paths, TLS and a default backend.
**For the exam**
- `kubectl create ingress NAME --class=C --rule='HOST/PATH*=SVC:PORT[,tls=SECRET]' [--default-backend=SVC:PORT]`.
- `pathType` is required; `Prefix` matches by path segment (`/api` matches `/api/v1`, not `/apix`).
- A backend port is the **Service's** port, by number or name.
- No address in `kubectl get ingress`? The controller is not running, or the class names another controller. Controller-specific behaviour lives in annotations (`nginx.ingress.kubernetes.io/rewrite-target`, `ssl-redirect`).
Next, prove it against the clock: **drill 5.5**.""",
}
# ---- 5.6 CoreDNS ------------------------------------------------------------------------------
L56 = {
"id": 9506, "objective": "5.6", "workers": 1,
"title": "Cluster DNS: names of Services and Pods, the Corefile, and a Pod's resolver",
"intro": r"""CoreDNS (Deployment `coredns`, Service `kube-dns` in `kube-system`) answers every Pod's DNS queries. The names it knows:
```
SERVICE.NAMESPACE.svc.cluster.local the ClusterIP (or, headless, the Pod IPs)
POD.SERVICE.NAMESPACE.svc.cluster.local one Pod of a headless Service - a StatefulSet's db-0.db...
A-B-C-D.NAMESPACE.pod.cluster.local a Pod by its IP, dots as dashes
```
Each Pod's `/etc/resolv.conf` points at `kube-dns` and searches its own Namespace first, so a bare `web` works inside the Namespace and `web.other` across. CoreDNS itself is configured by the **Corefile** in ConfigMap `coredns`, and reloads it on its own when it changes.
Namespace `dns-lab` has Service `web`, a StatefulSet `db` (2 replicas) behind headless Service `db`, and Pod `client`.""",
"setup": PODS + r"""
mkns dns-lab
mkdir -p "$WS/lesson-5.6"
srv dns-lab web; cli dns-lab client
$K -n dns-lab apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: db}
spec:
clusterIP: None
selector: {app: db}
ports: [{port: 5432}]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: db}
spec:
serviceName: db
replicas: 2
selector: {matchLabels: {app: db}}
template:
metadata: {labels: {app: db}}
spec:
containers: [{name: db, image: busybox:1.36, command: [sleep, '86400']}]
EOF
$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > "$PRIV/lesson-5.6.corefile"
ready dns-lab && $K -n dns-lab rollout status sts db --timeout=180s >/dev/null
""",
"steps": [
step("A Service by its full name",
r"""From `client`, look up `web` by its **fully qualified** name, and write the address you get to `{WS}/lesson-5.6/web.txt`. It is the Service's ClusterIP.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/web.txt" 2>/dev/null)" "$(j -n dns-lab svc web '{.spec.clusterIP}')" """,
hint=r"""`kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local`.""",
show=r"""kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local
kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local | awk '/^Name:/ {n=1} n && /^Address/ {print $NF; exit}' > {WS}/lesson-5.6/web.txt"""),
step("One Pod of a StatefulSet",
r"""A StatefulSet's Pods get stable names through their headless Service. Write the fully qualified DNS name of Pod `db-0` to `{WS}/lesson-5.6/db0.txt`, and check from `client` that it resolves to `db-0`'s IP.""",
check=DNS + r"""
eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/db0.txt" 2>/dev/null | sed 's/\.$//')" db-0.db.dns-lab.svc.cluster.local && resolves dns-lab client db-0.db.dns-lab.svc.cluster.local "$(j -n dns-lab pod db-0 '{.status.podIP}')" """,
hint=r"""`POD.SERVICE.NAMESPACE.svc.cluster.local` - the Service is the StatefulSet's `serviceName`.""",
show=r"""echo db-0.db.dns-lab.svc.cluster.local > {WS}/lesson-5.6/db0.txt
kubectl -n dns-lab exec client -- nslookup db-0.db.dns-lab.svc.cluster.local
kubectl -n dns-lab get pod db-0 -o wide"""),
step("Names of your own",
r"""Add a `hosts` block to CoreDNS's Corefile so that every Pod resolves **`legacy.internal`** to `10.96.0.250`, and every other name still resolves as before (`fallthrough`). Put it inside the `.:53 { ... }` server block, before `forward`.
CoreDNS reloads the changed ConfigMap by itself within a minute; `client` then resolves the new name.""",
check=DNS + r"""
$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep -q 'kubernetes cluster.local' || exit 1
resolves dns-lab client legacy.internal 10.96.0.250 && resolves dns-lab client web.dns-lab.svc.cluster.local "$(j -n dns-lab svc web '{.spec.clusterIP}')" """,
hint=r"""`kubectl -n kube-system edit cm coredns`. The block: `hosts {` / `10.96.0.250 legacy.internal` / `fallthrough` / `}`. A syntax error leaves CoreDNS on its old configuration: `kubectl -n kube-system logs -l k8s-app=kube-dns` says so.""",
show=r"""kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > {WS}/lesson-5.6/Corefile
sed -i '0,/^ forward/s// hosts {\n 10.96.0.250 legacy.internal\n fallthrough\n }\n forward/' {WS}/lesson-5.6/Corefile
cat {WS}/lesson-5.6/Corefile
kubectl -n kube-system create configmap coredns --from-file=Corefile={WS}/lesson-5.6/Corefile --dry-run=client -o yaml | kubectl apply -f -
sleep 40
kubectl -n dns-lab exec client -- nslookup legacy.internal"""),
step("A Pod's own resolver settings",
r"""Run Pod `tuned` in `dns-lab` (`busybox:1.36`, `sleep 86400`) whose resolver you set yourself: `dnsPolicy: None`, with the `kube-dns` Service's IP as its only nameserver, the search domains `dns-lab.svc.cluster.local` and `svc.cluster.local`, and the option `ndots: 2`. Its `/etc/resolv.conf` shows exactly that.""",
check=r"""
ip=$(j -n kube-system svc kube-dns '{.spec.clusterIP}')
r=$($K -n dns-lab exec tuned -- cat /etc/resolv.conf 2>/dev/null)
echo "$r" | grep -qx "nameserver $ip" && echo "$r" | grep -qx 'search dns-lab.svc.cluster.local svc.cluster.local' && echo "$r" | grep -q 'ndots:2' \
&& eq "$(j -n dns-lab pod tuned '{.spec.dnsPolicy}')" None""",
hint=r"""`spec.dnsPolicy: None` and `spec.dnsConfig: {nameservers: [IP], searches: [...], options: [{name: ndots, value: "2"}]}`. The other policies: `ClusterFirst` (the default), `Default` (the node's resolver), `ClusterFirstWithHostNet`.""",
show=r"""ip=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}')
kubectl -n dns-lab apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: tuned
spec:
dnsPolicy: None
dnsConfig:
nameservers: [$ip]
searches: [dns-lab.svc.cluster.local, svc.cluster.local]
options:
- {name: ndots, value: "2"}
containers:
- name: tuned
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n dns-lab wait --for=condition=Ready pod/tuned --timeout=120s
kubectl -n dns-lab exec tuned -- cat /etc/resolv.conf"""),
],
"outro": r"""You resolved a Service and a StatefulSet Pod by their full names, taught CoreDNS a name of your own, and set a Pod's resolver by hand.
**For the exam**
- Test DNS from a Pod: `kubectl run tmp --rm -it --restart=Never --image=busybox:1.36 -- nslookup NAME`.
- Nothing resolves: are the `coredns` Pods running and ready, does Service `kube-dns` have endpoints, does a NetworkPolicy block UDP/TCP 53, is the Corefile valid (`kubectl -n kube-system logs -l k8s-app=kube-dns`)?
- Cluster names resolve but outside names do not: the `forward` line of the Corefile, or the node's `/etc/resolv.conf`.
- The kubelet's `clusterDNS` and `clusterDomain` decide what goes into every Pod's resolv.conf.
Next, prove it against the clock: **drill 5.6**.""",
}
LESSONS = [L51, L52, L53, L54, L55, L56]

589
cka/lessons_storage.py Normal file
View File

@ -0,0 +1,589 @@
"""CKA lessons of the Storage domain (1.1-1.3). See cka/lessons.py for what a lesson is."""
from sim.bank import NS, step
# ---- 1.1 StorageClasses and dynamic provisioning ---------------------------------------------
L11 = {
"id": 9101, "objective": "1.1", "workers": 0,
"title": "StorageClasses: who makes the volume, when, and what happens after",
"intro": r"""Nobody creates PersistentVolumes by hand on a cluster with a **provisioner**. A claim names a **StorageClass**, and the provisioner behind that class makes a volume to fit the claim. The class decides three things you will be asked about:
```
provisioner who makes the volume rancher.io/local-path, ebs.csi.aws.com, ...
volumeBindingMode when it is made Immediate | WaitForFirstConsumer
reclaimPolicy what happens when the claim goes Delete | Retain
```
`WaitForFirstConsumer` holds a claim `Pending` until a Pod uses it, so the volume is made on the node the Pod lands on. `Immediate` provisions at once, wherever. The reclaim policy is copied into each volume when it is made: changing the class later does not change the volumes it already made.
One class may carry the annotation `storageclass.kubernetes.io/is-default-class: "true"`. A claim that names no class gets that one.
This cluster has kind's class `standard` (local-path provisioner) and an empty Namespace `store`.""",
"setup": NS + r"""
mkns store
mkdir -p "$WS/lesson-1.1"
""",
"steps": [
step("Read the default class",
r"""`kubectl get storageclass` marks the default with `(default)`. Write its name, its provisioner and its binding mode, separated by spaces, to `{WS}/lesson-1.1/default.txt` - for example `fast ebs.csi.aws.com Immediate`.""",
check=r"""
want="$(j sc standard '{.metadata.name} {.provisioner} {.volumeBindingMode}')"
[ -n "$want" ] && eq "$(tr -s ' \t' ' ' < "$WS/lesson-1.1/default.txt" 2>/dev/null | tr -d '\r\n' | sed 's/ *$//')" "$want" """,
hint=r"""`kubectl get sc` shows all three columns. For exactly the values: `-o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}'`.""",
show=r"""kubectl get sc
kubectl get sc standard -o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}' > {WS}/lesson-1.1/default.txt
cat {WS}/lesson-1.1/default.txt"""),
step("A class that keeps its volumes",
r"""Create StorageClass `keep`: the same provisioner as `standard` (`rancher.io/local-path`), binding mode `WaitForFirstConsumer`, and reclaim policy **`Retain`**, so a volume outlives its claim.
There is no `kubectl create storageclass`: write the manifest. `kubectl get sc standard -o yaml` is a good start - drop the default annotation when you copy it.""",
check=r"""
eq "$(j sc keep '{.provisioner} {.reclaimPolicy} {.volumeBindingMode}')" "rancher.io/local-path Retain WaitForFirstConsumer" \
&& [ "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""",
hint=r"""A StorageClass has no `spec`: `provisioner`, `reclaimPolicy` and `volumeBindingMode` sit at the top level, beside `metadata`. Its `apiVersion` is `storage.k8s.io/v1`.""",
show=r"""kubectl apply -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: keep
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
EOF
kubectl get sc"""),
step("A claim that waits",
r"""In Namespace `store`, create PersistentVolumeClaim `ledger`: class `keep`, `ReadWriteOnce`, `100Mi`.
It stays `Pending`, and that is correct. `kubectl -n store describe pvc ledger` says why in its events; write the event's **reason** (one word) to `{WS}/lesson-1.1/why.txt`.""",
check=r"""
eq "$(j -n store pvc ledger '{.spec.storageClassName}')" keep && eq "$(j -n store pvc ledger '{.spec.resources.requests.storage}')" 100Mi \
&& grep -qx 'WaitForFirstConsumer' <(tr -d ' \r' < "$WS/lesson-1.1/why.txt" 2>/dev/null)""",
hint=r"""The events at the bottom of `describe` have a Reason column. A claim is namespaced; the class is not.""",
show=r"""kubectl -n store apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger
spec:
storageClassName: keep
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 100Mi
EOF
kubectl -n store describe pvc ledger | tail -4
echo WaitForFirstConsumer > {WS}/lesson-1.1/why.txt"""),
step("Give it a consumer",
r"""Run Pod `bookkeeper` in `store` (image `busybox:1.36`) that mounts claim `ledger` at `/data` and runs `sh -c 'echo first > /data/entry.txt && sleep 86400'`.
As soon as the Pod is scheduled the provisioner makes a volume on its node and the claim turns `Bound`. Look at the new PersistentVolume: `kubectl get pv`. Its reclaim policy came from the class.""",
check=r"""
eq "$(j -n store pvc ledger '{.status.phase}')" Bound || exit 1
pv=$(j -n store pvc ledger '{.spec.volumeName}'); eq "$(j pv "$pv" '{.spec.persistentVolumeReclaimPolicy}')" Retain || exit 1
eq "$($K -n store exec bookkeeper -- cat /data/entry.txt 2>/dev/null)" first""",
hint=r"""`kubectl run bookkeeper --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'` gives you the Pod; add `volumes:` (with `persistentVolumeClaim.claimName`) at Pod level and `volumeMounts:` in the container.""",
show=r"""kubectl -n store apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: bookkeeper
spec:
containers:
- name: bookkeeper
image: busybox:1.36
command: [sh, -c, 'echo first > /data/entry.txt && sleep 86400']
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: ledger
EOF
kubectl -n store wait --for=condition=Ready pod/bookkeeper --timeout=120s
kubectl -n store get pvc ledger
kubectl get pv"""),
step("Delete the claim, keep the volume",
r"""Delete Pod `bookkeeper` and claim `ledger`. Because of `Retain`, the volume stays, with status `Released`: its data is still on disk, and no new claim can bind it until an admin says so.
Write the PersistentVolume's name to `{WS}/lesson-1.1/released.txt`.""",
check=r"""
[ -z "$(j -n store pvc ledger '{.metadata.name}')" ] || exit 1
pv=$(tr -d ' \r\n' < "$WS/lesson-1.1/released.txt" 2>/dev/null); [ -n "$pv" ] || exit 1
eq "$(j pv "$pv" '{.status.phase} {.spec.claimRef.namespace}/{.spec.claimRef.name}')" "Released store/ledger" """,
hint=r"""Read the volume name off the claim before you delete it (`kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}'`), or find it afterwards in `kubectl get pv`: the CLAIM column still names `store/ledger`.""",
show=r"""kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}' > {WS}/lesson-1.1/released.txt
kubectl -n store delete pod bookkeeper --grace-period=1
kubectl -n store delete pvc ledger
kubectl get pv"""),
step("Change the default class",
r"""Make `keep` the cluster's default class, and `standard` no longer the default. Two defaults would be ambiguous: set the annotation to `"true"` on one and `"false"` on the other.""",
check=r"""
eq "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" true \
&& [ "$(j sc standard '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""",
hint=r"""`kubectl annotate sc NAME storageclass.kubernetes.io/is-default-class=true --overwrite`, or `kubectl patch` with a `metadata.annotations` map.""",
show=r"""kubectl annotate sc standard storageclass.kubernetes.io/is-default-class=false --overwrite
kubectl annotate sc keep storageclass.kubernetes.io/is-default-class=true --overwrite
kubectl get sc"""),
],
"outro": r"""You read a class, made one that keeps its volumes, watched a claim wait for its Pod, kept a volume past its claim, and moved the default.
**For the exam**
- A StorageClass has no `spec`; `provisioner`, `reclaimPolicy`, `volumeBindingMode` and `allowVolumeExpansion` are top-level fields.
- `WaitForFirstConsumer`: a claim is `Pending` until a Pod uses it - not a fault. Look at `describe pvc` before you "fix" anything.
- The reclaim policy is copied into the volume when it is made. To keep an existing volume, patch the **PV**: `kubectl patch pv NAME -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'`.
- Growing a claim: the class needs `allowVolumeExpansion: true` and a driver that can expand, then raise `spec.resources.requests.storage` on the claim - claims only grow. The local-path provisioner can do neither that nor snapshots; a CSI driver such as the hostpath driver behind `csi-hostpath-sc` can, and lesson 1.3 grows, snapshots and restores a claim on it.
- One default class. `kubectl get sc` shows `(default)`.
Next, prove it against the clock: **drill 1.1**.""",
}
# ---- 1.2 volume types, access modes and reclaim policies -------------------------------------
L12 = {
"id": 9102, "objective": "1.2", "workers": 0,
"title": "Volume types, access modes and reclaim policies",
"intro": r"""A Pod's `volumes:` list says where data comes from; each container's `volumeMounts:` says where it appears. The types you meet on the exam:
```
emptyDir a scratch directory that lives and dies with the Pod (medium: Memory = tmpfs)
hostPath a directory of the node - tied to that node
configMap keys of a ConfigMap as files secret the same for a Secret
projected several of those, plus downwardAPI and a ServiceAccount token, in ONE directory
persistentVolumeClaim a claim, bound to a PersistentVolume
```
A PersistentVolume has **access modes** - `ReadWriteOnce` (one *node* at a time), `ReadOnlyMany`, `ReadWriteMany`, `ReadWriteOncePod` (one *Pod*) - and a **reclaim policy**, `Delete` or `Retain`. A claim binds a volume whose modes, size and class fit it.
Namespace `vols` has ConfigMap `app-config` (key `mode`) and Secret `app-secret` (key `password`).""",
"setup": NS + r"""
mkns vols
$K -n vols create configmap app-config --from-literal=mode=prod >/dev/null
$K -n vols create secret generic app-secret --from-literal=password=s3cr3t >/dev/null
mkdir -p "$WS/lesson-1.2"
""",
"steps": [
step("Two containers, one emptyDir",
r"""Create Pod `pair` in `vols` with an `emptyDir` volume `shared`, mounted at `/shared` in two `busybox:1.36` containers:
- `writer`: `sh -c 'echo hello > /shared/msg && sleep 86400'`
- `reader`: `sleep 86400`
The reader sees what the writer wrote: an emptyDir is shared by every container of its Pod.""",
check=r"""eq "$($K -n vols exec pair -c reader -- cat /shared/msg 2>/dev/null)" hello \
&& [ -n "$(j -n vols pod pair '{.spec.volumes[?(@.name=="shared")].emptyDir}')" ]""",
hint=r"""Both containers list the same `volumeMounts` entry (`name: shared`); the Pod lists the volume once, as `emptyDir: {}`.""",
show=r"""kubectl -n vols apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: pair
spec:
containers:
- name: writer
image: busybox:1.36
command: [sh, -c, 'echo hello > /shared/msg && sleep 86400']
volumeMounts:
- {name: shared, mountPath: /shared}
- name: reader
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- {name: shared, mountPath: /shared}
volumes:
- name: shared
emptyDir: {}
EOF
kubectl -n vols wait --for=condition=Ready pod/pair --timeout=120s
kubectl -n vols exec pair -c reader -- cat /shared/msg"""),
step("Scratch space in memory",
r"""Create Pod `scratch` in `vols` (`busybox:1.36`, `sleep 86400`) with an emptyDir at `/scratch` that is held in **memory**, at most `64Mi`.
Inside the Pod, `/proc/mounts` shows the difference: `/scratch` is a `tmpfs`. What is written there counts against the container's memory.""",
check=r"""
eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.medium}')" Memory && eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.sizeLimit}')" 64Mi \
&& $K -n vols exec scratch -- grep -q ' /scratch tmpfs ' /proc/mounts 2>/dev/null""",
hint=r"""`emptyDir: {medium: Memory, sizeLimit: 64Mi}`.""",
show=r"""kubectl -n vols apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: scratch
spec:
containers:
- name: scratch
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- {name: tmp, mountPath: /scratch}
volumes:
- name: tmp
emptyDir: {medium: Memory, sizeLimit: 64Mi}
EOF
kubectl -n vols wait --for=condition=Ready pod/scratch --timeout=120s
kubectl -n vols exec scratch -- grep /scratch /proc/mounts"""),
step("One directory from three sources",
r"""Create Pod `bundle` in `vols` (`busybox:1.36`, `sleep 86400`) with **one** `projected` volume mounted at `/etc/bundle`, made of:
- ConfigMap `app-config` (gives the file `mode`),
- Secret `app-secret` (gives `password`),
- the downward API: the Pod's own name as the file `podname`.
Separate volumes would need three mount points; a projected volume puts them side by side.""",
check=r"""
x() { $K -n vols exec bundle -- cat "/etc/bundle/$1" 2>/dev/null; }
[ -n "$(j -n vols pod bundle '{.spec.volumes[0].projected}')" ] && eq "$(x mode)" prod && eq "$(x password)" s3cr3t && eq "$(x podname)" bundle""",
hint=r"""`projected: {sources: [...]}` - each source is `configMap: {name: ...}`, `secret: {name: ...}` or `downwardAPI: {items: [{path: podname, fieldRef: {fieldPath: metadata.name}}]}`. `kubectl explain pod.spec.volumes.projected.sources` lists them.""",
show=r"""kubectl -n vols apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: bundle
spec:
containers:
- name: app
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- {name: bundle, mountPath: /etc/bundle, readOnly: true}
volumes:
- name: bundle
projected:
sources:
- configMap: {name: app-config}
- secret: {name: app-secret}
- downwardAPI:
items:
- path: podname
fieldRef: {fieldPath: metadata.name}
EOF
kubectl -n vols wait --for=condition=Ready pod/bundle --timeout=120s
kubectl -n vols exec bundle -- ls /etc/bundle"""),
step("A volume for exactly one Pod",
r"""Create PersistentVolume `archive`: `hostPath` `/srv/archive` (type `DirectoryOrCreate`), `200Mi`, access mode **`ReadWriteOncePod`**, reclaim policy `Retain`, and `storageClassName: ""`.
Then claim it: PVC `archive-claim` in `vols`, `ReadWriteOncePod`, `200Mi`, `storageClassName: ""`.
The empty class matters on both sides. A claim with no class at all gets the default class, and its provisioner would make a fresh volume instead of binding yours.""",
check=r"""
eq "$(j pv archive '{.spec.accessModes[0]} {.spec.persistentVolumeReclaimPolicy} {.spec.hostPath.path}')" "ReadWriteOncePod Retain /srv/archive" \
&& eq "$(j -n vols pvc archive-claim '{.status.phase} {.spec.volumeName}')" "Bound archive" """,
hint=r"""`storageClassName: ""` is spelled out, with the quotes, under `spec` of both objects. A PersistentVolume is cluster-wide: no namespace.""",
show=r"""kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: archive
spec:
capacity: {storage: 200Mi}
accessModes: [ReadWriteOncePod]
persistentVolumeReclaimPolicy: Retain
storageClassName: ""
hostPath: {path: /srv/archive, type: DirectoryOrCreate}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archive-claim
namespace: vols
spec:
storageClassName: ""
accessModes: [ReadWriteOncePod]
resources:
requests: {storage: 200Mi}
EOF
kubectl get pv archive
kubectl -n vols get pvc archive-claim"""),
step("Hand a Released volume to a new claim",
r"""Delete claim `archive-claim`. The volume becomes `Released`: it still remembers its old claim in `spec.claimRef`, and nothing binds it while it does.
Make it `Available` again by removing `spec.claimRef` from the PV, then create claim `archive-claim-2` in `vols` (same modes, size and empty class). It binds `archive`, and the data on it is still there.""",
check=r"""
[ -z "$(j -n vols pvc archive-claim '{.metadata.name}')" ] && eq "$(j -n vols pvc archive-claim-2 '{.status.phase} {.spec.volumeName}')" "Bound archive" """,
hint=r"""`kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'` - or `kubectl edit pv archive` and delete the `claimRef:` block.""",
show=r"""kubectl -n vols delete pvc archive-claim
kubectl get pv archive
kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archive-claim-2
namespace: vols
spec:
storageClassName: ""
accessModes: [ReadWriteOncePod]
resources:
requests: {storage: 200Mi}
EOF
kubectl get pv archive"""),
],
"outro": r"""You shared an emptyDir between containers, put one in memory, combined three sources in one projected volume, and took a `Retain` volume through `Bound`, `Released` and back.
**For the exam**
- `ReadWriteOnce` is one **node**, not one Pod; `ReadWriteOncePod` is one Pod. A claim's modes must be a subset of the volume's.
- `Retain` keeps the data and leaves the volume `Released`; to reuse it, remove `spec.claimRef` (or delete and re-create the PV). `Delete` removes the volume with its claim. `Recycle` is deprecated.
- `storageClassName: ""` binds a pre-made volume; leaving the field out means "the default class".
- configMap and secret volumes follow changes to their object after a short delay - but not when mounted with `subPath`, and environment variables never do.
- `kubectl explain pod.spec.volumes` lists every volume type; the docs page "Volumes" has an example of each.
Next, prove it against the clock: **drill 1.2**.""",
}
# ---- 1.3 PersistentVolumes and PersistentVolumeClaims ---------------------------------------
_PVS = r"""
$K apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-small
labels: {tier: bronze}
spec:
capacity: {storage: 100Mi}
accessModes: [ReadWriteOnce]
persistentVolumeReclaimPolicy: Retain
storageClassName: manual
hostPath: {path: /srv/pv-small, type: DirectoryOrCreate}
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-large
labels: {tier: gold}
spec:
capacity: {storage: 1Gi}
accessModes: [ReadWriteOnce]
persistentVolumeReclaimPolicy: Retain
storageClassName: manual
hostPath: {path: /srv/pv-large, type: DirectoryOrCreate}
EOF
"""
L13 = {
"id": 9103, "objective": "1.3", "workers": 0,
"title": "Claims: binding the right volume, protection, and claims that come and go",
"intro": r"""A claim binds **one** volume, and keeps it until the claim is deleted. Among the volumes that fit - access modes, class, at least the requested size - Kubernetes picks the smallest. You can steer it:
```
selector: matchLabels on the volume's labels "one of the gold volumes"
volumeName: the volume's name "exactly this volume"
```
Claims are also protected: one that a Pod still uses is not removed until the Pod is gone. And a StatefulSet makes claims of its own, one per replica, from `volumeClaimTemplates` - its `persistentVolumeClaimRetentionPolicy` says whether they go when replicas go.
Some things only a **CSI driver** can do for a claim: grow it, snapshot it, make a new claim from a snapshot or from another claim. This cluster has the CSI hostpath driver and its class `csi-hostpath-sc` (expandable), as the exam's clusters do, besides kind's `standard`.
Two volumes are ready, both of class `manual`: `pv-small` (100Mi, label `tier=bronze`) and `pv-large` (1Gi, `tier=gold`). Namespace `claims` has claim `growing` (`csi-hostpath-sc`, 10Mi), used by Pod `grower`.""",
"setup": NS + _PVS + r"""
ensure_csi_hostpath || { echo "could not install the CSI hostpath driver" >&2; exit 1; }
mkns claims
mkdir -p "$WS/lesson-1.3"
$K -n claims apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: growing}
spec: {storageClassName: csi-hostpath-sc, accessModes: [ReadWriteOnce], resources: {requests: {storage: 10Mi}}}
---
apiVersion: v1
kind: Pod
metadata: {name: grower}
spec:
containers:
- {name: grower, image: busybox:1.36, command: [sh, -c, 'echo kept > /data/note.txt; sync; sleep 86400'], volumeMounts: [{name: d, mountPath: /data}]}
volumes: [{name: d, persistentVolumeClaim: {claimName: growing}}]
EOF
$K -n claims wait --for=condition=Ready pod/grower --timeout=180s >/dev/null
""",
"steps": [
step("Bind by label",
r"""Create claim `gold-claim` in `claims`: class `manual`, `ReadWriteOnce`, `100Mi` - and a **selector** for `tier=gold`.
Without the selector it would bind `pv-small`, the smallest volume that fits. With it, it binds `pv-large`.""",
check=r"""eq "$(j -n claims pvc gold-claim '{.status.phase} {.spec.volumeName}')" "Bound pv-large" """,
hint=r"""`spec.selector.matchLabels: {tier: gold}` on the claim.""",
show=r"""kubectl -n claims apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gold-claim
spec:
storageClassName: manual
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 100Mi}
selector:
matchLabels: {tier: gold}
EOF
kubectl get pv"""),
step("Bind by name, and use it",
r"""Create claim `pinned` in `claims` that binds `pv-small` **by name** (class `manual`, `ReadWriteOnce`, `100Mi`), and Pod `user` (`busybox:1.36`, `sleep 86400`) that mounts it at `/data`.""",
check=r"""eq "$(j -n claims pvc pinned '{.status.phase} {.spec.volumeName}')" "Bound pv-small" && eq "$(j -n claims pod user '{.status.phase}')" Running \
&& eq "$(j -n claims pod user '{.spec.volumes[0].persistentVolumeClaim.claimName}')" pinned""",
hint=r"""`spec.volumeName: pv-small` on the claim. The volume's class and modes must still match the claim's.""",
show=r"""kubectl -n claims apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pinned
spec:
storageClassName: manual
volumeName: pv-small
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 100Mi}
---
apiVersion: v1
kind: Pod
metadata:
name: user
spec:
containers:
- name: user
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- {name: data, mountPath: /data}
volumes:
- name: data
persistentVolumeClaim: {claimName: pinned}
EOF
kubectl -n claims wait --for=condition=Ready pod/user --timeout=120s
kubectl -n claims get pvc"""),
step("A claim in use does not go",
r"""Delete claim `pinned` without waiting (`--wait=false`). It does not disappear: `kubectl -n claims get pvc` shows it `Terminating`, held by a **finalizer**.
Write that finalizer's name to `{WS}/lesson-1.3/finalizer.txt`.""",
check=r"""
[ -n "$(j -n claims pvc pinned '{.metadata.deletionTimestamp}')" ] \
&& eq "$(tr -d ' \r\n' < "$WS/lesson-1.3/finalizer.txt" 2>/dev/null)" kubernetes.io/pvc-protection""",
hint=r"""`kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers}'`. Without `--wait=false`, `kubectl delete` would sit there until the claim is really gone.""",
show=r"""kubectl -n claims delete pvc pinned --wait=false
kubectl -n claims get pvc pinned
kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers[0]}' > {WS}/lesson-1.3/finalizer.txt
cat {WS}/lesson-1.3/finalizer.txt"""),
step("Let it go",
r"""Delete Pod `user`. With no Pod using it any more, the protection lifts and claim `pinned` is removed. `pv-small` (reclaim policy `Retain`) is now `Released`.""",
check=r"""[ -z "$(j -n claims pod user '{.metadata.name}')" ] && [ -z "$(j -n claims pvc pinned '{.metadata.name}')" ] && eq "$(j pv pv-small '{.status.phase}')" Released""",
hint=r"""`kubectl -n claims delete pod user`, then `kubectl -n claims get pvc` and `kubectl get pv`.""",
show=r"""kubectl -n claims delete pod user --grace-period=1
kubectl -n claims get pvc
kubectl get pv pv-small"""),
step("Claims made by a StatefulSet",
r"""Create a StatefulSet `db` in `claims` with 2 replicas (`busybox:1.36`, `sleep 86400`) and its headless Service `db`. Give it a `volumeClaimTemplates` entry `data` (`ReadWriteOnce`, `50Mi`, class `standard`) mounted at `/var/lib/db`, and a `persistentVolumeClaimRetentionPolicy` that **deletes** a replica's claim when the StatefulSet is scaled down and **retains** the claims when the StatefulSet is deleted.
Each replica gets a claim of its own: `data-db-0`, `data-db-1`.""",
check=r"""
eq "$(j -n claims sts db '{.spec.persistentVolumeClaimRetentionPolicy.whenScaled} {.spec.persistentVolumeClaimRetentionPolicy.whenDeleted}')" "Delete Retain" || exit 1
eq "$(j -n claims sts db '{.status.readyReplicas}')" 2 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound && eq "$(j -n claims pvc data-db-1 '{.status.phase}')" Bound""",
hint=r"""`kubectl create deploy db --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400` is a start: change the kind, add `serviceName: db`, drop `strategy`, add `volumeClaimTemplates` beside `template`. The Service: `clusterIP: None`. `kubectl explain sts.spec.persistentVolumeClaimRetentionPolicy`.""",
show=r"""kubectl -n claims apply -f - <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: db
spec:
clusterIP: None
selector: {app: db}
ports:
- port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
serviceName: db
replicas: 2
selector:
matchLabels: {app: db}
persistentVolumeClaimRetentionPolicy:
whenScaled: Delete
whenDeleted: Retain
template:
metadata:
labels: {app: db}
spec:
containers:
- name: db
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- {name: data, mountPath: /var/lib/db}
volumeClaimTemplates:
- metadata:
name: data
spec:
storageClassName: standard
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
EOF
kubectl -n claims rollout status sts db --timeout=180s
kubectl -n claims get pvc"""),
step("Scale down, and the claim goes with the replica",
r"""Scale `db` to 1 replica. Pod `db-1` goes, and because `whenScaled` is `Delete`, so does `data-db-1`. `data-db-0` stays.
The default for both fields is `Retain`: without the policy, `data-db-1` would wait for the replica to come back.""",
check=r"""
eq "$(j -n claims sts db '{.spec.replicas}')" 1 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound || exit 1
for i in $(seq 1 10); do [ -z "$(j -n claims pvc data-db-1 '{.metadata.name}')" ] && exit 0; sleep 3; done; exit 1""",
hint=r"""`kubectl -n claims scale sts db --replicas=1`, then `kubectl -n claims get pvc -w`.""",
show=r"""kubectl -n claims scale sts db --replicas=1
kubectl -n claims rollout status sts db --timeout=120s
kubectl -n claims get pvc"""),
step("Grow a claim in use",
r"""Claim `growing` is full. Its class allows expansion (`kubectl get sc csi-hostpath-sc`: `ALLOWVOLUMEEXPANSION true`), so make it **50Mi** - while Pod `grower` keeps running. Raise the request; the driver does the rest, and `status.capacity` follows.""",
check=r"""eq "$(j -n claims pvc growing '{.spec.resources.requests.storage} {.status.capacity.storage}')" "50Mi 50Mi" && eq "$(j -n claims pod grower '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """,
hint=r"""`kubectl -n claims edit pvc growing`, or `kubectl patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}'`. A claim never shrinks; with `standard` (local-path) the same patch is refused.""",
show=r"""kubectl get sc csi-hostpath-sc
kubectl -n claims patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}'
sleep 10
kubectl -n claims get pvc growing"""),
step("Snapshot it, and restore the snapshot",
r"""Take VolumeSnapshot `growing-snap` of claim `growing` (snapshot class `csi-hostpath-snapclass`), wait until it is ready to use, then create claim `restored` (`csi-hostpath-sc`, 50Mi) **from** the snapshot: its `dataSource` names the VolumeSnapshot. The new volume holds what `growing` held when the snapshot was taken.""",
check=r"""
eq "$(j -n claims volumesnapshot growing-snap '{.spec.source.persistentVolumeClaimName} {.status.readyToUse}')" "growing true" && eq "$(j -n claims pvc restored '{.spec.dataSource.kind}/{.spec.dataSource.name} {.status.phase}')" "VolumeSnapshot/growing-snap Bound" """,
hint=r"""`apiVersion: snapshot.storage.k8s.io/v1`, `kind: VolumeSnapshot`, `spec: {volumeSnapshotClassName: ..., source: {persistentVolumeClaimName: growing}}`. The claim: `dataSource: {apiGroup: snapshot.storage.k8s.io, kind: VolumeSnapshot, name: growing-snap}`. A clone is the same with `dataSource: {kind: PersistentVolumeClaim, name: growing}`.""",
show=r"""kubectl -n claims apply -f - <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: growing-snap
spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: growing
EOF
kubectl -n claims wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/growing-snap --timeout=120s
kubectl -n claims apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: restored
spec:
storageClassName: csi-hostpath-sc
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: growing-snap
EOF
kubectl -n claims get volumesnapshot,pvc"""),
],
"outro": r"""You steered binding with a selector and with a volume's name, watched claim protection hold a claim until its Pod went, let a StatefulSet's claims follow its replicas, and grew, snapshotted and restored a claim on a CSI class.
**For the exam**
- A claim binds the smallest volume that fits, unless `selector` or `volumeName` say otherwise. The class must match on both sides.
- A `Pending` claim: `describe pvc` - no volume fits (size, modes, class, selector), the class has no provisioner, or it waits for its first consumer.
- `Terminating` claim: some Pod still uses it (`kubernetes.io/pvc-protection`). Find the Pod, not the finalizer.
- StatefulSet claims are named `TEMPLATE-STATEFULSET-ORDINAL` and kept by default; `persistentVolumeClaimRetentionPolicy` (`whenScaled`, `whenDeleted`) can delete them.
- A Pod's `ephemeral.volumeClaimTemplate` makes a claim that lives and dies with the Pod - a generic ephemeral volume, named `POD-VOLUME`.
- Expansion, snapshots and clones need a CSI driver that does them: `kubectl get sc` (ALLOWVOLUMEEXPANSION), `kubectl get volumesnapshotclass`, `kubectl get csidrivers`.
Next, prove it against the clock: **drill 1.3**.""",
}
LESSONS = [L11, L12, L13]

551
cka/lessons_tshoot.py Normal file
View File

@ -0,0 +1,551 @@
"""CKA lessons of the Troubleshooting domain (2.1-2.5). See cka/lessons.py for what a lesson is."""
from sim.bank import NS, step
# ---- 2.1 clusters and nodes -------------------------------------------------------------------
L21 = {
"id": 9201, "objective": "2.1", "workers": 1,
"title": "A node goes NotReady: the kubelet, its configuration, and pressure",
"intro": r"""A node is `Ready` while its **kubelet** reports in. When it stops, the control plane waits under a minute (its node monitor grace period) and marks the node `NotReady`; five minutes later - the default toleration for an unreachable node - its Pods are evicted. So for a `NotReady` node the questions go in this order:
```
kubectl get nodes / describe node what the control plane last heard
ssh NODE; sudo -i
systemctl status kubelet is it running, is it enabled?
journalctl -u kubelet -n 50 why it stopped, or why it will not start
/var/lib/kubelet/config.yaml its configuration (KubeletConfiguration)
/etc/kubernetes/kubelet.conf how it reaches the API server
/etc/systemd/system/kubelet.service.d/ its flags (drop-ins), and /var/lib/kubelet/kubeadm-flags.env
systemctl status containerd the runtime it needs
```
The kubelet also watches the node's resources. Below a **hard eviction threshold** - `memory.available`, `nodefs.available`, ... - it sets a condition (`MemoryPressure`, `DiskPressure`, `PIDPressure`) and evicts Pods to recover.
Something has happened to the kubelet on worker `{W1}`.""",
"setup": r"""
mkdir -p "$WS/lesson-2.1"
nx "$W1" 'systemctl disable --now kubelet' </dev/null >/dev/null 2>&1
# the control plane notices after its grace period; the lesson starts with the node NotReady
for i in $(seq 1 40); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" != True ] && exit 0; sleep 3; done
echo "$W1 is still Ready a minute after its kubelet was stopped" >&2; exit 1
""",
"steps": [
step("Which node, and since when",
r"""Find the node that is not `Ready` and write its name to `{WS}/lesson-2.1/node.txt`.
Then look at `kubectl describe node` for it: under **Conditions** every row says `Kubelet stopped posting node status` - the control plane has not heard from the kubelet, so it knows nothing more. The answer is on the node.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/node.txt" 2>/dev/null)" "$W1" """,
hint=r"""`kubectl get nodes`. `--no-headers` and `awk '$2 != "Ready" {print $1}'` give just the name.""",
show=r"""kubectl get nodes
kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt
kubectl describe node {W1} | grep -A8 Conditions"""),
step("Bring the kubelet back - for good",
r"""On `{W1}` (`ssh {W1}`, then `sudo -i`), look at the kubelet: `systemctl status kubelet` says `inactive (dead)` and `disabled`. Start it, and enable it so that it survives a reboot - a fix that is gone after the next reboot is not a fix.
Within a few seconds the node is `Ready` again.""",
check=r"""
nx "$W1" 'systemctl is-active -q kubelet && systemctl is-enabled -q kubelet' </dev/null || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""`systemctl enable --now kubelet` does both at once. `journalctl -u kubelet -n 20` tells you when it is running happily.""",
show=r"""ssh {W1} sudo bash <<'EOF'
systemctl status kubelet --no-pager | head -4
systemctl enable --now kubelet
systemctl is-active kubelet; systemctl is-enabled kubelet
EOF
kubectl get nodes"""),
step("Where the kubelet's configuration is",
r"""A kubelet reads its settings from the file named by its `--config` flag, and the kubeadm flags from `/var/lib/kubelet/kubeadm-flags.env`. On `{W1}`, find the file (`ps -o args= -C kubelet`, or the drop-in under `/etc/systemd/system/kubelet.service.d/`).
The kubelet also tells the API server what it actually runs with: `kubectl get --raw /api/v1/nodes/{W1}/proxy/configz`. Write the live value of its `cgroupDriver` to `{WS}/lesson-2.1/cgroup.txt`.""",
check=r"""
want=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["cgroupDriver"])')
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/cgroup.txt" 2>/dev/null)" "$want" """,
hint=r"""`configz` returns JSON: `| jq -r .kubeletconfig.cgroupDriver`. It must match containerd's `SystemdCgroup` setting, or Pods fail to start.""",
show=r"""ssh -n {W1} 'ps -o args= -C kubelet | tr " " "\n" | grep -- --config; sudo grep -n cgroupDriver /var/lib/kubelet/config.yaml'
kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt
cat {WS}/lesson-2.1/cgroup.txt"""),
step("Set the eviction thresholds",
r"""Look at `evictionHard` in `/var/lib/kubelet/config.yaml` on `{W1}`: kind sets every threshold to `0%`, which means "never evict" - fine for a test cluster, not for a real one.
On `{W1}`, make the kubelet evict when less than **`100Mi`** of memory or **`10%`** of the node filesystem is left: `memory.available: 100Mi` and `nodefs.available: 10%` under `evictionHard`. Restart the kubelet, and confirm it through `configz`.""",
check=r"""
$K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c '
import json, sys
e = json.load(sys.stdin)["kubeletconfig"].get("evictionHard") or {}
sys.exit(0 if e.get("memory.available") == "100Mi" and e.get("nodefs.available") == "10%" else 1)' || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""Edit the file with `vim`, or `sed`: change the `nodefs.available` line and add a `memory.available` line inside the `evictionHard:` block (two spaces of indent). A kubelet that does not come back after the restart: `journalctl -u kubelet -n 20` shows the line it could not parse.""",
show=r"""ssh {W1} sudo bash <<'EOF'
f=/var/lib/kubelet/config.yaml
grep -n -A4 '^evictionHard:' $f
sed -i 's/^ nodefs.available: .*/ nodefs.available: 10%/' $f
sed -i '/^evictionHard:/a\ memory.available: 100Mi' $f
grep -n -A5 '^evictionHard:' $f
systemctl restart kubelet
EOF
kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq .kubeletconfig.evictionHard"""),
step("Read the pressure conditions",
r"""The kubelet reports the result of those thresholds as node conditions. Write the status of `{W1}`'s three pressure conditions to `{WS}/lesson-2.1/pressure.txt`, one per line, as `TYPE STATUS` - for example `MemoryPressure False`.
`True` on any of them means the kubelet is evicting Pods, and the node gets a taint (`node.kubernetes.io/memory-pressure`, ...) so no new Pods land there.""",
check=r"""
want=$(for c in MemoryPressure DiskPressure PIDPressure; do echo "$c $(j node "$W1" "{.status.conditions[?(@.type==\"$c\")].status}")"; done | sort)
eq "$(tr -s ' \t' ' ' < "$WS/lesson-2.1/pressure.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """,
hint=r"""`kubectl describe node {W1}` lists them under Conditions. As a list: `-o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}'`, then keep the three `*Pressure` lines.""",
show=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt
cat {WS}/lesson-2.1/pressure.txt"""),
],
"outro": r"""You found a `NotReady` node, brought its kubelet back for good, read its configuration from the file and from the running kubelet, set eviction thresholds and read what the kubelet reports about them.
**For the exam**
- `NotReady`: the node is the place to look. `systemctl status kubelet`, then `journalctl -u kubelet`. A kubelet that restarts in a loop prints the reason a moment before each exit.
- Fixes stay fixed: `systemctl enable`, and edits in the files, not on the command line.
- Usual faults: kubelet stopped or disabled, a wrong path in `config.yaml` (CA file, static Pod path), a wrong flag in a drop-in or `kubeadm-flags.env` (`systemctl daemon-reload` after editing a unit), containerd down or the wrong runtime endpoint, an expired client certificate.
- `kubectl get --raw /api/v1/nodes/NODE/proxy/configz` is the kubelet's live configuration.
- Eviction: `evictionHard` (and `evictionSoft` with a grace period) in `config.yaml`. Pressure conditions taint the node; evicted Pods show `Failed` with reason `Evicted`.
Next, prove it against the clock: **drill 2.1**.""",
}
# ---- 2.2 cluster components -------------------------------------------------------------------
_BREAK_CP = r"""
mkns ops
mkdir -p "$WS/lesson-2.2"
gone() { local i; for i in $(seq 1 45); do [ -z "$(nx "$CP" "crictl ps --name $1 -q" </dev/null)" ] && return 0; sleep 2; done; return 1; }
# the controller manager first: it must be down before the Deployment exists, or it makes its Pods
nx "$CP" 'sed -i "s|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)$|\1-broken|" /etc/kubernetes/manifests/kube-controller-manager.yaml' </dev/null
gone kube-controller-manager || { echo "kube-controller-manager is still running" >&2; exit 1; }
$K -n ops create deploy queue --image=busybox:1.36 --replicas=2 -- sleep 86400 >/dev/null
nx "$CP" 'sed -i "s|^\( *\)- kube-scheduler$|\1- kube-schedulerr|" /etc/kubernetes/manifests/kube-scheduler.yaml' </dev/null
gone kube-scheduler || { echo "kube-scheduler is still running" >&2; exit 1; }
$K -n ops run waiting --image=busybox:1.36 -- sleep 86400 >/dev/null
"""
L22 = {
"id": 9202, "objective": "2.2", "workers": 1,
"title": "The control plane's own Pods: find what is down and why",
"intro": r"""On a kubeadm cluster the control plane runs as **static Pods**: the kubelet of the control-plane node starts whatever manifests it finds in `/etc/kubernetes/manifests`, and restarts them when a file changes. Each component has its own symptom when it is down:
```
kube-apiserver kubectl itself fails: "connection refused"
etcd the API server fails, then kubectl
kube-scheduler new Pods stay Pending, with no node and no event
kube-controller-manager Deployments get no ReplicaSet, ReplicaSets no Pods; nothing self-heals
```
When the API server is up, `kubectl -n kube-system get pods` and `describe` show the others. When it is not, go to the node: `crictl ps -a`, `crictl logs ID`, the files under `/var/log/pods/`, and `journalctl -u kubelet`.
Two things in Namespace `ops` do not work: Deployment `queue` has no Pods, and Pod `waiting` has no node.""",
"setup": NS + _BREAK_CP,
"steps": [
step("Which components are down",
r"""Find the control-plane components that are not running and write their names to `{WS}/lesson-2.2/down.txt`, one per line (for example `kube-proxy`).
`kubectl -n kube-system get pods` shows them; on the control-plane node, `sudo crictl ps -a` shows their containers, and `ls /etc/kubernetes/manifests` what the kubelet is told to run.""",
check=r"""eq "$(tr -d ' \r' < "$WS/lesson-2.2/down.txt" 2>/dev/null | grep . | sort | tr '\n' ' ')" "kube-controller-manager kube-scheduler " """,
hint=r"""A Pod that is not `Running` with all its containers ready. The Pod names end in the node's name; the component is the part before it.""",
show=r"""kubectl -n kube-system get pods
ls /etc/kubernetes/manifests
sudo crictl ps -a | grep kube-
printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt"""),
step("Repair the scheduler",
r"""Find out why `kube-scheduler` does not start, and fix its manifest. `kubectl -n kube-system describe pod kube-scheduler-{CP}` shows the error in its events, and so do `sudo crictl ps -a --name kube-scheduler` and `sudo journalctl -u kubelet`.
The kubelet notices the saved file on its own: no restart, no `kubectl apply`. Once the scheduler runs, Pod `waiting` gets a node.""",
check=r"""
nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml && [ -n "$(crictl ps --name kube-scheduler -q)" ]' </dev/null || exit 1
[ -n "$(j -n ops pod waiting '{.spec.nodeName}')" ]""",
hint=r"""Look at the first entry of the container's `command:` - the program the container runs. A static Pod's manifest is edited in place (`sudo vim`), never with `kubectl edit`.""",
show=r"""kubectl -n kube-system describe pod kube-scheduler-{CP} | tail -5
sudo grep -n -A2 'command:' /etc/kubernetes/manifests/kube-scheduler.yaml
sudo sed -i 's/- kube-schedulerr$/- kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
sudo crictl ps --name kube-scheduler
kubectl -n ops get pod waiting -o wide"""),
step("Repair the controller manager",
r"""Now `kube-controller-manager`. Its Pod is not crashing - its container never starts. `describe` on it tells you which step fails. Fix the manifest; once it runs, Deployment `queue` gets its ReplicaSet and both Pods.""",
check=r"""
nx "$CP" '! grep -q -- "-broken" /etc/kubernetes/manifests/kube-controller-manager.yaml && [ -n "$(crictl ps --name kube-controller-manager -q)" ]' </dev/null || exit 1
eq "$(j -n ops deploy queue '{.status.availableReplicas}')" 2""",
hint=r"""`ErrImagePull` / `ImagePullBackOff`: the image tag does not exist. Every control-plane image runs the cluster's version - compare with `kube-apiserver.yaml`.""",
show=r"""kubectl -n kube-system describe pod kube-controller-manager-{CP} | tail -5
sudo grep -n 'image:' /etc/kubernetes/manifests/*.yaml
sudo sed -i 's/-broken$//' /etc/kubernetes/manifests/kube-controller-manager.yaml
sudo crictl ps --name kube-controller-manager
kubectl -n ops get deploy queue"""),
step("Ask the API server how it feels",
r"""The API server has health endpoints of its own: `/livez` (restart me if this fails) and `/readyz` (send me traffic). With `?verbose` they list every check.
Save the output of `kubectl get --raw '/readyz?verbose'` in `{WS}/lesson-2.2/readyz.txt`.""",
check=r"""grep -q '^\[+\]etcd ok' "$WS/lesson-2.2/readyz.txt" 2>/dev/null && grep -q 'readyz check passed' "$WS/lesson-2.2/readyz.txt" """,
hint=r"""`kubectl get --raw` sends a GET to any API path, with your credentials. Quote the path: `?` means something to the shell.""",
show=r"""kubectl get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt | tail -3
kubectl get --raw '/livez'; echo"""),
],
"outro": r"""You read the control plane from both sides - `kubectl` and the node's container runtime - and repaired two static Pods by editing their manifests in place.
**For the exam**
- Static Pod manifests: `/etc/kubernetes/manifests` (the kubelet's `staticPodPath`). Edit the file; the kubelet restarts the Pod. Keep a copy *outside* that directory - a backup inside it is started too.
- Symptoms: no scheduler, Pods `Pending` with no events; no controller manager, Deployments without ReplicaSets; no API server, `kubectl` refuses to connect - then `crictl ps -a`, `crictl logs`, `/var/log/pods/kube-system_kube-apiserver-*/`.
- Typical faults: a wrong image tag, a typo in the command or a flag, a wrong path to a certificate, a volume mount that does not match a `hostPath`.
- `kubectl get --raw '/readyz?verbose'` and `/livez` show the API server's own checks, etcd's among them.
Next, prove it against the clock: **drill 2.2**.""",
}
# ---- 2.3 resource usage -----------------------------------------------------------------------
L23 = {
"id": 9203, "objective": "2.3", "workers": 1,
"title": "What uses the CPU and memory: kubectl top and the metrics pipeline",
"intro": r"""`kubectl top` shows what Pods and nodes **use** right now. The numbers come from **metrics-server**, which asks every kubelet every 15 seconds and serves the result as an API of its own, `metrics.k8s.io`:
```
kubelet (cAdvisor) ──▶ metrics-server ──▶ /apis/metrics.k8s.io ──▶ kubectl top, HorizontalPodAutoscaler
```
Without metrics-server there is no `kubectl top` and no autoscaling on CPU or memory. Usage is not the same as **requests**: requests are what the scheduler reserves, and `kubectl describe node` shows them under *Allocated resources*.
metrics-server runs on this cluster. Namespace `metrics` has a few Pods: one of them burns CPU, one holds a lot of memory, and Pod `duo` has two containers.""",
"setup": NS + r"""
ensure_metrics_server || { echo "metrics-server did not become ready" >&2; exit 1; }
mkns metrics
mkdir -p "$WS/lesson-2.3"
$K -n metrics apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: burner}
spec:
containers:
- name: burner
image: busybox:1.36
command: [sh, -c, 'while true; do :; done']
resources: {limits: {cpu: 300m, memory: 32Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: hoarder}
spec:
containers:
- name: hoarder
image: busybox:1.36
command: [sh, -c, 'dd if=/dev/zero of=/cache/blob bs=1M count=90 2>/dev/null; sleep 86400']
volumeMounts: [{name: cache, mountPath: /cache}]
volumes: [{name: cache, emptyDir: {medium: Memory}}]
---
apiVersion: v1
kind: Pod
metadata: {name: idler}
spec:
containers:
- {name: idler, image: busybox:1.36, command: [sleep, '86400']}
---
apiVersion: v1
kind: Pod
metadata: {name: duo}
spec:
containers:
- {name: app, image: busybox:1.36, command: [sleep, '86400']}
- name: cache
image: busybox:1.36
command: [sh, -c, 'dd if=/dev/zero of=/cache/blob bs=1M count=40 2>/dev/null; sleep 86400']
volumeMounts: [{name: cache, mountPath: /cache}]
volumes: [{name: cache, emptyDir: {medium: Memory}}]
EOF
$K -n metrics wait --for=condition=Ready pod --all --timeout=180s >/dev/null
# until metrics-server has a sample of every Pod
for i in $(seq 1 40); do [ "$($K top pod -n metrics --no-headers 2>/dev/null | grep -c .)" -ge 4 ] && exit 0; sleep 5; done
echo "kubectl top shows no metrics for the lesson's Pods" >&2; exit 1
""",
"steps": [
step("The heaviest Pod by memory",
r"""Write the name of the Pod in `metrics` that uses the most memory to `{WS}/lesson-2.3/memory.txt`. `kubectl top` can sort for you.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/memory.txt" 2>/dev/null)" hoarder""",
hint=r"""`kubectl top pod -n metrics --sort-by=memory`. `--no-headers | head -1 | awk '{print $1}'` keeps just the name.""",
show=r"""kubectl top pod -n metrics --sort-by=memory
kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt"""),
step("The busiest Pod in the cluster",
r"""Across **all** Namespaces, which Pod uses the most CPU? Write it as `NAMESPACE/NAME` to `{WS}/lesson-2.3/cpu.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/cpu.txt" 2>/dev/null)" metrics/burner""",
hint=r"""`kubectl top pod -A --sort-by=cpu`; with `-A` the first column is the Namespace.""",
show=r"""kubectl top pod -A --sort-by=cpu | head -5
kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt"""),
step("Inside one Pod",
r"""`kubectl top pod` adds up a Pod's containers. Which container of Pod `duo` uses the memory? Write its name to `{WS}/lesson-2.3/container.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/container.txt" 2>/dev/null)" cache""",
hint=r"""`--containers` splits the Pod into its containers.""",
show=r"""kubectl top pod duo -n metrics --containers
kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt"""),
step("Usage against what is reserved",
r"""`kubectl top node` shows use; `kubectl describe node {W1}`, under **Allocated resources**, shows what Pods have **requested** - which is what decides whether another Pod fits.
Write the memory that `{W1}` can give to Pods - its **allocatable** memory, as the API reports it (for example `8012345Ki`) - to `{WS}/lesson-2.3/allocatable.txt`.""",
check=r"""want=$(j node "$W1" '{.status.allocatable.memory}'); [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/allocatable.txt" 2>/dev/null)" "$want" """,
hint=r"""Allocatable is capacity minus what the node keeps for itself and for the eviction threshold: `.status.allocatable.memory`.""",
show=r"""kubectl top node
kubectl describe node {W1} | grep -A9 'Allocated resources'
kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt"""),
step("The metrics API itself",
r"""`kubectl top` only reads an API. Read it yourself: save what `/apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder` returns in `{WS}/lesson-2.3/raw.json`.
If this path answers `NotFound` or `ServiceUnavailable` on some cluster, `kubectl top` fails there too: look at the `metrics-server` Deployment in `kube-system`, and at `kubectl get apiservice v1beta1.metrics.k8s.io`.""",
check=r"""python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); sys.exit(0 if d.get("kind")=="PodMetrics" and d["metadata"]["name"]=="hoarder" and d.get("containers") else 1)' "$WS/lesson-2.3/raw.json" 2>/dev/null""",
hint=r"""`kubectl get --raw PATH`.""",
show=r"""kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json
jq '.containers' {WS}/lesson-2.3/raw.json
kubectl get apiservice v1beta1.metrics.k8s.io"""),
],
"outro": r"""You ranked Pods by memory and CPU, looked inside a Pod, compared use with what a node can give, and read the metrics API that `kubectl top` uses.
**For the exam**
- `kubectl top pod [-A] [-n NS] [-l SELECTOR] --sort-by=cpu|memory [--containers]`, `kubectl top node`. Write exactly what is asked - often just the name.
- No metrics: is metrics-server running, and is `apiservice v1beta1.metrics.k8s.io` `Available`? On kubeadm clusters it often needs `--kubelet-insecure-tls`.
- `top` is usage; `describe node` → *Allocated resources* is requests and limits. The scheduler goes by requests.
- Metrics lag by a few seconds and are averages: a Pod that has just started may not appear yet.
Next, prove it against the clock: **drill 2.3**.""",
}
# ---- 2.4 container output streams ------------------------------------------------------------
L24 = {
"id": 9204, "objective": "2.4", "workers": 1,
"title": "Logs, the previous container, the node's log files, and debug containers",
"intro": r"""Everything a container writes to **stdout** and **stderr** is kept by the node's runtime and served by `kubectl logs`:
```
kubectl logs POD [-c CONTAINER] one container (-c when the Pod has several)
--previous the instance before the last restart - where a crash is explained
-l app=api --prefix every Pod of a label, each line marked with its Pod
--since=10m --tail=50 -f a window, the end, follow
```
On the node the same output lives in `/var/log/pods/NAMESPACE_POD_UID/CONTAINER/N.log` (N counts restarts), linked from `/var/log/containers/`. `crictl logs` reads it too.
And when a container has no shell to `exec` into, `kubectl debug` adds an **ephemeral container** to the running Pod - your image, your tools, sharing the Pod's network and, with `--target`, its processes. `kubectl debug node/NAME` does the same for a node.
Namespace `logs` has Pod `flaky`, which crashed once; Pods `api-1` to `api-3`; and Pod `quiet`, built from an image with no shell at all.""",
"setup": NS + r"""
mkns logs
mkdir -p "$WS/lesson-2.4"
$K -n logs apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: flaky}
spec:
volumes: [{name: state, emptyDir: {}}]
containers:
- name: flaky
image: busybox:1.36
volumeMounts: [{name: state, mountPath: /state}]
command: [sh, -c, 'if [ ! -f /state/booted ]; then touch /state/booted; echo "FATAL config /etc/flaky.conf not found"; exit 1; fi; echo "INFO started on second attempt"; exec sleep 86400']
---
apiVersion: v1
kind: Pod
metadata: {name: quiet}
spec:
containers:
- {name: quiet, image: registry.k8s.io/pause:3.10.2}
EOF
for n in 1 2 3; do
msg='INFO request served'; [ $n = 2 ] && msg='ERROR upstream refused the request'
$K -n logs run api-$n --image=busybox:1.36 --labels=app=api -- sh -c "while true; do echo \"$msg\"; sleep 5; done" >/dev/null
done
$K -n logs wait --for=condition=Ready pod --all --timeout=180s >/dev/null
for i in $(seq 1 30); do [ "$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null && exit 0; sleep 3; done
echo "flaky did not restart" >&2; exit 1
""",
"steps": [
step("Why it crashed",
r"""Pod `flaky` is `Running` now, with one restart. The logs of the running container only say it started; the reason for the crash is in the logs of the instance **before** it.
Save those in `{WS}/lesson-2.4/previous.log`.""",
check=r"""grep -q 'FATAL config /etc/flaky.conf not found' "$WS/lesson-2.4/previous.log" 2>/dev/null && ! grep -q 'second attempt' "$WS/lesson-2.4/previous.log" """,
hint=r"""`kubectl logs --previous` (or `-p`).""",
show=r"""kubectl -n logs logs flaky
kubectl -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log
cat {WS}/lesson-2.4/previous.log"""),
step("The one replica that complains",
r"""Pods `api-1`, `api-2` and `api-3` all carry `app=api`. One of them logs errors. Read all three at once, find it, and write its name to `{WS}/lesson-2.4/erroring.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/erroring.txt" 2>/dev/null)" api-2""",
hint=r"""`kubectl logs -l app=api --prefix` puts `[pod/NAME/CONTAINER]` before each line. With a selector, kubectl shows only the last 10 lines of each Pod unless you say `--tail`.""",
show=r"""kubectl -n logs logs -l app=api --prefix --tail=2
kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt
cat {WS}/lesson-2.4/erroring.txt"""),
step("The log file on the node",
r"""The node keeps one file per container instance. Write the full path of the file that holds the output of `flaky`'s **current** instance to `{WS}/lesson-2.4/logfile.txt`.
Find the node with `kubectl -n logs get pod flaky -o wide`, then look under `/var/log/pods/` there. The directory name has the Pod's UID in it; the file name is the restart count.""",
check=r"""
p=$(tr -d ' \r\n' < "$WS/lesson-2.4/logfile.txt" 2>/dev/null)
uid=$(j -n logs pod flaky '{.metadata.uid}'); n=$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}'); node=$(j -n logs pod flaky '{.spec.nodeName}')
eq "$p" "/var/log/pods/logs_flaky_$uid/flaky/$n.log" && nx "$node" "grep -q 'second attempt' '$p'" </dev/null""",
hint=r"""`/var/log/pods/NAMESPACE_POD_UID/CONTAINER/RESTARTS.log`. `sudo ls -R /var/log/pods/ | grep flaky` on the node gets you close.""",
show=r"""kubectl -n logs get pod flaky -o wide
uid=$(kubectl -n logs get pod flaky -o jsonpath='{.metadata.uid}')
ssh -n {W1} "sudo ls /var/log/pods/logs_flaky_$uid/flaky/"
echo "/var/log/pods/logs_flaky_$uid/flaky/1.log" > {WS}/lesson-2.4/logfile.txt
ssh -n {W1} "sudo cat $(cat {WS}/lesson-2.4/logfile.txt)"""),
step("Debug a container that has no shell",
r"""`kubectl -n logs exec quiet -- sh` fails: the `pause` image has nothing but its one program. Add an ephemeral container to the running Pod instead: name `dbg`, image `busybox:1.36`, **targeting** container `quiet` so that it sees its processes, running `sleep 3600`.
Then run `ps` in `dbg` and write the command of the target's process - the first one, PID 1 - to `{WS}/lesson-2.4/target.txt`.""",
check=r"""
eq "$(j -n logs pod quiet '{.spec.ephemeralContainers[0].name} {.spec.ephemeralContainers[0].targetContainerName}')" "dbg quiet" || exit 1
grep -qx '/pause' <(tr -d ' \r' < "$WS/lesson-2.4/target.txt" 2>/dev/null)""",
hint=r"""`kubectl debug POD --image=IMAGE --target=CONTAINER -c NAME -- COMMAND`. Interactively you would add `-it` and use the shell straight away; without it, `kubectl exec POD -c dbg -- ps` reaches it once it runs. Ephemeral containers cannot be removed - the Pod keeps them until it goes.""",
show=r"""kubectl -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600
for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done
kubectl -n logs exec quiet -c dbg -- ps
kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt"""),
step("Debug the node itself",
r"""`kubectl debug node/{W1} --image=busybox:1.36` starts a Pod on that node with the node's whole filesystem mounted at `/host` - useful when you cannot `ssh` to a node, but can reach the API.
Use it to read the node's `/etc/os-release`, and write the value of its `PRETTY_NAME` to `{WS}/lesson-2.4/os.txt`.""",
check=r"""
want=$(nx "$W1" '. /etc/os-release; echo "$PRETTY_NAME"' </dev/null); [ -n "$want" ] || exit 1
eq "$(tr -d '\r"' < "$WS/lesson-2.4/os.txt" 2>/dev/null | sed 's/^PRETTY_NAME=//; s/ *$//')" "$want" \
&& [ -n "$($K get pods -A -o name 2>/dev/null | grep node-debugger)" ]""",
hint=r"""Without `-it`, give it a command and read its output with `kubectl logs`: the Pod is called `node-debugger-NODE-xxxxx`, in your current Namespace. Delete it afterwards.""",
show=r"""kubectl debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release
pod=$(kubectl get pods -o name | grep node-debugger | head -1)
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s
kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt
cat {WS}/lesson-2.4/os.txt"""),
],
"outro": r"""You read a crashed instance's logs, searched several Pods at once, found the log file on the node, and debugged both a Pod without a shell and a node.
**For the exam**
- A crash is explained in `kubectl logs --previous`. `describe pod` adds the exit code and the reason (`Error`, `OOMKilled`, `Completed`).
- Several containers: `-c NAME`, or `--all-containers`. Several Pods: `-l SELECTOR --prefix`, and mind the default `--tail=10`.
- Write logs to a file exactly as asked: `> file` for stdout. A container's stderr is in its logs too - `kubectl logs` does not separate them; the node's log file does, with `stdout` / `stderr` on each line.
- `kubectl debug -it POD --image=busybox --target=CONTAINER` for a container without tools; `kubectl debug node/NAME -it --image=busybox`, then `chroot /host`, for a node.
Next, prove it against the clock: **drill 2.4**.""",
}
# ---- 2.5 services and networking --------------------------------------------------------------
L25 = {
"id": 9205, "objective": "2.5", "workers": 1,
"title": "A Service that does not answer: selector, ports, policy, and kube-proxy",
"intro": r"""A request to a Service goes through several hands, and each can drop it:
```
client ──DNS──▶ ClusterIP:port ──kube-proxy rules──▶ endpoint PodIP:targetPort ──NetworkPolicy──▶ container
```
So check them in order:
- **Endpoints**: `kubectl get endpointslices -l kubernetes.io/service-name=SVC`. None? The selector matches no ready Pod.
- **Ports**: `targetPort` must be the port the container listens on (a number, or the name of a container port).
- **NetworkPolicy**: once a policy selects a Pod, only what it allows gets in.
- **kube-proxy** turns each Service into rules on every node - iptables chains named `KUBE-SVC-...` here.
In Namespace `shop`, Deployment `web` (nginx) has a Service `web` in front of it, and nothing gets through. Pods `client` and `stranger` are there to send requests from.""",
"setup": NS + r"""
mkns shop
mkdir -p "$WS/lesson-2.5"
$K -n shop create deploy web --image=nginx:1.27 --replicas=2 --port=80 >/dev/null
$K -n shop apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: web}
spec:
selector: {app: webapp}
ports:
- {port: 80, targetPort: 8080}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: lockdown}
spec:
podSelector: {matchLabels: {app: web}}
policyTypes: [Ingress]
EOF
$K -n shop run client --image=busybox:1.36 --labels=app=client -- sleep 86400 >/dev/null
$K -n shop run stranger --image=busybox:1.36 --labels=app=stranger -- sleep 86400 >/dev/null
$K -n shop rollout status deploy web --timeout=180s >/dev/null
$K -n shop wait --for=condition=Ready pod --all --timeout=180s >/dev/null
""",
"steps": [
step("No endpoints",
r"""Ask the Service from `client`: `kubectl -n shop exec client -- wget -qO- -T3 web`. It times out. Look at the Service's endpoints: there are none, because its selector matches no Pod.
Fix the **Service** so that it selects the Pods of Deployment `web`.""",
check=r"""
eq "$(j -n shop svc web '{.spec.selector.app}')" web || exit 1
n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' | grep -c .)
[ "$n" -eq 2 ]""",
hint=r"""Compare `kubectl -n shop get svc web -o wide` (SELECTOR) with `kubectl -n shop get pods --show-labels`. `kubectl set selector` or `kubectl edit svc`.""",
show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
kubectl -n shop get svc web -o wide
kubectl -n shop get pods --show-labels
kubectl -n shop set selector svc web app=web
kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""),
step("The wrong port",
r"""Now the Service has endpoints, but they point at port `8080` - and nginx listens on `80`. Make the Service send traffic to the port the container serves.""",
check=r"""
eq "$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{.items[0].ports[0].port}')" 80 && eq "$(j -n shop svc web '{.spec.ports[0].port}')" 80""",
hint=r"""`targetPort` is the container's port; `port` is the Service's own. `kubectl -n shop edit svc web`, or a JSON patch on `/spec/ports/0/targetPort`.""",
show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]'
kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""),
step("Let the right client in",
r"""Still nothing: NetworkPolicy `lockdown` selects the `web` Pods and allows no ingress at all.
Keep it - do not delete it - and let **`client`** (label `app=client`) reach the `web` Pods on TCP 80. `stranger` must still be refused.""",
check=r"""
[ -n "$(j -n shop netpol lockdown '{.metadata.name}')" ] || exit 1
ok() { $K -n shop exec "$1" -- wget -qO- -T 3 web >/dev/null 2>&1; }
for i in 1 2 3 4 5; do ok client && break; sleep 3; done; ok client || exit 1
! ok stranger""",
hint=r"""Add an `ingress` rule to `lockdown`: `from: [{podSelector: {matchLabels: {app: client}}}]` with `ports: [{protocol: TCP, port: 80}]`. A NetworkPolicy's ports are the Pod's, not the Service's.""",
show=r"""kubectl -n shop apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: lockdown
spec:
podSelector:
matchLabels: {app: web}
policyTypes: [Ingress]
ingress:
- from:
- podSelector:
matchLabels: {app: client}
ports:
- {protocol: TCP, port: 80}
EOF
kubectl -n shop exec client -- wget -qO- -T3 web | head -4
kubectl -n shop exec stranger -- wget -qO- -T3 web || echo refused"""),
step("How kube-proxy routes it",
r"""kube-proxy turned Service `web` into rules on every node. Find out how: its mode is in ConfigMap `kube-proxy` in `kube-system`. Write the mode to `{WS}/lesson-2.5/mode.txt`.
Then, on the control-plane node, find the iptables chain that kube-proxy made for this Service - the `KUBE-SVC-...` chain that its ClusterIP jumps to - and write the chain's name to `{WS}/lesson-2.5/chain.txt`.""",
check=r"""
mode=$($K -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}'); mode=${mode:-iptables}
eq "$(tr -d ' \r\n"' < "$WS/lesson-2.5/mode.txt" 2>/dev/null)" "$mode" || exit 1
ip=$(j -n shop svc web '{.spec.clusterIP}')
want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" </dev/null)
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.5/chain.txt" 2>/dev/null)" "$want" """,
hint=r"""`kubectl -n kube-system get cm kube-proxy -o yaml | grep mode`. On the node: `sudo iptables-save -t nat | grep CLUSTER-IP` - the line for the Service jumps (`-j`) to its chain. In `nftables` mode the rules are in `sudo nft list ruleset` instead.""",
show=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt
ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32"
sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt
sudo iptables-save -t nat | grep "^-A $(cat {WS}/lesson-2.5/chain.txt)" """),
],
"outro": r"""You took a Service from "times out" to answering by fixing its selector, its target port and a NetworkPolicy, and followed it into kube-proxy's rules.
**For the exam**
- Endpoints first: `kubectl get endpointslices -l kubernetes.io/service-name=SVC` (or `kubectl get endpoints SVC`). Empty means the selector, or Pods that are not Ready.
- `port` is the Service's, `targetPort` the container's; a named `targetPort` must match a `containerPort` name.
- Test from a Pod: `kubectl exec POD -- wget -qO- -T3 SVC:PORT`, or a throwaway `kubectl run tmp --rm -it --restart=Never --image=busybox:1.36 -- ...`.
- A NetworkPolicy never blocks what it does not select; once one selects a Pod, everything not allowed is dropped - and egress policies need DNS (UDP/TCP 53) allowed.
- kube-proxy: a DaemonSet in `kube-system`, configured by ConfigMap `kube-proxy`; `iptables-save -t nat | grep KUBE-SVC` or `nft list ruleset` on a node.
Next, prove it against the clock: **drill 2.5**.""",
}
LESSONS = [L21, L22, L23, L24, L25]

672
cka/lessons_work.py Normal file
View File

@ -0,0 +1,672 @@
"""CKA lessons of the Workloads and Scheduling domain (3.1-3.5). See cka/lessons.py for what a lesson is."""
from sim.bank import NS, step
# ---- 3.1 rolling updates and rollbacks --------------------------------------------------------
L31 = {
"id": 9301, "objective": "3.1", "workers": 1,
"title": "Rollouts: strategy, a release that fails, undo, and pause",
"intro": r"""Every change to a Deployment's Pod **template** is a new **revision**: a new ReplicaSet, filled while the old one is emptied. Changes elsewhere - `replicas`, the strategy - are not revisions.
```
strategy: RollingUpdate maxSurge: extra Pods allowed maxUnavailable: missing Pods allowed
strategy: Recreate all old Pods go first, then the new ones come - a gap, on purpose
kubectl rollout status | history [--revision=N] | undo [--to-revision=N] | pause | resume | restart
```
A failed release does not roll itself back: the rollout stops, and with `maxUnavailable: 0` the old Pods keep serving until you act.
Namespace `rollout` runs Deployment `api`: `nginx:1.26`, 4 replicas.""",
"setup": NS + r"""
mkns rollout
mkdir -p "$WS/lesson-3.1"
$K -n rollout create deploy api --image=nginx:1.26 --replicas=4 >/dev/null
$K -n rollout rollout status deploy api --timeout=180s >/dev/null
""",
"steps": [
step("Never fewer than four",
r"""Change the update strategy of `api` so that an update first adds **one** Pod and never takes one away before its replacement is ready: `maxSurge: 1`, `maxUnavailable: 0`.""",
check=r"""eq "$(j -n rollout deploy api '{.spec.strategy.type} {.spec.strategy.rollingUpdate.maxSurge} {.spec.strategy.rollingUpdate.maxUnavailable}')" "RollingUpdate 1 0" """,
hint=r"""`kubectl -n rollout edit deploy api`, or `kubectl patch` with `{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}`.""",
show=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
kubectl -n rollout get deploy api -o jsonpath='{.spec.strategy}'; echo"""),
step("A release with a reason",
r"""Update `api` to `nginx:1.27`, and record why: the annotation `kubernetes.io/change-cause` on the Deployment, set to `upgrade to 1.27`. `kubectl rollout history` shows it beside the revision.""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1
eq "$(j -n rollout deploy api '{.metadata.annotations.kubernetes\.io/change-cause}')" "upgrade to 1.27" || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas}')" "4 4" """,
hint=r"""`kubectl set image deploy/api nginx=nginx:1.27` (the container is called `nginx`), then `kubectl annotate deploy api kubernetes.io/change-cause='...'`. The `--record` flag is gone.""",
show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27
kubectl -n rollout annotate deploy api kubernetes.io/change-cause='upgrade to 1.27' --overwrite
kubectl -n rollout rollout status deploy api --timeout=180s
kubectl -n rollout rollout history deploy api"""),
step("A release that cannot start",
r"""Now a mistake: set the image of `api` to `nginx:1.27-broken`, a tag that does not exist.
Watch what happens: one new Pod (the surge) fails to pull, and the rollout stops there. Because `maxUnavailable` is `0`, all four `nginx:1.27` Pods keep serving. `kubectl rollout status` would wait for ever (here, until `progressDeadlineSeconds` marks it failed).""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27-broken || exit 1
[ "$(j -n rollout deploy api '{.status.availableReplicas}')" = 4 ] || exit 1
$K -n rollout get pods -l app=api --no-headers 2>/dev/null | grep -Eq 'ErrImagePull|ImagePullBackOff' """,
hint=r"""`kubectl set image` again. `kubectl -n rollout get pods` shows the Pod that cannot pull; `kubectl -n rollout get rs` shows the two ReplicaSets side by side.""",
show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27-broken
sleep 10
kubectl -n rollout get pods
kubectl -n rollout get rs"""),
step("Undo",
r"""Roll `api` back to the revision before - `nginx:1.27` - and let the rollout finish.""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
hint=r"""`kubectl rollout undo deploy/api`. It does not create a ReplicaSet: it scales the old one back up, and that revision gets a new number.""",
show=r"""kubectl -n rollout rollout undo deploy/api
kubectl -n rollout rollout status deploy api --timeout=180s
kubectl -n rollout rollout history deploy api"""),
step("Back to a revision by number",
r"""Look at the history, then go back to the revision that ran **`nginx:1.26`**, by its number. `kubectl rollout history deploy api --revision=N` shows what a revision's template was.""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.26 || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
hint=r"""Revision 1 is where the Deployment started. `kubectl rollout undo deploy/api --to-revision=1`.""",
show=r"""kubectl -n rollout rollout history deploy api
kubectl -n rollout rollout history deploy api --revision=1 | grep Image
kubectl -n rollout rollout undo deploy/api --to-revision=1
kubectl -n rollout rollout status deploy api --timeout=180s"""),
step("Two changes, one rollout",
r"""Each `set` command starts a rollout of its own. To make two changes at once, **pause** the Deployment, make them - image `nginx:1.27`, and an environment variable `MODE=prod` in the container - then **resume**: one new revision, one rollout.""",
check=r"""
[ "$(j -n rollout deploy api '{.spec.paused}')" != true ] || exit 1
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].env[?(@.name=="MODE")].value}')" prod || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
hint=r"""`kubectl rollout pause deploy/api`, `kubectl set image ...`, `kubectl set env deploy/api MODE=prod`, `kubectl rollout resume deploy/api`.""",
show=r"""kubectl -n rollout rollout pause deploy/api
kubectl -n rollout set image deploy/api nginx=nginx:1.27
kubectl -n rollout set env deploy/api MODE=prod
kubectl -n rollout rollout resume deploy/api
kubectl -n rollout rollout status deploy api --timeout=180s
kubectl -n rollout rollout history deploy api"""),
],
"outro": r"""You tuned the strategy, released with a recorded cause, watched a broken release stall safely, undid it, went back to a numbered revision, and batched two changes into one rollout.
**For the exam**
- `kubectl set image deploy/NAME CONTAINER=IMAGE` - the container's name, not the image's.
- `rollout undo` goes to the previous revision; `--to-revision=N` to any in `rollout history`. The history keeps `revisionHistoryLimit` old ReplicaSets (10 by default).
- `maxSurge` / `maxUnavailable` take numbers or percentages; `maxUnavailable: 0` keeps full capacity during an update. `Recreate` has no such fields.
- `rollout restart` makes new Pods with an unchanged spec (it stamps an annotation into the template).
- A stuck rollout: `kubectl rollout status` waits, `describe deploy` shows `Progressing=False` after `progressDeadlineSeconds`.
Next, prove it against the clock: **drill 3.1**.""",
}
# ---- 3.2 ConfigMaps and Secrets ---------------------------------------------------------------
_ENC_CHECK = r"""
nx "$CP" 'ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/conf/db --print-value-only 2>/dev/null | head -c 64' </dev/null | grep -aq '^k8s:enc:aescbc:v1:'"""
L32 = {
"id": 9302, "objective": "3.2", "workers": 1,
"title": "ConfigMaps and Secrets: create, consume, update, and encrypt",
"intro": r"""A ConfigMap holds configuration, a Secret holds the confidential part of it - and both reach a container the same three ways:
```
env: - name: X valueFrom: {configMapKeyRef | secretKeyRef} one key, renamed if you like
envFrom: - configMapRef | secretRef every key, as it is called
volumes: - configMap | secret every key as a file
```
Only a mounted volume follows later changes (after a short delay, and never with `subPath`); environment variables are read once, when the container starts.
A Secret is **base64-encoded, not encrypted**: anyone who may read it, or read etcd, has it. Encryption at rest is the API server's job, set up with an `EncryptionConfiguration`.
Files for this lesson are in `{WS}/lesson-3.2/`; Namespace `conf` is empty.""",
"setup": NS + r"""
mkns conf
mkdir -p "$WS/lesson-3.2"
printf 'log.level=info\ncache.size=128\n' > "$WS/lesson-3.2/app.properties"
printf 'DB_USER=ledger\nDB_PASS=t0ps3cret\n' > "$WS/lesson-3.2/db.env"
""",
"steps": [
step("A ConfigMap from a file and a literal",
r"""Create ConfigMap `app-config` in `conf` with two keys: `app.properties`, holding the file `{WS}/lesson-3.2/app.properties`, and `MODE` with the value `prod`.""",
check=r"""
eq "$(j -n conf cm app-config '{.data.MODE}')" prod && $K -n conf get cm app-config -o jsonpath='{.data.app\.properties}' | grep -q '^cache.size=128$'""",
hint=r"""`kubectl create configmap NAME --from-file=PATH --from-literal=KEY=VALUE`. With `--from-file`, the key is the file's name.""",
show=r"""kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod
kubectl -n conf describe cm app-config"""),
step("A Secret from an env file",
r"""Create Secret `db` (type `Opaque`) in `conf` from `{WS}/lesson-3.2/db.env`, one key per line of the file: `DB_USER` and `DB_PASS`. Then read `DB_PASS` back, decoded.""",
check=r"""
eq "$(j -n conf secret db '{.type}')" Opaque && eq "$(j -n conf secret db '{.data.DB_USER}' | base64 -d)" ledger && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""",
hint=r"""`kubectl create secret generic NAME --from-env-file=PATH`. Reading one key: `-o jsonpath='{.data.DB_PASS}' | base64 -d`.""",
show=r"""kubectl -n conf create secret generic db --from-env-file={WS}/lesson-3.2/db.env
kubectl -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d; echo"""),
step("Consume both",
r"""Run Pod `app` in `conf` (`busybox:1.36`, `sleep 86400`) that gets:
- every key of Secret `db` as an environment variable (`envFrom`),
- the variable `APP_MODE` from key `MODE` of ConfigMap `app-config`,
- ConfigMap `app-config` as files under `/etc/app`.""",
check=r"""
x() { $K -n conf exec app -- "$@" 2>/dev/null; }
eq "$(x printenv DB_PASS)" t0ps3cret && eq "$(x printenv APP_MODE)" prod && x cat /etc/app/app.properties | grep -q '^log.level=info$'""",
hint=r"""`envFrom: [{secretRef: {name: db}}]`, `env: [{name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: MODE}}}]`, and a `configMap` volume.""",
show=r"""kubectl -n conf apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: busybox:1.36
command: [sleep, '86400']
envFrom:
- secretRef: {name: db}
env:
- name: APP_MODE
valueFrom:
configMapKeyRef: {name: app-config, key: MODE}
volumeMounts:
- {name: config, mountPath: /etc/app}
volumes:
- name: config
configMap: {name: app-config}
EOF
kubectl -n conf wait --for=condition=Ready pod/app --timeout=120s
kubectl -n conf exec app -- sh -c 'printenv DB_USER APP_MODE; cat /etc/app/app.properties'"""),
step("Change the configuration, live",
r"""Change `cache.size` to `256` in ConfigMap `app-config` - edit the file and re-create the ConfigMap's key from it, or `kubectl edit`.
Then watch `/etc/app/app.properties` inside Pod `app`: within a minute the kubelet replaces the file. `APP_MODE` would not change, even if you edited `MODE`: the environment is fixed at start.""",
check=r"""$K -n conf exec app -- cat /etc/app/app.properties 2>/dev/null | grep -q '^cache.size=256$'""",
hint=r"""Regenerate and apply: `kubectl create configmap app-config --from-file=... --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f -` - the usual way to update a ConfigMap from files.""",
show=r"""sed -i 's/^cache.size=.*/cache.size=256/' {WS}/lesson-3.2/app.properties
kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f -
sleep 30
kubectl -n conf exec app -- cat /etc/app/app.properties"""),
step("A TLS Secret",
r"""Make a self-signed certificate for `shop.example.com` (`openssl` is on this host), and store it as Secret `shop-tls` in `conf`, of type `kubernetes.io/tls` - the type an Ingress or Gateway expects. Its keys are `tls.crt` and `tls.key`.""",
check=r"""
eq "$(j -n conf secret shop-tls '{.type}')" kubernetes.io/tls || exit 1
j -n conf secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'shop.example.com'""",
hint=r"""`openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com`, then `kubectl create secret tls NAME --cert=tls.crt --key=tls.key`.""",
show=r"""cd {WS}/lesson-3.2
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
kubectl -n conf create secret tls shop-tls --cert=tls.crt --key=tls.key
kubectl -n conf get secret shop-tls
cd"""),
step("Encrypt Secrets at rest",
r"""Read Secret `db` straight out of etcd on `{CP}`, and you see `t0ps3cret` in plain text. Make the API server encrypt Secrets before it stores them:
1. Write an `EncryptionConfiguration` to `/etc/kubernetes/enc/enc.yaml`: for resource `secrets`, provider `aescbc` with one key `key1` (32 random bytes, base64), then `identity: {}` so that what is still unencrypted can be read.
2. Start the API server with `--encryption-provider-config=/etc/kubernetes/enc/enc.yaml`, and mount `/etc/kubernetes/enc` into its Pod (a `hostPath` volume): it runs in a container and sees only what is mounted.
3. Once it is back, rewrite Secret `db`, so that it is stored again - encrypted this time.
Edit a copy of the manifest outside `/etc/kubernetes/manifests` and copy it back: the kubelet runs every file in that directory, backups too.""",
check=_ENC_CHECK + r""" && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""",
hint=r"""The docs page "Encrypting Confidential Data at Rest" has the file. Stored values of encrypted Secrets start with `k8s:enc:aescbc:v1:key1:`: `sudo ETCDCTL_API=3 etcdctl ... get /registry/secrets/conf/db | od -c | head` (or `hexdump -C`). Rewriting: `kubectl get secret db -o json | kubectl replace -f -`. If the API server does not come back, `sudo crictl ps -a` and the logs under `/var/log/pods/kube-system_kube-apiserver-*` say why.""",
show=r"""sudo mkdir -p /etc/kubernetes/enc
sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
providers:
- aescbc:
keys:
- name: key1
secret: $(head -c 32 /dev/urandom | base64)
- identity: {}
EOF
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
| .spec.containers[0].volumeMounts += [{"name": "enc", "mountPath": "/etc/kubernetes/enc", "readOnly": true}]
| .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' /tmp/kube-apiserver.yaml
sudo cp /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
sleep 20
for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
kubectl -n conf get secret db -o json | kubectl replace -f -
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
get /registry/secrets/conf/db | od -c | head -4"""),
],
"outro": r"""You built a ConfigMap and Secrets from files, consumed them three ways, updated configuration in a running Pod, and turned on encryption at rest for Secrets.
**For the exam**
- `kubectl create configmap|secret generic NAME --from-literal --from-file --from-env-file`; `kubectl create secret tls|docker-registry`. Generate, then apply: `--dry-run=client -o yaml | kubectl apply -f -`.
- `immutable: true` on a ConfigMap or Secret: no more changes, only delete and re-create - and Pods must restart to see the new one.
- Mounted keys follow updates; `subPath` mounts and environment variables do not.
- Encryption at rest: `--encryption-provider-config` and a mount for the file. The **first** provider encrypts; the others are tried when reading. `identity` first means "not encrypted". Existing Secrets stay as they are until rewritten (`kubectl get secrets -A -o json | kubectl replace -f -`).
Next, prove it against the clock: **drill 3.2**.""",
}
# ---- 3.3 autoscaling --------------------------------------------------------------------------
L33 = {
"id": 9303, "objective": "3.3", "workers": 1,
"title": "Autoscaling: the HorizontalPodAutoscaler, and resizing a Pod in place",
"intro": r"""Three kinds of autoscaling exist, and only one is built in:
```
HorizontalPodAutoscaler more or fewer Pods, from metrics built in (needs metrics-server)
VerticalPodAutoscaler bigger or smaller requests per Pod an add-on, with its own CRDs
Cluster Autoscaler more or fewer nodes an add-on, per cloud
```
An HPA compares each Pod's usage with its **request**: 50% CPU means half of what the Pod requested. A Pod without a request gives it nothing to compare against. The HPA checks every 15 seconds and waits before scaling down (`behavior.scaleDown.stabilizationWindowSeconds`, five minutes unless you say otherwise).
On v1.37 a running Pod's CPU and memory can also be changed **in place**, through its `resize` subresource, without a new Pod.
metrics-server runs here. In Namespace `scale`, Deployment `api` (nginx) requests 100m CPU and idles; Deployment `cruncher` requests 50m and burns CPU.""",
"setup": NS + r"""
ensure_metrics_server || { echo "metrics-server did not become ready" >&2; exit 1; }
mkns scale
$K -n scale apply -f - >/dev/null <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api}
spec:
replicas: 1
selector: {matchLabels: {app: api}}
template:
metadata: {labels: {app: api}}
spec:
containers:
- name: nginx
image: nginx:1.27
resources: {requests: {cpu: 100m, memory: 64Mi}}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: cruncher}
spec:
replicas: 1
selector: {matchLabels: {app: cruncher}}
template:
metadata: {labels: {app: cruncher}}
spec:
containers:
- name: cruncher
image: busybox:1.36
command: [sh, -c, 'while true; do :; done']
resources: {requests: {cpu: 50m, memory: 16Mi}, limits: {cpu: 150m, memory: 32Mi}}
EOF
$K -n scale rollout status deploy api --timeout=180s >/dev/null && $K -n scale rollout status deploy cruncher --timeout=180s >/dev/null
""",
"steps": [
step("An HPA in one command",
r"""Make `api` scale between **2** and **6** replicas, aiming at **50%** CPU. Within a minute the HPA shows a current value - `kubectl -n scale get hpa` - and `api` runs at least 2 replicas.""",
check=r"""
eq "$(j -n scale hpa api '{.spec.minReplicas} {.spec.maxReplicas} {.spec.metrics[0].resource.name} {.spec.metrics[0].resource.target.averageUtilization}')" "2 6 cpu 50" || exit 1
[ -n "$(j -n scale hpa api '{.status.currentMetrics[0].resource.current.averageUtilization}')" ] && [ "$(j -n scale deploy api '{.status.readyReplicas}')" -ge 2 ] 2>/dev/null""",
hint=r"""`kubectl autoscale deploy api --min=2 --max=6 --cpu=50%` (kubectl before 1.34: `--cpu-percent=50`). `<unknown>` in the TARGETS column means no metrics yet - or a Pod without a CPU request.""",
show=r"""kubectl -n scale autoscale deploy api --min=2 --max=6 --cpu=50%
sleep 45
kubectl -n scale get hpa api"""),
step("Scale down calmly",
r"""Give the HPA of `api` a `behavior`: when scaling **down**, wait **120** seconds of stable recommendations (`stabilizationWindowSeconds`), and remove at most **1** Pod per **60** seconds.
`kubectl autoscale` cannot do that; edit the HPA (`autoscaling/v2`).""",
check=r"""
eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.stabilizationWindowSeconds}')" 120 || exit 1
eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.policies[0].type} {.spec.behavior.scaleDown.policies[0].value} {.spec.behavior.scaleDown.policies[0].periodSeconds}')" "Pods 1 60" """,
hint=r"""Under `spec`: `behavior: {scaleDown: {stabilizationWindowSeconds: 120, policies: [{type: Pods, value: 1, periodSeconds: 60}]}}`. `kubectl explain hpa.spec.behavior.scaleDown`.""",
show=r"""kubectl -n scale patch hpa api --type merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":120,"policies":[{"type":"Pods","value":1,"periodSeconds":60}]}}}}'
kubectl -n scale describe hpa api | grep -A4 Behavior"""),
step("Watch it scale up",
r"""Write an HPA for `cruncher` as a manifest (`autoscaling/v2`): 1 to **4** replicas at **60%** average CPU. `cruncher` uses about three times its request, so the HPA soon adds Pods: wait until it wants more than one.""",
check=r"""
eq "$(j -n scale hpa cruncher '{.spec.maxReplicas} {.spec.metrics[0].resource.target.averageUtilization}')" "4 60" || exit 1
[ "$(j -n scale hpa cruncher '{.status.desiredReplicas}')" -gt 1 ] 2>/dev/null""",
hint=r"""`kubectl autoscale ... --dry-run=client -o yaml` writes the manifest for you. `kubectl -n scale get hpa cruncher -w` shows it react; `describe hpa` lists its decisions as events.""",
show=r"""kubectl -n scale apply -f - <<'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: cruncher
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: cruncher
minReplicas: 1
maxReplicas: 4
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
EOF
sleep 60
kubectl -n scale get hpa cruncher"""),
step("Resize a running Pod",
r"""Run Pod `sizer` in `scale` (`busybox:1.36`, `sleep 86400`) with CPU request `100m` and limit `200m`, memory request and limit `64Mi`.
Then give it more CPU **without replacing it**: request `150m`, limit `300m`, through the Pod's `resize` subresource. The container keeps running - its restart count stays 0 - and the Pod's status shows the new values once the kubelet has applied them.""",
check=r"""
eq "$(j -n scale pod sizer '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.limits.cpu}')" "150m 300m" || exit 1
eq "$(j -n scale pod sizer '{.status.containerStatuses[0].resources.limits.cpu} {.status.containerStatuses[0].restartCount}')" "300m 0" """,
hint=r"""`kubectl patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{...}}]}}'`. A plain `kubectl edit` of a Pod's resources is refused; the subresource is the way in. A container's `resizePolicy` decides whether a change needs a restart.""",
show=r"""kubectl -n scale run sizer --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"sizer","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"200m","memory":"64Mi"}}}]}}'
kubectl -n scale wait --for=condition=Ready pod/sizer --timeout=120s
kubectl -n scale patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{"requests":{"cpu":"150m"},"limits":{"cpu":"300m"}}}]}}'
sleep 5
kubectl -n scale get pod sizer -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}{.status.containerStatuses[0].restartCount}{"\n"}'"""),
],
"outro": r"""You made HPAs both ways, slowed down their scale-down, watched one scale up, and resized a running Pod without replacing it.
**For the exam**
- `kubectl autoscale deploy NAME --min --max --cpu=N%` for the simple case; `autoscaling/v2` YAML for memory, several metrics, or `behavior`.
- The HPA needs metrics-server **and** requests on the target's containers. `kubectl describe hpa` explains `<unknown>`.
- Utilization is relative to the request; `averageValue` is absolute (`target: {type: AverageValue, averageValue: 200Mi}`).
- In-place resize: `kubectl patch pod NAME --subresource resize ...`; `resizePolicy` (`NotRequired` / `RestartContainer`) per resource. Deployments still replace Pods when their template changes.
- VPA and the Cluster Autoscaler are add-ons: know what they do, and that the VPA should not fight an HPA over the same metric.
Next, prove it against the clock: **drill 3.3**.""",
}
# ---- 3.4 robust, self-healing deployments ------------------------------------------------------
L34 = {
"id": 9304, "objective": "3.4", "workers": 1,
"title": "Self-healing: ReplicaSets, probes, disruption budgets, Jobs",
"intro": r"""Kubernetes heals by comparing what is with what should be, over and over:
```
ReplicaSet too few Pods? make more (a Deployment manages them)
livenessProbe container stuck? restart it
readinessProbe not ready? take it out of the Service's endpoints
startupProbe slow to start? hold the other probes until it is up
PodDisruptionBudget how many Pods a voluntary disruption (drain, eviction) may take
Job / CronJob run to completion, retry on failure, on a schedule
```
Namespace `heal` runs Deployment `web` (nginx, 3 replicas) and Deployment `api` (nginx, 2 replicas), whose Pods never become ready.""",
"setup": NS + r"""
mkns heal
mkdir -p "$WS/lesson-3.4"
$K -n heal create deploy web --image=nginx:1.27 --replicas=3 >/dev/null
$K -n heal apply -f - >/dev/null <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: api}
spec:
replicas: 2
selector: {matchLabels: {app: api}}
template:
metadata: {labels: {app: api}}
spec:
containers:
- name: nginx
image: nginx:1.27
readinessProbe:
httpGet: {path: /healthz, port: 80}
periodSeconds: 5
EOF
$K -n heal rollout status deploy web --timeout=180s >/dev/null
$K -n heal get pods -l app=web -o jsonpath='{.items[*].metadata.uid}' > "$PRIV/lesson-3.4.uids"
""",
"steps": [
step("Delete a Pod, get a Pod",
r"""Delete one Pod of Deployment `web`. Its ReplicaSet notices that it has 2 of 3, and makes a new one at once.
Write the name of that ReplicaSet to `{WS}/lesson-3.4/rs.txt`. `kubectl get pod POD -o yaml` names it under `ownerReferences`.""",
check=r"""
rs=$(tr -d ' \r\n' < "$WS/lesson-3.4/rs.txt" 2>/dev/null); [ -n "$rs" ] || exit 1
eq "$(j -n heal rs "$rs" '{.metadata.ownerReferences[0].name} {.status.readyReplicas}')" "web 3" || exit 1
now=$(j -n heal pods -l app=web '{.items[*].metadata.uid}'); for u in $now; do grep -q "$u" "$PRIV/lesson-3.4.uids" || exit 0; done; exit 1""",
hint=r"""`kubectl -n heal delete pod $(kubectl -n heal get pod -l app=web -o name | head -1)`. Pods of a Deployment are named DEPLOYMENT-RSHASH-RANDOM; the middle part is the ReplicaSet's suffix.""",
show=r"""kubectl -n heal delete $(kubectl -n heal get pod -l app=web -o name | head -1)
kubectl -n heal get pods -l app=web
kubectl -n heal get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].name}' > {WS}/lesson-3.4/rs.txt
kubectl -n heal get rs"""),
step("A readiness probe that asks the wrong thing",
r"""The Pods of `api` run, but none is `Ready`: `kubectl -n heal describe pod -l app=api` shows the readiness probe failing with `404`. nginx has no `/healthz`. Make the probe ask for `/` instead, and both Pods become ready.""",
check=r"""eq "$(j -n heal deploy api '{.spec.template.spec.containers[0].readinessProbe.httpGet.path}')" / && eq "$(j -n heal deploy api '{.status.readyReplicas}')" 2""",
hint=r"""A JSON patch on `/spec/template/spec/containers/0/readinessProbe/httpGet/path`, or `kubectl edit deploy api`. The fix is a change of template: a rollout.""",
show=r"""kubectl -n heal describe pod -l app=api | grep -m2 Readiness
kubectl -n heal patch deploy api --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]'
kubectl -n heal rollout status deploy api --timeout=120s"""),
step("A liveness probe that restarts",
r"""Run Pod `fragile` in `heal` (`busybox:1.36`) with the command `sh -c 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400'` and a **liveness** probe that runs `cat /tmp/healthy` every 5 seconds, starting after 5.
After 20 seconds the file is gone, the probe fails three times, and the kubelet restarts the container - which makes the file again. Wait for the first restart.""",
check=r"""
eq "$(j -n heal pod fragile '{.spec.containers[0].livenessProbe.exec.command[1]}')" /tmp/healthy || exit 1
[ "$(j -n heal pod fragile '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null""",
hint=r"""`livenessProbe: {exec: {command: [cat, /tmp/healthy]}, initialDelaySeconds: 5, periodSeconds: 5}`. `kubectl -n heal get pod fragile -w` shows RESTARTS go up; `describe` shows `Liveness probe failed`.""",
show=r"""kubectl -n heal apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: fragile
spec:
containers:
- name: fragile
image: busybox:1.36
command: [sh, -c, 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400']
livenessProbe:
exec:
command: [cat, /tmp/healthy]
initialDelaySeconds: 5
periodSeconds: 5
EOF
sleep 50
kubectl -n heal get pod fragile"""),
step("A budget for disruptions",
r"""Create PodDisruptionBudget `web-pdb` in `heal`: at least **2** Pods of `app=web` must stay available.
Delete Pod `fragile` first: it has done its job, and `drain` refuses a Pod that no controller would re-create (unless `--force`, which deletes it for good).
Then try to drain `{W1}`, where all three `web` Pods run: `kubectl drain {W1} --ignore-daemonsets --timeout=30s`. It evicts what it may and refuses the rest - "would violate the pod's disruption budget". Save its output in `{WS}/lesson-3.4/drain.txt`, then make `{W1}` schedulable again.""",
check=r"""
eq "$(j -n heal pdb web-pdb '{.spec.minAvailable} {.spec.selector.matchLabels.app}')" "2 web" || exit 1
grep -qi 'disruption budget' "$WS/lesson-3.4/drain.txt" 2>/dev/null && [ "$(j node "$W1" '{.spec.unschedulable}')" != true ]""",
hint=r"""`kubectl create pdb web-pdb --selector=app=web --min-available=2`. Drain prints the refusals on stderr: `2>&1 | tee FILE`. `kubectl uncordon {W1}` afterwards - drain cordons first.""",
show=r"""kubectl -n heal create pdb web-pdb --selector=app=web --min-available=2
kubectl -n heal get pdb web-pdb
kubectl -n heal delete pod fragile --grace-period=1
kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 | tee {WS}/lesson-3.4/drain.txt | tail -3
kubectl uncordon {W1}"""),
step("A Job that runs to completion",
r"""Create Job `batch` in `heal`: `busybox:1.36` running `sh -c 'echo processed; sleep 2'`, **3** completions, **2** at a time, giving up after **2** retries (`backoffLimit`). Wait until it is complete.""",
check=r"""eq "$(j -n heal job batch '{.spec.completions} {.spec.parallelism} {.spec.backoffLimit} {.status.succeeded}')" "3 2 2 3" """,
hint=r"""`kubectl create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'`, then add `completions`, `parallelism` and `backoffLimit` under `spec`. `kubectl wait --for=condition=complete job/batch`.""",
show=r"""kubectl -n heal create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo processed; sleep 2' > {WS}/lesson-3.4/batch.yaml
yq -i '.spec.completions = 3 | .spec.parallelism = 2 | .spec.backoffLimit = 2' {WS}/lesson-3.4/batch.yaml
kubectl -n heal apply -f {WS}/lesson-3.4/batch.yaml
kubectl -n heal wait --for=condition=complete job/batch --timeout=120s
kubectl -n heal get pods -l job-name=batch"""),
step("A CronJob, and a run on demand",
r"""Create CronJob `report` in `heal`: every 30 minutes (`*/30 * * * *`), `busybox:1.36` running `date`, never two runs at once (`concurrencyPolicy: Forbid`), keeping the last **2** successful Jobs.
Do not wait half an hour: start a Job `report-now` from the CronJob yourself, and let it finish.""",
check=r"""
eq "$(j -n heal cronjob report '{.spec.schedule} {.spec.concurrencyPolicy} {.spec.successfulJobsHistoryLimit}')" "*/30 * * * * Forbid 2" || exit 1
eq "$(j -n heal job report-now '{.metadata.annotations.cronjob\.kubernetes\.io/instantiate} {.status.succeeded}')" "manual 1" """,
hint=r"""`kubectl create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date`, then the two fields. `kubectl create job report-now --from=cronjob/report`.""",
show=r"""kubectl -n heal create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date > {WS}/lesson-3.4/report.yaml
yq -i '.spec.concurrencyPolicy = "Forbid" | .spec.successfulJobsHistoryLimit = 2' {WS}/lesson-3.4/report.yaml
kubectl -n heal apply -f {WS}/lesson-3.4/report.yaml
kubectl -n heal create job report-now --from=cronjob/report
kubectl -n heal wait --for=condition=complete job/report-now --timeout=120s"""),
],
"outro": r"""You watched a ReplicaSet replace a Pod, fixed a readiness probe, saw a liveness probe restart a container, stopped a drain with a disruption budget, and ran a Job and a CronJob.
**For the exam**
- Readiness decides **traffic** (endpoints); liveness decides **restarts**; a startup probe holds both off until the app has started. Probes: `httpGet`, `tcpSocket`, `exec`, `grpc`.
- A PDB has `minAvailable` **or** `maxUnavailable`. It stops evictions - a drain, or anything else that uses the Eviction API - never a `kubectl delete pod`, and never a node that just dies.
- Jobs: `completions`, `parallelism`, `backoffLimit`, `activeDeadlineSeconds`, `ttlSecondsAfterFinished`; a Job's Pods use `restartPolicy: Never` or `OnFailure`.
- CronJobs: `schedule` in cron syntax (UTC unless `timeZone`), `concurrencyPolicy`, `suspend`, history limits; `kubectl create job NAME --from=cronjob/CJ` to test one.
Next, prove it against the clock: **drill 3.4**.""",
}
# ---- 3.5 Pod admission and scheduling ---------------------------------------------------------
L35 = {
"id": 9305, "objective": "3.5", "workers": 2,
"title": "Scheduling: QoS, affinity, spreading, taints, defaults, and nodeName",
"intro": r"""The scheduler puts a Pod on a node that passes every **filter** and scores best:
```
requests does it fit? (requests, not limits, and not usage)
nodeSelector / nodeAffinity may it go there? (node labels)
podAffinity / podAntiAffinity near or away from other Pods (their labels, per topologyKey)
taints and tolerations is the node closed to it? cordon = the unschedulable taint
```
`nodeName` skips the scheduler entirely. Requests and limits also give each Pod a **QoS class**, which decides who is evicted first when a node runs short: `BestEffort`, then `Burstable`, then `Guaranteed`. A **LimitRange** fills in requests and limits for Pods that bring none.
This cluster has two workers, `{W1}` and `{W2}`. Namespace `sched` is empty.""",
"setup": NS + r"""
mkns sched
""",
"steps": [
step("Three QoS classes",
r"""Create three Pods in `sched`, each `busybox:1.36` running `sleep 86400`:
- `gold`: requests **equal** to limits, for CPU (`100m`) and memory (`64Mi`) → `Guaranteed`
- `silver`: only a memory request of `32Mi` → `Burstable`
- `bronze`: no resources at all → `BestEffort`""",
check=r"""
for p in gold:Guaranteed silver:Burstable bronze:BestEffort; do eq "$(j -n sched pod "${p%%:*}" '{.status.qosClass}')" "${p##*:}" || exit 1; done""",
hint=r"""`kubectl run NAME --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400`, then `resources:` in the container. `.status.qosClass` shows the result.""",
show=r"""kubectl -n sched run bronze --image=busybox:1.36 -- sleep 86400
kubectl -n sched run silver --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"silver","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"memory":"32Mi"}}}]}}'
kubectl -n sched run gold --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"gold","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"64Mi"}}}]}}'
kubectl -n sched get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass"""),
step("Only on SSD nodes",
r"""Label `{W2}` with `disk=ssd`. Then run Pod `fast` in `sched` (`busybox:1.36`, `sleep 86400`) with a **required** node affinity for `disk In (ssd)`. It lands on `{W2}` - and if no node had the label, it would stay `Pending`.""",
check=r"""
eq "$(j node "$W2" '{.metadata.labels.disk}')" ssd && eq "$(j -n sched pod fast '{.spec.nodeName}')" "$W2" \
&& eq "$(j -n sched pod fast '{.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key}')" disk""",
hint=r"""`kubectl label node {W2} disk=ssd`. The affinity block: `affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[].matchExpressions[]` with `key`, `operator: In`, `values`. The docs page "Assign Pods to Nodes using Node Affinity" has it to copy.""",
show=r"""kubectl label node {W2} disk=ssd
kubectl -n sched apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: fast
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- {key: disk, operator: In, values: [ssd]}
containers:
- name: fast
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n sched get pod fast -o wide"""),
step("One replica per node",
r"""Create Deployment `spread` in `sched` (2 replicas, `busybox:1.36`, `sleep 86400`, label `app=spread`) whose Pods **refuse** to share a node with each other: a required pod anti-affinity on `app=spread`, with `topologyKey: kubernetes.io/hostname`. One Pod lands on each worker; a third replica would stay `Pending`.""",
check=r"""
eq "$(j -n sched deploy spread '{.status.readyReplicas}')" 2 || exit 1
eq "$(j -n sched deploy spread '{.spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey}')" kubernetes.io/hostname || exit 1
[ "$($K -n sched get pods -l app=spread -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 2 ]""",
hint=r"""`podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution: [{labelSelector: {matchLabels: {app: spread}}, topologyKey: kubernetes.io/hostname}]` in the Pod template. `topologySpreadConstraints` is the softer, counting way to spread.""",
show=r"""kubectl -n sched apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: spread
spec:
replicas: 2
selector:
matchLabels: {app: spread}
template:
metadata:
labels: {app: spread}
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels: {app: spread}
topologyKey: kubernetes.io/hostname
containers:
- name: spread
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n sched rollout status deploy spread --timeout=120s
kubectl -n sched get pods -l app=spread -o wide"""),
step("A node for batch work only",
r"""Taint `{W1}` with `dedicated=batch:NoSchedule`: from now on only Pods that tolerate it are scheduled there (Pods already running stay - `NoExecute` would evict them).
Then run Pod `nightly` in `sched` (`busybox:1.36`, `sleep 86400`) that tolerates the taint **and** is placed on `{W1}` with a `nodeSelector` on `kubernetes.io/hostname`. A toleration only opens the door; it does not lead the Pod there.""",
check=r"""
[ -n "$($K get node "$W1" -o jsonpath='{range .spec.taints[*]}{.key}={.value}:{.effect}{"\n"}{end}' | grep -x 'dedicated=batch:NoSchedule')" ] || exit 1
eq "$(j -n sched pod nightly '{.spec.nodeName} {.status.phase}')" "$W1 Running" && eq "$(j -n sched pod nightly '{.spec.tolerations[?(@.key=="dedicated")].value}')" batch""",
hint=r"""`kubectl taint node {W1} dedicated=batch:NoSchedule`. The toleration: `{key: dedicated, operator: Equal, value: batch, effect: NoSchedule}`.""",
show=r"""kubectl taint node {W1} dedicated=batch:NoSchedule
kubectl -n sched apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: nightly
spec:
nodeSelector:
kubernetes.io/hostname: {W1}
tolerations:
- {key: dedicated, operator: Equal, value: batch, effect: NoSchedule}
containers:
- name: nightly
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n sched get pod nightly -o wide"""),
step("Defaults for Pods that bring none",
r"""Create LimitRange `defaults` in `sched`: containers that say nothing get a **request** of `50m` CPU and `32Mi` memory and a **limit** of `200m` and `128Mi`.
Then run Pod `plain` (`busybox:1.36`, `sleep 86400`, no resources) and look at what it got. A LimitRange acts when a Pod is created: Pods that already exist keep what they had.""",
check=r"""
eq "$(j -n sched limitrange defaults '{.spec.limits[0].type}')" Container || exit 1
eq "$(j -n sched pod plain '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.requests.memory} {.spec.containers[0].resources.limits.cpu} {.spec.containers[0].resources.limits.memory}')" "50m 32Mi 200m 128Mi" """,
hint=r"""`spec.limits: [{type: Container, defaultRequest: {cpu: 50m, memory: 32Mi}, default: {cpu: 200m, memory: 128Mi}}]` - `default` is the limit.""",
show=r"""kubectl -n sched apply -f - <<'EOF'
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
spec:
limits:
- type: Container
defaultRequest: {cpu: 50m, memory: 32Mi}
default: {cpu: 200m, memory: 128Mi}
EOF
kubectl -n sched run plain --image=busybox:1.36 -- sleep 86400
kubectl -n sched get pod plain -o jsonpath='{.spec.containers[0].resources}'; echo"""),
step("Past the scheduler",
r"""Cordon `{W2}`. Then create Pod `manual` in `sched` (`busybox:1.36`, `sleep 86400`) with **`nodeName: {W2}`**. It runs there anyway: `nodeName` hands the Pod straight to that node's kubelet, and cordoning only tells the *scheduler* to stay away.
Leave `{W2}` cordoned.""",
check=r"""eq "$(j node "$W2" '{.spec.unschedulable}')" true && eq "$(j -n sched pod manual '{.spec.nodeName} {.status.phase}')" "$W2 Running" """,
hint=r"""`kubectl cordon {W2}`, then a Pod manifest with `spec.nodeName`. Without a running scheduler, this is also how you place a Pod by hand.""",
show=r"""kubectl cordon {W2}
kubectl -n sched run manual --image=busybox:1.36 --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 86400
kubectl -n sched get pod manual -o wide
kubectl get nodes"""),
],
"outro": r"""You produced all three QoS classes, steered Pods with node affinity, kept replicas apart, reserved a node with a taint, set defaults with a LimitRange, and bypassed the scheduler.
**For the exam**
- `Pending` with `FailedScheduling`: the event lists why each node was rejected - insufficient cpu/memory, a taint not tolerated, affinity not matched, unschedulable.
- Requests decide placement; limits are enforced at run time (CPU is throttled, memory over the limit is `OOMKilled`).
- `required...` affinity filters, `preferred...` only scores. Pod (anti-)affinity needs a `topologyKey`.
- Taint effects: `NoSchedule`, `PreferNoSchedule`, `NoExecute` (evicts running Pods without a toleration). Remove one with a trailing `-`: `kubectl taint node N dedicated=batch:NoSchedule-`.
- ResourceQuota caps a Namespace's total and then **requires** requests/limits on new Pods; a LimitRange supplies them. PriorityClasses decide who is preempted.
Next, prove it against the clock: **drill 3.5**.""",
}
LESSONS = [L31, L32, L33, L34, L35]

18
cka/questions.py
View File

@ -21,9 +21,9 @@ TRACK = {
"minutes": 120,
"pass_pct": 66,
# Kubernetes minor of the real exam environment, which picks the kind cluster (shared with CKS).
# LF FAQ / "Important Instructions: CKA and CKAD", checked 2026-10-02: "The CKA environment is
# currently running Kubernetes v1.35". The LF moves each exam on its own schedule.
"k8s": "v1.35",
# LF FAQ / "Important Instructions: CKA and CKAD", checked 2026-10-19: "The CKA environment is
# currently running Kubernetes v1.37". The LF moves each exam on its own schedule.
"k8s": "v1.37",
# Worker nodes of a task's cluster unless the task says otherwise ("workers", or WORKERS below).
# One, so the control-plane node keeps its NoSchedule taint and workloads land on a worker, as
# in the exam; a task that names a worker ($W1, {W2}, ...) gets the three the bank was written for.
@ -34,7 +34,8 @@ TRACK = {
# fixed simulator sets (./ksim sim c|d) - excluded from random sampling. Sets A and B were
# retired on 2026-09-22; their last (fixed, verified) versions are in git history.
"sets": {"c": "Simulator Set C", "d": "Simulator Set D (killer.sh-style)"},
# CKA_Curriculum_v1.35.pdf (cncf/curriculum) - unchanged since the February 2025 revision
# CKA_Curriculum_v1.37.pdf (cncf/curriculum) - the same document as v1.35, unchanged since the
# February 2025 revision
"domains": {
"ARCH": "Cluster Architecture, Installation & Configuration (25%)",
"WORK": "Workloads & Scheduling (15%)",
@ -97,7 +98,7 @@ Details you need:
- "ssh to the control-plane node" is: `{SHELL_CP}`
- `etcdctl` and `etcdutl` are installed on that node, in `/usr/local/bin`
Verify the snapshot with `etcdutl snapshot status` (etcd 3.6, which v1.35 runs, removed `etcdctl snapshot status`).
Verify the snapshot with `etcdutl snapshot status` (etcd 3.6 removed `etcdctl snapshot status`; v1.37 runs etcd 3.7).
""",
"setup": r"""
nx "$CP" 'mkdir -p /opt/cka; rm -f /opt/cka/etcd-snapshot.db'
@ -655,11 +656,12 @@ $K -n default delete pod sched-probe --wait=false >/dev/null 2>&1
# ---- 68-85 / 86-102 in the two fixed simulator sets, 103-113 (pool tasks for the objectives the
# ---- curriculum map found thin, see curriculum.py) in bank_objectives, and the pool tasks that
# ---- bring every domain to 17 in bank_storage (114-124), bank_net (125-128) and bank_tshoot
# ---- (129-131). Numbering is contiguous - keep it that way.
from cka import bank_extra, bank_hard, bank_net, bank_objectives, bank_sim_c, bank_sim_d, bank_storage, bank_tshoot
# ---- (129-131); what a full CKA course teaches beyond that in bank_course (132-147).
# ---- Numbering is contiguous - keep it that way.
from cka import bank_course, bank_extra, bank_hard, bank_net, bank_objectives, bank_sim_c, bank_sim_d, bank_storage, bank_tshoot
QUESTIONS = (QUESTIONS + bank_extra.QUESTIONS + bank_hard.QUESTIONS
+ bank_sim_c.QUESTIONS + bank_sim_d.QUESTIONS + bank_objectives.QUESTIONS
+ bank_storage.QUESTIONS + bank_net.QUESTIONS + bank_tshoot.QUESTIONS)
+ bank_storage.QUESTIONS + bank_net.QUESTIONS + bank_tshoot.QUESTIONS + bank_course.QUESTIONS)
QUESTIONS.sort(key=lambda q: q["id"])
_ids = [q["id"] for q in QUESTIONS]
assert len(_ids) == len(set(_ids)), "duplicate question ids"

4
cka/solutions.py
View File

@ -104,6 +104,9 @@ spec:
schedule: {type: string}
target: {type: string}
EOF
# the new type is served a moment after the CRD is created: a Backup sent at once can fail
# with "no matches for kind Backup"
k wait --for condition=established crd/backups.data.example.com
# Step 2: Create a custom resource instance of Backup
cat <<EOF | k apply -f -
@ -418,6 +421,7 @@ spec:
environment: {type: string, enum: [prod, stage, dev], default: dev}
image: {type: string}
EOF
k wait --for condition=established crd/$(echo %{kind} | tr A-Z a-z)s.deploy.example.com
# Step 2: A valid object
k -n default apply -f - <<EOF

36
cka/solve.py
View File

@ -719,6 +719,38 @@ kubectl -n oslo-secure run vault-reader --image=busybox:1.36 --dry-run=client -o
{"name": "APP_PASS", "valueFrom": {"secretKeyRef": {"name": "db-login", "key": "pass"}}}]' | kubectl apply -f -
kubectl -n oslo-secure wait --for=condition=Ready pod/vault-reader --timeout=180s
""",
# ---- 132-145: bank_course.py (the others are solved by their solution as written, below)
132: r"""
apt_get() { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq -y -o Dpkg::Options::=--force-confold "$@" </dev/null >/dev/null; }
sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt_get update
sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null
sudo kubeadm upgrade apply v1.37.1 -y </dev/null | tail -2
kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl >/dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null
sudo systemctl daemon-reload && sudo systemctl restart kubelet
for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done
kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data
ssh {W1} sudo bash <<'EOF'
export DEBIAN_FRONTEND=noninteractive
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq </dev/null
apt-mark unhold kubeadm kubelet kubectl >/dev/null
apt-get install -y -qq kubeadm=1.37.1-1.1 </dev/null >/dev/null
kubeadm upgrade node </dev/null | tail -1
apt-get install -y -qq kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 </dev/null >/dev/null
apt-mark hold kubeadm kubelet kubectl >/dev/null
systemctl daemon-reload && systemctl restart kubelet
EOF
kubectl uncordon {W1}
""",
139: r"""
ssh {W1} sudo bash <<'EOF'
yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
systemctl restart kubelet
EOF
for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
}
# ---- 1-113: the banks carried over from cka-sim.
@ -726,7 +758,9 @@ kubectl -n oslo-secure wait --for=condition=Ready pod/vault-reader --timeout=180
# text the candidate is shown - and a solution that stops working is found the next time it runs.
AS_WRITTEN = [1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
40, 42, 43, 44, 47, 48, 49, 50, 52, 53, 54, 55, 56, 58, 62, 63, 64, 68, 69, 72, 73, 74, 75, 78, 80, 82, 83,
85, 89, 98, 99, 100, 101, 103, 104, 109, 113]
85, 89, 98, 99, 100, 101, 103, 104, 109, 113,
# bank_course.py: written to run as they stand
133, 134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 145, 146, 147]
def _as_written():

646
ckad/lessons.py Normal file
View File

@ -0,0 +1,646 @@
"""CKAD lessons: guided labs, one per curriculum objective, on a cluster of your own and without a
clock. A lesson explains, then walks you through steps that you do on its host; each step has a
check, a hint and the commands that do it. It ends by sending you to the objective's timed drill.
Lessons write nothing to the history or the scores - only a "lesson done" mark on the objective.
./ksim lesson the lessons of this track, and which you have done
./ksim lesson 2.1 start the lesson of objective 2.1 (id or key)
./ksim lesson check check the step you are on; the next one follows
./ksim lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after
A lesson's host is named after its id - lesson 9201 runs on `ckad9201` - which never clashes with
a task's. Its files go under {WS}/lesson-<objective>/. A step's `show` commands are what the
learner types; the audit runs exactly them, as `candidate`, so what a lesson shows always works.
Pilot (2026-10-16): the Application Deployment domain, 2.1 to 2.4.
"""
from sim.bank import NS, step
# ---- 2.1 blue/green and canary ---------------------------------------------------------------
_WEB = r"""
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-%{v}
labels: {app: web, version: %{v}}
spec:
replicas: 2
selector:
matchLabels: {app: web, version: %{v}}
template:
metadata:
labels: {app: web, version: %{v}}
spec:
containers:
- name: web
image: busybox:1.36
command: [sh, -c, 'mkdir -p /www && echo %{v} > /www/index.html && exec httpd -f -p 80 -h /www']
ports:
- containerPort: 80
"""
# a request through Service web from Pod client, retried while endpoints settle
_ASK = r"""
ask() { local i; for i in 1 2 3 4 5 6; do [ "$($K -n shop exec client -- wget -qO- -T 3 web 2>/dev/null)" = "$1" ] && return 0; sleep 2; done; return 1; }
"""
L21 = {
"id": 9201, "objective": "2.1", "workers": 0,
"title": "Blue/green and canary with plain Deployments",
"intro": r"""A Deployment replaces its Pods by rolling them over. Sometimes you want more control than that: run the new version **beside** the old one, try it, and decide when it gets the traffic. Nothing beyond plain Deployments and a Service is needed, because the **Service's selector** decides which Pods receive requests.
**Blue/green**: two Deployments, one Service. Changing the Service's selector moves all traffic at once, and changing it back is your rollback:
```
┌─▶ web-blue (blue)
Service web ─selector─┤
└╌▶ web-green (green)
```
**Canary**: let the Service select **both** versions, and set the split with replica counts. With 4 stable Pods and 1 canary Pod, about one request in five reaches the canary.
In Namespace `shop` you will find Deployment `web-blue` (each of its Pods answers `blue`), Service `web` in front of it, and a Pod `client` to send requests from.""",
"setup": NS + r"""
mkns shop
$K -n shop apply -f - >/dev/null <<EOF
""" + _WEB.replace("%{v}", "blue") + r"""
---
apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector: {app: web, version: blue}
ports:
- port: 80
targetPort: 80
EOF
$K -n shop run client --image=busybox:1.36 --command -- sleep 86400 >/dev/null
$K -n shop rollout status deploy web-blue --timeout=180s >/dev/null
$K -n shop wait --for=condition=Ready pod/client --timeout=180s >/dev/null
mkdir -p "$WS/lesson-2.1"
""",
"steps": [
step("Ask the Service who answers",
r"""Before you change anything, see what users get today. The Pod `client` has `wget`, and inside the Namespace a Service is reachable by its name:
```
kubectl -n shop exec client -- wget -qO- web
```
Save the answer in `{WS}/lesson-2.1/before.txt`.""",
check=r"""tr -d '\r' < "$WS/lesson-2.1/before.txt" 2>/dev/null | grep -qx blue""",
hint=r"""`kubectl exec POD -- COMMAND` runs a command in a running Pod; `wget -qO-` prints the page instead of saving it. Pipe it through `tee FILE`, or redirect with `> FILE`.""",
show=r"""kubectl -n shop exec client -- wget -qO- web | tee {WS}/lesson-2.1/before.txt"""),
step("Start green beside blue",
r"""Create Deployment `web-green` next to `web-blue`: 2 replicas, the same container, except that each Pod answers `green`. Its Pods carry the labels `app=web` and `version=green`.
Copying blue's manifest is the quickest start: `kubectl -n shop get deploy web-blue -o yaml`. Change the name, the `version` label everywhere and the word `blue` in the command, and leave out `status`, `uid`, `resourceVersion` and `creationTimestamp`. Wait until both Pods are ready.
The Service still selects `version=blue`, so no user sees green yet: you can try it first.""",
check=r"""
eq "$(j -n shop deploy web-green '{.status.availableReplicas}')" 2 \
&& eq "$(j -n shop deploy web-green '{.spec.template.metadata.labels.app}')" web \
&& eq "$(j -n shop deploy web-green '{.spec.template.metadata.labels.version}')" green || exit 1
ip=$(j -n shop pods -l app=web,version=green '{.items[0].status.podIP}')
[ -n "$ip" ] && eq "$($K -n shop exec client -- wget -qO- -T 3 "$ip" 2>/dev/null)" green""",
hint=r"""Labels appear three times in a Deployment: `metadata.labels` (optional), `spec.selector.matchLabels` and `spec.template.metadata.labels` - the selector must match the template. `kubectl -n shop rollout status deploy web-green` waits until it is ready, and `kubectl -n shop get pods -l version=green -o wide` shows the Pod IPs to try.""",
show=r"""kubectl -n shop apply -f - <<'EOF'
""" + _WEB.replace("%{v}", "green").strip("\n") + r"""
EOF
kubectl -n shop rollout status deploy web-green
kubectl -n shop get pods -l app=web --show-labels"""),
step("Cut over to green",
r"""Point Service `web` at green. Changing the selector **is** the cutover: within a second every new request goes to green.
Careful: `kubectl set selector` replaces the **whole** selector, so name both `app` and `version`. Then ask the Service again from `client` - the answer should be `green`.""",
check=_ASK + r"""
eq "$(j -n shop svc web '{.spec.selector.app}')" web && eq "$(j -n shop svc web '{.spec.selector.version}')" green && ask green""",
hint=r"""`kubectl set selector svc web 'app=web,version=green'` - or `kubectl patch svc web -p '{"spec":{"selector":{"version":"green"}}}'`, which merges and keeps `app`. `kubectl -n shop get svc web -o wide` shows the selector.""",
show=r"""kubectl -n shop set selector svc web 'app=web,version=green'
kubectl -n shop get svc web -o wide
kubectl -n shop exec client -- wget -qO- web"""),
step("Roll back in one command",
r"""Say green misbehaves. Rolling back a blue/green release is the same move in reverse: point the Service at blue again. Blue never stopped running, so this is instant - no new rollout, no image to pull.
Switch back, and ask the Service once more.""",
check=_ASK + r"""
eq "$(j -n shop svc web '{.spec.selector.app}')" web && eq "$(j -n shop svc web '{.spec.selector.version}')" blue && ask blue""",
hint=r"""The command you used for the cutover, with `version=blue`.""",
show=r"""kubectl -n shop set selector svc web 'app=web,version=blue'
kubectl -n shop exec client -- wget -qO- web"""),
step("Canary: one Service, both versions",
r"""Now a canary. Let Service `web` select **every** Pod of the app, blue and green, by dropping `version` from its selector. Requests spread over all ready endpoints, so with 2 blue and 2 green Pods about half of them reach green.
Ask the Service a few times and watch the answers change.""",
check=r"""
eq "$(j -n shop svc web '{.spec.selector.app}')" web && [ -z "$(j -n shop svc web '{.spec.selector.version}')" ] || exit 1
for i in 1 2 3 4 5 6; do
n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' 2>/dev/null | grep -c .)
[ "$n" -eq 4 ] && exit 0; sleep 2
done; exit 1""",
hint=r"""`kubectl set selector svc web 'app=web'` leaves only `app`. `kubectl -n shop get endpointslices -l kubernetes.io/service-name=web` lists the Pods that receive traffic.""",
show=r"""kubectl -n shop set selector svc web 'app=web'
kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
for i in 1 2 3 4 5 6 7 8; do kubectl -n shop exec client -- wget -qO- web; done"""),
step("Weight the canary to 20%",
r"""Half the traffic is too much for a canary. The split follows the replica counts, so run **4** blue Pods and **1** green Pod: about one request in five reaches green.
When green has proved itself you would scale it up and blue down, then remove blue - that is the whole canary.""",
check=r"""
eq "$(j -n shop deploy web-blue '{.status.availableReplicas}')" 4 && eq "$(j -n shop deploy web-green '{.spec.replicas}')" 1 \
&& eq "$(j -n shop deploy web-green '{.status.availableReplicas}')" 1 \
&& eq "$(j -n shop svc web '{.spec.selector.app}')" web && [ -z "$(j -n shop svc web '{.spec.selector.version}')" ]""",
hint=r"""`kubectl scale deploy NAME --replicas=N`, once for each Deployment.""",
show=r"""kubectl -n shop scale deploy web-blue --replicas=4
kubectl -n shop scale deploy web-green --replicas=1
kubectl -n shop rollout status deploy web-blue
kubectl -n shop get deploy"""),
],
"outro": r"""You did a blue/green cutover, an instant rollback and a weighted canary with nothing but Deployments, labels and one Service.
**For the exam**
- The Service's selector is the switch. `kubectl get svc NAME -o wide` shows it; `kubectl get endpointslices -l kubernetes.io/service-name=NAME` shows who receives traffic.
- `kubectl set selector` replaces the whole selector; `kubectl patch` merges into it.
- Keep blue running until a task says otherwise: it is your rollback.
- A canary's share is its Pods divided by all the Pods behind the Service.
Next, prove it against the clock: **drill 2.1** gives you this objective's tasks with a timer, and no solution until a task is graded.""",
}
# ---- 2.2 rolling updates and rollbacks -------------------------------------------------------
L22 = {
"id": 9202, "objective": "2.2", "workers": 0,
"title": "Rolling updates, a bad release, and the way back",
"intro": r"""A Deployment never edits a running Pod. Change its Pod template - a new image, a new environment variable - and it creates a new ReplicaSet, then moves Pods over from the old one a few at a time. Two numbers decide how:
```
maxSurge extra Pods during an update
maxUnavailable Pods that may be missing
```
Each template change is a **revision**, and you can list revisions and go back to one.
In Namespace `rollout`, Deployment `api` runs `nginx:1.25` with 4 replicas.""",
"setup": NS + r"""
mkns rollout
$K -n rollout create deploy api --image=nginx:1.25 --replicas=4 >/dev/null
$K -n rollout rollout status deploy api --timeout=180s >/dev/null 2>&1
mkdir -p "$WS/lesson-2.2"
""",
"steps": [
step("Read the update strategy",
r"""Find the Deployment's `maxSurge` and `maxUnavailable`, and write them separated by a space to `{WS}/lesson-2.2/strategy.txt` - for example `25% 25%`.""",
check=r"""eq "$(tr -d '\r\n' < "$WS/lesson-2.2/strategy.txt" 2>/dev/null | tr -s ' ')" "25% 25%" """,
hint=r"""Both live under `.spec.strategy.rollingUpdate`. `kubectl explain deploy.spec.strategy.rollingUpdate` describes them, and `-o jsonpath='{...}'` prints only the values.""",
show=r"""kubectl -n rollout get deploy api -o jsonpath='{.spec.strategy.rollingUpdate.maxSurge} {.spec.strategy.rollingUpdate.maxUnavailable}' > {WS}/lesson-2.2/strategy.txt
cat {WS}/lesson-2.2/strategy.txt"""),
step("Never drop below four Pods",
r"""25% of 4 Pods means one Pod may be missing during an update. Make updates add a Pod before they remove one: set `maxUnavailable` to `0` and `maxSurge` to `1`.""",
check=r"""eq "$(j -n rollout deploy api '{.spec.strategy.rollingUpdate.maxUnavailable}')" 0 && eq "$(j -n rollout deploy api '{.spec.strategy.rollingUpdate.maxSurge}')" 1""",
hint=r"""A merge patch with `kubectl patch deploy api -p '{"spec":{"strategy":{...}}}'`, or `kubectl edit deploy api`.""",
show=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
kubectl -n rollout get deploy api -o jsonpath='{.spec.strategy}'; echo"""),
step("Roll out nginx 1.27, with a reason",
r"""Update the container `nginx` to image `nginx:1.27`, and record why with the annotation `kubernetes.io/change-cause` set to `nginx 1.27`: `kubectl rollout history` shows it in the CHANGE-CAUSE column. Follow the update with `kubectl rollout status` until it finishes.""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 \
&& eq "$(j -n rollout deploy api '{.metadata.annotations.kubernetes\.io/change-cause}')" "nginx 1.27" \
&& eq "$(j -n rollout deploy api '{.status.updatedReplicas}')" 4 && eq "$(j -n rollout deploy api '{.status.availableReplicas}')" 4 \
&& eq "$(j -n rollout deploy api '{.status.replicas}')" 4""",
hint=r"""`kubectl set image deploy/api nginx=nginx:1.27` - the name before `=` is the **container's**. Then `kubectl annotate deploy/api kubernetes.io/change-cause='nginx 1.27'`.""",
show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27
kubectl -n rollout annotate deploy/api kubernetes.io/change-cause='nginx 1.27' --overwrite
kubectl -n rollout rollout status deploy/api
kubectl -n rollout rollout history deploy/api"""),
step("Ship a bad release",
r"""Now a mistake: update to image `nginx:9.99-nope`, which does not exist. Run `kubectl rollout status` with `--timeout=20s` and look at the Pods: one new Pod cannot pull its image, and because of `maxUnavailable: 0` all four old Pods keep serving.
A rollout that cannot finish does not take your app down - it stalls.""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:9.99-nope || exit 1
eq "$(j -n rollout deploy api '{.status.availableReplicas}')" 4 || exit 1
for i in $(seq 1 20); do
$K -n rollout get pods -o jsonpath='{range .items[*]}{.status.containerStatuses[0].state.waiting.reason}{"\n"}{end}' 2>/dev/null \
| grep -Eq 'ErrImagePull|ImagePullBackOff' && exit 0
sleep 3
done; exit 1""",
hint=r"""`kubectl set image` once more. `kubectl get pods` shows `ErrImagePull` or `ImagePullBackOff`, and `kubectl describe pod NAME` says why.""",
show=r"""kubectl -n rollout set image deploy/api nginx=nginx:9.99-nope
kubectl -n rollout rollout status deploy/api --timeout=20s
kubectl -n rollout get pods"""),
step("Roll back",
r"""Undo the bad release: `kubectl rollout undo` returns to the previous revision, `nginx:1.27`, and the stuck Pod goes away. Wait until the rollout is complete.""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 \
&& eq "$(j -n rollout deploy api '{.status.updatedReplicas}')" 4 && eq "$(j -n rollout deploy api '{.status.availableReplicas}')" 4 \
&& eq "$(j -n rollout deploy api '{.status.replicas}')" 4""",
hint=r"""`kubectl rollout undo deploy/api` goes back one revision; `--to-revision=N` goes back to any one that `kubectl rollout history` lists.""",
show=r"""kubectl -n rollout rollout undo deploy/api
kubectl -n rollout rollout status deploy/api"""),
step("Read the history right",
r"""List the revisions with `kubectl rollout history deploy/api`. The undo did not go back to revision 2: it made the 1.27 template the **newest** revision, and revision 2 is gone from the list.
Write the current revision number to `{WS}/lesson-2.2/revision.txt`.""",
check=r"""
cur=$(j -n rollout deploy api '{.metadata.annotations.deployment\.kubernetes\.io/revision}')
[ -n "$cur" ] && eq "$(tr -dc '0-9' < "$WS/lesson-2.2/revision.txt" 2>/dev/null)" "$cur" """,
hint=r"""The last line of `kubectl rollout history` - or the Deployment's annotation `deployment.kubernetes.io/revision`.""",
show=r"""kubectl -n rollout rollout history deploy/api
kubectl -n rollout get deploy api -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}' > {WS}/lesson-2.2/revision.txt
cat {WS}/lesson-2.2/revision.txt"""),
],
"outro": r"""You read and tuned an update strategy, rolled out a new image with a recorded reason, watched a broken release stall without an outage, and rolled it back.
**For the exam**
- `kubectl set image deploy/NAME CONTAINER=IMAGE`: the **container** name goes before `=`.
- `kubectl rollout status` waits, `rollout history` lists revisions, `rollout undo [--to-revision=N]` goes back, and `rollout pause` / `resume` gather several changes into one rollout.
- `maxUnavailable: 0` with a `maxSurge` keeps full capacity during an update, and a broken image only stalls it.
- The `kubernetes.io/change-cause` annotation is what `rollout history` shows as CHANGE-CAUSE (`--record` no longer exists).
Next, prove it against the clock: **drill 2.2**.""",
}
# ---- 2.3 Helm -----------------------------------------------------------------------------------
L23 = {
"id": 9203, "objective": "2.3", "workers": 0,
"title": "Helm: install, upgrade, roll back",
"intro": r"""Helm installs an application from a **chart**: templates plus default values. One installation is a **release**, and every change to a release - an upgrade or a rollback - is a new **revision** that you can list and return to.
```
chart ── helm install shop ──▶ revision 1
── helm upgrade shop ──▶ revision 2, 3 ..
```
In the exam a chart usually comes from a repository (`helm repo add`, `helm search repo`). Here it is local, which works the same: a chart directory or a packaged `.tgz` can go anywhere a chart name can. In `{WS}/lesson-2.3` you will find a chart `webapp` 0.1.0 as a directory, `chart/`, and version 0.2.0 packaged as `webapp-0.2.0.tgz`.""",
"setup": r"""
d="$WS/lesson-2.3"; rm -rf "$d"; mkdir -p "$d"
src=$(mktemp -d)
for v in 0.1.0 0.2.0; do
c="$src/$v/webapp"; mkdir -p "$c/templates"
cat > "$c/Chart.yaml" <<EOF
apiVersion: v2
name: webapp
description: A small web app for the Helm lesson
version: $v
appVersion: "1.27"
EOF
cat > "$c/values.yaml" <<'EOF'
replicaCount: 1
image:
repository: nginx
tag: "1.27"
service:
type: ClusterIP
port: 80
EOF
cat > "$c/templates/deployment.yaml" <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-webapp
labels: {app.kubernetes.io/instance: {{ .Release.Name }}, helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels: {app.kubernetes.io/instance: {{ .Release.Name }}}
template:
metadata:
labels: {app.kubernetes.io/instance: {{ .Release.Name }}}
spec:
containers:
- name: web
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
ports:
- containerPort: 80
EOF
cat > "$c/templates/service.yaml" <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: {{ .Release.Name }}-webapp
spec:
type: {{ .Values.service.type }}
selector: {app.kubernetes.io/instance: {{ .Release.Name }}}
ports:
- port: {{ .Values.service.port }}
targetPort: 80
EOF
done
cp -r "$src/0.1.0/webapp" "$d/chart"
helm package "$src/0.2.0/webapp" -d "$d" >/dev/null
rm -rf "$src"
command -v helm >/dev/null || echo "warn: helm is not installed on this machine - the checks need it"
""",
"steps": [
step("Look inside before you install",
r"""See what a chart would do without touching the cluster. `helm show values` lists its knobs and their defaults; `helm template` renders the manifests Helm would apply.
Render the chart in `{WS}/lesson-2.3/chart` as release `shop` into `{WS}/lesson-2.3/rendered.yaml`, and read it.""",
check=r"""f="$WS/lesson-2.3/rendered.yaml"; grep -q 'kind: Deployment' "$f" 2>/dev/null && grep -q 'kind: Service' "$f" && grep -q 'name: shop-webapp' "$f" """,
hint=r"""`helm template RELEASE CHART` prints to stdout: redirect it with `> FILE`.""",
show=r"""helm show values {WS}/lesson-2.3/chart
helm template shop {WS}/lesson-2.3/chart > {WS}/lesson-2.3/rendered.yaml
cat {WS}/lesson-2.3/rendered.yaml"""),
step("Install it",
r"""Install the chart as release `shop` into Namespace `charts` - let Helm create the Namespace - with **2** replicas instead of the default 1.""",
check=r"""
helm -n charts status shop -o json 2>/dev/null | python3 -c 'import sys,json; sys.exit(0 if json.load(sys.stdin)["info"]["status"]=="deployed" else 1)' || exit 1
eq "$(j -n charts deploy shop-webapp '{.status.availableReplicas}')" 2""",
hint=r"""`helm install RELEASE CHART -n NAMESPACE --create-namespace --set replicaCount=2`. `helm list -n charts` shows the release.""",
show=r"""helm -n charts install shop {WS}/lesson-2.3/chart --create-namespace --set replicaCount=2
helm -n charts list
kubectl -n charts rollout status deploy shop-webapp"""),
step("Upgrade with a values file",
r"""`--set` is fine for one value; for several, a values file. Write `{WS}/lesson-2.3/prod-values.yaml` with `replicaCount: 3` and a Service of type `NodePort`, and upgrade `shop` with it.
`helm history` then lists a second revision.""",
check=r"""
grep -Eq '^replicaCount: *3' "$WS/lesson-2.3/prod-values.yaml" 2>/dev/null || exit 1
eq "$(j -n charts deploy shop-webapp '{.status.availableReplicas}')" 3 && eq "$(j -n charts svc shop-webapp '{.spec.type}')" NodePort || exit 1
[ "$(helm -n charts history shop -o json 2>/dev/null | python3 -c 'import sys,json; print(max(r["revision"] for r in json.load(sys.stdin)))')" -ge 2 ]""",
hint=r"""A values file has the shape of `values.yaml`: `service:` on one line, `type: NodePort` indented under it. Then `helm upgrade shop CHART -n charts -f FILE`.""",
show=r"""cat > {WS}/lesson-2.3/prod-values.yaml <<'EOF'
replicaCount: 3
service:
type: NodePort
EOF
helm -n charts upgrade shop {WS}/lesson-2.3/chart -f {WS}/lesson-2.3/prod-values.yaml
kubectl -n charts rollout status deploy shop-webapp
helm -n charts history shop"""),
step("A new chart version, with your values",
r"""Upgrade `shop` to chart version 0.2.0, `{WS}/lesson-2.3/webapp-0.2.0.tgz`, **keeping the values you set**.
Careful: `helm upgrade` without values starts from the chart's defaults again, so your 3 replicas and the NodePort would be gone. `--reuse-values` keeps them (or pass `-f prod-values.yaml` once more).""",
check=r"""
helm -n charts list -o json 2>/dev/null | python3 -c 'import sys,json; r=[x for x in json.load(sys.stdin) if x["name"]=="shop"]; sys.exit(0 if r and r[0]["chart"]=="webapp-0.2.0" else 1)' || exit 1
eq "$(j -n charts deploy shop-webapp '{.spec.replicas}')" 3 && eq "$(j -n charts svc shop-webapp '{.spec.type}')" NodePort""",
hint=r"""`helm upgrade shop {WS}/lesson-2.3/webapp-0.2.0.tgz -n charts --reuse-values`. `helm list -n charts` shows the chart version, `helm get values shop -n charts` the values in use.""",
show=r"""helm -n charts upgrade shop {WS}/lesson-2.3/webapp-0.2.0.tgz --reuse-values
helm -n charts list
helm -n charts get values shop"""),
step("Roll back",
r"""Go back to revision **2**: chart 0.1.0 with your production values. Look at `helm history` before and after - a rollback is a new revision too, it does not rewind the counter.""",
check=r"""
helm -n charts history shop -o json 2>/dev/null | python3 -c '
import sys, json
h = json.load(sys.stdin); last = max(h, key=lambda r: r["revision"])
sys.exit(0 if last["revision"] >= 4 and last["chart"] == "webapp-0.1.0" and last["status"] == "deployed" else 1)' || exit 1
eq "$(j -n charts deploy shop-webapp '{.spec.replicas}')" 3""",
hint=r"""`helm history shop -n charts`, then `helm rollback shop 2 -n charts`.""",
show=r"""helm -n charts history shop
helm -n charts rollback shop 2
helm -n charts history shop"""),
step("Clean up",
r"""Uninstall release `shop`. Helm deletes everything the release created - the Deployment and the Service - and its history too, unless you add `--keep-history`. The Namespace stays.""",
check=r"""! helm -n charts status shop >/dev/null 2>&1 && [ -z "$(j -n charts deploy shop-webapp '{.metadata.name}')" ] && [ -z "$(j -n charts svc shop-webapp '{.metadata.name}')" ]""",
hint=r"""`helm uninstall shop -n charts`. `helm list -a -n charts` would still show a release uninstalled with `--keep-history`.""",
show=r"""helm -n charts uninstall shop
kubectl -n charts get all"""),
],
"outro": r"""You rendered a chart, installed it, upgraded it with a values file and to a new chart version, rolled it back, and removed it.
**For the exam**
- `helm repo add NAME URL`, `helm repo update` and `helm search repo` find charts; a directory or a `.tgz` works wherever a chart name does.
- `helm show values` and `helm template` let you look before you install; `helm get values RELEASE` shows what a release runs with.
- `--set` for one value, `-f values.yaml` for several. On `upgrade`, values you do not pass go back to the chart's defaults unless you add `--reuse-values`.
- `helm history` and `helm rollback RELEASE REVISION`: every change, a rollback included, is a new revision.
- Always give the Namespace: `-n NAMESPACE`, and `--create-namespace` on install.
Next, prove it against the clock: **drill 2.3**.""",
}
# ---- 2.4 Kustomize ------------------------------------------------------------------------------
_RENDER = r"""
# rendered NAME: the overlay's objects as JSON, one per line, checked without applying anything
rendered() {
$K kustomize "$WS/lesson-2.4/overlays/$1" 2>/dev/null | $K create --dry-run=client -f - -o json 2>/dev/null | python3 -c '
import sys, json
t, dec, i = sys.stdin.read(), json.JSONDecoder(), 0
while True: # one List, or one JSON document per object - kubectl has printed both
while i < len(t) and t[i].isspace(): i += 1
if i >= len(t): break
o, i = dec.raw_decode(t, i)
for x in (o.get("items", []) if o.get("kind") == "List" else [o]): print(json.dumps(x))'
}
"""
L24 = {
"id": 9204, "objective": "2.4", "workers": 0,
"title": "Kustomize: one base, two environments",
"intro": r"""Kustomize builds plain YAML from a directory that holds a `kustomization.yaml`. There are no templates and no variables: a **base** holds the manifests, and an **overlay** points at the base and changes it - Namespace, names, labels, images, replicas, or anything else through a patch - without editing it.
```
base/ web: Deployment + Service
overlays/staging/ staging-web in staging
overlays/prod/ prod-web in prod, 3 x 1.27
```
`kubectl` has Kustomize built in: `kubectl kustomize DIR` renders, `kubectl apply -k DIR` applies. In `{WS}/lesson-2.4/base` you will find a Deployment `web` (`nginx:1.25`, 1 replica), a Service `web` and their `kustomization.yaml`. The Namespaces `staging` and `prod` exist already.""",
"setup": NS + r"""
mkns staging; mkns prod
d="$WS/lesson-2.4"; rm -rf "$d"; mkdir -p "$d/base"
cat > "$d/base/deployment.yaml" <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
labels: {app: web}
spec:
replicas: 1
selector:
matchLabels: {app: web}
template:
metadata:
labels: {app: web}
spec:
containers:
- name: web
image: nginx:1.25
ports:
- containerPort: 80
EOF
cat > "$d/base/service.yaml" <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector: {app: web}
ports:
- port: 80
targetPort: 80
EOF
cat > "$d/base/kustomization.yaml" <<'EOF'
resources:
- deployment.yaml
- service.yaml
EOF
( cd "$d/base" && sha256sum deployment.yaml service.yaml kustomization.yaml ) > "$PRIV/lesson-2.4-base.sum"
""",
"steps": [
step("Render the base",
r"""See what Kustomize makes of the base before you change anything: render `{WS}/lesson-2.4/base` into `{WS}/lesson-2.4/base.yaml`, and read it.""",
check=r"""f="$WS/lesson-2.4/base.yaml"; grep -q 'kind: Deployment' "$f" 2>/dev/null && grep -q 'kind: Service' "$f" && grep -q 'image: nginx:1.25' "$f" """,
hint=r"""`kubectl kustomize DIR` prints the result - the same as `kustomize build DIR`. Redirect it with `> FILE`.""",
show=r"""kubectl kustomize {WS}/lesson-2.4/base > {WS}/lesson-2.4/base.yaml
cat {WS}/lesson-2.4/base.yaml"""),
step("An overlay for staging",
r"""Create `{WS}/lesson-2.4/overlays/staging/kustomization.yaml`. It uses the base `../../base`, puts everything into Namespace `staging`, prefixes every name with `staging-`, and gives every resource the label `env: staging`.
For the label, use `labels:` with `includeSelectors: true`, which adds it to the selectors too. The older `commonLabels` does the same but is deprecated. Render the overlay to check it.""",
check=_RENDER + r"""
rendered staging | python3 -c '
import sys, json
items = [json.loads(line) for line in sys.stdin if line.strip()]
by = {(o["kind"], o["metadata"]["name"]): o for o in items}
dep, svc = by.get(("Deployment", "staging-web")), by.get(("Service", "staging-web"))
ok = bool(dep and svc) and all(o["metadata"].get("namespace") == "staging" and o["metadata"].get("labels", {}).get("env") == "staging" for o in (dep, svc))
sys.exit(0 if ok and dep["spec"]["template"]["metadata"]["labels"].get("env") == "staging" else 1)'""",
hint=r"""```
resources:
- ../../base
namespace: staging
namePrefix: staging-
labels:
- pairs:
env: staging
includeSelectors: true
```""",
show=r"""mkdir -p {WS}/lesson-2.4/overlays/staging
cat > {WS}/lesson-2.4/overlays/staging/kustomization.yaml <<'EOF'
resources:
- ../../base
namespace: staging
namePrefix: staging-
labels:
- pairs:
env: staging
includeSelectors: true
EOF
kubectl kustomize {WS}/lesson-2.4/overlays/staging"""),
step("Apply staging",
r"""Apply the staging overlay with kubectl's built-in Kustomize, then look at what you got.""",
check=r"""
eq "$(j -n staging deploy staging-web '{.status.availableReplicas}')" 1 && eq "$(j -n staging deploy staging-web '{.metadata.labels.env}')" staging \
&& [ -n "$(j -n staging svc staging-web '{.metadata.name}')" ]""",
hint=r"""`kubectl apply -k DIR` - `-k` instead of `-f`.""",
show=r"""kubectl apply -k {WS}/lesson-2.4/overlays/staging
kubectl -n staging get deploy,svc --show-labels"""),
step("A prod overlay: replicas and image",
r"""Production differs in more than its name. Create the overlay `{WS}/lesson-2.4/overlays/prod`: the same base, Namespace `prod`, prefix `prod-`, label `env: prod`, **3** replicas and image `nginx:1.27`.
Use Kustomize's own fields for the last two, `replicas:` and `images:` - do not copy the Deployment. Render it to check.""",
check=_RENDER + r"""
rendered prod | python3 -c '
import sys, json
items = [json.loads(line) for line in sys.stdin if line.strip()]
by = {(o["kind"], o["metadata"]["name"]): o for o in items}
dep = by.get(("Deployment", "prod-web"))
ok = bool(dep) and dep["metadata"].get("namespace") == "prod" and dep["metadata"].get("labels", {}).get("env") == "prod"
ok = ok and dep["spec"].get("replicas") == 3 and dep["spec"]["template"]["spec"]["containers"][0]["image"] == "nginx:1.27"
sys.exit(0 if ok and ("Service", "prod-web") in by else 1)'""",
hint=r"""Besides `resources`, `namespace`, `namePrefix` and `labels`:
```
replicas:
- name: web
count: 3
images:
- name: nginx
newTag: "1.27"
```
`replicas` names the Deployment by its base name; `images` names the image, not the container.""",
show=r"""mkdir -p {WS}/lesson-2.4/overlays/prod
cat > {WS}/lesson-2.4/overlays/prod/kustomization.yaml <<'EOF'
resources:
- ../../base
namespace: prod
namePrefix: prod-
labels:
- pairs:
env: prod
includeSelectors: true
replicas:
- name: web
count: 3
images:
- name: nginx
newTag: "1.27"
EOF
kubectl kustomize {WS}/lesson-2.4/overlays/prod"""),
step("Patch what the fields cannot reach",
r"""For anything else there is a patch. Give the `web` container in prod a memory limit of `128Mi` with a patch file in the prod overlay, listed under `patches:` - then apply the prod overlay.""",
check=r"""
eq "$(j -n prod deploy prod-web '{.status.availableReplicas}')" 3 \
&& eq "$(j -n prod deploy prod-web '{.spec.template.spec.containers[0].image}')" nginx:1.27 \
&& eq "$(j -n prod deploy prod-web '{.spec.template.spec.containers[0].resources.limits.memory}')" 128Mi \
&& eq "$(j -n prod deploy prod-web '{.metadata.labels.env}')" prod""",
hint=r"""A strategic-merge patch looks like a small Deployment that names only what changes - by the **base** name `web`:
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
template:
spec:
containers:
- name: web
resources:
limits:
memory: 128Mi
```
and in `kustomization.yaml`: `patches:` with `- path: limits.yaml`.""",
show=r"""cat > {WS}/lesson-2.4/overlays/prod/limits.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
template:
spec:
containers:
- name: web
resources:
limits:
memory: 128Mi
EOF
cat >> {WS}/lesson-2.4/overlays/prod/kustomization.yaml <<'EOF'
patches:
- path: limits.yaml
EOF
kubectl apply -k {WS}/lesson-2.4/overlays/prod
kubectl -n prod rollout status deploy prod-web"""),
step("One base, two environments",
r"""Look at both environments side by side: `kubectl get deploy -A -l app=web -o wide`. They come from one base that you never edited - the exam checks that too.
Write the image that **staging** runs to `{WS}/lesson-2.4/staging-image.txt`.""",
check=r"""
( cd "$WS/lesson-2.4/base" && sha256sum -c --quiet "$PRIV/lesson-2.4-base.sum" >/dev/null 2>&1 ) || exit 1
eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/staging-image.txt" 2>/dev/null)" nginx:1.25 || exit 1
[ -n "$(j -n staging deploy staging-web '{.metadata.name}')" ] && [ -n "$(j -n prod deploy prod-web '{.metadata.name}')" ]""",
hint=r"""`-o jsonpath='{.spec.template.spec.containers[0].image}'` prints just the image.""",
show=r"""kubectl get deploy -A -l app=web -o wide
kubectl -n staging get deploy staging-web -o jsonpath='{.spec.template.spec.containers[0].image}' > {WS}/lesson-2.4/staging-image.txt
cat {WS}/lesson-2.4/staging-image.txt"""),
],
"outro": r"""You rendered a base, built two overlays on it with Kustomize's own fields and a patch, and applied both - and the base stayed as it was.
**For the exam**
- `kubectl kustomize DIR` renders, `kubectl apply -k DIR` applies, `kubectl delete -k DIR` removes.
- An overlay lists its base under `resources:` and never edits it.
- Built-in fields first - `namespace`, `namePrefix` / `nameSuffix`, `labels`, `images`, `replicas` - and a patch for anything else. A patch names objects by their **base** names.
- `configMapGenerator` and `secretGenerator` make ConfigMaps and Secrets whose names change with their content, so a Deployment that uses one rolls when the data changes - drill 2.4 has one.
Next, prove it against the clock: **drill 2.4**.""",
}
LESSONS = [L21, L22, L23, L24]

47
cks/bank_harden.py
View File

@ -41,9 +41,12 @@ print(json.dumps(p) if isinstance(p, (list, dict)) else p)' "$1" 2>/dev/null; }
# older than the node's for the three packages, held as kubeadm's documentation asks.
APT = r"""
K8S_LIST='-o Dir::Etc::sourcelist=/etc/apt/sources.list.d/kubernetes.list -o Dir::Etc::sourceparts=- -o APT::Get::List-Cleanup=0'
# hold_back NODE VERSION: put NODE's kubelet, kubeadm and kubectl on VERSION and restart its kubelet
# hold_back NODE VERSION: put NODE's kubelet, kubeadm and kubectl on VERSION and restart its kubelet.
# pkgs.k8s.io has one repository per minor, so the node's repository line moves to VERSION's minor
# first, and stays there - as on a node installed with that minor.
hold_back() {
nx "$1" "apt-get update -qq $K8S_LIST >/dev/null 2>&1
nx "$1" "sed -i 's#/core:/stable:/v[0-9.]*/deb/#/core:/stable:/v${2%.*}/deb/#' /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq $K8S_LIST >/dev/null 2>&1
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --allow-downgrades --allow-change-held-packages -o Dpkg::Options::=--force-confold kubelet=$2-1.1 kubeadm=$2-1.1 kubectl=$2-1.1 >/dev/null 2>&1 || exit 1
apt-mark hold kubelet kubeadm kubectl >/dev/null; systemctl daemon-reload; systemctl restart kubelet" </dev/null || return 1
local i; for i in $(seq 1 40); do [ "$(j node "$1" '{.status.nodeInfo.kubeletVersion}')" = "v$2" ] && return 0; sleep 3; done; return 1
@ -715,7 +718,7 @@ openssl x509 -in %{user}.crt -noout -subject -dates
# ================================================================= 2.4 Upgrade ====
{
"id": 37, "domain": "HARDEN", "title": "A worker on a vulnerable kubelet", "workers": 1, "difficulty": "hard",
"params": lambda r: {"ns": ns_pick(r, "web"), "old": r.choice(["1.35.4", "1.35.5", "1.35.6"])},
"params": lambda r: {"ns": ns_pick(r, "web"), "old": r.choice(["1.36.3", "1.36.4", "1.36.5"])},
"task": """
Node `{W1}` still runs Kubernetes **v%{old}**, a release with known vulnerabilities. The control
plane is newer (`k get nodes`).
@ -752,29 +755,33 @@ eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" "$(j node "$CP" '{.stat
for i in $(seq 1 20); do [ "$(j -n %{ns} deploy web '{.status.availableReplicas}')" = 2 ] && exit 0; sleep 3; done; exit 1"""),
],
"solution": """
k get nodes # {W1}: v%{old}; the control plane's version is the target, say 1.35.X
k get nodes # {W1}: v%{old}; the control plane's version is the target, say 1.37.X
k drain {W1} --ignore-daemonsets --delete-emptydir-data
{SHELL_W1}
sudo -i
cat /etc/apt/sources.list.d/kubernetes.list # .../core:/stable:/v1.36/deb/ - the node's minor
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-cache madison kubeadm | head -5 # the package versions on offer
apt-mark unhold kubeadm && apt-get install -y kubeadm=1.35.X-1.1 && apt-mark hold kubeadm
apt-mark unhold kubeadm && apt-get install -y kubeadm=1.37.X-1.1 && apt-mark hold kubeadm
kubeadm upgrade node # a worker: `node`, not `apply`
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.35.X-1.1 kubectl=1.35.X-1.1 && apt-mark hold kubelet kubectl
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.37.X-1.1 kubectl=1.37.X-1.1 && apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet
exit; exit
k uncordon {W1}
k get nodes # both on the same version, Ready
# docs: "Upgrading kubeadm clusters" -> "Upgrade worker nodes". kubeadm first, because it rewrites
# the kubelet's configuration for the new version; the kubelet package last, then the restart -
# `k get nodes` shows the version of the running kubelet, not of the package.
# docs: "Upgrading kubeadm clusters" -> "Changing the package repository", then "Upgrade worker
# nodes". {W1} is a minor behind, and pkgs.k8s.io serves one repository per minor: until the line
# names v1.37, apt offers nothing newer than 1.36. kubeadm first, because it rewrites the kubelet's
# configuration for the new version; the kubelet package last, then the restart - `k get nodes`
# shows the version of the running kubelet, not of the package.
""",
},
{
"id": 38, "domain": "HARDEN", "title": "Upgrade the control plane", "workers": 0, "difficulty": "hard",
"task": """
A vulnerability in the API server was fixed in a newer patch release of Kubernetes v1.35.
A vulnerability in the API server was fixed in a newer patch release of Kubernetes v1.37.
Upgrade this cluster — a single control-plane node, `{CP}` — to the **newest v1.35 patch release**
Upgrade this cluster — a single control-plane node, `{CP}` — to the **newest v1.37 patch release**
that the node's package repository offers:
1. Upgrade the `kubeadm` package, look at `kubeadm upgrade plan`, and apply the upgrade.
@ -790,7 +797,7 @@ t=$(cat "$PRIV/q38.target"); cur=$(pkgs "$CP" | cut -d' ' -f1)
[ "$t" != "$cur" ] || { echo "the repository has nothing newer than $cur: there is no upgrade to do" >&2; exit 1; }
""",
"checks": [
C("the API server runs the newest v1.35 patch release", 4, API + r"""
C("the API server runs the newest v1.37 patch release", 4, API + r"""
t=$(cat "$PRIV/q38.target"); wait_api 60 || exit 1
eq "$($K version -o json 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["serverVersion"]["gitVersion"])')" "v${t%%-*}" """),
C("the node's kubelet does too, and the node is Ready and schedulable", 3, API + r"""
@ -806,13 +813,13 @@ j -n kube-system pod "kube-controller-manager-$CP" '{.spec.containers[0].image}'
],
"solution": """
sudo -i
apt-get update && apt-cache madison kubeadm | head -3 # newest first, say 1.35.X-1.1
apt-mark unhold kubeadm && apt-get install -y kubeadm=1.35.X-1.1 && apt-mark hold kubeadm
apt-get update && apt-cache madison kubeadm | head -3 # newest first, say 1.37.X-1.1
apt-mark unhold kubeadm && apt-get install -y kubeadm=1.37.X-1.1 && apt-mark hold kubeadm
kubeadm version -o short
kubeadm upgrade plan # what would change, and to which version
kubeadm upgrade apply v1.35.X # a few minutes; answers y
kubeadm upgrade apply v1.37.X # a few minutes; answers y
kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.35.X-1.1 kubectl=1.35.X-1.1 && apt-mark hold kubelet kubectl
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.37.X-1.1 kubectl=1.37.X-1.1 && apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon {CP}
kubectl get nodes; kubectl version
@ -824,7 +831,7 @@ kubectl get nodes; kubectl version
},
{
"id": 39, "domain": "HARDEN", "title": "Prepare an upgrade", "workers": 2,
"params": lambda r: {"old": r.choice(["1.35.4", "1.35.5", "1.35.6"]), "n": r.choice([1, 2])},
"params": lambda r: {"old": r.choice(["1.36.3", "1.36.4", "1.36.5"]), "n": r.choice([1, 2])},
"task": """
A CVE affects every kubelet older than the version this cluster's control plane runs. Prepare the
upgrade — do **not** upgrade anything yet.
@ -833,8 +840,8 @@ upgrade — do **not** upgrade anything yet.
`{WS}/q39/outdated.txt`, one per line.
2. Cordon those nodes, so that no new Pod lands on a vulnerable kubelet. Do not drain them, and
leave the other nodes schedulable.
3. Write the newest v1.35 package version of `kubeadm` that the repository on `{CP}` offers
(as apt prints it, for example `1.35.2-1.1`) to `{WS}/q39/target.txt`.
3. Write the newest v1.37 package version of `kubeadm` that the repository on `{CP}` offers
(as apt prints it, for example `1.37.2-1.1`) to `{WS}/q39/target.txt`.
4. Save the output of `kubeadm upgrade plan` on `{CP}` to `{WS}/q39/plan.txt`.
""",
"setup": API + APT + r"""
@ -867,7 +874,7 @@ sudo kubeadm upgrade plan | tee {WS}/q39/plan.txt
# cordon only marks the node unschedulable: running Pods stay. drain = cordon + evict.
# `upgrade plan` reads the cluster and changes nothing: it lists the current and the target version
# of each component and which kubelets it found. The kubelet may be up to three minor versions
# behind the API server, never ahead - but an old patch release still carries its bugs.
# behind the API server, never ahead - but an old release still carries its bugs.
""",
},
]

2
cks/bank_runtime.py
View File

@ -327,7 +327,7 @@ t = datetime.datetime(2026, 9, 14, 2, 0, 0)
ACTOR, IP = "system:serviceaccount:%{ns}:%{sa}", "%{ip}"
out = []
def ev(user, groups, ip, verb, res, ns, name, code, sub=None, group="", agent="kubectl/v1.35.8"):
def ev(user, groups, ip, verb, res, ns, name, code, sub=None, group="", agent="kubectl/v1.37.0"):
global t
t += datetime.timedelta(seconds=r.randint(1, 9), microseconds=r.randint(0, 999999))
uri = ("/api/v1" if not group else "/apis/" + group + "/v1") + ("/namespaces/" + ns if ns else "") + "/" + res + ("/" + name if name else "") + ("/" + sub if sub else "")

8
cks/curriculum.py
View File

@ -245,15 +245,15 @@ k config set-credentials jane --client-certificate=jane.crt --client-key=jane.ke
],
"fast": """
kubeadm upgrade plan
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm='1.35.x-*' && apt-mark hold kubeadm
kubeadm upgrade apply v1.35.x # workers: kubeadm upgrade node
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm='1.37.x-*' && apt-mark hold kubeadm
kubeadm upgrade apply v1.37.x # workers: kubeadm upgrade node
k drain NODE --ignore-daemonsets --delete-emptydir-data
apt-mark unhold kubelet kubectl && apt-get install -y kubelet='1.35.x-*' kubectl='1.35.x-*' && apt-mark hold kubelet kubectl
apt-mark unhold kubelet kubectl && apt-get install -y kubelet='1.37.x-*' kubectl='1.37.x-*' && apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet; k uncordon NODE
k get nodes; kubelet --version
""",
"traps": [
"The package repository is per minor version (pkgs.k8s.io/core:/stable:/v1.35): a new minor needs the repository line changed first.",
"The package repository is per minor version (pkgs.k8s.io/core:/stable:/v1.37): a new minor needs the repository line changed first.",
"`kubectl get nodes` shows the kubelet's version, so a node looks old until its kubelet is restarted.",
"Forgetting to uncordon leaves the node unschedulable.",
],

15
cks/questions.py
View File

@ -15,10 +15,12 @@ TRACK = {
"name": "Certified Kubernetes Security Specialist",
"minutes": 120,
"pass_pct": 67,
# Kubernetes minor of the real exam environment, which picks the kind cluster (shared with CKA).
# LF "Important Instructions: CKS", checked 2026-10-02: "The CKS environment is currently running
# Kubernetes v1.35". (cncf/curriculum still names the PDF "CKS_Curriculum v1.34".)
"k8s": "v1.35",
# Kubernetes minor of the cluster, which picks the kind node image (shared with CKA). Ahead of
# the real exam on purpose: LF "Important Instructions: CKS", checked 2026-10-19, still says "The
# CKS environment is currently running Kubernetes v1.35", but CKA and CKAD are on v1.37 and
# CKS is expected to follow by the time it is taken. (cncf/curriculum still names the PDF
# "CKS_Curriculum v1.34".)
"k8s": "v1.37",
# Worker nodes of a task's cluster unless the task says otherwise ("workers", or WORKERS below)
"workers": 1,
# CKS hosts are the common exam host plus the scanners, gVisor, istioctl and Kubernetes as apt
@ -239,8 +241,9 @@ $K -n %{ns} create deploy %{dep} --image=nginx:1.27 >/dev/null
C("%{dep} uses %{dep}-sa, which does mount a token", 3, r"""
eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.serviceAccountName}')" %{dep}-sa \
&& [ "$(j -n %{ns} sa %{dep}-sa '{.automountServiceAccountToken}')" != false ]"""),
C("%{dep} pods have a projected token mounted", 3, r"""
p=$(j -n %{ns} pod -l app=%{dep} '{.items[0].metadata.name}'); [ -n "$p" ] || exit 1
C("%{dep} pods run as %{dep}-sa, with a projected token mounted", 3, r"""
# a Pod of the new template: untouched, the Deployment's Pods run as default - with a token
p=$(j -n %{ns} pod -l app=%{dep} '{.items[?(@.spec.serviceAccountName=="%{dep}-sa")].metadata.name}' | awk '{print $1}'); [ -n "$p" ] || exit 1
$K -n %{ns} exec "$p" -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null"""),
C("a pod on the default ServiceAccount has no token", 3, r"""
cleanup() { $K -n %{ns} delete pod plain --wait=false >/dev/null 2>&1; }

2
cks/solve.py
View File

@ -482,7 +482,7 @@ kubectl -n %{ns} create rolebinding auditors-view --clusterrole=view --group=aud
37: r"""
V=$(kubectl get node {CP} -o jsonpath='{.status.nodeInfo.kubeletVersion}' | tr -d v)
kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data --force
ssh -n {W1} "sudo apt-get update -qq; sudo apt-mark unhold kubeadm kubelet kubectl >/dev/null; sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$V-1.1 >/dev/null && sudo kubeadm upgrade node && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$V-1.1 kubectl=$V-1.1 >/dev/null; sudo apt-mark hold kubeadm kubelet kubectl; sudo systemctl daemon-reload; sudo systemctl restart kubelet"
ssh -n {W1} "sudo sed -i 's|/v[0-9.]*/deb/|/v${V%.*}/deb/|' /etc/apt/sources.list.d/kubernetes.list; sudo apt-get update -qq; sudo apt-mark unhold kubeadm kubelet kubectl >/dev/null; sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$V-1.1 >/dev/null && sudo kubeadm upgrade node && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$V-1.1 kubectl=$V-1.1 >/dev/null; sudo apt-mark hold kubeadm kubelet kubectl; sudo systemctl daemon-reload; sudo systemctl restart kubelet"
kubectl uncordon {W1}
kubectl -n %{ns} rollout status deploy web --timeout=180s
sleep 15

18
images/desktop/Containerfile
View File

@ -1,8 +1,20 @@
# The exam's remote desktop: XFCE with a terminal and Firefox, served to the browser over noVNC.
# It is the "base" machine of a session - no kubeconfig here; every task says which host to ssh to.
FROM docker.io/library/debian:trixie-slim
# The exam's remote desktop: XFCE on Ubuntu 24.04 with a terminal and Firefox, served to the
# browser over noVNC. It is the "base" machine of a session - no kubeconfig here; every task says
# which host to ssh to.
FROM docker.io/library/ubuntu:24.04
ENV DEBIAN_FRONTEND=noninteractive LANG=C.UTF-8
# Ubuntu's own firefox package only installs the snap, which cannot run in a container: Firefox ESR
# comes from Mozilla's APT repository instead, pinned above Ubuntu's. Ubuntu's image has a user
# `ubuntu` on uid 1000, which is the candidate's.
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl \
&& install -d -m 0755 /etc/apt/keyrings \
&& curl -fsSL -o /etc/apt/keyrings/packages.mozilla.org.asc https://packages.mozilla.org/apt/repo-signing-key.gpg \
&& echo 'deb [signed-by=/etc/apt/keyrings/packages.mozilla.org.asc] https://packages.mozilla.org/apt mozilla main' \
> /etc/apt/sources.list.d/mozilla.list \
&& printf 'Package: *\nPin: origin packages.mozilla.org\nPin-Priority: 1000\n' > /etc/apt/preferences.d/mozilla \
&& userdel -r ubuntu \
&& apt-get update \
&& apt-get install -y --no-install-recommends \
xfce4 xfce4-terminal dbus-x11 elementary-xfce-icon-theme greybird-gtk-theme \
fonts-dejavu fonts-liberation \

32
images/node-cks/Containerfile
View File

@ -1,38 +1,15 @@
# CKS exam host: the common exam host (images/node) plus what the CKS hosts carry - the scanners,
# a sandbox runtime, a service mesh client, and Kubernetes installed as real packages so that an
# upgrade is `apt-get install` and `kubeadm upgrade`, as on the day. Built by `./ksim build cks`.
# a sandbox runtime and a service mesh client. Built by `./ksim build cks`.
ARG BASE
FROM ${BASE}
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
python3-minimal bsdextrautils binutils xxd strace file bzip2 gnupg conntrack tcpdump lsof kmod vsftpd \
python3-minimal binutils strace file bzip2 conntrack tcpdump lsof kmod vsftpd \
&& rm -rf /var/lib/apt/lists/* \
# vsftpd is here to be found and removed in a task; it must not run on every host
&& (systemctl disable vsftpd >/dev/null 2>&1 || true) \
&& rm -f /etc/systemd/system/multi-user.target.wants/vsftpd.service
# --- Kubernetes as packages -------------------------------------------------------------------
# A kind node carries kubelet, kubeadm and kubectl as loose binaries. Installing the same version
# from pkgs.k8s.io puts them under dpkg, so a task can hold a node back on an older patch release
# and the candidate upgrades it the documented way. kind's own kubelet unit and drop-ins live in
# /etc/systemd and keep taking precedence over the package's.
ARG K8S_VERSION
ARG K8S_MINOR
RUN mkdir -p /etc/apt/keyrings \
&& curl -fsSL "https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/Release.key" | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg \
&& echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/ /" > /etc/apt/sources.list.d/kubernetes.list \
# kubeadm refuses to upgrade a node without the cpuset cgroup controller, which a rootless
# container does not get. The packaged binary is diverted and a wrapper (rootfs/usr/bin/kubeadm)
# passes --ignore-preflight-errors=SystemVerification to `kubeadm upgrade`, so the candidate
# types the same commands as on a real node. The diversion survives package upgrades.
&& dpkg-divert --add --rename --divert /usr/bin/kubeadm.real /usr/bin/kubeadm \
&& apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends -o Dpkg::Options::=--force-confold \
"kubelet=${K8S_VERSION}-*" "kubeadm=${K8S_VERSION}-*" "kubectl=${K8S_VERSION}-*" \
&& apt-mark hold kubelet kubeadm kubectl \
&& rm -rf /var/lib/apt/lists/* \
&& test -x /usr/bin/kubeadm.real
# --- CKS tools --------------------------------------------------------------------------------
ARG KUBE_BENCH=0.16.0
ARG TRIVY=0.75.0
@ -56,9 +33,8 @@ RUN cd /tmp && G=https://github.com \
&& chown -R root:root /usr/local/bin /etc/kube-bench
COPY rootfs/ /
RUN chmod 755 /usr/bin/kubeadm \
# gVisor as a second runtime handler ("runsc") beside runc; a RuntimeClass selects it
&& printf '\n[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]\n runtime_type = "io.containerd.runsc.v1"\n [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]\n TypeUrl = "io.containerd.runsc.v1.options"\n ConfigPath = "/etc/containerd/runsc.toml"\n' >> /etc/containerd/config.toml \
# gVisor as a second runtime handler ("runsc") beside runc; a RuntimeClass selects it
RUN printf '\n[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]\n runtime_type = "io.containerd.runsc.v1"\n [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]\n TypeUrl = "io.containerd.runsc.v1.options"\n ConfigPath = "/etc/containerd/runsc.toml"\n' >> /etc/containerd/config.toml \
# trivy reads /etc/trivy.yaml (the Docker Hub cache as a mirror, one cache directory for all
# users), for logins and for sudo alike
&& install -d -m 1777 /var/lib/trivy \

101
images/node/Containerfile
View File

@ -1,23 +1,101 @@
# Exam host: a kind node (kubeadm, kubelet, containerd, static-pod control plane) that you reach
# with `ssh <host>` as `candidate`, carrying the tools the real exam hosts have. Built once per
# Kubernetes minor by `./ksim build`; every question gets its own cluster from this image.
# Exam host: a kind node (kubeadm, kubelet, containerd, static-pod control plane) on Ubuntu 24.04,
# as the exam's hosts are, that you reach with `ssh <host>` as `candidate`, carrying the tools the
# real exam hosts have. Built once per Kubernetes minor by `./ksim build`; every question gets its
# own cluster from this image.
ARG BASE
ARG ETCD_IMAGE
FROM ${ETCD_IMAGE} AS etcd
FROM ${BASE} AS kind
# kind's node image is Debian: kind's images/base/Dockerfile (v0.33.0) on debian:trixie-slim, plus
# Kubernetes. This is the same node on Ubuntu. kind's packages and systemd trimming are done here;
# what does not depend on the distribution is copied from kind's image: its scripts and units,
# containerd, runc, crictl, the CNI plugins and the preloaded images. Kubernetes itself comes from
# pkgs.k8s.io below, as on a node installed with kubeadm - and with it the kubernetes-cni package,
# whose plugins (the same release) take the place of kind's.
FROM docker.io/library/ubuntu:24.04
# Ubuntu's container image is minimized: no man pages, and `man` a stub. The exam hosts have them
# ("man and man pages for further documentation"), so they come back the way `unminimize` does it;
# docs and translations stay out. Its user `ubuntu` holds uid 1000, which is the candidate's.
RUN printf 'path-exclude=/usr/share/doc/*\npath-include=/usr/share/doc/*/copyright\npath-exclude=/usr/share/locale/*/LC_MESSAGES/*.mo\n' \
> /etc/dpkg/dpkg.cfg.d/excludes \
&& rm -f /usr/bin/man && dpkg-divert --quiet --remove --rename /usr/bin/man \
&& userdel -r ubuntu \
&& apt-get update \
&& dpkg -S /usr/share/man/ | sed 's|, |\n|g;s|: [^:]*$||' | DEBIAN_FRONTEND=noninteractive xargs apt-get install --reinstall -y -qq \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
systemd \
conntrack iptables nftables iproute2 ethtool util-linux mount kmod \
libseccomp2 pigz fuse-overlayfs \
nfs-common open-iscsi \
e2fsprogs \
bash ca-certificates curl jq procps \
man-db manpages \
&& rm -rf /var/lib/apt/lists/*
COPY --from=kind /kind/ /kind/
COPY --from=kind /LICENSES/ /LICENSES/
COPY --from=kind /usr/local/bin/ /usr/local/bin/
COPY --from=kind /usr/local/sbin/runc /usr/local/sbin/
COPY --from=kind /opt/cni/ /opt/cni/
COPY --from=kind /etc/containerd/ /etc/containerd/
COPY --from=kind /etc/crictl.yaml /etc/
COPY --from=kind /etc/default/kubelet /etc/default/
COPY --from=kind /etc/sysctl.d/10-network-magic.conf /etc/sysctl.d/10-network-security.conf /etc/sysctl.d/
COPY --from=kind /etc/systemd/system/containerd.service /etc/systemd/system/containerd-fuse-overlayfs.service \
/etc/systemd/system/kubelet.service /etc/systemd/system/kubelet.slice /etc/systemd/system/undo-mount-hacks.service \
/etc/systemd/system/
COPY --from=kind /etc/systemd/system/kubelet.service.d/ /etc/systemd/system/kubelet.service.d/
COPY --from=kind /var/lib/containerd/ /var/lib/containerd/
RUN find /lib/systemd/system/sysinit.target.wants/ -name "systemd-tmpfiles-setup.service" -delete \
&& rm -f /lib/systemd/system/multi-user.target.wants/* \
&& rm -f /etc/systemd/system/*.wants/* \
&& rm -f /lib/systemd/system/local-fs.target.wants/* \
&& rm -f /lib/systemd/system/sockets.target.wants/*udev* \
&& rm -f /lib/systemd/system/sockets.target.wants/*initctl* \
&& rm -f /lib/systemd/system/basic.target.wants/* \
&& echo "ReadKMsg=no" >> /etc/systemd/journald.conf \
&& ln -sf /usr/lib/systemd/systemd /sbin/init \
&& systemctl enable kubelet.service containerd.service undo-mount-hacks.service \
&& systemctl mask systemd-binfmt.service \
&& mkdir -p /etc/kubernetes/manifests
ENV container=docker
STOPSIGNAL SIGRTMIN+3
ENTRYPOINT ["/usr/local/bin/entrypoint", "/sbin/init"]
FROM ${BASE}
# etcdctl/etcdutl at exactly the version of the cluster's etcd (the etcd image itself is distroless)
COPY --from=etcd /usr/local/bin/etcdctl /usr/local/bin/etcdutl /usr/local/bin/
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
openssh-server dbus sudo vim nano less jq tmux bash-completion curl wget ca-certificates \
iputils-ping dnsutils netcat-openbsd psmisc tree openssl git \
openssh-server dbus sudo vim nano less jq tmux bash-completion curl wget ca-certificates gnupg \
iputils-ping dnsutils netcat-openbsd psmisc tree openssl git bsdextrautils xxd \
podman netavark aardvark-dns \
&& rm -rf /var/lib/apt/lists/* \
# Debian's podman ships a CNI config. The kubelet would take it for the cluster's pod network
# A podman package may ship a CNI config. The kubelet would take it for the cluster's pod network
# until kindnet has written its own, and the first Pods would land on podman's bridge.
&& rm -f /etc/cni/net.d/87-podman-bridge.conflist \
&& touch /etc/containers/nodocker
# --- Kubernetes as packages -------------------------------------------------------------------
# kubelet, kubeadm and kubectl of the cluster's version from pkgs.k8s.io, held, with the repository
# of their minor - as on an exam node. An upgrade is then `apt-get install` and `kubeadm upgrade`,
# as on the day, and a task can hold a node back on an older release. kind's kubelet unit and
# drop-ins live in /etc/systemd and keep taking precedence over the package's.
ARG K8S_VERSION
ARG K8S_MINOR
RUN mkdir -p /etc/apt/keyrings \
&& curl -fsSL "https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/Release.key" | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg \
&& echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${K8S_MINOR}/deb/ /" > /etc/apt/sources.list.d/kubernetes.list \
# kubeadm refuses to upgrade a node without the cpuset cgroup controller, which a rootless
# container does not get. The packaged binary is diverted and a wrapper (rootfs/usr/bin/kubeadm)
# passes --ignore-preflight-errors=SystemVerification to `kubeadm upgrade`, so the candidate
# types the same commands as on a real node. The diversion survives package upgrades.
&& dpkg-divert --add --rename --divert /usr/bin/kubeadm.real /usr/bin/kubeadm \
&& apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends -o Dpkg::Options::=--force-confold \
"kubelet=${K8S_VERSION}-*" "kubeadm=${K8S_VERSION}-*" "kubectl=${K8S_VERSION}-*" \
&& apt-mark hold kubelet kubeadm kubectl \
&& rm -rf /var/lib/apt/lists/* \
&& test -x /usr/bin/kubeadm.real
ARG HELM_VERSION
RUN arch="$(dpkg --print-architecture)" \
&& curl -fsSL "https://get.helm.sh/helm-${HELM_VERSION}-linux-${arch}.tar.gz" \
@ -26,15 +104,18 @@ RUN arch="$(dpkg --print-architecture)" \
&& chmod +x /usr/local/bin/yq
COPY rootfs/ /
# uid 1000, passwordless sudo: the exam gives you `candidate` and expects `sudo -i` for node work
RUN useradd -m -s /bin/bash -u 1000 candidate \
RUN chmod 755 /usr/bin/kubeadm \
&& useradd -m -s /bin/bash -u 1000 candidate \
&& echo 'candidate ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/candidate && chmod 440 /etc/sudoers.d/candidate \
&& cp /etc/skel/.vimrc /root/.vimrc \
&& echo '. /etc/exam.bashrc' >> /etc/bash.bashrc \
&& systemctl enable ssh \
# sshd as a service of its own: Ubuntu 24.04 would start it from ssh.socket on the first
# connection, and a host that is up should be listening
&& (systemctl disable ssh.socket 2>/dev/null || true) && systemctl enable ssh.service \
# registry mirrors are read from /etc/containerd/certs.d (see docker.io/hosts.toml there)
&& printf '\n[plugins."io.containerd.grpc.v1.cri".registry]\n config_path = "/etc/containerd/certs.d"\n' >> /etc/containerd/config.toml \
# Installing dbus stamps the image with a D-Bus machine id, and systemd-machine-id-setup (run by
# kind's entrypoint on every boot) copies that into /etc/machine-id instead of making a new one:
# every node would share one machine id, derive the same eth0 MAC from it, and be unreachable from
# its neighbours. Point it at /etc/machine-id, as a stock Debian does, so each node gets its own.
# its neighbours. Point it at /etc/machine-id, as a stock Ubuntu does, so each node gets its own.
&& rm -f /var/lib/dbus/machine-id && ln -s /etc/machine-id /var/lib/dbus/machine-id

2
images/node-cks/rootfs/usr/bin/kubeadm → images/node/rootfs/usr/bin/kubeadm
View File

@ -1,6 +1,6 @@
#!/bin/sh
# kubeadm on an exam host. The packaged binary is /usr/bin/kubeadm.real (a dpkg diversion, see
# images/node-cks/Containerfile). `kubeadm upgrade` checks the node first and stops when the cpuset
# images/node/Containerfile). `kubeadm upgrade` checks the node first and stops when the cpuset
# cgroup controller is missing - which it always is on a rootless container node, and never on a
# real one. This wrapper lets that one check pass and changes nothing else.
if [ "$1" = upgrade ]; then

2
images/vm/guest/provision.sh
View File

@ -14,7 +14,7 @@ export DEBIAN_FRONTEND=noninteractive
modprobe 9p; modprobe 9pnet_virtio
mkdir -p /mnt/build && mount -t 9p -o trans=virtio,version=9p2000.L,ro build /mnt/build
. /mnt/build/env # K8S_VERSION=1.35.8 K8S_MINOR=v1.35
. /mnt/build/env # K8S_VERSION=1.37.0 K8S_MINOR=v1.37
# --- nothing that works behind the candidate's back
systemctl disable --now unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer 2>/dev/null || true

426
ksim
View File

@ -22,7 +22,7 @@ from collections.abc import Callable
from datetime import datetime
from typing import Any
VERSION = "2026.10.15.3"
VERSION = "2026.10.19.1"
ROOT = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, ROOT)
from sim import desktop, envs # noqa: E402
@ -47,8 +47,9 @@ WEAK_FILE = ""
PROGRESS_FILE = "" # study-<track>.json: when each objective was done, your focus, what you marked known
OBJECTIVES = [] # the track's curriculum, objective by objective (<track>/curriculum.py), if it has one
STUDY_ORDER = [] # domains in the order the path walks them, when that differs from the curriculum's own
K8S = "" # Kubernetes minor the track's real exam runs, e.g. "v1.35": picks the exam host image
K8S = "" # Kubernetes minor the track's real exam runs, e.g. "v1.37": picks the exam host image
PENDING = {} # tasks of the bank that are held back, id -> why (PENDING in <track>/questions.py)
LESSONS = [] # guided labs, one per objective (<track>/lessons.py), each run like a task on a host of its own
def load_track(track_id):
@ -83,6 +84,12 @@ def load_track(track_id):
if e.name != f"{track_id}.curriculum":
raise
objectives, study_order = [], [] # no objective map for this track: drill by domain only
try:
lessons = [lesson_task(l, objectives) for l in importlib.reload(importlib.import_module(f"{track_id}.lessons")).LESSONS]
except ModuleNotFoundError as e:
if e.name != f"{track_id}.lessons":
raise
lessons = []
for q in questions:
q["weight"] = sum(c["pts"] for c in q["checks"])
q.setdefault("difficulty", "normal")
@ -107,10 +114,19 @@ def load_track(track_id):
HISTORY_FILE=os.path.join(ROOT, f"history-{track_id}.jsonl"),
WEAK_FILE=os.path.join(ROOT, f"weak-spots-{track_id}.md"),
PROGRESS_FILE=os.path.join(ROOT, f"study-{track_id}.json"), OBJECTIVES=objectives, STUDY_ORDER=study_order,
K8S=track["k8s"], PENDING=pending,
K8S=track["k8s"], PENDING=pending, LESSONS=lessons,
)
def lesson_task(l, objectives):
"""A lesson as the engine runs it: a task-shaped dict, so its host, setup and checks go through
the same code as a task's (host_of, provision, run). Its steps' checks are its checks."""
o = next((o for o in objectives if o["id"] == l["objective"]), None)
return dict(l, lesson=True, domain=o["domain"] if o else "", difficulty="normal", set=None,
checks=[{"name": s["title"], "pts": 1, "script": s["check"]} for s in l["steps"]], weight=len(l["steps"]),
task=l["intro"], solution="\n\n".join(s["show"] for s in l["steps"] if s["show"]))
def set_questions(name):
return [q for q in QUESTIONS if q.get("set") == name]
@ -128,9 +144,12 @@ PARALLEL = max(1, int(os.environ.get("KSIM_PARALLEL") or max(2, min(8, (os.cpu_c
# ---------------------------------------------- a task's own cluster ----
_WORKER_REF = re.compile(r"(?:\$\{?|\{(?:SHELL_)?)W([123])\b")
def node_image():
"""The exam host image of the current track: its Kubernetes minor, in the track's own flavour if it has one."""
return envs.node_image(K8S, TRACK.get("image"))
def node_image(q=None):
"""The exam host image of a task: its track's Kubernetes minor, in the track's own flavour if it
has one - unless the task names a minor ("k8s") or a flavour ("image") of its own, as a task
that upgrades its cluster to the track's minor does."""
q = q or {}
return envs.node_image(q.get("k8s", K8S), q.get("image", TRACK.get("image")))
def is_vm(q):
@ -142,7 +161,7 @@ def make_host(q):
"""Create the host of a task: a kind cluster of containers, or a VM. Raises envs.EnvError."""
if is_vm(q):
return envs.create_vm(host_of(q), K8S)
return envs.create(host_of(q), node_image(), workers_of(q))
return envs.create(host_of(q), node_image(q), workers_of(q))
def workers_of(q):
@ -209,7 +228,7 @@ ensure_gateway_crds() {
}
# ensure_metrics_server -> a working metrics API for `kubectl top`. kind ships none, and kind's
# kubelet serving certs have no IP SANs, so it only becomes Ready with --kubelet-insecure-tls.
# Pinned: v0.9.x supports Kubernetes 1.34+, so it serves the v1.35 and v1.37 hosts alike.
# Pinned: v0.9.x supports Kubernetes 1.34+, so it serves the v1.37 hosts.
METRICS_SERVER_URL=https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.0/components.yaml
ensure_metrics_server() {
if ! $K -n kube-system get deploy metrics-server >/dev/null 2>&1; then
@ -221,6 +240,43 @@ ensure_metrics_server() {
fi
$K -n kube-system rollout status deploy metrics-server --timeout=150s >/dev/null 2>&1
}
# ensure_csi_hostpath -> the CSI hostpath driver with snapshots, and StorageClass csi-hostpath-sc
# (expandable), as the real CKA's clusters have it: what local-path cannot do - expanding a claim,
# VolumeSnapshots, cloning. Pinned to csi-driver-host-path v1.18.0 (its deploy/kubernetes-1.34
# directory, which kubernetes-1.35 and -latest link to); each sidecar's RBAC at the version of its
# image there, and the snapshot CRDs and controller of external-snapshotter at the snapshotter's.
CSI_GH=https://raw.githubusercontent.com/kubernetes-csi
ensure_csi_hostpath() {
$K get csidriver hostpath.csi.k8s.io >/dev/null 2>&1 && $K get sc csi-hostpath-sc >/dev/null 2>&1 \
&& $K -n default get pod csi-hostpathplugin-0 -o jsonpath='{.status.containerStatuses[*].ready}' 2>/dev/null | grep -qv false && return 0
local f s=$CSI_GH/external-snapshotter/v8.6.0
for f in volumesnapshotclasses volumesnapshotcontents volumesnapshots; do
curl -sfL --max-time 60 "$s/client/config/crd/snapshot.storage.k8s.io_$f.yaml" | $K apply -f - >/dev/null 2>&1 || return 1
done
$K wait --for condition=established --timeout=60s crd/volumesnapshotclasses.snapshot.storage.k8s.io \
crd/volumesnapshotcontents.snapshot.storage.k8s.io crd/volumesnapshots.snapshot.storage.k8s.io >/dev/null 2>&1 || return 1
for f in $s/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml $s/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml \
$CSI_GH/external-provisioner/v6.3.0/deploy/kubernetes/rbac.yaml $CSI_GH/external-attacher/v4.12.0/deploy/kubernetes/rbac.yaml \
$s/deploy/kubernetes/csi-snapshotter/rbac-csi-snapshotter.yaml $CSI_GH/external-resizer/v2.2.1/deploy/kubernetes/rbac.yaml \
$CSI_GH/external-health-monitor/v0.18.0/deploy/kubernetes/external-health-monitor-controller/rbac.yaml; do
curl -sfL --max-time 60 "$f" | $K apply -f - >/dev/null 2>&1 || return 1
done
for f in driverinfo plugin snapshotclass; do
curl -sfL --max-time 60 "$CSI_GH/csi-driver-host-path/v1.18.0/deploy/kubernetes-1.34/hostpath/csi-hostpath-$f.yaml" | $K apply -f - >/dev/null 2>&1 || return 1
done
$K apply -f - >/dev/null 2>&1 <<'CSI_SC' || return 1
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-hostpath-sc
provisioner: hostpath.csi.k8s.io
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
CSI_SC
$K -n kube-system rollout status deploy snapshot-controller --timeout=180s >/dev/null 2>&1 \
&& $K -n default rollout status sts csi-hostpathplugin --timeout=240s >/dev/null 2>&1
}
"""
def _session_tracks():
@ -290,6 +346,8 @@ def paused_total(st: dict[str, Any] | None) -> float:
def remaining(st: dict[str, Any] | None) -> float:
if not st:
return 0.0
if st.get("mode") == "lesson": # a lesson teaches: it has no clock
return float("inf")
return float(st.get("started_at", 0)) + float(st.get("duration", EXAM_SECONDS)) + paused_total(st) - time.time()
def set_paused(st: dict[str, Any], on: bool) -> None:
@ -335,11 +393,11 @@ def session_questions(st: dict[str, Any] | None) -> list[dict[str, Any]]:
# so compose_exam's rng.shuffle never reached the candidate and every exam arrived grouped by
# domain and difficulty - base tasks first, then hard, then the sim sets. Real exams are not
# ordered by difficulty, and practising triage on a sorted list teaches the wrong instinct.
by_id = {q["id"]: q for q in QUESTIONS}
by_id = {q["id"]: q for q in QUESTIONS + LESSONS}
return [by_id[i] for i in ids if i in by_id]
def q_by_id(n: int) -> dict[str, Any]:
for q in QUESTIONS:
for q in QUESTIONS + LESSONS: # a lesson has a host like a task: ./ksim ssh 9201 reaches it
if q["id"] == n:
return q
# ids run 1..N without gaps today, but print the real ranges rather than "1-<count>" so a
@ -401,19 +459,21 @@ def run(q, script, timeout=180, params=None):
# environment points KUBECONFIG at admin.conf, which only root can read and sshd never passes on.
AS_CANDIDATE = ["-u", "candidate", "-w", "/home/candidate", "-e", "KUBECONFIG=/home/candidate/.kube/config"]
def solve_on_host(q, params, timeout=600):
"""Run a task's scripted solution the way a candidate would type it: as `candidate`, in an
interactive shell on the task's host, so the host's aliases and profile apply."""
def solve_on_host(q, params, timeout=600, script=None):
"""Run a task's scripted solution (or another script: a lesson step's commands) the way a
candidate would type it: as `candidate`, in an interactive shell on the task's host, so the
host's aliases and profile apply."""
into = ["ssh", "candidate@vm", "bash -i"] if is_vm(q) else ["bash", "-i"]
try:
p = subprocess.run([envs.RUNTIME, "exec", "-i", *([] if is_vm(q) else AS_CANDIDATE), nodes_of(q)["CP"], *into],
input=render(q["solve"], params, q), capture_output=True, text=True, timeout=timeout)
input=render(q["solve"] if script is None else script, params, q), capture_output=True, text=True,
errors="replace", timeout=timeout) # a script may print binary (an etcd value, say)
except subprocess.TimeoutExpired:
return 1, f"solution script timed out after {timeout}s"
return p.returncode, p.stdout[-400:]
def k8s_version():
"""The Kubernetes release the track's exam hosts run, e.g. v1.35.8 (from the pinned node image)."""
"""The Kubernetes release the track's exam hosts run, e.g. v1.37.0 (from the pinned node image)."""
m = re.search(r":(v[\d.]+)", envs.BASE_IMAGES.get(K8S, ""))
return m.group(1) if m else K8S
@ -824,6 +884,7 @@ def load_progress():
objectives objective id -> when it was first done
focus domains the path walks first, in that order (./ksim focus)
known domains and objectives you declared known, which the path leaves out (./ksim skip)
lessons objective id -> when its lesson was first done (every step checked)
tasks what the study sessions of earlier versions solved - kept as it was, read by nothing.
Results against the clock live in the history; this file only steers the path."""
try:
@ -834,14 +895,16 @@ def load_progress():
oids = {o["id"] for o in OBJECTIVES}
return {"tasks": dict(d.get("tasks", {})), "objectives": dict(d.get("objectives", {})),
"focus": [x for x in d.get("focus", []) if x in DOMAINS],
"known": [x for x in dict.fromkeys(d.get("known", [])) if x in DOMAINS or x in oids]}
"known": [x for x in dict.fromkeys(d.get("known", [])) if x in DOMAINS or x in oids],
"lessons": dict(d.get("lessons", {}))}
def save_progress(prog):
with open(PROGRESS_FILE, "w") as f:
json.dump({**({"tasks": prog["tasks"]} if prog.get("tasks") else {}),
"objectives": {k: prog["objectives"][k] for k in sorted(prog["objectives"])},
**({"focus": prog["focus"]} if prog.get("focus") else {}),
**({"known": prog["known"]} if prog.get("known") else {})}, f, indent=1)
**({"known": prog["known"]} if prog.get("known") else {}),
**({"lessons": {k: prog["lessons"][k] for k in sorted(prog["lessons"])}} if prog.get("lessons") else {})}, f, indent=1)
f.write("\n")
def load_known():
@ -861,6 +924,60 @@ def known_task(q, known):
objs = objectives_of(q["id"])
return bool(objs) and all(is_known(o, known) for o in objs)
# ------------------------------------------------------------ lessons ----
# A lesson (<track>/lessons.py) is a guided lab on one objective: an introduction, steps that you do
# on a host of your own - each with a check, a hint and the commands that do it - and a wrap-up that
# sends you to the objective's timed drill. It runs as a session of mode "lesson": no clock, its
# steps checked one at a time and in order, nothing written to the history or the statistics; a
# lesson whose every step was checked is stamped done in study-<track>.json.
def find_lesson(arg):
"""A lesson by its objective (2.1, or the objective's key) or by its id (9201); None if none fits."""
o = find_objective(arg)
return next((l for l in LESSONS if (o and l["objective"] == o["id"]) or str(l["id"]) == str(arg)), None)
def lesson_of(st):
"""The lesson a session is, or None."""
if not st or st.get("mode") != "lesson":
return None
return next((l for l in LESSONS if l["id"] in st.get("ids", [])), None)
def lessons_done():
"""objective id -> when its lesson was first done."""
return load_progress()["lessons"]
def check_step(l, params, i):
"""Run the check of step i on the lesson's host."""
try:
rc, out = run(l, l["steps"][i]["check"], timeout=240, params=params)
except subprocess.TimeoutExpired:
rc, out = 1, "timed out"
return {"pass": rc == 0, "out": out[-600:], "at": time.time()}
def open_step(st, l):
"""The first step not checked yet, or None when every one is."""
done = st.get("steps", {})
return next((i for i in range(len(l["steps"])) if not done.get(str(i), {}).get("pass")), None)
def lesson_check(st, i):
"""Check step i of the lesson in progress and keep the result. Steps count in order - a step
builds on the one before - so only the first open step or one already done can be checked, and
a pass stays a pass when a later step changes what an earlier check looked at.
Returns (result, None) or (None, why not)."""
l = lesson_of(st)
if not l or st.get("ended_at"):
return None, "No lesson in progress."
if not 0 <= i < len(l["steps"]):
return None, f"The lesson has steps 1 to {len(l['steps'])}."
first = open_step(st, l)
if first is not None and i > first:
return None, f"Do step {first + 1} first: steps count in order."
r = check_step(l, session_params(st, l), i)
steps = st.setdefault("steps", {})
if r["pass"] or not steps.get(str(i), {}).get("pass"):
steps[str(i)] = r
save_state(st)
return r, None
def task_state(qid, agg):
"""passed = the latest attempt against the clock was a full pass; failed = it was not;
new = never attempted."""
@ -1273,7 +1390,7 @@ def preflight(qs):
("kubectl", "seeds and grades the tasks"), ("ssh-keygen", "makes the key the desktop logs in with")):
if not shutil.which(tool):
die(f"'{tool}' is not installed - it {why}.")
missing = [i for i in (node_image(), envs.DESKTOP_IMAGE) if not envs.image_exists(i)]
missing = [i for i in dict.fromkeys([node_image(q) for q in qs if not is_vm(q)] + [envs.DESKTOP_IMAGE]) if not envs.image_exists(i)]
if missing:
die(f"Not built yet: {', '.join(missing)}\n Build the exam host and desktop images (once, a few minutes): ./ksim build")
if any(is_vm(q) for q in qs) and not (envs.vm_ready(K8S) and envs.image_exists(envs.VM_IMAGE)):
@ -1296,7 +1413,7 @@ def _log_msg(*args: Any, **kwargs: Any) -> None:
def provision(q, params, log: Callable[..., Any] = _log_msg):
"""One task, start to finish: its cluster, its scenario, and the hand-over to `candidate`.
Returns None, or what went wrong."""
host, tag = host_of(q), f"Q{q['id']:02d} {q['title']}"
host, tag = host_of(q), (f"lesson {q['objective']} {q['title']}" if q.get("lesson") else f"Q{q['id']:02d} {q['title']}")
try:
make_host(q)
except envs.EnvError as e:
@ -1402,25 +1519,54 @@ def begin_session(qs, mode, duration, force, log: Callable[..., Any] = _log_msg,
st = {"mode": mode, "track": TRACK_ID, "hard": hard, "ids": [q["id"] for q in qs],
"params": params_by_id, "duration": duration, "started_at": time.time(), "ended_at": None,
"k8s": k8s_version(), "results": {}, "flags": [], "notes": "",
"objective": objective} # the curriculum objective a drill was started for, if any
"objective": objective} # the curriculum objective a drill or a lesson was started for, if any
lesson = mode == "lesson"
if lesson:
st["steps"] = {} # step index -> its latest check (a pass sticks)
save_state(st)
log("")
log(bold("=" * 64))
log(bold(f" {TRACK_ID.upper()} {'EXAM' if mode == 'exam' else 'DRILL'}{' · HARD' if hard else ''} - clock started"))
log(bold(f" {TRACK_ID.upper()} {'LESSON' if lesson else 'EXAM' if mode == 'exam' else 'DRILL'}{' · HARD' if hard else ''} - "
+ ("no clock: take your time" if lesson else "clock started")))
log(f" Hosts : {hosts[0]}" + (f" .. {len(hosts)} in all" if len(hosts) > 1 else "")
+ f" (Kubernetes {st['k8s']}, as the real {TRACK_ID.upper()} runs {K8S})")
if lesson:
log(f" Lesson : {objective} {qs[0]['title']} - {len(qs[0]['steps'])} steps")
else:
log(f" Ends : {datetime.fromtimestamp(st['started_at'] + duration).strftime('%H:%M:%S')} ({duration // 60} min)")
log(f" Tasks : {', '.join('Q%02d' % q['id'] for q in qs) if mode == 'drill' else len(qs)} Pass mark: {PASS_PCT}%")
if problems:
log(yellow(f" Seeding had problems on: {', '.join(problems)} - ./ksim reset N rebuilds one task's host"))
log(bold("=" * 64))
log(" ./ksim web the exam in the browser: task pane, remote desktop, clock\n"
" ./ksim list / show N / grade N / timer / end terminal workflow\n"
+ f" Every task names its host: ssh {hosts[0]} from the desktop's terminal, then sudo -i when you need root.\n"
+ f" Files a task asks for go under {WS_DISPLAY}/ on its host.")
log(" ./ksim web " + ("the lesson in the browser: its steps beside the remote desktop\n" if lesson else
"the exam in the browser: task pane, remote desktop, clock\n")
+ (" ./ksim lesson step / check / hint / show, ./ksim end terminal workflow\n" if lesson else
" ./ksim list / show N / grade N / timer / end terminal workflow\n")
+ f" {'The lesson runs on' if lesson else 'Every task names its host:'} ssh {hosts[0]} from the desktop's terminal, then sudo -i when you need root.\n"
+ f" Files go under {WS_DISPLAY}/ on the host.")
return st
def finish_lesson(st):
"""End a lesson: nothing is graded or recorded in the history; a lesson whose every step was
checked is stamped done on its objective (the first time). Returns the lines to show."""
l = lesson_of(st)
st["ended_at"] = time.time()
n, done = len(l["steps"]), sum(1 for i in range(len(l["steps"])) if st.get("steps", {}).get(str(i), {}).get("pass"))
if done == n:
prog = load_progress()
prog["lessons"].setdefault(l["objective"], datetime.now().isoformat(timespec="minutes"))
save_progress(prog)
lines = [f"✓ Lesson {l['objective']} done - {n}/{n} steps checked. Now prove it against the clock: ./ksim drill {l['objective']}"]
else:
lines = [f"Lesson {l['objective']} finished with {done}/{n} steps checked - take it again any time: ./ksim lesson {l['objective']}"]
st["milestones"] = lines
save_state(st)
return lines
def finish_session(st, log: Callable[..., Any] = _log_msg):
if st.get("mode") == "lesson":
finish_lesson(st)
return None
qs = session_questions(st)
over = remaining(st) <= 0
set_paused(st, False)
@ -1529,6 +1675,163 @@ def cmd_drill(args):
begin_session(kept, "drill", (minutes * 60) if minutes else per_task * len(kept), force, hard=hard,
objective=objective if len(args) == 1 else None)
def cli_md(text):
"""A lesson's Markdown for a terminal: fenced blocks indented, `code` highlighted."""
out, fence = [], False
for line in text.strip("\n").split("\n"):
if line.strip().startswith("```"):
fence = not fence
continue
out.append(" " + line if fence else paint_code(line))
return "\n".join(out)
def print_step(l, params, i, st=None):
s = l["steps"][i]
done = bool(st and st.get("steps", {}).get(str(i), {}).get("pass"))
print(bold(f"Step {i + 1} of {len(l['steps'])}: {render(s['title'], params, l)}") + (green(" ✓ checked") if done else ""))
print(cli_md(render(s["text"], params, l)))
print(dim(" ./ksim lesson check when it is done ./ksim lesson hint / show if you are stuck"))
def print_outro(l, params):
print(bold("Every step is checked.")); print("-" * 64)
print(cli_md(render(l["outro"], params, l))); print("-" * 64)
print(dim(" ./ksim end finishes the lesson"))
def cmd_lesson(args):
"""Lessons (<track>/lessons.py): guided labs on a host of your own, with no clock.
./ksim lesson | lesson OBJ | lesson step|check|hint|show [N] | lesson audit [OBJ..]"""
sub = args[0] if args else ""
if sub == "audit":
return lesson_audit(args[1:])
st = load_state()
live = lesson_of(st) if st and not st.get("ended_at") else None
if sub in ("check", "step", "hint", "show"):
if not live:
die("No lesson in progress. ./ksim lesson lists them; ./ksim lesson 2.1 starts one.")
params = session_params(st, live)
num = [a for a in args[1:] if a.isdigit()]
i = int(num[0]) - 1 if num else open_step(st, live)
if i is None:
return print_outro(live, params)
if not 0 <= i < len(live["steps"]):
die(f"The lesson has steps 1 to {len(live['steps'])}.")
s = live["steps"][i]
if sub == "step":
return print_step(live, params, i, st)
if sub in ("hint", "show"):
print(bold(f"{'Hint' if sub == 'hint' else 'The commands'} - step {i + 1}: {render(s['title'], params, live)}"))
text = render(s[sub], params, live)
print((cli_md(text) if sub == "hint" else text) or dim("Nothing for this step."))
return
r, why = lesson_check(st, i)
if why:
die(why)
if not r["pass"]:
print(red(f"✗ Step {i + 1} is not done yet: {render(s['title'], params, live)}"))
if r["out"]:
print(dim(" " + r["out"][-300:].replace("\n", "\n ")))
print(dim(" ./ksim lesson hint a hint ./ksim lesson show the commands"))
return
print(green(f"✓ Step {i + 1} done: {render(s['title'], params, live)}")); print()
nxt = open_step(load_state(), live)
return print_outro(live, params) if nxt is None else print_step(live, params, nxt)
if sub:
l = find_lesson(sub)
if not l:
die(f"No lesson for '{sub}'. The {TRACK_ID.upper()} lessons: "
+ (", ".join(f"{x['objective']} ({x['title']})" for x in LESSONS) or "none yet") + " (./ksim lesson lists them)")
st = begin_session([l], "lesson", 0, "--force" in args, objective=l["objective"])
params = session_params(st, l)
print(); print(bold(f"Lesson {l['objective']} - {l['title']}")); print("-" * 64)
print(cli_md(render(l["intro"], params, l))); print("-" * 64)
print_step(l, params, 0, st)
return
if not LESSONS:
print(f"No lessons for {TRACK_ID.upper()} yet.")
return
done = lessons_done()
print(bold(f"{TRACK['name']} - lessons") + dim(" guided labs on a host of your own, with no clock"))
for l in LESSONS:
print(f" {green('✓') if l['objective'] in done else dim('·')} {l['objective']:<5} {l['title']}"
+ dim(f" {len(l['steps'])} steps · {DOMAINS.get(l['domain'], l['domain'])}"))
print(dim(f"\n ./ksim lesson {LESSONS[0]['objective']} starts that lesson ./ksim lesson audit proves them on real clusters"))
def lesson_audit(args):
"""Prove lessons on real clusters: each is seeded on a host of its own; then every step must
fail its check, have its commands run as `candidate`, and pass its check - so a lesson can be
done exactly as it shows, and no step is done before you do it."""
for t in TRACKS:
try:
with open(os.path.join(STATE_DIR, f"state-{t}.json")) as f:
if not json.load(f).get("ended_at"):
die(f"A {t.upper()} session is open, and a new session would delete the audit's hosts. End it first.")
except (OSError, ValueError):
pass
names = [a for a in args if not a.startswith("--")]
ls = list(LESSONS) if not names or names == ["all"] else [find_lesson(n) for n in names]
if not ls or None in ls:
die(f"usage: ./ksim lesson audit [all|OBJECTIVE..] [--keep] (lessons: {', '.join(l['objective'] for l in LESSONS) or 'none'})")
preflight(ls[:PARALLEL])
envs.ensure_registry()
lock = os.path.join(STATE_DIR, f"audit-{os.getpid()}.lock") # keeps sessions from starting meanwhile
os.makedirs(STATE_DIR, exist_ok=True)
open(lock, "w").close()
import atexit
atexit.register(lambda: os.path.exists(lock) and os.remove(lock))
print(bold(f"Lesson audit of {len(ls)} {TRACK_ID.upper()} lesson(s), {PARALLEL} at a time - each step must fail, be done by its own commands, then pass."))
def one(l):
params = make_params(l, random.Random(l["id"])); host = host_of(l); t0 = time.time()
res = {"id": l["id"], "objective": l["objective"], "title": l["title"], "host": host, "steps": len(l["steps"]), "proven": 0}
try:
make_host(l)
except envs.EnvError as e:
res.update(verdict="no cluster", detail=str(e)[-400:], secs=int(time.time() - t0))
return res
try:
rc, out = run(l, l["setup"], params=params, timeout=300)
except subprocess.TimeoutExpired:
rc, out = 1, "setup timed out after 300s"
envs.hand_over(host, None)
if rc != 0:
res.update(verdict="seed failed", detail=out[-400:])
else:
for i, s in enumerate(l["steps"]):
if check_step(l, params, i)["pass"]:
res.update(verdict=f"step {i + 1} passes before its commands")
break
_, sout = solve_on_host(l, params, script=s["show"])
# what the commands created may still be starting: a learner would watch it come up
r, waited = check_step(l, params, i), 0
while not r["pass"] and waited < 90:
time.sleep(10); waited += 10
r = check_step(l, params, i)
if not r["pass"]:
res.update(verdict=f"step {i + 1} fails after its commands", detail=(r["out"] + "\n--- the commands printed:\n" + sout)[-600:])
break
res["proven"] = i + 1
else:
res["verdict"] = "ok"
if "--keep" not in args:
envs.destroy(host)
res["secs"] = int(time.time() - t0)
return res
results = []
with concurrent.futures.ThreadPoolExecutor(max_workers=PARALLEL) as pool:
for fut in concurrent.futures.as_completed([pool.submit(one, l) for l in ls]):
res = fut.result(); results.append(res)
col = green if res["verdict"] == "ok" else red
print(f" {col(res['verdict']):<44} {res['objective']:<5}{res['proven']}/{res['steps']} steps {res['secs']:>4}s {res['title']}", flush=True)
if res.get("detail"):
print(dim(" " + res["detail"].replace("\n", "\n ")), flush=True)
results.sort(key=lambda r: r["id"])
out_dir = os.path.join(STATE_DIR, "audit"); os.makedirs(out_dir, exist_ok=True)
path = os.path.join(out_dir, f"lessons-{TRACK_ID}.json")
with open(path, "w") as f:
json.dump({"k8s": k8s_version(), "at": datetime.now().isoformat(timespec="minutes"), "results": results}, f, indent=1)
bad = [r for r in results if r["verdict"] != "ok"]
print(bold(f"\n{len(results) - len(bad)}/{len(results)} lessons proven step by step.") + (red(f" {len(bad)} need work: " + ", ".join(r["objective"] for r in bad)) if bad else ""))
print(dim(f" report: {os.path.relpath(path, ROOT)}"))
def cmd_sim(args):
"""A fixed simulator set, full exam rules and clock."""
force = "--force" in args
@ -1589,6 +1892,8 @@ def cmd_grade(args):
# meant to rehearse the real thing (no feedback until you submit) must not do by default.
if st.get("mode") == "exam" and not st.get("ended_at") and not force:
die("No feedback during an exam - the real one gives none. Submit with ./ksim end (--force to peek anyway).")
if st.get("mode") == "lesson":
die("A lesson is checked step by step: ./ksim lesson check")
ids = [q["id"] for q in session_questions(st)] if args[0] == "all" else [int(a) for a in args]
for n in ids:
q = q_by_id(n); r = grade_one(q)
@ -1614,6 +1919,10 @@ def print_score(st):
print(f"\n{bold('Score:')} {earned}/{total} = {pct}% -> {verdict} (pass mark {PASS_PCT}%)")
def status_line(st):
l = lesson_of(st)
if l:
n = len(l["steps"]); done = n if open_step(st, l) is None else open_step(st, l)
return green("lesson - no clock") + dim(f" [{done}/{n} steps checked]")
r = remaining(st)
if st.get("paused_at"):
return yellow(f"⏸ PAUSED - {fmt(r)} remaining") + dim(" ./ksim resume")
@ -1629,10 +1938,15 @@ def cmd_status(args):
print(status_line(st))
print(dim(f"{sum(1 for v in st['results'].values() if v['pass'])} passed / {len(st['results'])} graded / {len(session_questions(st))} in session"))
def no_clock(st):
if st.get("mode") == "lesson":
die("A lesson has no clock - take your time. (./ksim lesson step shows where you are)")
def cmd_timer(args):
st = require_started()
if st.get("ended_at"):
die("Session already ended.")
no_clock(st)
try:
while True:
st = load_state(); r = remaining(st)
@ -1652,10 +1966,12 @@ def cmd_pause(args):
st = require_started()
if st.get("ended_at"):
die("Session already ended.")
no_clock(st)
set_paused(st, True); print(status_line(st))
def cmd_resume(args):
st = require_started()
no_clock(st)
set_paused(st, False); print(status_line(st))
def cmd_restart(args):
@ -1664,6 +1980,7 @@ def cmd_restart(args):
st = require_started()
if st.get("ended_at"):
die("Session already ended - nothing to restart. Start a new one: ./ksim drill, ./ksim start, ./ksim next")
no_clock(st)
restart_clock(st)
print(status_line(st) + dim(" clock restarted - the hosts and your work are as they were"))
@ -1671,6 +1988,11 @@ def cmd_end(args):
st = require_started()
if st.get("ended_at"):
die("Session already ended and recorded.")
if st.get("mode") == "lesson": # nothing to grade: the steps were checked as you went
for line in finish_lesson(st):
print(green(line) if line.startswith("✓") else line)
print(dim("A lesson writes nothing to the history or the scores. Its host stays up until the next session."))
return
down = hosts_down(st)
if down:
die(down_message(down))
@ -1823,7 +2145,12 @@ class Api(http.server.BaseHTTPRequestHandler):
ulog = lambda line: Api.log_lines.append(ansi.sub("", line))
def work():
try:
if body.get("sim") in SETS:
if body.get("lesson"):
l = find_lesson(body["lesson"])
if not l:
die(f"No lesson for {body['lesson']}.")
begin_session([l], "lesson", 0, True, log=ulog, other_ok=False, objective=l["objective"])
elif body.get("sim") in SETS:
begin_session(set_questions(body["sim"]), "exam", EXAM_SECONDS, True, log=ulog, hard=True, other_ok=False)
elif mode == "exam":
begin_session(compose_exam(hard, rng), "exam", EXAM_SECONDS, True, log=ulog, hard=hard, other_ok=False)
@ -1862,9 +2189,15 @@ class Api(http.server.BaseHTTPRequestHandler):
return self._json({"seeding": True})
if not st:
return self._json({"error": "no session"}, 400)
if st.get("mode") == "lesson" and p.path in ("/api/pause", "/api/restart", "/api/grade"):
return self._json({"error": "A lesson has no clock and is checked step by step."}, 400)
if p.path == "/api/flag":
qid = int(body["id"]); flags = set(st.get("flags", []))
flags.symmetric_difference_update({qid}); st["flags"] = sorted(flags); save_state(st)
elif p.path == "/api/lesson/check": # one step of the lesson in progress
_, why = lesson_check(st, int(body.get("step", -1)))
if why:
return self._json({"error": why}, 400)
elif p.path == "/api/pause":
set_paused(st, bool(body.get("paused", True)))
elif p.path == "/api/restart":
@ -1888,7 +2221,7 @@ class Api(http.server.BaseHTTPRequestHandler):
st["results"][str(q["id"])] = grade_one(q, session_params(st, q)); save_state(st)
elif p.path == "/api/end":
if not st.get("ended_at"):
down = hosts_down(st) # grading hosts that are not running would record zeros
down = hosts_down(st) if st.get("mode") != "lesson" else [] # grading hosts that are not running would record zeros
if down:
return self._json({"error": down_message(down)}, 503)
finish_session(st, log=lambda *_: None)
@ -1929,16 +2262,31 @@ class Api(http.server.BaseHTTPRequestHandler):
"objective": st.get("objective"), "milestones": st.get("milestones", []), "aborted": bool(st.get("aborted")),
"results": st["results"], "score": {"earned": earned, "total": total, "pct": pct, "pass_pct": PASS_PCT},
"domains": self.domain_scores(st), "server_time": time.time()}
l = lesson_of(st)
if l:
payload["session"]["steps"] = st.get("steps", {})
payload["lesson"] = self.lesson_payload(l, session_params(st, l))
return payload
def lesson_payload(self, l: dict[str, Any], params: dict[str, Any]) -> dict[str, Any]:
"""A lesson as the page shows it: introduction, steps (with their hint and commands, which the
page keeps behind a click), wrap-up, its host and the objective's drill."""
r = lambda t: render(t, params, l).strip("\n")
return {"id": l["id"], "objective": l["objective"], "title": l["title"], "host": host_of(l), "nodes": 1 + workers_of(l),
"intro": r(l["intro"]), "outro": r(l["outro"]),
"steps": [{"title": r(s["title"]), "text": r(s["text"]), "hint": r(s["hint"]), "show": r(s["show"])} for s in l["steps"]]}
def path_payload(self) -> dict[str, Any]:
"""The curriculum as the UI shows it: every objective with its notes, tasks and state, how
many are done, and the next step on the path. Empty for a track without one."""
if not OBJECTIVES:
return {"objectives": [], "focus": [], "known": load_known(), "path": None}
agg, known = stats(), load_known()
agg, known, done = stats(), load_known(), lessons_done()
lesson = {l["objective"]: l for l in LESSONS}
objs = [{"id": o["id"], "key": o["key"], "domain": o["domain"], "text": o["text"], "state": objective_state(o, agg, known),
"know": o["know"], "fast": o["fast"].strip("\n"), "traps": o["traps"], "docs": o["docs"],
"lesson": ({"id": lesson[o["id"]]["id"], "title": lesson[o["id"]]["title"], "steps": len(lesson[o["id"]]["steps"]),
"done": o["id"] in done} if o["id"] in lesson else None),
"tasks": [{"id": q["id"], "title": q["title"], "set": q.get("set"), "state": task_state(q["id"], agg)}
for q in objective_tasks(o, True)]} for o in OBJECTIVES]
step = next_step(agg); full, fixed = mock_scores()
@ -2014,13 +2362,17 @@ def cmd_build(args):
for t in tracks:
if t not in TRACKS:
die(f"usage: ./ksim build [{'|'.join(TRACKS)}|all] [--vm] (default: the current track, {TRACK_ID})")
minors, variants = [], []
minors, variants, vm_minors = [], [], []
for t in tracks:
load_track(t)
if K8S not in minors:
minors.append(K8S)
if TRACK.get("image") and (K8S, TRACK["image"]) not in variants:
variants.append((K8S, TRACK["image"]))
if any(is_vm(q) for q in QUESTIONS) and K8S not in vm_minors:
vm_minors.append(K8S)
for q in [{}] + [q for q in QUESTIONS + LESSONS if not is_vm(q)]: # the track's own, then any a task names
k8s, variant = q.get("k8s", K8S), q.get("image", TRACK.get("image"))
if k8s not in minors:
minors.append(k8s)
if variant and (k8s, variant) not in variants:
variants.append((k8s, variant))
print(bold(f"Building images for {', '.join(t.upper() for t in tracks)} - the first build downloads a few hundred MB..."))
try:
for k8s in minors:
@ -2029,7 +2381,7 @@ def cmd_build(args):
envs.build_variant_image(k8s, variant)
envs.build_desktop_image()
if "--vm" in args: # the VM exam host, for tasks that need a kernel of their own
for k8s in minors:
for k8s in vm_minors:
envs.build_vm_image(k8s)
except envs.EnvError as e:
die(str(e))
@ -2226,6 +2578,10 @@ def cmd_help(args):
" as passed, and a mock still draws from it (skip clear / unskip X bring it back)\n"
" notes [OBJ|DOMAIN|all] what to know, fastest commands, traps and docs for an objective (3.2 or probes)\n"
" drill OBJ an objective's tasks against the clock, those not yet passed first\n")
objective_help += "" if not LESSONS else (
f" lesson [OBJ] the track's lessons ({len(LESSONS)}), or start one: a guided lab on a host of your own, no clock\n"
" lesson step|check|hint|show [N] the step you are on, check it, a hint, the commands; ./ksim end finishes\n"
" lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after them\n")
print(f"""usage: ./ksim [--track cka|ckad|cks] <command> (current track: {TRACK_ID}, {len(QUESTIONS)} tasks)
Every task runs on its own cluster at the exam's Kubernetes version ({TRACK_ID}: {k8s_version()}),
reached with `ssh <host>` from the remote desktop.
@ -2267,7 +2623,7 @@ COMMANDS = {"start": cmd_start, "drill": cmd_drill, "list": cmd_list, "show": cm
"reset": cmd_reset, "web": cmd_web, "abort": cmd_abort, "progress": cmd_progress, "solution": cmd_solution, "sim": cmd_sim, "version": cmd_version, "help": cmd_help,
"build": cmd_build, "hosts": cmd_hosts, "ssh": cmd_ssh, "down": cmd_down, "audit": cmd_audit, "coverage": cmd_coverage,
"objectives": cmd_objectives, "curriculum": cmd_objectives, "notes": cmd_notes, "next": cmd_next, "focus": cmd_focus,
"skip": cmd_skip, "known": cmd_skip, "unskip": cmd_unskip}
"skip": cmd_skip, "known": cmd_skip, "unskip": cmd_unskip, "lesson": cmd_lesson, "lessons": cmd_lesson}
if __name__ == "__main__":
cmd = ARGV[0] if ARGV else "help"

6
sim/bank.py
View File

@ -19,6 +19,12 @@ mkns() {
def C(name, pts, script):
return {"name": name, "pts": pts, "script": script}
def step(title, text, check, hint="", show=""):
"""One step of a lesson (<track>/lessons.py): what to do (Markdown), the check that proves it
(bash, exit 0 = done, run like a task's checks), a hint, and the commands that do it - shown on
request, and run by ./ksim lesson audit to prove the step can be done as shown."""
return {"title": title, "text": text, "check": check, "hint": hint, "show": show}
def ns_pick(r, base):
"""Namespace name with a random suffix so the same task never lands on the same namespace twice."""
return f"{base}-{r.choice(['alpha', 'bravo', 'delta', 'echo', 'kilo', 'nova', 'omega', 'zulu'])}"

29
sim/envs.py
View File

@ -26,10 +26,11 @@ COURSE = "/opt/course" # a task's files, as its host sees them
# cka0037-control-plane, cka0037-worker, ... Only names of this shape are ever deleted.
HOST_RE = re.compile(r"^(cka|ckad|cks)\d{4}$")
# The kind node image each exam host image is built from, one per Kubernetes minor an exam runs.
# Pinned by digest: kind only guarantees an image built for its own release (these are v0.33.0's).
# The kind node image each exam host image is built from, one per Kubernetes minor an exam runs -
# and v1.36, for the tasks that upgrade their cluster to v1.37 ("k8s": "v1.36"). Pinned by digest:
# kind only guarantees an image built for its own release (these are v0.33.0's).
BASE_IMAGES = {
"v1.35": "docker.io/kindest/node:v1.35.8@sha256:07b2536e30b803ed61d1677a79df6115f798ce64c80f9e22f6ed45afd09323c0",
"v1.36": "docker.io/kindest/node:v1.36.4@sha256:099e049362a1526b2db71494e1947aae99bd16290d7c895f2b7ea312e3cbfaed",
"v1.37": "docker.io/kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5",
}
HELM_VERSION = "v4.3.0"
@ -83,17 +84,22 @@ def image_exists(name):
def build_node_image(k8s, log=print):
"""The exam host image of a Kubernetes minor (images/node): Ubuntu, with kind's runtime and
preloaded images from the pinned kind node image, and Kubernetes of that release as packages."""
base = BASE_IMAGES.get(k8s)
if not base:
m = re.search(r":v([\d.]+)@", base or "")
if not m:
raise EnvError(f"no kind node image is pinned for Kubernetes {k8s} (BASE_IMAGES in sim/envs.py)")
log(f" {node_image(k8s)}: from {base.split('@')[0]}")
log(f" {node_image(k8s)}: Ubuntu 24.04, Kubernetes {m.group(1)} as packages, kind's runtime from {base.split('@')[0]}")
# etcdctl/etcdutl come out of the etcd image this kubeadm would deploy, so they match the server
rc, out = sh([RUNTIME, "run", "--rm", "--entrypoint", "sh", base, "-c",
'kubeadm config images list --kubernetes-version "$(cat /kind/version)" 2>/dev/null | grep /etcd:'])
etcd = out.strip().splitlines()[-1] if rc == 0 and out.strip() else ""
# (the line that names it: when the image is pulled first, podman's progress comes before it)
etcd = next((l.strip() for l in reversed(out.splitlines()) if "/etcd:" in l), "") if rc == 0 else ""
if not etcd:
raise EnvError(f"could not read the etcd image of {base}:\n{out[-400:]}")
rc, out = sh([RUNTIME, "build", "-t", node_image(k8s), "--build-arg", f"BASE={base}", "--build-arg", f"ETCD_IMAGE={etcd}",
"--build-arg", f"K8S_VERSION={m.group(1)}", "--build-arg", f"K8S_MINOR={k8s}",
"--build-arg", f"HELM_VERSION={HELM_VERSION}", os.path.join(ROOT, "images", "node")], timeout=1800)
if rc != 0:
raise EnvError(f"building {node_image(k8s)} failed:\n{out[-1500:]}")
@ -105,14 +111,11 @@ def build_variant_image(k8s, variant, log=print):
ctx = os.path.join(ROOT, "images", f"node-{variant}")
if not os.path.isdir(ctx):
raise EnvError(f"no image directory for the '{variant}' exam host (images/node-{variant})")
rc, out = sh([RUNTIME, "run", "--rm", "--entrypoint", "cat", node_image(k8s), "/kind/version"])
version = out.strip().splitlines()[-1].lstrip("v") if rc == 0 and out.strip() else ""
if not re.match(r"^\d+\.\d+\.\d+$", version):
raise EnvError(f"could not read the Kubernetes version of {node_image(k8s)}:\n{out[-300:]}")
if not image_exists(node_image(k8s)):
raise EnvError(f"{node_image(k8s)} is not built yet; the '{variant}' exam host is built on top of it")
name = node_image(k8s, variant)
log(f" {name}: from {node_image(k8s)} (Kubernetes {version} as packages, {variant.upper()} tools)")
rc, out = sh([RUNTIME, "build", "-t", name, "--build-arg", f"BASE={node_image(k8s)}",
"--build-arg", f"K8S_VERSION={version}", "--build-arg", f"K8S_MINOR={k8s}", ctx], timeout=3600)
log(f" {name}: from {node_image(k8s)}, with the {variant.upper()} tools")
rc, out = sh([RUNTIME, "build", "-t", name, "--build-arg", f"BASE={node_image(k8s)}", ctx], timeout=3600)
if rc != 0:
raise EnvError(f"building {name} failed:\n{out[-1500:]}")
log(f" {name}: built")

231
web/index.html
View File

@ -236,7 +236,8 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
color: #fff; border-bottom: 1px solid var(--head-line); }
.xtitlebox { display: flex; align-items: center; gap: 15px; min-width: 0; }
.xtitle { font-size: 1.25rem; font-weight: 700; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.timer { padding: .25rem .5rem; background: rgba(255,255,255,.1); border-radius: .25rem; color: #fff; font-weight: 500; font-variant-numeric: tabular-nums; transition: background-color .3s, color .3s; }
.timer { padding: .25rem .5rem; background: rgba(255,255,255,.1); border-radius: .25rem; color: #fff; font-weight: 500; font-variant-numeric: tabular-nums; transition: background-color .3s, color .3s;
white-space: nowrap; flex-shrink: 0; }
.timer.warning { background: rgba(220,53,69,.25); color: var(--yellow); animation: pulse 1.4s ease-in-out infinite; }
.timer.paused { color: #bbb; }
@keyframes pulse { 0%, 100% { background: rgba(220,53,69,.25); } 50% { background: rgba(220,53,69,.5); } }
@ -276,6 +277,20 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
.qcontent hr { border: 0; border-top: 1px solid var(--line); margin: 1rem 0; }
.qtitle { font-size: 18px; margin: 0 0 .6rem; color: var(--ink); }
.qbody p { margin: 0 0 12px; white-space: pre-wrap; }
.qbody ul, .notes-md ul { margin: 0 0 12px; padding-left: 20px; } .qbody li { margin: 3px 0; }
/* a fenced block in a task or a lesson: kept as written, and a click copies it */
pre.code { margin: 0 0 12px; padding: 10px 12px; font-size: 12.5px; line-height: 1.5; background: var(--pre-bg); border: 1px solid var(--line);
border-radius: 6px; color: var(--text); overflow-x: auto; white-space: pre; cursor: copy; }
pre.code:hover { border-color: var(--link); }
pre.code.copied { border-color: var(--ok-text); }
/* lessons */
.lstate { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; margin: 16px 0 4px; }
.lstate .ok { color: var(--ok-text); font-weight: 600; } .lstate .no { color: var(--bad-text); font-weight: 600; }
.lstate .lock { color: var(--muted); font-size: 13px; }
.lmark { font: 600 11px var(--sans); text-transform: uppercase; letter-spacing: .04em; padding: 1px 7px; border-radius: 50px;
border: 1px solid var(--line); color: var(--muted); margin-right: 6px; white-space: nowrap; }
.lmark.done { border-color: var(--ok-text); color: var(--ok-text); }
.stepsum { margin: 0 0 14px; color: var(--muted); font-size: 13px; text-transform: uppercase; letter-spacing: .05em; }
.checkrow { display: flex; gap: 10px; align-items: center; flex-wrap: wrap; margin-top: 16px; font-size: 13px; }
.checks { margin-top: 14px; background: var(--surface-2); border: 1px solid var(--line); border-radius: 6px; padding: 12px 14px; font-size: 14px; }
.checks div { padding: 3px 0; }
@ -321,7 +336,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
.score-box { background: var(--score-bg); color: #fff; font-size: 22px; font-weight: 700; padding: 8px 18px; border-radius: 4px; font-variant-numeric: tabular-nums; }
.pct { font-size: 30px; font-weight: 800; color: var(--res-head); }
.rank-badge { padding: 8px 16px; border-radius: 4px; font-weight: 700; color: #fff; }
.rank-high { background: var(--ok); } .rank-low { background: var(--bad); }
.rank-high { background: var(--ok); } .rank-low { background: var(--bad); } .rank-mid { background: var(--grey); }
.bars { margin: 0 0 18px; }
.bar { display: grid; grid-template-columns: 76px 1fr 64px; gap: 10px; align-items: center; margin: 6px 0; font-size: 14px; }
.bar .track { height: 8px; background: var(--track); border-radius: 5px; overflow: hidden; }
@ -387,7 +402,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
<h2 class="section-title">Your path</h2>
<p class="section-sub">One objective at a time against the clock, then a mock exam, then the fixed simulator sets.</p>
<div class="ladder" id="ladder"></div>
<div class="pathgo"><button class="btn btn-primary" id="pathGo">Continue</button><span class="why" id="pathWhy"></span></div>
<div class="pathgo"><button class="btn btn-primary" id="pathGo">Continue</button><button class="btn btn-success hidden" id="pathLesson">Lesson first</button><span class="why" id="pathWhy"></span></div>
</section>
<section class="card" id="domainCard">
<h2 class="section-title">Drill a domain</h2>
@ -450,6 +465,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
<div class="qcontent" id="qContent"></div>
<div class="qactions">
<button class="btn btn-outline-warning" id="btnFlag"></button>
<button class="btn btn-secondary hidden" id="btnBack"><span aria-hidden="true">←</span> Back</button>
<button class="btn btn-success" id="btnSatisfied">Satisfied with answer <span aria-hidden="true">→</span></button>
</div>
<details class="notepad" id="notesWrap"><summary>Notepad - kept with the attempt, not graded</summary><textarea id="notes" placeholder="Your notes"></textarea></details>
@ -470,10 +486,10 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
<div class="res-actions">
<button class="btn btn-secondary" data-act="dashboard">Dashboard</button>
<button class="btn btn-primary" id="btnDrillMissed">Drill what I missed</button>
<button class="btn btn-success" data-act="answers">View answers</button>
<button class="btn btn-outline-primary" data-act="review">Review tasks</button>
<button class="btn btn-success" id="btnAnswers" data-act="answers">View answers</button>
<button class="btn btn-outline-primary" id="btnReview" data-act="review">Review tasks</button>
<button class="btn btn-outline-primary" data-act="history">History</button>
<button class="btn btn-danger" data-act="terminate">Delete the hosts</button>
<button class="btn btn-danger" id="btnDelete" data-act="terminate">Delete the hosts</button>
<div class="dd">
<button class="btn btn-outline-primary dd-toggle" aria-expanded="false">Theme</button>
<div class="dd-menu"><button class="dd-item" data-theme-set="auto">Auto - follow the system</button><button class="dd-item" data-theme-set="light">Light</button><button class="dd-item" data-theme-set="dark">Dark</button></div>
@ -482,7 +498,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
<div class="res-wrap">
<div class="res-header"><h2 id="resTitle">Results</h2><div class="meta" id="resMeta"></div></div>
<div class="summary">
<div><h3>Points scored</h3><div class="score-row"><span class="score-box"><span id="resEarned">0</span> / <span id="resTotal">0</span></span><span class="pct" id="resPct"></span></div></div>
<div><h3 id="resLabel">Points scored</h3><div class="score-row"><span class="score-box"><span id="resEarned">0</span> / <span id="resTotal">0</span></span><span class="pct" id="resPct"></span></div></div>
<span class="rank-badge" id="resBadge"></span>
</div>
<div class="bars" id="resDomains"></div>
@ -498,6 +514,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
<div class="mb">
<div class="field"><label class="lbl" for="pkTrack">Certification</label><select class="select" id="pkTrack"></select></div>
<div class="field"><label class="lbl" for="pkSession">Session</label><select class="select" id="pkSession"></select></div>
<div class="field hidden" id="pkLessonRow"><label class="lbl" for="pkLesson">Lesson</label><select class="select" id="pkLesson"></select></div>
<div class="field hidden" id="pkObjRow"><label class="lbl" for="pkObjective">Objective</label><select class="select" id="pkObjective"></select></div>
<div class="field hidden" id="pkDomRow"><label class="lbl" for="pkDomain">Domain</label><select class="select" id="pkDomain"></select></div>
<div class="field hidden" id="pkPickRow"><label class="lbl">Tasks</label><div class="picker" id="pkTasks"></div></div>
@ -562,7 +579,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
<script>
(() => {
const $ = s => document.querySelector(s), $$ = s => [...document.querySelectorAll(s)];
let S = null, cur = null, skew = 0, clockT = null, autoSubmitted = false, view = 'landing', polling = false, hard = false;
let S = null, cur = null, skew = 0, clockT = null, autoSubmitted = false, view = 'landing', polling = false, hard = false, lpage = null;
try { hard = localStorage.getItem('ksim.hard') === '1'; } catch (e) { /* storage blocked: start without it */ }
const api = (p, body) => fetch(p, body ? {method: 'POST', headers: {'Content-Type': 'application/json'}, body: JSON.stringify(body)} : {})
.then(async r => { const j = await r.json(); if (!r.ok) throw new Error(j.error || r.statusText); return j; });
@ -576,12 +593,19 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
const closeModal = id => $(id).classList.add('hidden');
const live = () => !!(S && S.session && !S.session.ended_at);
// minimal markdown: paragraphs, **bold**, `code` (click to copy). Lists stay as literal lines.
const render = md => md.split(/\n\s*\n/).map(p => '<p>' + esc(p).replace(/\*\*(.+?)\*\*/g, '<b>$1</b>')
.replace(/`([^`]+)`/g, (m, c) => `<code class="ic" title="click to copy">${c}</code>`) + '</p>').join('');
// minimal markdown: paragraphs, **bold**, `code` (click to copy), ```fenced blocks``` (kept as
// written, click to copy), and a paragraph made only of "- " lines as a list. Other lists stay as
// literal lines.
const inlineMd = s => esc(s).replace(/\*\*(.+?)\*\*/g, '<b>$1</b>')
.replace(/`([^`]+)`/g, (m, c) => `<code class="ic" title="click to copy">${c}</code>`);
const render = md => md.split(/^```[^\n]*\n([\s\S]*?)\n?^```[ \t]*$/m).map((part, i) => i % 2
? `<pre class="code" title="click to copy">${esc(part)}</pre>`
: part.split(/\n\s*\n/).filter(p => p.trim()).map(p => /^(- .*(\n|$))+$/.test(p.trim() + '\n')
? '<ul>' + p.trim().split('\n').map(l => `<li>${inlineMd(l.slice(2))}</li>`).join('') + '</ul>'
: `<p>${inlineMd(p.replace(/^\n+|\n+$/g, ''))}</p>`).join('')).join('');
// a click copies to the remote desktop's clipboard (paste there with Ctrl+Shift+V) and to yours
function copyable(root) {
root.querySelectorAll('code.ic').forEach(c => c.onclick = () => {
root.querySelectorAll('code.ic, pre.code').forEach(c => c.onclick = () => {
const done = () => { c.classList.add('copied'); setTimeout(() => c.classList.remove('copied'), 700); };
api('/api/clip', {text: c.textContent}).then(done, done);
if (navigator.clipboard) navigator.clipboard.writeText(c.textContent).catch(() => {});
@ -657,8 +681,11 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
$('#hardHero').checked = hard;
const ended = sess && sess.ended_at;
$('#navResults').classList.toggle('hidden', !(ended && !sess.aborted));
const lessonLine = () => { const n = S.lesson.steps.length, c = Object.values(sess.steps || {}).filter(r => r.pass).length;
return `Your last lesson, ${esc(S.lesson.objective)}, ended with ${c}/${n} steps checked. <a href="#" id="lastResults">View its summary</a>`; };
$('#lastLine').innerHTML = !ended ? 'Every task gets its own cluster: a drill is ready in about a minute, a full exam in three or four.'
: sess.aborted ? 'Your last session was terminated: nothing was graded or recorded.'
: sess.mode === 'lesson' && S.lesson ? lessonLine()
: `Your last ${sess.mode === 'drill' ? 'drill' : 'exam'} ended with ${sess.score.pct}%. <a href="#" id="lastResults">View its results</a>`;
const lr = $('#lastResults'); if (lr) lr.onclick = e => { e.preventDefault(); go('results'); };
paintPath(); paintDomains();
@ -678,6 +705,10 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
}).join('');
$('#pathGo').textContent = 'Continue · ' + P.next.cmd.replace('./ksim ', '');
$('#pathWhy').textContent = P.next.why + '.';
// the next objective has a lesson you have not done: offer it before its drill
const no = P.next.objective && S.objectives.find(o => o.id === P.next.objective), lessonFirst = !!(no && no.lesson && !no.lesson.done);
$('#pathLesson').classList.toggle('hidden', !lessonFirst);
if (lessonFirst) { $('#pathLesson').textContent = `Lesson first · ${no.id}`; $('#pathLesson').title = no.lesson.title; $('#pathLesson').onclick = () => startLesson(no.id); }
// the objective list: notes, tasks, and what you said you know
const chip = t => `<span class="st-${t.state}" title="Q${t.id} ${esc(t.title)}${t.set ? ' (simulator set)' : ''}">${MARK[t.state]}${t.id}</span>`;
// pool tasks, then the simulator-set ones in one pair of brackets (they stay out of objective sessions)
@ -689,11 +720,13 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
const knownTag = d => known.includes(d)
? ` <span class="st-known" title="./ksim skip ${d}: you said you are good with this domain, so the path leaves it out. Nothing is recorded as passed, and a mock exam still draws from it.">· known, off the path</span><a href="#" class="touch" data-touch="${d}" title="./ksim drill ${d}:3 — three of its tasks, each from another objective, against the clock: to keep a domain you know warm">3 of its questions</a><a href="#" class="known" data-known="${d}" data-on="0" title="./ksim unskip ${d}">put it back</a>`
: `<a href="#" class="known" data-known="${d}" data-on="1" title="./ksim skip ${d}: leave this domain out of the path. Nothing is recorded as passed, and a mock exam still draws from it.">I know this — skip it</a>`;
const lmark = o => !o.lesson ? '' : `<span class="lmark${o.lesson.done ? ' done' : ''}" title="${o.lesson.done ? 'Lesson done' : 'A lesson teaches this objective'}: ${esc(o.lesson.title)}">lesson${o.lesson.done ? ' ✓' : ''}</span>`;
$('#objList').innerHTML = doms.map(d => `<h3>${esc(S.track.domains[d])}${focus.includes(d) ? ` <span class="st-focus" title="./ksim focus: the path walks this domain first">· focus ${focus.indexOf(d) + 1}</span>` : ''}${knownTag(d)}</h3>` + S.objectives.filter(o => o.domain === d).map(o =>
`<details class="obj"><summary>${mk(o.state)}<span class="oid">${o.id}</span><span class="otext">${esc(o.text)}</span><span class="chips">${chips(o)}</span></summary>
`<details class="obj"><summary>${mk(o.state)}<span class="oid">${o.id}</span><span class="otext">${esc(o.text)}</span><span class="chips">${lmark(o)}${chips(o)}</span></summary>
<div class="notes">${notesHtml(o)}</div>
<div class="objrow"><button class="btn btn-primary btn-sm" data-drill="${o.id}">Against the clock</button></div>
<div class="objrow">${o.lesson ? `<button class="btn btn-success btn-sm" data-lesson="${o.id}" title="${esc(o.lesson.title)}">Lesson · ${o.lesson.steps} steps, no clock</button> ` : ''}<button class="btn btn-primary btn-sm" data-drill="${o.id}">Against the clock</button></div>
</details>`).join('')).join('');
$$('#objList [data-lesson]').forEach(b => b.onclick = () => startLesson(b.dataset.lesson));
$$('#objList [data-drill]').forEach(b => b.onclick = () => start('drill', null, false, false, null, b.dataset.drill));
$$('#objList a.touch').forEach(a => a.onclick = e => { e.preventDefault(); start('drill', null, false, false, null, null, [[a.dataset.touch, 3]]); });
$$('#objList a.known').forEach(a => a.onclick = async e => {
@ -732,6 +765,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
function sessionOptions() {
const o = [];
if (S.path) o.push(['path', `Continue the path — ${S.path.next.cmd.replace('./ksim ', '')}`]);
if ((S.objectives || []).some(x => x.lesson)) o.push(['lesson', 'Take a lesson — a guided lab, no clock']);
if ((S.objectives || []).length) o.push(['objective', 'Drill an objective']);
o.push(['domain', 'Drill a domain'], ['next', `Drill the next domain — ${S.track.domains[nextDomain()]}`],
['all', 'Drill the whole bank'], ['pick', 'Pick the tasks to drill']);
@ -746,6 +780,9 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
$('#pkSession').value = opts.some(([v]) => v === keep) ? keep : opts[0][0];
$('#pkObjective').innerHTML = (S.objectives || []).map(o => `<option value="${o.id}">${MARK[o.state] || '·'} ${o.id} ${esc(o.text)}</option>`).join('');
if (S.path && S.path.next.objective) $('#pkObjective').value = S.path.next.objective;
const withLesson = (S.objectives || []).filter(o => o.lesson);
$('#pkLesson').innerHTML = withLesson.map(o => `<option value="${o.id}">${o.lesson.done ? '✓' : '·'} ${o.id} ${esc(o.lesson.title)}</option>`).join('');
const firstOpen = withLesson.find(o => !o.lesson.done); if (firstOpen) $('#pkLesson').value = firstOpen.id;
$('#pkDomain').innerHTML = Object.entries(S.track.domains).map(([d, n]) => `<option value="${d}">${d} — ${esc(n)}${(S.drilled || {})[d] ? ' ✓' : (S.known || []).includes(d) ? ' »' : ''}</option>`).join('');
$('#pkTasks').dataset.filled = '';
$('#hardPicker').checked = hard; $('#pkErr').classList.add('hidden');
@ -754,10 +791,15 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
async function pickerChanged() {
const v = $('#pkSession').value, T = S.track, per = hard ? 5 : 6;
$('#pkObjRow').classList.toggle('hidden', v !== 'objective');
$('#pkLessonRow').classList.toggle('hidden', v !== 'lesson');
$('#pkDomRow').classList.toggle('hidden', v !== 'domain');
$('#pkPickRow').classList.toggle('hidden', v !== 'pick');
let desc = '', warn = '';
if (v === 'path') desc = S.path.next.why + '.';
else if (v === 'lesson') {
const o = (S.objectives || []).find(x => x.id === $('#pkLesson').value);
desc = `A guided lab on a host of your own: an introduction, then ${o ? o.lesson.steps : 'a few'} steps, each with a Check button, a hint and the commands. No clock, and nothing goes into your history or scores. It ends by sending you to the objective's timed drill.`;
}
else if (v === 'objective') desc = `Its tasks that have not passed against the clock yet, up to five in a sitting, ${per} minutes each. Feedback per task while you work; a task's solution once it is graded.`;
else if (v === 'domain') desc = `Five tasks of that domain, ${per} minutes each, the ones you have seen least recently first. Feedback per task while you work.`;
else if (v === 'next') desc = `${T.domains[nextDomain()]} — 5 tasks, ${per} min each. Drills rotate through all domains before the exam.`;
@ -774,7 +816,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
}
$('#pkDesc').textContent = desc;
$('#pkWarn').textContent = warn; $('#pkWarn').classList.toggle('hidden', !warn);
$('#pkStart').textContent = warn ? 'Start anyway' : 'Start exam';
$('#pkStart').textContent = warn ? 'Start anyway' : v === 'lesson' ? 'Start lesson' : 'Start exam';
}
async function fillTaskPicker() {
const box = $('#pkTasks'), want = hard ? 'hard' : 'normal';
@ -788,6 +830,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
function pickerStart() {
const v = $('#pkSession').value;
if (v === 'path') return continuePath();
if (v === 'lesson') return startLesson($('#pkLesson').value);
if (v === 'objective') return start('drill', null, false, false, null, $('#pkObjective').value);
if (v === 'domain') return start('drill', null, false, false, $('#pkDomain').value);
if (v === 'next') return start('drill', null, true);
@ -823,6 +866,8 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
const start = (mode, ids, next, campaign, domain, objective, touch) =>
launch({mode, ids, next: !!next, campaign: !!campaign, domain: domain || null, objective: objective || null, touch: touch || null, hard});
const startSim = name => launch({mode: 'exam', sim: name, hard: true});
const startLesson = obj => launch({lesson: obj});
const isLesson = () => !!(S && S.lesson && S.session && S.session.mode === 'lesson');
async function resumeSeeding() {
polling = true; openModal('#mPrep');
try { await waitSeeded(); if (live()) ready(false); else route(); }
@ -831,8 +876,21 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
}
function ready(fresh) {
skew = Date.now() / 1000 - S.session.server_time; autoSubmitted = false; $('#vnc').dataset.ready = '';
cur = S.questions[0] && S.questions[0].id; go('exam');
cur = S.questions[0] && S.questions[0].id; lpage = fresh ? 0 : null; go('exam');
if (!fresh) return;
if (isLesson()) { // no clock to start: the dialog only says how a lesson goes
const L = S.lesson;
$('#startTitle').textContent = `Lesson ${L.objective}: ${L.title}`;
$('#startBody').innerHTML = `<div class="alert alert-info"><b>No clock.</b> Take your time: a lesson teaches, the drills test.</div>
<ul>
<li>it runs on a host of its own: <code class="ic">${esc('ssh ' + L.host)}</code> in the desktop's terminal;</li>
<li>read the introduction, then do the ${L.steps.length} steps - each has a <b>Check step</b> button, a hint, and the commands if you are stuck;</li>
<li>steps count in order, and nothing goes into your history or scores;</li>
<li>the wrap-up sends you to the objective's timed drill.</li>
</ul>`;
openModal('#mStart');
return;
}
// as on the day: the clock starts when you say so, not when the last cluster came up
const sess = S.session, drill = sess.mode === 'drill';
$('#startTitle').textContent = drill ? 'Ready to begin your drill' : 'Ready to begin your exam';
@ -858,6 +916,17 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
}
function paintMenus() {
const sess = S.session;
if (isLesson()) {
$('#mnControls').innerHTML = live()
? `<button class="dd-item" data-act="finishLesson">Finish lesson</button>
<div class="dd-sep"></div>
<button class="dd-item danger" data-act="terminate">Terminate lesson</button>`
: `<button class="dd-item" data-act="results">Back to the summary</button>
<button class="dd-item" data-act="dashboard">Back to the dashboard</button>
<div class="dd-sep"></div>
<button class="dd-item danger" data-act="terminate">Delete the host</button>`;
return;
}
$('#mnControls').innerHTML = live()
? `<button class="dd-item" data-act="submit">Submit ${sess.mode === 'drill' ? 'drill' : 'exam'}</button>
<button class="dd-item" data-act="pause">${sess.paused ? 'Resume the clock' : 'Pause the clock'}</button>
@ -870,7 +939,10 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
<button class="dd-item danger" data-act="terminate">Delete the hosts</button>`;
}
function paintExam() {
if (isLesson()) return paintLesson();
const sess = S.session, on = live();
$('#btnBack').classList.add('hidden');
$('#btnSatisfied').innerHTML = 'Satisfied with answer <span aria-hidden="true">→</span>';
$('#xTitle').textContent = `${S.track_id.toUpperCase()} ${on ? (sess.mode === 'drill' ? 'Drill' : 'Exam') : 'Review'}`
+ (sess.hard ? ' · HARD' : '') + (sess.objective ? ' · objective ' + sess.objective : '');
document.title = `${$('#xTitle').textContent} · Kubestronaut exam simulator`;
@ -908,8 +980,11 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
$('#btnFlag').className = 'btn ' + (fl ? 'btn-warning' : 'btn-outline-warning') + (on ? '' : ' hidden');
$('#btnFlag').innerHTML = `${FLAG} ${fl ? 'Flagged' : 'Flag for review'}`;
$('#btnSatisfied').disabled = i >= n - 1;
if (document.activeElement !== $('#notes')) $('#notes').value = sess.notes || '';
// the remote desktop
paintDesk();
}
// what the exam and a lesson share: the notepad, the remote desktop, the clock
function paintDesk() {
if (document.activeElement !== $('#notes')) $('#notes').value = S.session.notes || '';
const vnc = $('#vnc');
if (!vnc.dataset.ready) {
vnc.dataset.ready = 1;
@ -925,12 +1000,86 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
try { S = await api('/api/grade', {id: cur}); paintExam(); }
catch (e) { alert(e.message); b.disabled = false; b.textContent = 'Check this task'; }
}
function step(d) { const i = S.questions.findIndex(q => q.id === cur), j = i + d; if (j >= 0 && j < S.questions.length) { cur = S.questions[j].id; paintExam(); $('#qContent').scrollTop = 0; } }
function step(d) {
if (isLesson()) { lpage = Math.max(0, Math.min(lpage + d, S.lesson.steps.length + 1)); paintExam(); $('#qContent').scrollTop = 0; return; }
const i = S.questions.findIndex(q => q.id === cur), j = i + d;
if (j >= 0 && j < S.questions.length) { cur = S.questions[j].id; paintExam(); $('#qContent').scrollTop = 0; }
}
// ================= lesson screen =================
// Pages: 0 the introduction, 1..n the steps, n+1 the wrap-up. Same frame as the exam screen.
const stepDone = (k, sess = S.session) => !!((sess.steps || {})[k] && sess.steps[k].pass);
function lessonResume() { // where a reloaded lesson picks up: the first step not checked yet
const n = S.lesson.steps.length, k = [...Array(n).keys()].find(i => !stepDone(i));
return k === undefined ? n + 1 : k + 1;
}
function paintLesson() {
const L = S.lesson, sess = S.session, on = live(), n = L.steps.length;
if (lpage === null) lpage = lessonResume();
lpage = Math.max(0, Math.min(lpage, n + 1));
$('#xTitle').textContent = `${S.track_id.toUpperCase()} Lesson · ${L.objective} ${L.title}`;
document.title = `Lesson ${L.objective} · Kubestronaut exam simulator`;
paintMenus();
const checked = [...Array(n).keys()].filter(k => stepDone(k)).length, firstOpen = [...Array(n).keys()].find(k => !stepDone(k));
const name = p => p === 0 ? 'Introduction' : p === n + 1 ? 'Wrap-up' : `Step ${p} of ${n}`;
$('#qDrop').textContent = name(lpage);
$('#qPrev').disabled = lpage <= 0; $('#qNext').disabled = lpage >= n + 1;
$('#qMenu').innerHTML = [...Array(n + 2).keys()].map(p => `<button class="dd-item ${p >= 1 && p <= n && stepDone(p - 1) ? 'pass' : ''}" data-lp="${p}">`
+ (p === 0 ? 'Introduction' : p === n + 1 ? `Wrap-up · ${checked}/${n} steps checked` : `Step ${p} · ${esc(L.steps[p - 1].title)}`) + '</button>').join('');
$$('#qMenu [data-lp]').forEach(b => b.onclick = () => { lpage = +b.dataset.lp; closeMenus(); paintExam(); $('#qContent').scrollTop = 0; });
let body = `<div class="qhead"><div class="line"><b>This lesson runs on:</b> <code class="ic" title="click to copy">${esc('ssh ' + L.host)}</code></div>
<div class="line sub">${L.nodes}-node cluster · files in ${esc(S.workspace)} on that host · no clock</div></div><hr>`;
if (lpage === 0) {
body += `<div class="stepsum">Lesson ${esc(L.objective)} · ${n} steps</div><h2 class="qtitle">${esc(L.title)}</h2><div class="qbody">${render(L.intro)}</div>`;
} else if (lpage === n + 1) {
body += `<div class="stepsum">Wrap-up · ${checked} of ${n} steps checked</div><h2 class="qtitle">${esc(L.title)}</h2><div class="qbody">${render(L.outro)}</div>`
+ (on ? `<div class="lstate"><button class="btn btn-primary" id="btnFinish">Finish lesson</button>`
+ (checked < n ? `<span class="lock">${n - checked} step(s) not checked yet - the lesson counts as done only when every step is.</span>` : '') + '</div>' : '');
} else {
const k = lpage - 1, s = L.steps[k], r = (sess.steps || {})[k], locked = firstOpen !== undefined && k > firstOpen;
body += `<div class="stepsum">Step ${lpage} of ${n}</div><h2 class="qtitle">${esc(s.title)}</h2><div class="qbody">${render(s.text)}</div>`;
if (on) {
body += `<div class="lstate"><button class="btn btn-outline-primary" id="btnStep"${locked ? ' disabled' : ''}>Check step</button>`
+ (r && r.pass ? '<span class="ok">✓ Done</span>' : r ? '<span class="no">✗ Not done yet</span>' : '')
+ (locked ? `<span class="lock">Check step ${firstOpen + 1} first: steps count in order.</span>` : '') + '</div>'
+ (r && !r.pass && r.out ? `<details class="fold"><summary>What the check saw</summary><pre>${esc(r.out)}</pre></details>` : '');
} else if (r && r.pass) body += '<div class="lstate"><span class="ok">✓ Done</span></div>';
if (s.hint) body += `<details class="fold"><summary>Hint</summary><div class="qbody" style="margin-top:10px">${render(s.hint)}</div></details>`;
if (s.show) body += `<details class="fold"><summary>Show me the commands</summary><pre class="code" title="click to copy" style="margin:10px 0 0">${esc(s.show)}</pre></details>`;
}
$('#qContent').innerHTML = body;
copyable($('#qContent'));
const bs = $('#btnStep'); if (bs) bs.onclick = () => checkStep(lpage - 1);
const bf = $('#btnFinish'); if (bf) bf.onclick = finishLesson;
$('#btnFlag').className = 'btn btn-outline-warning hidden';
$('#btnBack').classList.remove('hidden'); $('#btnBack').disabled = lpage <= 0;
$('#btnSatisfied').innerHTML = (lpage === 0 ? 'Start: step 1' : lpage === n ? 'Wrap-up' : lpage === n + 1 ? 'Wrap-up' : 'Next step') + ' <span aria-hidden="true">→</span>';
$('#btnSatisfied').disabled = lpage >= n + 1;
paintDesk();
}
async function checkStep(k) {
const b = $('#btnStep'); b.disabled = true; b.textContent = 'Checking…';
try { S = await api('/api/lesson/check', {step: k}); }
catch (e) { alert(e.message === 'not found' ? oldServer('lessons') : e.message); }
paintExam();
}
async function finishLesson() {
const n = S.lesson.steps.length, c = [...Array(n).keys()].filter(k => stepDone(k)).length;
if (c < n && !(await ask({title: 'Finish the lesson?', ok: 'Finish lesson',
body: `<p class="mb0">${c} of ${n} steps are checked. The lesson counts as done only when every step is - you can take it again any time.</p>`}))) return;
$('#busyTitle').textContent = 'Finishing the lesson';
$('#busyBody').innerHTML = '<p class="mb0">Nothing is graded: the steps were checked as you went.</p>';
openModal('#mBusy');
try { S = await api('/api/end', {}); }
catch (e) { closeModal('#mBusy'); alert(e.message); return; }
closeModal('#mBusy'); go('results');
}
function tickClock() {
clearInterval(clockT);
const sess = S.session, el = $('#timer');
if (!sess || sess.ended_at) { el.textContent = sess ? 'ended' : '--:--'; el.className = 'timer'; return; }
if (sess.mode === 'lesson') { el.textContent = 'no clock'; el.className = 'timer paused'; return; }
const tick = () => {
const now = Date.now() / 1000 - skew;
const pausedTotal = sess.paused_total + (sess.paused ? (now - sess.server_time) : 0);
@ -973,8 +1122,11 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
catch (e) { alert(e.message === 'not found' ? oldServer('restart') : e.message); }
}
async function terminate() {
const on = live();
const ok = await ask(on
const on = live(), what = isLesson() ? 'lesson' : 'session';
const ok = await ask(on && isLesson()
? {title: 'Terminate the lesson', ok: 'Terminate lesson', okClass: 'btn-danger',
body: '<p class="mb0">This deletes the lesson\'s host right away and ends the lesson. Nothing is recorded; you can take it again any time.</p>'}
: on
? {title: 'Terminate session', ok: 'Terminate session', okClass: 'btn-danger',
body: `<div class="alert alert-warning"><p><b>Warning!</b> Terminating this session will:</p><ul><li>delete every host of this session right away;</li><li>end it without grading it - nothing is written to your history.</li></ul></div><p class="mb0">This cannot be undone. Terminate the session?</p>`}
: {title: 'Delete the hosts', ok: 'Delete the hosts', okClass: 'btn-danger',
@ -986,7 +1138,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
try { S = await api('/api/terminate', {}); }
catch (e) { alert(e.message === 'not found' ? oldServer('Terminate session') : e.message); return; }
finally { closeModal('#mBusy'); }
if (on) { go('landing'); lobbyNote('Session terminated: nothing was graded or recorded, and its hosts are deleted.'); }
if (on) { go('landing'); lobbyNote(`${what === 'lesson' ? 'Lesson' : 'Session'} terminated: nothing was graded or recorded, and its hosts are deleted.`); }
else { route(); alert('The hosts are deleted. Your results and your history stay.'); }
}
function toggleFullscreen() {
@ -996,7 +1148,33 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
function reconnect() { const f = $('#term'); if (f) f.src = S.desktop; }
// ================= results =================
function resultButtons(lesson) {
$('#btnAnswers').dataset.act = lesson ? 'retake' : 'answers'; $('#btnAnswers').textContent = lesson ? 'Take it again' : 'View answers';
$('#btnReview').textContent = lesson ? 'Review the lesson' : 'Review tasks';
$('#btnDelete').textContent = lesson ? 'Delete the host' : 'Delete the hosts';
$('#resLabel').textContent = lesson ? 'Steps checked' : 'Points scored';
}
function paintLessonResults() {
const L = S.lesson, s = S.session, n = L.steps.length, c = [...Array(n).keys()].filter(k => stepDone(k)).length, all = c === n;
document.title = `Lesson ${L.objective} · Kubestronaut exam simulator`;
resultButtons(true);
$('#resTitle').textContent = `Lesson ${L.objective}`;
$('#resMeta').innerHTML = `${esc(L.title)}<br>Finished ${new Date(s.ended_at * 1000).toLocaleString()}<br>No clock - nothing went into your history`;
$('#resEarned').textContent = c; $('#resTotal').textContent = n; $('#resPct').textContent = all ? 'done' : '';
$('#resBadge').textContent = all ? 'Lesson done ✓' : `${n - c} step(s) to go`;
$('#resBadge').className = 'rank-badge ' + (all ? 'rank-high' : 'rank-mid');
$('#resDomains').innerHTML = '';
$('#resPath').innerHTML = (s.milestones || []).map(m => `<div class="${m.startsWith('✓') ? 'milestone' : ''}">${esc(m)}</div>`).join('');
$('#resQuestions').innerHTML = `<div class="qcard ${all ? 't-pass' : 't-partial'}"><div class="qcard-h"><h3>${esc(L.title)}</h3><span class="qs">${c} of ${n}</span></div>
<ul class="vitems">${L.steps.map((st, k) => `<li class="vitem"><span>Step ${k + 1} · ${esc(st.title)}</span><span class="vstat ${stepDone(k) ? 'ok' : 'no'}">${stepDone(k) ? '✓' : '✗'}</span></li>`).join('')}</ul><div class="pad"></div></div>`;
$('#btnDrillMissed').disabled = false;
$('#btnDrillMissed').textContent = `Drill ${L.objective} against the clock`;
$('#btnDrillMissed').onclick = () => start('drill', null, false, false, null, L.objective);
}
function paintResults() {
if (isLesson()) return paintLessonResults();
resultButtons(false);
$('#btnDrillMissed').textContent = 'Drill what I missed';
const s = S.session, sc = s.score, pass = sc.pct >= sc.pass_pct;
document.title = `Results · Kubestronaut exam simulator`;
$('#resTitle').textContent = `${s.mode === 'drill' ? 'Drill' : 'Exam'} results`;
@ -1055,7 +1233,8 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
submit: confirmSubmit, pause: togglePause, restart: restartClock, terminate, fullscreen: toggleFullscreen, reconnect,
resetPanels: () => { $('#qpanel').style.width = ''; try { localStorage.removeItem('ksim.split'); } catch (e) { /* fine */ } },
desktopFullscreen: () => { const f = $('#term'); if (f && f.requestFullscreen) f.requestFullscreen().catch(() => {}); },
history: showHistory, answers: showAnswers, results: () => go('results'), dashboard: () => go('landing'), review: () => go('exam'),
history: showHistory, answers: showAnswers, results: () => go('results'), dashboard: () => go('landing'), review: () => { lpage = null; go('exam'); },
finishLesson, retake: () => startLesson(S.lesson.objective),
};
document.addEventListener('click', e => {
const t = e.target.closest('.dd-toggle');
@ -1084,16 +1263,20 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
$('#hardHero').onchange = e => setHard(e.target.checked);
$('#hardPicker').onchange = e => setHard(e.target.checked);
$('#pkSession').onchange = pickerChanged;
$('#pkLesson').onchange = pickerChanged;
$('#pkTrack').onchange = async e => { const keep = $('#pkSession').value; if (await switchTrack(e.target.value)) openPicker(keep); else e.target.value = S.track_id; };
$('#pkStart').onclick = pickerStart;
$('#btnBegin').onclick = async () => {
closeModal('#mStart'); toggleFullscreen();
closeModal('#mStart');
if (isLesson()) return; // a lesson has no clock to start
toggleFullscreen();
try { S = await api('/api/restart', {}); skew = Date.now() / 1000 - S.session.server_time; route(); }
catch (e) { /* a ./ksim web from before restart existed: the clock keeps the time it has */ }
};
$('#qPrev').onclick = () => step(-1);
$('#qNext').onclick = () => step(1);
$('#btnSatisfied').onclick = () => step(1);
$('#btnBack').onclick = () => step(-1);
$('#btnFlag').onclick = async () => { S = await api('/api/flag', {id: cur}); paintExam(); };
let nt; $('#notes').oninput = () => { clearTimeout(nt); nt = setTimeout(() => api('/api/notes', {notes: $('#notes').value}).catch(() => {}), 600); };
// the divider between the task and the desktop