vumast3r/kubestronaut-sim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: vumast3r:e182ab45a13887130198e8088d389970c418b1fd
Branches Tags
vumast3r:main
vumast3r:v2026.10.27.2
vumast3r:v2026.10.27.1
vumast3r:v2026.10.26.4
vumast3r:v2026.10.26.3
vumast3r:v2026.10.26.2
...
compare: vumast3r:e713a16a784f87bba0a24553c93dd67746e65bd6
Branches Tags
vumast3r:main
vumast3r:v2026.10.27.2
vumast3r:v2026.10.27.1
vumast3r:v2026.10.26.4
vumast3r:v2026.10.26.3
vumast3r:v2026.10.26.2

2 Commits
e182ab45a1 ... e713a16a78

Author SHA1 Message Date
vumas3r
e713a16a78 chore: release 2026.10.27.1 
Bump VERSION to 2026.10.27.1 and date its CHANGELOG.md entry: every
CKA reference solution shown the exam way and audited by its own parts,
./ksim solution lint, from e761d53.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNSZLLsGXTMBsw17kDunkQ
 2026-10-27 20:57:02 -06:00
vumas3r
e761d53a96 feat: show every CKA reference solution the way the exam is done 
105 of the 147 CKA reference solutions had you type what nobody types
in the exam: heredoc manifests, sed, JSON patches, kubectl wait, loops,
command substitution, jsonpath into files. Each is rewritten as the
lessons are: a generator and the lines to add in vim, a manifest pasted
from the docs page it names, kubectl edit, ssh NODE and sudo -i, or a
value read off the screen and echoed. The other 42 were commands typed
as they stand already.

A solution can now be parts (sim/bank.py cmd, hand) as a lesson step's
show is. The page and ./ksim solution N render it as Markdown, each
thing to type a block of its own that a click copies, with the edit
beneath it. A solution in parts is its own audit script: load_track
runs its parts, and for each edit by hand the script beside it, so the
solution you read is the one proven. Their separate scripts in
cka/solve.py are dropped, and a task that has both is refused.

./ksim solution lint [N|DOMAIN..] reads what the solutions have you
type against TYPING_TRAPS: the CKA bank is clean. Q49's params gain
the plural stem and shortName the solution spells out (no new draws);
Q103, ported from CKAD Q23, takes a solution of its own.

Audited from a copy with renamed hosts: Q3-Q67 (52 tasks) all solved by
their parts, after two waits were added to the audit scripts (Q43's
DNS record, Q53's Pods); Q68-Q147 still running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNSZLLsGXTMBsw17kDunkQ
 2026-10-27 20:56:51 -06:00
14 changed files with 3781 additions and 2653 deletions
Show Stats Expand all files Collapse all files

20
CHANGELOG.md
View File

@ -4,6 +4,26 @@ Notable changes to kubestronaut-sim, newest first. A version is the `VERSION` st
(year.month.day.n); `./ksim help` and the top bar of the web page show the one you are running.
Every entry names the commit it came in.
## 2026.10.27.1 — 2026-10-27
Everything below came in `e761d53`.
### Changed
- **Every CKA reference solution shows the exam way**, as the lessons do: a generator and the few
lines to add in `vim`, a manifest pasted from the docs page it names, `kubectl edit`, `ssh NODE`,
or a value you read and `echo` - no heredocs, `sed`, JSON patches, `kubectl wait` or loops. 105
of the 147 were rewritten; the other 42 were commands you type already. The page and `./ksim
solution N` show each thing to type as a block of its own, which a click copies.
- A solution in parts (`cmd`, `hand`) is its own audit script: `./ksim audit` runs what it shows,
and for each edit by hand the script beside it, so the solution you read is the one proven. Their
separate scripts in `cka/solve.py` are gone.
### Added
- `./ksim solution lint [N|DOMAIN..]`: what the reference solutions have you type, against the
list `./ksim lesson lint` uses.
## 2026.10.26.4 — 2026-10-26
Everything below came in `226ef19`.

18
README.md
View File

@ -200,7 +200,7 @@ Each track is a directory: `cka/`, `ckad/`, `cks/`.
| `bank_*.py` | more tasks; `bank_sim_*` are the fixed simulator sets |
| `curriculum.py` | the exam's objectives in order, each with study notes and the tasks that practise it |
| `solutions.py` | reference solutions (CKA keeps them apart; the others inline them) |
| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved |
| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved - for the tasks whose reference solution is not in parts |
| `lessons.py` | guided labs, one per objective (CKA: all, one `lessons_*.py` per domain · CKAD: 2.1–2.4): introduction, steps with their check, hint and commands, wrap-up |
| Track | Kubernetes | Bank today | Target (17–20 pool tasks per domain) |
@ -242,7 +242,8 @@ v1.36), CKAD Q1 and CKS Q60, Q61 and Q64 passed again on the rebuilt images.
"title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
"setup": "bash that seeds the scenario on the task's own cluster",
"checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
"solution": "...",
"solution": [cmd("k ... --dry-run=client -o yaml > f.yaml"), # or one string, a script as it stands
hand("vim f.yaml", "what to add, Markdown", run="the same, scripted: yq, a heredoc")],
"workers": 2, # optional: worker nodes of its cluster
"k8s": "v1.36"} # optional: a minor of its own (an upgrade task)
```
@ -287,9 +288,16 @@ and the untouched grade, which catches a check that only passes once the seeded
until a stopped kubelet shows as NotReady; `--keep` leaves the host up to look at
(`KUBECONFIG=.state/envs/<host>/kubeconfig`, `./ksim down` removes it).
In `cka/solve.py`, `AS_WRITTEN` lists the tasks whose reference solution is a script as it stands:
for those the audit runs exactly the text the candidate is shown, so a solution that stops working
with a new Kubernetes release fails the next audit instead of misleading you after a drill.
A reference solution in parts - `cmd()` and `hand()`, as a lesson step's `show` - is its own
script: the audit runs what is typed as it stands, and for each part done by hand its `run`, so the
solution the candidate reads after a drill is the one that is proven, and one that stops working
with a new Kubernetes release fails the next audit. 105 of the 147 CKA solutions are written so
(the page and `./ksim solution N` show each thing to type as a block of its own, which a click
copies, with the edit to make in `vim` beneath it); the other 42 are commands typed as they stand,
each with its script in `solve.py`. `./ksim solution lint` reads what they have you type, against
the same list of things you would not type in the exam as `./ksim lesson lint`. In `cka/solve.py`,
`AS_WRITTEN` lists the few tasks whose reference solution is still one script, which the audit
runs as it stands.
**Writing a lesson.** A lesson in `<track>/lessons.py` is a dict with `id` (9000 plus the objective,
`9201` for 2.1 — its host is `ckad9201`, which no task's can be), `objective`, `title`, `workers`,

732
cka/bank_course.py
View File

@ -12,6 +12,8 @@ operator with broken RBAC do its job, and 146-147 grow, snapshot, restore and cl
the CSI hostpath driver's class `csi-hostpath-sc`, as the real exam's clusters have it
(ensure_csi_hostpath).
"""
from sim.bank import cmd, hand
from cka.common import C, CAN, NET, NS, ONE, backup_operator, ns_pick, name_pick
_ETCD = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
@ -59,38 +61,65 @@ for n in "$CP" "$W1"; do
done
eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" v1.37.1 && eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" 2"""),
],
"solution": """
# the control plane - this host
sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list # one repository per minor
sudo apt-get update
"solution": [hand(r"""sudo vim /etc/apt/sources.list.d/kubernetes.list""",
r"""The docs page *Upgrading kubeadm clusters* starts here: pkgs.k8s.io has one repository per minor, so 1.37 packages exist for apt only once the line says `/v1.37/` instead of `/v1.36/`.""",
run=r"""apt_get() { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq -y -o Dpkg::Options::=--force-confold "$@" </dev/null >/dev/null; }
sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list"""),
hand(r"""sudo apt-get update
apt-cache madison kubeadm | head -3
sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.1 -y # a few minutes
k drain {CP} --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
k uncordon {CP}
# the worker
k drain {W1} --ignore-daemonsets --delete-emptydir-data
{SHELL_W1}
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.37.1-1.1
sudo apt-mark hold kubeadm""",
r"""The control plane - this host - first, and kubeadm first: it drives the upgrade.""",
run=r"""apt_get update
sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null"""),
hand(r"""sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.1""",
r"""Answer `y`; it takes a few minutes, and the API server restarts on the new version.""",
run=r"""sudo kubeadm upgrade apply v1.37.1 -y </dev/null | tail -2"""),
cmd(r"""k drain {CP} --ignore-daemonsets --delete-emptydir-data"""),
hand(r"""sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet""",
r"""Then the node's kubelet and kubectl, held again.""",
run=r"""sudo apt-mark unhold kubelet kubectl >/dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null
sudo systemctl daemon-reload && sudo systemctl restart kubelet"""),
hand(r"""k uncordon {CP}
k get nodes""",
r"""Again until `k` answers - the kubelet is restarting. `{CP}` shows `v1.37.1`.""",
run=r"""for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""k drain {W1} --ignore-daemonsets --delete-emptydir-data"""),
hand(r"""{SHELL_W1}
sudo -i
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update
vim /etc/apt/sources.list.d/kubernetes.list""",
r"""`/v1.36/` to `/v1.37/` here too.""",
run=r""""""),
hand(r"""apt-get update
apt-mark unhold kubeadm kubelet kubectl
apt-get install -y kubeadm=1.37.1-1.1
kubeadm upgrade node # a worker: node, not apply
kubeadm upgrade node
apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
apt-mark hold kubeadm kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
exit
exit""",
r"""A worker runs `kubeadm upgrade node`, not `apply`: it hosts no control plane, only its kubelet's configuration to update.""",
run=r"""ssh {W1} sudo bash <<'EOF'
export DEBIAN_FRONTEND=noninteractive
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq </dev/null
apt-mark unhold kubeadm kubelet kubectl >/dev/null
apt-get install -y -qq kubeadm=1.37.1-1.1 </dev/null >/dev/null
kubeadm upgrade node </dev/null | tail -1
apt-get install -y -qq kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 </dev/null >/dev/null
apt-mark hold kubeadm kubelet kubectl >/dev/null
systemctl daemon-reload && systemctl restart kubelet
exit; exit
k uncordon {W1}
k get nodes
# docs: "Upgrading kubeadm clusters". pkgs.k8s.io has one repository per minor, so 1.37 packages
# appear only once the line names v1.37. kubeadm first - it drives the upgrade and rewrites the
# kubelet's configuration; kubelet and kubectl after `upgrade apply`, then the restart: `k get
# nodes` shows each kubelet's running version. Workers run `kubeadm upgrade node`.
""",
EOF"""),
cmd(r"""k uncordon {W1}
k get nodes""", r"""Both `Ready`, both `v1.37.1` - `VERSION` is each kubelet's.""")],
},
# ------------------------------------------------------------------ 133 ----
{
@ -196,19 +225,16 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.ephemeralContainers[?(@.name=="debugger")].t
&& eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q134.uid") 0" """),
C("mode.txt holds the value the process runs with", 3, ONE + r"""eq "$(one "$WS/q134/mode.txt")" %{mode}"""),
],
"solution": """
k -n %{ns} exec %{pod} -- sh # fails: no shell in the image
k -n %{ns} get cm %{pod}-env -o yaml # MODE=open - but that is the ConfigMap NOW
k -n %{ns} debug %{pod} --image=busybox:1.36 --target=app -c debugger --profile=general -- sleep 3600
for i in $(seq 1 30); do k -n %{ns} exec %{pod} -c debugger -- true 2>/dev/null && break; sleep 2; done
k -n %{ns} exec %{pod} -c debugger -- ps # PID 1 is /pause, the app's process
k -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\\0" "\\n" < /proc/1/environ' | grep '^MODE='
k -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\\0" "\\n" < /proc/1/environ' | sed -n 's/^MODE=//p' > {WS}/q134/mode.txt
# Environment variables from a ConfigMap are read once, when the container starts: editing the
# ConfigMap changes nothing for a running process. --target puts the debug container into app's
# process namespace, so /proc/1 is app's process; the `general` profile gives it SYS_PTRACE,
# which reading another user's /proc/PID/environ needs. Interactively: `k debug -it ... -- sh`.
""",
"solution": [cmd(r"""k -n %{ns} exec %{pod} -- sh
k -n %{ns} get cm %{pod}-env -o yaml""", r"""No shell in the image. And the ConfigMap says `MODE=open` - now; the environment was read once, when the container started."""),
cmd(r"""k -n %{ns} debug %{pod} --image=busybox:1.36 --target=app -c debugger --profile=general -- sleep 3600""", r"""`--target` puts the debug container into `app`'s process namespace; the `general` profile gives it `SYS_PTRACE`, which reading another process's environment needs. Interactively: `k debug -it ... -- sh`."""),
hand(r"""k -n %{ns} exec %{pod} -c debugger -- ps""",
r"""Again until it answers: PID 1 is `/pause`, `app`'s process.""",
run=r"""for i in $(seq 1 30); do kubectl -n %{ns} exec %{pod} -c debugger -- true 2>/dev/null && break; sleep 2; done"""),
cmd(r"""k -n %{ns} exec %{pod} -c debugger -- cat /proc/1/environ | tr '\0' '\n'""", r"""`MODE=%{mode}`: what the process runs with."""),
hand(r"""echo %{mode} > {WS}/q134/mode.txt""",
r"""With the value after `MODE=`.""",
run=r"""kubectl -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\0" "\n" < /proc/1/environ' | sed -n 's/^MODE=//p' > {WS}/q134/mode.txt""")],
},
# ------------------------------------------------------------------ 135 ----
{
@ -253,19 +279,19 @@ tok = [v for v in s.get("volumes", []) if any("serviceAccountToken" in src for s
sys.exit(1 if tok else 0)' || exit 1
done"""),
],
"solution": """
k -n %{ns} create serviceaccount pipeline
"solution": [cmd(r"""k -n %{ns} create serviceaccount pipeline
k -n %{ns} create role pipeline --verb=get,list,create,update,patch --resource=deployments.apps
k -n %{ns} create rolebinding pipeline --role=pipeline --serviceaccount=%{ns}:pipeline
k auth can-i delete deployments -n %{ns} --as=system:serviceaccount:%{ns}:pipeline # no
k -n %{ns} create token pipeline --duration=%{h}h > {WS}/q135/pipeline.token
k -n %{ns} patch deploy frontend -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":false}}}}'
k -n %{ns} rollout status deploy frontend
# `create token` asks the TokenRequest API for a short-lived, bound token - nothing is stored in a
# Secret. automountServiceAccountToken: false on the Pod template (or on the ServiceAccount; the
# Pod's setting wins) leaves out the projected volume kube-api-access-*. To try a token alone, use
# no kubeconfig: a client certificate in yours would be checked first.
""",
k auth can-i delete deployments -n %{ns} --as=system:serviceaccount:%{ns}:pipeline""", r"""`no`."""),
cmd(r"""k -n %{ns} create token pipeline --duration=%{h}h > {WS}/q135/pipeline.token""", r"""The TokenRequest API issues a short-lived, bound token; nothing is stored in a Secret."""),
hand(r"""k -n %{ns} edit deploy frontend""",
r"""In the Pod template's `spec:`, beside `containers:` and indented like it, then `:wq`:
```
automountServiceAccountToken: false
```
It leaves out the projected volume `kube-api-access-*`. (On the ServiceAccount it works too; the Pod's setting wins.)""",
run=r"""kubectl -n %{ns} patch deploy frontend -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":false}}}}'"""),
cmd(r"""k -n %{ns} rollout status deploy frontend""")],
},
# ------------------------------------------------------------------ 136 ----
{
@ -298,22 +324,20 @@ eq "$(x theme)" dark && eq "$(x locale)" en_GB && eq "$(x db/password)" pw-%{mem
x() { $K -n %{ns} exec %{pod} -- cat "/etc/portal/$1" 2>/dev/null; }
x labels | grep -qx 'app="%{pod}"' && eq "$(x mem-limit)" %{mem}"""),
],
"solution": """
k -n %{ns} apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: %{pod}
labels: {app: %{pod}}
spec:
containers:
- name: portal
image: busybox:1.36
command: [sleep, '86400']
"solution": [cmd(r"""k -n %{ns} run %{pod} --image=busybox:1.36 --labels=app=%{pod} --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Replace the container's `resources: {}` with the limit and the mount:
```
resources:
limits: {memory: %{mem}Mi}
limits:
memory: %{mem}Mi
volumeMounts:
- {name: portal, mountPath: /etc/portal, readOnly: true}
- name: portal
mountPath: /etc/portal
readOnly: true
```
and add to the Pod's `spec:`, right below `spec:` (the docs page *Projected Volumes* has an example with all three kinds):
```
volumes:
- name: portal
projected:
@ -323,20 +347,33 @@ spec:
- secret:
name: portal-db
items:
- {key: password, path: db/password}
- key: password
path: db/password
- downwardAPI:
items:
- path: labels
fieldRef: {fieldPath: metadata.labels}
fieldRef:
fieldPath: metadata.labels
- path: mem-limit
resourceFieldRef: {containerName: portal, resource: limits.memory, divisor: 1Mi}
EOF
k -n %{ns} wait --for=condition=Ready pod/%{pod}
k -n %{ns} exec %{pod} -- sh -c 'ls -R /etc/portal; cat /etc/portal/labels /etc/portal/mem-limit'
# `items` picks keys and renames them: only `password` from the Secret, as db/password. A
# resourceFieldRef needs the container's name in a volume (in an env var it is implied), and the
# divisor turns bytes into MiB.
""",
resourceFieldRef:
containerName: %{pod}
resource: limits.memory
divisor: 1Mi
```
`items` picks keys and renames them; in a volume a `resourceFieldRef` needs the container's name, and the divisor turns bytes into MiB.""",
run=r"""yq -i '.spec.containers[0].resources = {"limits": {"memory": "%{mem}Mi"}}
| .spec.containers[0].volumeMounts = [{"name": "portal", "mountPath": "/etc/portal", "readOnly": true}]
| .spec.volumes = [{"name": "portal", "projected": {"sources": [
{"configMap": {"name": "portal-config"}},
{"secret": {"name": "portal-db", "items": [{"key": "password", "path": "db/password"}]}},
{"downwardAPI": {"items": [{"path": "labels", "fieldRef": {"fieldPath": "metadata.labels"}},
{"path": "mem-limit", "resourceFieldRef": {"containerName": "%{pod}", "resource": "limits.memory", "divisor": "1Mi"}}]}}]}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod %{pod}""",
r"""Again until it is `Running`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=120s"""),
cmd(r"""k -n %{ns} exec %{pod} -- ls -R /etc/portal
k -n %{ns} exec %{pod} -- cat /etc/portal/labels /etc/portal/mem-limit""")],
},
# ------------------------------------------------------------------ 137 ----
{
@ -404,18 +441,19 @@ eq "$(j -n %{ns} deploy critical '{.status.availableReplicas} {.status.updatedRe
eq "$(j -n %{ns} deploy critical '{.spec.template.spec.containers[0].resources.requests.cpu} {.spec.template.spec.containers[0].resources.limits.cpu} {.spec.template.spec.containers[0].resources.requests.memory} {.spec.template.spec.containers[0].resources.limits.memory}')" "200m 200m 128Mi 128Mi" || exit 1
for q in $($K -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'); do [ "$q" = Guaranteed ] || exit 1; done"""),
],
"solution": """
k -n %{ns} get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
k -n %{ns} get pods alpha beta delta gamma -o jsonpath='{range .items[*]}{.metadata.name} {.status.qosClass}{"\\n"}{end}' > {WS}/q137/qos.txt
echo gamma > {WS}/q137/first.txt
k -n %{ns} set resources deploy critical --requests=cpu=200m,memory=128Mi --limits=cpu=200m,memory=128Mi
"solution": [cmd(r"""k -n %{ns} get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass""", r"""Guaranteed: every container has requests equal to limits, CPU and memory. BestEffort: no container has any. Everything else is Burstable - `delta` too, as one of its containers sets nothing."""),
hand(r"""vim {WS}/q137/qos.txt""",
r"""```
alpha Guaranteed
beta Burstable
gamma BestEffort
delta Burstable
```""",
run=r"""printf 'alpha Guaranteed\nbeta Burstable\ngamma BestEffort\ndelta Burstable\n' > {WS}/q137/qos.txt"""),
cmd(r"""echo gamma > {WS}/q137/first.txt""", r"""Under memory pressure BestEffort goes first, then Burstable Pods using more than they requested, Guaranteed last."""),
cmd(r"""k -n %{ns} set resources deploy critical --requests=cpu=200m,memory=128Mi --limits=cpu=200m,memory=128Mi
k -n %{ns} rollout status deploy critical
k -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'; echo
# Guaranteed: every container has requests equal to limits for both CPU and memory. BestEffort: no
# container has any. Everything else is Burstable - delta too, because one container sets nothing.
# Under memory pressure the kubelet evicts BestEffort first, then Burstable Pods using more than
# they requested; Guaranteed Pods last.
""",
k -n %{ns} get pods -l app=critical -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass""")],
},
# ------------------------------------------------------------------ 138 ----
{
@ -452,15 +490,21 @@ eq "$(j -n %{ns} pod %{pod} '{.status.containerStatuses[0].resources.requests.cp
eq "$(j -n %{ns} pod %{pod} '{.spec.containers[0].resources.limits.cpu}')" %{lim} \
&& eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q138.uid") 0" """),
],
"solution": """
k -n %{ns} get pod %{pod} -o jsonpath='{.spec.containers[0].resources}'; echo
k -n %{ns} patch pod %{pod} --subresource resize --patch '{"spec":{"containers":[{"name":"encoder","resources":{"requests":{"cpu":"%{req}"},"limits":{"cpu":"%{lim}"}}}]}}'
k -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].resources}{"\\n"}{.status.containerStatuses[0].restartCount}{"\\n"}'
# A running Pod's CPU and memory can be changed in place, through the `resize` subresource - a
# plain `k edit` or `k apply` of a Pod's resources is refused. The spec
# says what you asked for; status.containerStatuses[].resources what the kubelet has applied.
# A container's resizePolicy decides whether a resource's change needs a restart (CPU: no).
""",
"solution": [cmd(r"""k -n %{ns} get pod %{pod} -o yaml | grep -A6 resources:"""),
hand(r"""k -n %{ns} edit pod %{pod} --subresource resize""",
r"""Through the `resize` subresource - a plain `k edit` of a Pod's resources is refused. Set the CPU, leave the memory, then `:wq`:
```
resources:
limits:
cpu: %{lim}
memory: 48Mi
requests:
cpu: %{req}
memory: 48Mi
```""",
run=r"""kubectl -n %{ns} patch pod %{pod} --subresource resize --patch '{"spec":{"containers":[{"name":"encoder","resources":{"requests":{"cpu":"%{req}"},"limits":{"cpu":"%{lim}"}}}]}}'"""),
cmd(r"""k -n %{ns} get pod %{pod} -o yaml | grep -A6 resources:
k -n %{ns} get pod %{pod}""", r"""The spec says what you asked for, `status.containerStatuses[].resources` what the kubelet applied; `RESTARTS 0` - a container's `resizePolicy` decides, and CPU needs no restart.""")],
},
# ------------------------------------------------------------------ 139 ----
{
@ -495,24 +539,33 @@ sys.exit(0 if s == "%{soft}Mi" and secs == 90 else 1)'"""),
nx "$W1" 'grep -q "%{soft}Mi" /var/lib/kubelet/config.yaml && grep -q "%{mem}Mi" /var/lib/kubelet/config.yaml && grep -q "^staticPodPath:" /var/lib/kubelet/config.yaml && systemctl is-active -q kubelet' </dev/null || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True"""),
],
"solution": """
{SHELL_W1}
"solution": [hand(r"""{SHELL_W1}
sudo -i
grep -n -A4 '^eviction' /var/lib/kubelet/config.yaml # kind sets the hard thresholds to 0%
yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%"
| .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
grep -n -A4 ^eviction /var/lib/kubelet/config.yaml
vim /var/lib/kubelet/config.yaml""",
r"""Under `evictionHard:` set the two hard thresholds, and add the soft one with its grace period at the top level - a soft threshold without a grace period for the same signal, and the kubelet refuses to start:
```
evictionHard:
memory.available: "%{mem}Mi"
nodefs.available: "%{fs}%"
evictionSoft:
memory.available: "%{soft}Mi"
evictionSoftGracePeriod:
memory.available: "1m30s"
```
Keep the other `evictionHard` entries as they are.""",
run=r"""ssh {W1} sudo bash <<'EOF'
yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
systemctl restart kubelet
EOF"""),
hand(r"""systemctl restart kubelet
journalctl -u kubelet -n 5 --no-pager
exit; exit
k get --raw /api/v1/nodes/{W1}/proxy/configz | jq '.kubeletconfig | {evictionHard, evictionSoft, evictionSoftGracePeriod}'
# A soft threshold needs a grace period for the same signal, or the kubelet refuses to start. By
# hand: under evictionHard add `memory.available: "%{mem}Mi"` and change nodefs.available; then
# evictionSoft:
# memory.available: "%{soft}Mi"
# evictionSoftGracePeriod:
# memory.available: "1m30s"
# configz shows what the running kubelet uses.
""",
exit
exit""",
r"""No complaint about the configuration; then back on the host.""",
run=r"""for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done"""),
cmd(r"""k get node {W1}
k get --raw /api/v1/nodes/{W1}/proxy/configz | tr ',' '\n' | grep -i eviction""", r"""`Ready`; `configz` shows the configuration the running kubelet uses.""")],
},
# ------------------------------------------------------------------ 140 ----
{
@ -548,19 +601,24 @@ ch=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A
n=$(nx "$CP" "iptables-save -t nat | grep -- '^-A $ch ' | grep -c -- '-j KUBE-SEP-'" </dev/null)
eq "$n" %{n} && eq "$(one "$WS/q140/endpoints.txt")" "$n" """),
],
"solution": """
k -n kube-system get cm kube-proxy -o jsonpath='{.data.config\\.conf}' | grep '^mode' # iptables
echo iptables > {WS}/q140/mode.txt
ip=$(k -n %{ns} get svc %{svc} -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32" # KUBE-SERVICES ... -j KUBE-SVC-XXXX
chain=$(sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1)
echo $chain > {WS}/q140/chain.txt
sudo iptables-save -t nat | grep -- "^-A $chain " # one -j KUBE-SEP-... per endpoint
sudo iptables-save -t nat | grep -- "^-A $chain " | grep -c -- '-j KUBE-SEP-' > {WS}/q140/endpoints.txt
# KUBE-SERVICES matches ClusterIP:port and jumps to the Service's KUBE-SVC chain; that picks an
# endpoint chain at random (statistic mode random), and each KUBE-SEP chain DNATs to one Pod.
# One KUBE-SEP per ready endpoint. In nftables mode: `sudo nft list ruleset | grep -A5 %{svc}`.
""",
"solution": [cmd(r"""k -n kube-system get cm kube-proxy -o yaml | grep mode""", r"""`mode: iptables` - an empty mode means the default, iptables too."""),
hand(r"""echo iptables > {WS}/q140/mode.txt""",
r"""The mode it printed.""",
run=r"""m=$(kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"')
echo ${m:-iptables} > {WS}/q140/mode.txt"""),
cmd(r"""k -n %{ns} get svc %{svc}""", r"""Note its `CLUSTER-IP`."""),
hand(r"""sudo iptables-save -t nat | grep 10.96.12.34""",
r"""With that IP: `KUBE-SERVICES` matches the ClusterIP and port and jumps (`-j`) to the Service's own `KUBE-SVC-...` chain.""",
run=r""""""),
hand(r"""echo KUBE-SVC-ABCDEFGHIJKLMNOP > {WS}/q140/chain.txt
sudo iptables-save -t nat | grep -- '-A KUBE-SVC-ABCDEFGHIJKLMNOP '""",
r"""With the chain's name. Its rules pick one endpoint chain at random (`statistic mode random`); each `KUBE-SEP-...` DNATs to one Pod - count them. In nftables mode: `sudo nft list ruleset | grep -A5 %{svc}`.""",
run=r"""ip=$(kubectl -n %{ns} get svc %{svc} -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/q140/chain.txt"""),
hand(r"""echo 3 > {WS}/q140/endpoints.txt""",
r"""With the number of `-j KUBE-SEP-` rules: one per ready endpoint.""",
run=r"""chain=$(cat {WS}/q140/chain.txt)
sudo iptables-save -t nat | grep -- "^-A $chain " | grep -c -- '-j KUBE-SEP-' > {WS}/q140/endpoints.txt""")],
},
# ------------------------------------------------------------------ 141 ----
{
@ -593,36 +651,65 @@ eq "$(j -n %{ns} endpointslice billing-1 '{.metadata.labels.kubernetes\.io/servi
C("wget billing from client returns the API's answer", 3, r"""
for i in 1 2 3 4 5; do [ "$($K -n %{ns} exec client -- wget -qO- -T 3 billing 2>/dev/null)" = billing-api-%{port} ] && exit 0; sleep 3; done; exit 1"""),
],
"solution": """
cat {WS}/q141/billing.txt # ADDRESS:PORT
addr=$(cut -d: -f1 {WS}/q141/billing.txt); port=$(cut -d: -f2 {WS}/q141/billing.txt)
k -n %{ns} apply -f - <<EOF
"solution": [cmd(r"""cat {WS}/q141/billing.txt""", r"""`ADDRESS:PORT`."""),
hand(r"""vim billing.yaml""",
r"""A Service with no selector - no controller writes its endpoints - and the EndpointSlice you maintain, tied to it by the label `kubernetes.io/service-name` (the docs page *Service* has both under *Services without selectors*). With the address and port from the file:
```
apiVersion: v1
kind: Service
metadata:
name: billing
namespace: %{ns}
spec:
ports:
- {port: 80, protocol: TCP}
- port: 80
protocol: TCP
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: billing-1
namespace: %{ns}
labels:
kubernetes.io/service-name: billing
addressType: IPv4
endpoints:
- addresses: ["$addr"]
- addresses:
- 172.18.0.3
ports:
- {port: $port, protocol: TCP}
EOF
k -n %{ns} exec client -- wget -qO- -T3 billing
# Without a selector, no controller writes endpoints for the Service: the label
# kubernetes.io/service-name ties a slice to it. The slice's port is where traffic goes (the
# Service's targetPort plays no part), and its name must match the Service port's - here both
# unnamed. ExternalName would need a DNS name; an address needs this.
""",
- port: %{port}
protocol: TCP
```
The slice's port is where traffic goes - the Service has no `targetPort` to speak of - and its name must match the Service port's: here both unnamed.""",
run=r"""addr=$(cut -d: -f1 {WS}/q141/billing.txt); port=$(cut -d: -f2 {WS}/q141/billing.txt)
cat > billing.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
name: billing
namespace: %{ns}
spec:
ports:
- port: 80
protocol: TCP
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: billing-1
namespace: %{ns}
labels:
kubernetes.io/service-name: billing
addressType: IPv4
endpoints:
- addresses:
- "$addr"
ports:
- port: $port
protocol: TCP
EOF"""),
cmd(r"""k apply -f billing.yaml
k -n %{ns} exec client -- wget -qO- -T3 billing""", r"""`billing-api-%{port}`. (`ExternalName` would need a DNS name; an address needs this.)""")],
},
# ------------------------------------------------------------------ 142 ----
{
@ -660,8 +747,24 @@ $K -n %{ns} create secret generic ksim-probe --from-literal=x=y >/dev/null 2>&1
nx "$CP" '""" + _ETCD + r""" get /registry/secrets/%{ns}/ksim-probe --print-value-only 2>/dev/null | head -c 64' </dev/null | grep -aq '^k8s:enc:aescbc:v1:'; r=$?
$K -n %{ns} delete secret ksim-probe --ignore-not-found >/dev/null 2>&1; exit $r"""),
],
"solution": """
sudo mkdir -p /etc/kubernetes/enc
"solution": [cmd(r"""head -c 32 /dev/urandom | base64""", r"""The host is the control-plane node. Copy the output: that is the key."""),
hand(r"""sudo mkdir -p /etc/kubernetes/enc
sudo vim /etc/kubernetes/enc/enc.yaml""",
r"""Paste the example from the docs page *Encrypting Confidential Data at Rest* (search for `aescbc`) and cut it down to:
```
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
providers:
- aescbc:
keys:
- name: key1
secret: <your key>
- identity: {}
```
The first provider encrypts; `identity` after it still reads what was stored before.""",
run=r"""sudo mkdir -p /etc/kubernetes/enc
sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
@ -673,23 +776,31 @@ resources:
- name: key1
secret: $(head -c 32 /dev/urandom | base64)
- identity: {}
EOF
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak # a backup OUTSIDE manifests/
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
EOF"""),
cmd(r"""sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/""", r"""Edit a copy: the kubelet acts on every save in `manifests/`, half-done edits too."""),
hand(r"""sudo vim ~/kube-apiserver.yaml""",
r"""Add three lines, each beside its kind (`yy` and `p` on an existing `k8s-certs` mount or volume, then change it):
```
- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
```
in the `command:` list,
```
- {name: enc, mountPath: /etc/kubernetes/enc, readOnly: true}
```
under `volumeMounts:`, and
```
- {name: enc, hostPath: {path: /etc/kubernetes/enc, type: DirectoryOrCreate}}
```
under `volumes:` - the API server runs in a container, and sees only what is mounted.""",
run=r"""sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
| .spec.containers[0].volumeMounts += [{"name": "enc", "mountPath": "/etc/kubernetes/enc", "readOnly": true}]
| .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' /tmp/kube-apiserver.yaml
sudo cp /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
sleep 20; for i in $(seq 1 40); do k get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
k -n %{ns} get secret %{sec} -o json | k replace -f - # stored again: encrypted now
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \\
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \\
get /registry/secrets/%{ns}/%{sec} | od -c | head -3 # k 8 s : e n c : a e s c b c : v 1 : k e y 1
# By hand: in kube-apiserver.yaml add the flag to the command, a volumeMount for /etc/kubernetes/enc
# and a hostPath volume of the same path - the API server runs in a container and sees only what is
# mounted. The FIRST provider encrypts; `identity` after it reads what was stored before. Every
# Secret: `k get secrets -A -o json | k replace -f -`.
""",
| .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' ~/kube-apiserver.yaml"""),
cmd(r"""sudo cp ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
hand(r"""k -n kube-system get pods""",
r"""Again until it answers: the API server is back in a minute or so.""",
run=r"""sleep 20; for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""k -n %{ns} get secret %{sec} -o yaml | k replace -f -""", r"""Written again, the Secret is stored encrypted; every Secret at once: `k get secrets -A -o json | k replace -f -`."""),
cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/%{ns}/%{sec} | head -c 80""", r"""`k8s:enc:aescbc:v1:key1:` and no plain text.""")],
},
# ------------------------------------------------------------------ 143 ----
{
@ -730,20 +841,39 @@ sys.exit(0 if ok else 1)'"""),
eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas} {.status.updatedReplicas} {.status.replicas}')" "3 3 3" || exit 1
[ "$($K -n %{ns} get pods -l app=%{dep} --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 3 ]"""),
],
"solution": """
k label node {W2} {W3} tier=edge
k -n %{ns} patch deploy %{dep} --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}},"template":{"spec":{"affinity":{
"solution": [cmd(r"""k label node {W2} {W3} tier=edge"""),
hand(r"""k -n %{ns} edit deploy %{dep}""",
r"""In the Pod template's `spec:`, above `containers:` (the docs page *Assigning Pods to Nodes* has both kinds):
```
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: %{dep}
topologyKey: kubernetes.io/hostname
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: tier
operator: In
values:
- edge
```
and in the same edit, under `strategy:`, let the rollout replace one Pod at a time:
```
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
```
With 3 workers and one Pod each, a surge Pod has nowhere to go, and 25% of 3 unavailable rounds down to 0: the default rollout would wait for ever. Then `:wq`.""",
run=r"""kubectl -n %{ns} patch deploy %{dep} --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}},"template":{"spec":{"affinity":{
"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app":"%{dep}"}},"topologyKey":"kubernetes.io/hostname"}]},
"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"tier","operator":"In","values":["edge"]}]}}]}}}}}}'
k -n %{ns} rollout status deploy %{dep}
k -n %{ns} get pods -o wide
# The anti-affinity's labelSelector picks the Pods to stay away from - its own, by its template's
# labels - and topologyKey says per what: one per hostname = one per node. With 3 workers a 4th
# replica would stay Pending - and so does the rolling update's surge Pod, since every node already
# has one of the old Pods (same labels). The default maxUnavailable of 25% is 0 Pods for 3
# replicas, so the rollout would wait for ever: let it replace one Pod at a time instead
# (maxSurge 0, maxUnavailable 1), in the same change.
""",
"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"tier","operator":"In","values":["edge"]}]}}]}}}}}}'"""),
cmd(r"""k -n %{ns} rollout status deploy %{dep}
k -n %{ns} get pods -o wide""", r"""Three nodes, one Pod each. The anti-affinity's selector names the Pods to keep away from - its own - and `topologyKey` per what: one per hostname, one per node.""")],
},
# ------------------------------------------------------------------ 144 ----
{
@ -787,15 +917,18 @@ eq "$(j -n %{ns} pod %{sts}-0 '{.spec.containers[0].image}') $(j -n %{ns} pod %{
eq "$(for i in 0 1; do j -n %{ns} pod %{sts}-$i '{.metadata.uid}'; echo; done)" "$(cat "$PRIV/q144.uids")" \
&& eq "$(j -n %{ns} sts %{sts} '{.spec.template.spec.containers[0].image}')" nginx:1.27"""),
],
"solution": """
k -n %{ns} patch sts %{sts} -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'
k -n %{ns} set image sts/%{sts} nginx=nginx:1.27
"solution": [hand(r"""k -n %{ns} edit sts %{sts}""",
r"""The partition first - a StatefulSet updates from the highest ordinal down, and without one the new image would reach all three within moments. Under `updateStrategy:`, then `:wq`:
```
updateStrategy:
rollingUpdate:
partition: 2
type: RollingUpdate
```""",
run=r"""kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'"""),
cmd(r"""k -n %{ns} set image sts %{sts} nginx=nginx:1.27
k -n %{ns} rollout status sts %{sts}
k -n %{ns} get pods -l app=%{sts} -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image
# The partition FIRST: a StatefulSet updates from the highest ordinal down, and with no partition
# the new image would reach -1 and -0 within moments. Pods with an ordinal >= partition get the new
# template; the others keep theirs. Next step of the canary: partition 1, then 0.
""",
k -n %{ns} get pods -l app=%{sts} -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image""", r"""Ordinals at or above the partition get the new template: `%{sts}-2` alone. The next steps of the canary: partition 1, then 0.""")],
},
# ------------------------------------------------------------------ 145 ----
{
@ -825,32 +958,53 @@ eq "$(j crd backups.data.example.com '{.status.conditions[?(@.type=="Established
C("Backup %{name} is Scheduled: the controller may update its status", 3, r"""
eq "$(j -n %{ns} backup %{name} '{.status.phase}')" Scheduled && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller"""),
],
"solution": """
grep -E '^kind:|^ name:' {WS}/q145/operator.yaml
k apply -f {WS}/q145/operator.yaml
k wait --for condition=established crd/backups.data.example.com
k -n backup-operator rollout status deploy controller
k create ns %{ns}
k -n %{ns} apply -f - <<'EOF'
"solution": [cmd(r"""grep -e ^kind: -e '^ name:' {WS}/q145/operator.yaml
k apply -f {WS}/q145/operator.yaml"""),
hand(r"""k get crd backups.data.example.com
k -n backup-operator get pods""",
r"""Again until the controller is `Running`.""",
run=r"""kubectl wait --for condition=established crd/backups.data.example.com --timeout=60s
kubectl -n backup-operator rollout status deploy controller --timeout=180s"""),
cmd(r"""k create ns %{ns}"""),
hand(r"""vim backup.yaml""",
r"""```
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: %{name}
namespace: %{ns}
spec:
schedule: "30 1 * * *"
EOF
sleep 10
k -n %{ns} get cm backup-%{name} # made - but:
k -n %{ns} get backup %{name} -o jsonpath='{.status}'; echo # no phase
k -n backup-operator logs deploy/controller | tail -3 # ... cannot patch resource "backups/status" ...
k auth can-i patch backups.data.example.com --subresource=status --as=system:serviceaccount:backup-operator:controller -A
k patch clusterrole backup-operator --type json -p '[{"op":"add","path":"/rules/0/resources/-","value":"backups/status"}]'
sleep 10
k -n %{ns} get backup %{name} -o jsonpath='{.status.phase}'; echo
# The CRD has a status subresource, and a subresource is a resource of its own for RBAC:
# `backups/status`. The controller could read Backups and write ConfigMaps, but not set a phase.
# The log of an operator's controller is where such a gap shows - here as `forbidden`.
""",
```""",
run=r"""cat > backup.yaml <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: %{name}
namespace: %{ns}
spec:
schedule: "30 1 * * *"
EOF"""),
cmd(r"""k apply -f backup.yaml"""),
hand(r"""k -n %{ns} get cm backup-%{name}
k -n %{ns} get backup %{name} -o yaml | grep -A1 status
k -n backup-operator logs deploy/controller | tail -3""",
r"""The ConfigMap is made, but no `phase`: the log says `cannot patch resource "backups/status"`. The CRD has a status subresource, and for RBAC a subresource is a resource of its own.""",
run=r"""sleep 10"""),
cmd(r"""k auth can-i patch backups.data.example.com --subresource=status --as=system:serviceaccount:backup-operator:controller -A""", r"""`no`."""),
hand(r"""k edit clusterrole backup-operator""",
r"""Add `backups/status` to the first rule's resources, then `:wq` - the one thing wrong:
```
- apiGroups:
- data.example.com
resources:
- backups
- backups/status
```""",
run=r"""kubectl patch clusterrole backup-operator --type json -p '[{"op":"add","path":"/rules/0/resources/-","value":"backups/status"}]'"""),
hand(r"""k -n %{ns} get backup %{name} -o yaml | grep -A1 status""",
r"""Again until `phase: Scheduled` - the controller tries every 5 seconds.""",
run=r"""for i in $(seq 1 20); do [ "$(kubectl -n %{ns} get backup %{name} -o jsonpath='{.status.phase}')" = Scheduled ] && break; sleep 3; done""")],
},
# ------------------------------------------------------------------ 146 ----
{
@ -881,41 +1035,66 @@ eq "$(j -n %{ns} pod web-server "{.spec.volumes[?(@.name==\"$v\")].persistentVol
eq "$(j -n %{ns} pvc pv-volume '{.spec.resources.requests.storage} {.status.capacity.storage}')" "%{size} %{size}" || exit 1
eq "$(j -n %{ns} pod web-server '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """),
],
"solution": """
k get sc csi-hostpath-sc # ALLOWVOLUMEEXPANSION: true - a claim of this class may grow
k -n %{ns} apply -f - <<'EOF'
"solution": [cmd(r"""k get sc csi-hostpath-sc""", r"""`ALLOWVOLUMEEXPANSION true`: a claim of this class may grow."""),
hand(r"""vim pvc.yaml""",
r"""```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pv-volume
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
accessModes: [ReadWriteOnce]
accessModes:
- ReadWriteOnce
resources:
requests: {storage: 10Mi}
---
requests:
storage: 10Mi
```""",
run=r"""cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: Pod
kind: PersistentVolumeClaim
metadata:
name: web-server
name: pv-volume
namespace: %{ns}
spec:
containers:
- name: web-server
image: nginx:1.27
storageClassName: csi-hostpath-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Mi
EOF"""),
cmd(r"""k apply -f pvc.yaml
k -n %{ns} run web-server --image=nginx:1.27 --dry-run=client -o yaml > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: web-server`:
```
volumeMounts:
- {name: html, mountPath: /usr/share/nginx/html}
- name: html
mountPath: /usr/share/nginx/html
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: html
persistentVolumeClaim: {claimName: pv-volume}
EOF
k -n %{ns} wait --for=condition=Ready pod/web-server
k -n %{ns} patch pvc pv-volume -p '{"spec":{"resources":{"requests":{"storage":"%{size}"}}}}'
k -n %{ns} get pvc pv-volume # CAPACITY follows once the driver has resized it
# `k edit pvc pv-volume` works just as well. A claim only grows, and only when its class has
# allowVolumeExpansion: true and its driver can expand; spec.resources.requests is what you ask
# for, status.capacity what you have. Filesystem volumes of this driver grow online - the Pod
# keeps running.
""",
persistentVolumeClaim:
claimName: pv-volume
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "html", "mountPath": "/usr/share/nginx/html"}]
| .spec.volumes = [{"name": "html", "persistentVolumeClaim": {"claimName": "pv-volume"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod web-server""",
r"""Again until it is `Running`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/web-server --timeout=180s"""),
hand(r"""k -n %{ns} edit pvc pv-volume""",
r"""Under `spec.resources.requests`, then `:wq` - a claim only grows:
```
storage: %{size}
```""",
run=r"""kubectl -n %{ns} patch pvc pv-volume -p '{"spec":{"resources":{"requests":{"storage":"%{size}"}}}}'"""),
cmd(r"""k -n %{ns} get pvc pv-volume
k -n %{ns} get pod web-server""", r"""Again until `CAPACITY %{size}`: `spec.resources.requests` is what you asked for, `status.capacity` what the driver has done. This driver grows filesystems online - the Pod keeps running.""")],
},
# ------------------------------------------------------------------ 147 ----
{
@ -963,71 +1142,132 @@ x() { $K -n %{ns} exec auditor -- cat "$1/ledger.txt" 2>/dev/null; }
eq "$(x /restore)" entries-%{n} && eq "$(x /copy)" entries-%{n} \
&& eq "$(j -n %{ns} pod auditor '{.spec.volumes[*].persistentVolumeClaim.claimName}' | tr ' ' '\n' | sort | tr '\n' ' ')" "ledger-copy ledger-restore " """),
],
"solution": """
k get volumesnapshotclass # csi-hostpath-snapclass, driver hostpath.csi.k8s.io
k -n %{ns} apply -f - <<'EOF'
"solution": [cmd(r"""k get volumesnapshotclass""", r"""`csi-hostpath-snapclass`, driver `hostpath.csi.k8s.io`."""),
hand(r"""vim snap.yaml""",
r"""The docs page *Volume Snapshots* has the VolumeSnapshot:
```
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: ledger-snap
namespace: %{ns}
spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: ledger-data
EOF
k -n %{ns} wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/ledger-snap --timeout=120s
k -n %{ns} apply -f - <<'EOF'
```""",
run=r"""cat > snap.yaml <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: ledger-snap
namespace: %{ns}
spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: ledger-data
EOF"""),
cmd(r"""k apply -f snap.yaml"""),
hand(r"""k -n %{ns} get volumesnapshot ledger-snap""",
r"""Again until `READYTOUSE true`.""",
run=r"""kubectl -n %{ns} wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/ledger-snap --timeout=120s"""),
hand(r"""vim claims.yaml""",
r"""`dataSource` makes a claim from a snapshot (*Volume Snapshots*) or from another claim of the same class and namespace - a clone (*CSI Volume Cloning*; no `apiGroup`, claims are core). Both at least as big as the source:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger-restore
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: ledger-snap
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger-copy
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
dataSource:
kind: PersistentVolumeClaim
name: ledger-data
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
```""",
run=r"""cat > claims.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger-restore
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: ledger-snap
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
---
apiVersion: v1
kind: Pod
kind: PersistentVolumeClaim
metadata:
name: auditor
name: ledger-copy
namespace: %{ns}
spec:
containers:
- name: auditor
image: busybox:1.36
command: [sleep, '86400']
storageClassName: csi-hostpath-sc
dataSource:
kind: PersistentVolumeClaim
name: ledger-data
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
EOF"""),
cmd(r"""k apply -f claims.yaml
k -n %{ns} run auditor --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: auditor`:
```
volumeMounts:
- {name: restore, mountPath: /restore}
- {name: copy, mountPath: /copy}
- name: restore
mountPath: /restore
- name: copy
mountPath: /copy
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: restore
persistentVolumeClaim: {claimName: ledger-restore}
persistentVolumeClaim:
claimName: ledger-restore
- name: copy
persistentVolumeClaim: {claimName: ledger-copy}
EOF
k -n %{ns} wait --for=condition=Ready pod/auditor --timeout=180s
k -n %{ns} exec auditor -- cat /restore/ledger.txt /copy/ledger.txt
# A snapshot is taken by the CSI driver (VolumeSnapshotClass -> driver); `dataSource` makes a new
# claim from it - apiGroup snapshot.storage.k8s.io - or from another claim of the same class and
# Namespace, a clone (no apiGroup: claims are core). Both need a CSI driver that can do it, and
# the new claim must be at least as big as the source.
""",
persistentVolumeClaim:
claimName: ledger-copy
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "restore", "mountPath": "/restore"}, {"name": "copy", "mountPath": "/copy"}]
| .spec.volumes = [{"name": "restore", "persistentVolumeClaim": {"claimName": "ledger-restore"}}, {"name": "copy", "persistentVolumeClaim": {"claimName": "ledger-copy"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pvc,pod""",
r"""Again until both claims are `Bound` and `auditor` `Running`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/auditor --timeout=180s"""),
cmd(r"""k -n %{ns} exec auditor -- cat /restore/ledger.txt /copy/ledger.txt""", r"""`entries-%{n}`, twice.""")],
},
]

4
cka/bank_hard.py
View File

@ -186,7 +186,9 @@ $K delete ns %{ns} --ignore-not-found --wait=false >/dev/null
},
{
"id": 49, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults",
"params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])},
# low, short: the plural's stem and the shortName, which the solution spells out
"params": lambda r: (lambda p: dict(p, low=p["kind"].lower(), short=p["kind"][:2].lower()))(
{"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])}),
"task": CTX + """
Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the kind in lowercase plus
`s`, shortName = the first two letters of the kind in lowercase) with schema:

151
cka/bank_net.py
View File

@ -5,6 +5,8 @@ Written for the per-task hosts: a task may rewrite the cluster's CoreDNS configu
resolution and hostPorts are tested for real; Ingress and Gateway objects are graded as objects,
as no controller for them runs on the hosts.
"""
from sim.bank import cmd, hand
from cka.common import C, DNS, NS, ONE, PODS, PY, ns_pick, name_pick
QUESTIONS = [
@ -153,12 +155,15 @@ parents == [("%{gw}", "%{ns}")] and s.get("hostnames") == ["%{host}.example.com"
out=$($K -n %{ns} get ingress %{host} 2>&1); case "$out" in *NotFound*) ;; *) exit 1 ;; esac
eq "$(j -n %{ns} secret %{host}-tls '{.metadata.uid}')" "$(cat "$PRIV/q126.uid")" """),
],
"solution": """
k -n %{ns} get ingress %{host} -o yaml # host, the TLS Secret (%{host}-tls), /api -> api:8080, / -> web:80
cat <<EOF | k apply -f -
"solution": [cmd(r"""k -n %{ns} get ingress %{host} -o yaml""", r"""The host, the TLS Secret `%{host}-tls`, `/api` to `api:8080` and `/` to `web:80`."""),
hand(r"""vim gw.yaml""",
r"""What the Ingress keeps in one object, the Gateway API splits: the listener - port, hostname, certificate - goes in the Gateway, the routing in the HTTPRoute. The docs page *Gateway API* has both; `k explain gateway.spec.listeners.tls` the TLS fields:
```
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: %{gw}, namespace: %{ns}}
metadata:
name: %{gw}
namespace: %{ns}
spec:
gatewayClassName: %{gc}
listeners:
@ -168,28 +173,84 @@ spec:
hostname: %{host}.example.com
tls:
mode: Terminate
certificateRefs: [{kind: Secret, name: %{host}-tls}]
certificateRefs:
- kind: Secret
name: %{host}-tls
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: %{host}, namespace: %{ns}}
metadata:
name: %{host}
namespace: %{ns}
spec:
parentRefs: [{name: %{gw}}]
hostnames: [%{host}.example.com]
parentRefs:
- name: %{gw}
hostnames:
- %{host}.example.com
rules:
- matches: [{path: {type: PathPrefix, value: /api}}]
backendRefs: [{name: api, port: 8080}]
- matches: [{path: {type: PathPrefix, value: /}}]
backendRefs: [{name: web, port: 80}]
EOF
- matches:
- path:
type: PathPrefix
value: /api
backendRefs:
- name: api
port: 8080
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: web
port: 80
```""",
run=r"""cat > gw.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: %{gw}
namespace: %{ns}
spec:
gatewayClassName: %{gc}
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: %{host}.example.com
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: %{host}-tls
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: %{host}
namespace: %{ns}
spec:
parentRefs:
- name: %{gw}
hostnames:
- %{host}.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /api
backendRefs:
- name: api
port: 8080
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: web
port: 80
EOF"""),
cmd(r"""k apply -f gw.yaml
k -n %{ns} get gateway,httproute
k -n %{ns} delete ingress %{host}
# What an Ingress keeps in one object, the Gateway API splits by role: the Gateway is the
# listener - port, hostname, certificate - and belongs to whoever runs the edge; the HTTPRoute is
# the routing and belongs to the application. spec.tls of the Ingress becomes tls.certificateRefs
# on the listener, each path becomes a rule with a PathPrefix match, and a Service backend becomes
# a backendRef. `k explain gateway.spec.listeners.tls` and the Gateway API docs are allowed.
""",
k -n %{ns} delete ingress %{host}""")],
},
# ------------------------------------------------------------------ 127 ----
{
@ -399,23 +460,37 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.hostNetwork}')" true || exit 1
pip=$(j -n %{ns} pod %{pod} '{.status.podIP}'); hip=$(j -n %{ns} pod %{pod} '{.status.hostIP}')
[ -n "$pip" ] && eq "$pip" "$hip" && eq "$(tr -s ' \t\r\n' ' ' < "$WS/q128/ips.txt" 2>/dev/null | sed 's/^ //; s/ $//')" "$pip $hip" """),
],
"solution": """
k -n %{ns} edit ds %{ds} # under the container:
# ports: [{containerPort: 8080, hostPort: %{hp}}]
k -n %{ns} rollout status ds %{ds}
k get nodes -o wide # INTERNAL-IP
curl -s http://<a worker's INTERNAL-IP>:%{hp}/ # agent on <that node>
k -n %{ns} logs %{pod} --tail=2 # bad address %{svc}
k -n %{ns} get pod %{pod} -o jsonpath='{.spec.dnsPolicy}'; echo # ClusterFirst - which a hostNetwork Pod does not get
vi {WS}/q128/collector.yaml # spec.dnsPolicy: ClusterFirstWithHostNet
k -n %{ns} delete pod %{pod}; k apply -f {WS}/q128/collector.yaml
k -n %{ns} logs %{pod} --tail=2 # resolved %{svc}
k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt
# hostPort maps one port of the node to one Pod on it (the CNI's portmap plugin writes the rule);
# hostNetwork puts the whole Pod into the node's network namespace - its IP IS the node's IP, and
# it uses the node's /etc/resolv.conf, which knows nothing of cluster.local, unless dnsPolicy is
# ClusterFirstWithHostNet. dnsPolicy cannot be changed on a running Pod: recreate it.
""",
"solution": [hand(r"""k -n %{ns} edit ds %{ds}""",
r"""Give the container's port a `hostPort`, then `:wq`:
```
ports:
- containerPort: 8080
hostPort: %{hp}
protocol: TCP
```
`hostPort` maps one port of the node to the Pod on it (the CNI's `portmap` plugin writes the rule); `hostNetwork` would put the whole Pod in the node's network.""",
run=r"""kubectl -n %{ns} patch ds %{ds} --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/ports/0/hostPort", "value": %{hp}}]'"""),
cmd(r"""k -n %{ns} rollout status ds %{ds}
k get nodes -o wide""", r"""Note a worker's `INTERNAL-IP`."""),
hand(r"""curl -s http://172.18.0.3:%{hp}/""",
r"""With that IP: `agent on <that node>`.""",
run=r""""""),
cmd(r"""k -n %{ns} logs %{pod} --tail=2
k -n %{ns} get pod %{pod} -o yaml | grep -e hostNetwork -e dnsPolicy""", r"""`bad address`: a `hostNetwork` Pod uses the node's `/etc/resolv.conf`, which knows nothing of `cluster.local` - unless its `dnsPolicy` is `ClusterFirstWithHostNet`."""),
hand(r"""vim {WS}/q128/collector.yaml""",
r"""In the Pod's `spec:`, below `hostNetwork: true`:
```
dnsPolicy: ClusterFirstWithHostNet
```""",
run=r"""yq -i '.spec.dnsPolicy = "ClusterFirstWithHostNet"' {WS}/q128/collector.yaml"""),
cmd(r"""k replace --force -f {WS}/q128/collector.yaml""", r"""A running Pod's `dnsPolicy` cannot change: `--force` deletes it and creates it again."""),
hand(r"""k -n %{ns} logs %{pod} --tail=2
k -n %{ns} get pod %{pod} -o wide""",
r"""Again until `resolved %{svc}`. Its IP is its node's: note it.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=180s
sleep 8"""),
hand(r"""echo "172.18.0.3 172.18.0.3" > {WS}/q128/ips.txt""",
r"""The Pod's IP and its node's - the same address.""",
run=r"""kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt""")],
},
]

220
cka/bank_objectives.py
View File

@ -20,7 +20,7 @@ kubeadm, etcd, the runtime. All normal difficulty, all in the random pool; nothi
node or a control-plane component (109 cordons the control-plane node while you work on it and has
you uncordon it; the session baseline uncordons it if you do not).
"""
from sim.bank import C, NS, ns_pick, name_pick
from sim.bank import C, NS, ns_pick, name_pick, cmd, hand
from ckad import bank_extra as _ckad_extra, bank_more as _ckad_more
@ -30,18 +30,34 @@ def _port(bank, src_id, new_id, domain, **over):
old, new = f"q{src_id}", f"q{new_id}"
q = dict(src, id=new_id, domain=domain, **over)
for k in ("task", "setup", "solution"):
if k not in over: # a solution of its own (in parts) is the CKA one already
q[k] = src[k].replace(old, new)
q["checks"] = [dict(c, script=c["script"].replace(old, new), name=c["name"].replace(old, new)) for c in src["checks"]]
return q
# 103's own solution: the CKAD task's, done as in the exam
_SOL103 = [hand(r"""k -n %{ns} top pod""",
r"""Again until it shows all three: metrics-server needs about 30 seconds after the Pods start. Without it, `Metrics API not available`.""",
run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} top pod --no-headers 2>/dev/null | wc -l)" -ge 3 ] && break; sleep 5; done"""),
cmd(r"""k -n %{ns} top pod --sort-by=cpu
k -n %{ns} top pod --sort-by=memory""", r"""The first line of each is the heaviest."""),
hand(r"""echo %{cpu} > {WS}/q103/cpu.txt
echo %{mem} > {WS}/q103/memory.txt""",
r"""With the names on those first lines.""",
run=r"""kubectl -n %{ns} top pod --sort-by=cpu --no-headers | head -1 | awk '{print $1}' > {WS}/q103/cpu.txt
kubectl -n %{ns} top pod --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/q103/memory.txt"""),
cmd(r"""echo 'kubectl top node' > {WS}/q103/nodes.sh
echo 'kubectl top pod -A --sort-by=memory' > {WS}/q103/pods.sh""", r"""`k top` reads the metrics API (`metrics.k8s.io`) that metrics-server serves; `--containers` breaks a Pod down per container.""")]
ETCDCTL = ("etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
"--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key")
QUESTIONS = [
# ---------------------------------------------------------------- TSHOOT ----
# needs a working metrics API, which Q23 breaks on purpose: never in one session with it
_port(_ckad_extra, 23, 103, "TSHOOT", breaks="metrics-server"),
_port(_ckad_extra, 23, 103, "TSHOOT", breaks="metrics-server", solution=_SOL103),
_port(_ckad_more, 64, 104, "TSHOOT"),
# ------------------------------------------------------------------ WORK ----
{
@ -183,32 +199,44 @@ cg=$(nx "$W3" "awk '/^cgroupDriver:/{print \$2}' /var/lib/kubelet/config.yaml" <
want=$(printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s' "$fwd" "$swap" "$fso" "$cg")
eq "$(tr -d ' \t\r' < "$WS/q107/node.txt" 2>/dev/null | grep .)" "$want" """),
],
"solution": """
{SHELL_W3}
"solution": [hand(r"""{SHELL_W3}
sudo -i
printf 'overlay\\nbr_netfilter\\n' > /etc/modules-load.d/%{name}.conf
cat > /etc/sysctl.d/%{name}.conf <<EOF
vim /etc/modules-load.d/%{name}.conf""",
r"""The docs page *Container Runtimes* has both files under *Install and configure prerequisites*. One module per line:
```
overlay
br_netfilter
```""",
run=r"""ssh -n {W3} 'printf "overlay\nbr_netfilter\n" | sudo tee /etc/modules-load.d/%{name}.conf >/dev/null'"""),
hand(r"""vim /etc/sysctl.d/%{name}.conf""",
r"""```
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# on a real machine you now load and apply them without a reboot:
# modprobe overlay && modprobe br_netfilter && sysctl --system
sysctl -n net.ipv4.ip_forward # 1
cat /proc/swaps # any line under the header = swap is on (swapon --show)
grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml
exit; exit
cat > {WS}/q107/node.txt <<EOF
ip_forward=1
swap=<on|off>
fail-swap-on=<true|false>
cgroup-driver=<systemd|cgroupfs>
EOF
# `sysctl -w` and `modprobe` alone are gone after a reboot - the files are what makes them stick.
# kubeadm's preflight refuses a node with swap on unless the kubelet is told to tolerate it
# (failSwapOn: false), and the kubelet's cgroup driver has to match the container runtime's
# (SystemdCgroup = true in /etc/containerd/config.toml goes with cgroupDriver: systemd).
""",
```
On a real machine they would now be loaded and applied without a reboot: `modprobe overlay`, `modprobe br_netfilter`, `sysctl --system`. `sysctl -w` and `modprobe` alone are gone after a reboot; the files make them stick.""",
run=r"""ssh -n {W3} 'printf "net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\n" | sudo tee /etc/sysctl.d/%{name}.conf >/dev/null'"""),
hand(r"""sysctl -n net.ipv4.ip_forward
cat /proc/swaps
grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml
exit
exit""",
r"""Any line below the header of `/proc/swaps` means swap is on. No `failSwapOn` line means the default, `true`.""",
run=r""""""),
hand(r"""vim {WS}/q107/node.txt""",
r"""What the node printed:
```
ip_forward=<0 or 1>
swap=<on or off>
fail-swap-on=<true or false>
cgroup-driver=<systemd or cgroupfs>
```
kubeadm's preflight refuses a node with swap on unless the kubelet tolerates it (`failSwapOn: false`); and the kubelet's cgroup driver has to match the runtime's (`SystemdCgroup = true` in containerd goes with `cgroupDriver: systemd`).""",
run=r"""IPF=$(ssh -n {W3} 'sudo sysctl -n net.ipv4.ip_forward')
SW=off; [ "$(ssh -n {W3} 'sed 1d /proc/swaps | grep -c .')" -gt 0 ] && SW=on
FS=$(ssh -n {W3} "sudo awk '/^failSwapOn:/ {print \$2}' /var/lib/kubelet/config.yaml"); [ -n "$FS" ] || FS=true
CG=$(ssh -n {W3} "sudo awk '/^cgroupDriver:/ {print \$2}' /var/lib/kubelet/config.yaml")
printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s\n' "$IPF" "$SW" "$FS" "$CG" > {WS}/q107/node.txt""")],
},
{
"id": 108, "domain": "ARCH", "title": "The container runtime behind a Pod",
@ -263,29 +291,35 @@ cid=$($K -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].cont
log=$(nx "$W2" "crictl inspect -o go-template --template '{{.status.logPath}}' $cid 2>/dev/null" </dev/null | tr -d '[:space:]')
[ -n "$cid" ] && [ -n "$log" ] && tr -d ' \t\r' < "$WS/q108/runtime.txt" 2>/dev/null | grep -qx "log=$log" """),
],
"solution": """
k -n %{ns} get pod %{pod} -o wide # on {W2}
{SHELL_W2}
"solution": [cmd(r"""k -n %{ns} get pod %{pod} -o wide""", r"""On `{W2}`."""),
hand(r"""{SHELL_W2}
sudo -i
crictl version # RuntimeName: containerd RuntimeVersion: v2.x.y
grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml # unix:///run/containerd/containerd.sock
crictl pods --name %{pod} # the Pod sandbox
crictl ps --no-trunc --name app --pod $(crictl pods --name %{pod} -q) # full container ID
crictl inspect <id> | grep -E '"pid"|logPath' # .info.pid and .status.logPath
# or: crictl inspect -o go-template --template '{{.info.pid}} {{.status.logPath}}' <id>
exit; exit
cat > {WS}/q108/runtime.txt <<EOF
crictl version
grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
crictl ps --no-trunc --name app""",
r"""`RuntimeName` and `RuntimeVersion`; the CRI socket the kubelet uses - crictl speaks the same protocol, which is why it works even when the API server does not; and the container's full ID (`--no-trunc`), the line whose `POD` is `%{pod}`.""",
run=r""""""),
hand(r"""crictl inspect 0123abcd... | grep -e '"pid"' -e logPath
exit
exit""",
r"""With that ID: `.info.pid` and `.status.logPath`. The log is the file `kubectl logs` reads, under `/var/log/pods`.""",
run=r""""""),
hand(r"""vim {WS}/q108/runtime.txt""",
r"""The values you read:
```
runtime=containerd <version>
socket=unix:///run/containerd/containerd.sock
container-id=<64 hex characters>
pid=<number>
container-id=<the 64 hex characters>
pid=<the number>
log=/var/log/pods/%{ns}_%{pod}_<pod uid>/app/0.log
EOF
# The kubelet drives the runtime over the CRI socket, and crictl speaks the same protocol - which
# is why it still works when the API server does not. `kubectl get pod -o
# jsonpath='{.status.containerStatuses[0].containerID}'` shows the same ID with a containerd://
# prefix; /var/log/pods/... is the file `kubectl logs` reads.
""",
```""",
run=r"""V=$(ssh -n {W2} 'sudo crictl version 2>/dev/null')
RN=$(echo "$V" | awk '/^RuntimeName:/ {print $2}'); RV=$(echo "$V" | awk '/^RuntimeVersion:/ {print $2}')
SOCK=$(ssh -n {W2} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
CID=$(kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#^[a-z-]*://##')
PID=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.info.pid}}' $CID 2>/dev/null")
LOG=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.status.logPath}}' $CID 2>/dev/null")
printf 'runtime=%s %s\nsocket=%s\ncontainer-id=%s\npid=%s\nlog=%s\n' "$RN" "$RV" "$SOCK" "$CID" "$PID" "$LOG" > {WS}/q108/runtime.txt""")],
},
{
# drains the control-plane node: CoreDNS moves off it, so never beside the tasks that break DNS
@ -468,29 +502,33 @@ c=$(j -n kube-system lease kube-controller-manager '{.spec.holderIdentity}' | cu
f=$(tr -d ' \t\r' < "$WS/q111/ha.txt" 2>/dev/null)
[ -n "$s" ] && [ -n "$c" ] && echo "$f" | grep -qx "scheduler-leader=$s" && echo "$f" | grep -qx "controller-manager-leader=$c" """),
],
"solution": """
k get nodes -l node-role.kubernetes.io/control-plane # how many
k -n kube-system get pods -l component=etcd -o wide # an etcd static Pod on the control-plane node = stacked
k -n kube-system get lease kube-scheduler kube-controller-manager # HOLDER is <node>_<uuid>
sudo -i # the task's host is the control-plane node: no hop needed
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \\
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl member list -w table # one line per member: ID, NAME, peer and client URLs
etcdctl endpoint status -w table # IS LEADER (add --cluster to ask every member)
exit
cat > {WS}/q111/ha.txt <<EOF
"solution": [cmd(r"""k get nodes -l node-role.kubernetes.io/control-plane
k -n kube-system get pods -l component=etcd -o wide
k -n kube-system get lease kube-scheduler kube-controller-manager""", r"""An etcd static Pod on the control-plane node: stacked. A Lease's `HOLDER` is `<node>_<uuid>`."""),
hand(r"""sudo -i
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl member list -w table
etcdctl endpoint status -w table
exit""",
r"""The host is the control-plane node. One line per member; `IS LEADER` in the status (`--cluster` asks every member).""",
run=r""""""),
hand(r"""vim {WS}/q111/ha.txt""",
r"""```
control-plane-nodes=1
etcd-topology=stacked
etcd-members=1
etcd-leader={CP}
scheduler-leader={CP}
controller-manager-leader={CP}
EOF
# In an HA control plane every kube-apiserver serves at once, but only one kube-scheduler and one
# kube-controller-manager act: they elect a leader through a Lease in kube-system, and the others
# wait. etcd elects its own leader and needs a majority of members to stay writable - three
# members survive the loss of one, five of two, and an even number adds nothing.
""",
```
In an HA control plane every API server serves at once, but one scheduler and one controller manager act, through the leader Lease. etcd elects its own leader and needs a majority to stay writable: three members survive the loss of one, five of two.""",
run=r"""E='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
N=$(kubectl get nodes -l node-role.kubernetes.io/control-plane --no-headers | wc -l)
M=$($E member list | wc -l)
L=$($E member list | awk -F', ' 'NR==1 {print $3}')
S=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
C=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
printf 'control-plane-nodes=%s\netcd-topology=stacked\netcd-members=%s\netcd-leader=%s\nscheduler-leader=%s\ncontroller-manager-leader=%s\n' "$N" "$M" "$L" "$S" "$C" > {WS}/q111/ha.txt""")],
},
{
"id": 112, "domain": "ARCH", "title": "CNI, CSI and CRI: what this cluster plugs in",
@ -532,30 +570,35 @@ prov=$($K get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclas
f=$(tr -d ' \t\r' < "$WS/q112/interfaces.txt" 2>/dev/null)
echo "$f" | grep -qx "csi-drivers=$n" && echo "$f" | grep -qx "default-provisioner=$prov" """),
],
"solution": """
{SHELL_W1}
"solution": [hand(r"""{SHELL_W1}
sudo -i
ls /etc/cni/net.d/ # 10-kindnet.conflist - the kubelet's runtime takes the first file
cat /etc/cni/net.d/10-kindnet.conflist # "plugins": [{"type": "ptp", "ipam": {"type": "host-local"}}, {"type": "portmap"}]
ls /opt/cni/bin # the plugin binaries those types name
ls /etc/cni/net.d
cat /etc/cni/net.d/10-kindnet.conflist
grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
exit; exit
k get csidrivers # No resources found = 0
k get sc # the (default) class and its PROVISIONER
cat > {WS}/q112/interfaces.txt <<EOF
exit
exit""",
r"""The runtime takes the first file of `/etc/cni/net.d`. Its `plugins` chain, in order - each `type` names a binary in `/opt/cni/bin` - and the `ipam` type inside one of them.""",
run=r""""""),
cmd(r"""k get csidrivers
k get sc""", r"""`No resources found`: no CSI driver - the local-path provisioner is not one. The `(default)` class's `PROVISIONER`."""),
hand(r"""vim {WS}/q112/interfaces.txt""",
r"""The values you read - here:
```
cni-config=/etc/cni/net.d/10-kindnet.conflist
cni-plugins=ptp,portmap
cni-ipam=host-local
cri-socket=unix:///run/containerd/containerd.sock
csi-drivers=0
default-provisioner=rancher.io/local-path
EOF
# CRI: the kubelet talks to the runtime over that socket (crictl uses the same one). CNI: the
# runtime calls the plugins of the chain in order for every Pod sandbox - here ptp wires the veth
# pair, host-local hands out an address from the node's Pod CIDR, portmap serves hostPorts.
# CSI: a storage driver registers a CSIDriver object and a StorageClass names it as provisioner;
# this cluster's local-path provisioner is not a CSI driver, hence none.
""",
```
`ptp` wires each Pod's veth pair, `host-local` hands out addresses from the node's Pod CIDR, `portmap` serves hostPorts.""",
run=r"""CONF=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
PL=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[].type] | join(",")')
IPAM=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[] | select(.ipam) | .ipam.type][0]')
SOCK=$(ssh -n {W1} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
N=$(kubectl get csidrivers --no-headers 2>/dev/null | wc -l)
PROV=$(kubectl get sc -o json | jq -r '.items[] | select(.metadata.annotations["storageclass.kubernetes.io/is-default-class"] == "true") | .provisioner')
printf 'cni-config=%s\ncni-plugins=%s\ncni-ipam=%s\ncri-socket=%s\ncsi-drivers=%s\ndefault-provisioner=%s\n' "$CONF" "$PL" "$IPAM" "$SOCK" "$N" "$PROV" > {WS}/q112/interfaces.txt""")],
},
# ------------------------------------------------------------------- NET ----
{
@ -623,19 +666,18 @@ eq "$(tr -s ' \t' ' ' < "$WS/q113/pod-cidrs.txt" 2>/dev/null | tr -d '\r' | sed
C("outside.txt names %{edge}, why.txt names hostNetwork", 2, r"""
eq "$(tr -d '[:space:]' < "$WS/q113/outside.txt" 2>/dev/null | sed 's#^pod/##')" %{edge} && grep -qi 'hostNetwork' "$WS/q113/why.txt" 2>/dev/null"""),
],
"solution": """
k -n %{ns} get pods -o wide # IP and NODE of every Pod
k -n %{ns} get pod %{web} -o jsonpath='{.status.podIP} {.spec.nodeName}' > {WS}/q113/web.txt
k -n %{ns} exec %{client} -- wget -qO- -T3 http://$(cut -d' ' -f1 {WS}/q113/web.txt) > {WS}/q113/direct.html
k get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\\n"}{end}' > {WS}/q113/pod-cidrs.txt
# %{edge}'s IP is its node's own address (compare with `k get nodes -o wide`), not one from 10.244.x.0/24
echo %{edge} > {WS}/q113/outside.txt
"solution": [cmd(r"""k -n %{ns} get pods -o wide
k get nodes -o wide""", r"""The `IP` and `NODE` of every Pod. `%{edge}`'s IP is its node's own `INTERNAL-IP`, not one from a Pod CIDR."""),
hand(r"""echo "10.244.1.5 {W1}" > {WS}/q113/web.txt""",
r"""With `%{web}`'s IP and node.""",
run=r"""kubectl -n %{ns} get pod %{web} -o jsonpath='{.status.podIP} {.spec.nodeName}' > {WS}/q113/web.txt"""),
hand(r"""k -n %{ns} exec %{client} -- wget -qO- -T3 http://10.244.1.5 > {WS}/q113/direct.html""",
r"""With that IP: Pod to Pod, across nodes, no Service and no NAT - the network model the CNI makes true.""",
run=r"""kubectl -n %{ns} exec %{client} -- wget -qO- -T3 http://$(kubectl -n %{ns} get pod %{web} -o jsonpath='{.status.podIP}') > {WS}/q113/direct.html"""),
cmd(r"""k get nodes -o custom-columns=NAME:.metadata.name,CIDR:.spec.podCIDR --no-headers > {WS}/q113/pod-cidrs.txt
cat {WS}/q113/pod-cidrs.txt
k -n %{ns} get pod %{edge} -o yaml | grep hostNetwork
echo hostNetwork > {WS}/q113/why.txt
# Every Pod gets its own address from its node's Pod CIDR, and any Pod reaches any other Pod on
# any node at that address without NAT - that is the whole network model, and the CNI plugin is
# what makes it true. A hostNetwork Pod opts out: it lives in the node's network namespace and
# shares its address and ports. Pod addresses change with every new Pod, which is why Services exist.
""",
echo %{edge} > {WS}/q113/outside.txt
echo hostNetwork > {WS}/q113/why.txt""", r"""A `hostNetwork` Pod lives in its node's network namespace and shares the node's address and ports.""")],
},
]

641
cka/bank_sim_c.py
View File

@ -12,7 +12,7 @@ Two scenarios are deliberately adapted so they cannot wreck the practice cluster
Run with: ./ksim sim c
"""
from sim.bank import C, NS
from sim.bank import C, NS, cmd, hand
Q = []
@ -62,42 +62,87 @@ eq "$(j -n geo-fence pod geofence-reader '{.status.phase}')" Running \
&& eq "$(j -n geo-fence pod geofence-reader '{.spec.volumes[0].persistentVolumeClaim.claimName}')" geofence-pvc \
&& j -n geo-fence pod geofence-reader '{.spec.containers[0].volumeMounts[*].mountPath}' | grep -q /geo-data"""),
],
solution="""
cat <<EOF | k apply -f -
solution=[hand(r"""vim pv.yaml""",
r"""The PersistentVolume and PersistentVolumeClaim from the docs page *Configure a Pod to Use a PersistentVolume for Storage*, `---` between them. `volumeName` pins the claim to this PV: without it the binder takes any PV whose size, access mode and class fit.
```
apiVersion: v1
kind: PersistentVolume
metadata: {name: geofence-pv}
metadata:
name: geofence-pv
spec:
capacity: {storage: 250Mi}
accessModes: [ReadWriteOnce]
hostPath: {path: /data/geofence}
storageClassName: manual
capacity:
storage: 250Mi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/geofence
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: geofence-pvc, namespace: geo-fence}
metadata:
name: geofence-pvc
namespace: geo-fence
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 250Mi}}
storageClassName: manual
volumeName: geofence-pv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 250Mi
```""",
run=r"""cat > pv.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: geofence-pv
spec:
storageClassName: manual
capacity:
storage: 250Mi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/geofence
---
apiVersion: v1
kind: Pod
metadata: {name: geofence-reader, namespace: geo-fence}
kind: PersistentVolumeClaim
metadata:
name: geofence-pvc
namespace: geo-fence
spec:
volumes: [{name: geo, persistentVolumeClaim: {claimName: geofence-pvc}}]
containers:
- name: reader
image: busybox:1.36
command: [sh, -c, "ls /geo-data && sleep 3600"]
volumeMounts: [{name: geo, mountPath: /geo-data}]
EOF
storageClassName: manual
volumeName: geofence-pv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 250Mi
EOF"""),
cmd(r"""k apply -f pv.yaml
k -n geo-fence get pvc
# volumeName pins the claim to one PV by name. Without it the binder picks ANY PV whose capacity,
# accessModes and storageClassName match - which on a busy cluster is rarely the one you meant.
""",
k -n geo-fence run geofence-reader --image=busybox:1.36 --dry-run=client -o yaml --command -- sh -c "ls /geo-data && sleep 3600" > pod.yaml""", r"""`Bound` to `geofence-pv`."""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: geofence-reader`:
```
volumeMounts:
- name: geo
mountPath: /geo-data
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: geo
persistentVolumeClaim:
claimName: geofence-pvc
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "geo", "mountPath": "/geo-data"}]
| .spec.volumes = [{"name": "geo", "persistentVolumeClaim": {"claimName": "geofence-pvc"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml
k -n geo-fence get pod geofence-reader""")],
)
# ------------------------------------------------------------------ 69 ----
@ -260,15 +305,19 @@ want=$(j -n dispatch-hub svc dispatch-service '{.spec.clusterIP}')
got=$(tr -d '[:space:]' < "$WS/q71/service-ip.txt" 2>/dev/null)
[ -n "$want" ] && eq "$got" "$want" """),
],
solution="""
k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
k -n dispatch-hub get endpointslices -l kubernetes.io/service-name=dispatch-service # must not be empty
k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- \\
wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt
k -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt
# `expose` copies the Deployment's selector for you, which is what makes the endpoints appear.
# Write the IP with jsonpath, not by eyeballing `get svc` - the grader compares it literally.
""",
solution=[cmd(r"""k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
k -n dispatch-hub get svc,endpointslices""", r"""`expose` copies the Deployment's selector, which is what makes the endpoints appear. Note the Service's `CLUSTER-IP`."""),
hand(r"""k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt
cat {WS}/q71/test-result.txt""",
r"""nginx's welcome page, fetched from a Pod in the cluster.""",
run=r"""kubectl -n dispatch-hub rollout status deploy dispatch-api --timeout=240s
for i in 1 2 3 4 5; do
kubectl -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt </dev/null 2>/dev/null
grep -qi nginx {WS}/q71/test-result.txt && break; kubectl -n dispatch-hub delete pod probe --ignore-not-found >/dev/null 2>&1; sleep 3
done"""),
hand(r"""echo 10.96.123.45 > {WS}/q71/service-ip.txt""",
r"""With the `CLUSTER-IP` that `get svc` showed - the address alone.""",
run=r"""kubectl -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt""")],
)
# ------------------------------------------------------------------ 72 ----
@ -358,54 +407,102 @@ j -n ride-history deploy ride-archive '{.spec.template.spec.volumes[*].persisten
j -n ride-history deploy ride-archive '{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' | grep -qw ride-archive-pvc-new || exit 1
for i in $(seq 1 20); do [ "$(j -n ride-history deploy ride-archive '{.status.availableReplicas}')" -ge 1 ] 2>/dev/null && exit 0; sleep 5; done; exit 1"""),
],
solution="""
k -n ride-history get pvc,pv # note reclaimPolicy Delete on the old PV
cat <<EOF | k apply -f -
solution=[cmd(r"""k -n ride-history get pvc
k get pv ride-archive-pv""", r"""`RECLAIM POLICY Delete` on the old PV: losing the claim loses the data."""),
hand(r"""vim new.yaml""",
r"""The new PV and its claim, pinned to it by name (the docs page *Configure a Pod to Use a PersistentVolume for Storage* has both):
```
apiVersion: v1
kind: PersistentVolume
metadata: {name: ride-archive-pv-new}
metadata:
name: ride-archive-pv-new
spec:
capacity: {storage: 500Mi}
accessModes: [ReadWriteOnce]
hostPath: {path: /data/ride-archive-new}
storageClassName: manual
capacity:
storage: 500Mi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/ride-archive-new
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: ride-archive-pvc-new, namespace: ride-history}
metadata:
name: ride-archive-pvc-new
namespace: ride-history
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 500Mi}}
storageClassName: manual
volumeName: ride-archive-pv-new
EOF
k -n ride-history scale deploy ride-archive --replicas=0 # release the RWO claim first
cat <<EOF | k apply -f -
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
```""",
run=r"""cat > new.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: data-migrator, namespace: ride-history}
kind: PersistentVolume
metadata:
name: ride-archive-pv-new
spec:
restartPolicy: Never
volumes:
- {name: old, persistentVolumeClaim: {claimName: ride-archive-pvc}}
- {name: new, persistentVolumeClaim: {claimName: ride-archive-pvc-new}}
containers:
- name: migrator
image: busybox:1.36
command: [sh, -c, "cp -a /old-data/. /new-data/ && echo migration-complete && sleep 3600"]
storageClassName: manual
capacity:
storage: 500Mi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/ride-archive-new
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ride-archive-pvc-new
namespace: ride-history
spec:
storageClassName: manual
volumeName: ride-archive-pv-new
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
EOF"""),
cmd(r"""k apply -f new.yaml
k -n ride-history scale deploy ride-archive --replicas=0
k -n ride-history run data-migrator --image=busybox:1.36 --restart=Never --dry-run=client -o yaml -- sh -c "cp -a /old-data/. /new-data/ && echo migration-complete && sleep 3600" > m.yaml""", r"""Scale to 0 first: the running Pod holds the old claim. `cp -a /old-data/. /new-data/` copies the contents, dotfiles too, rather than the directory itself."""),
hand(r"""vim m.yaml""",
r"""Both claims: add to the container, below `name: data-migrator`,
```
volumeMounts:
- {name: old, mountPath: /old-data}
- {name: new, mountPath: /new-data}
EOF
k -n ride-history logs data-migrator # migration-complete
k -n ride-history patch deploy ride-archive --type=json \\
-p '[{"op":"replace","path":"/spec/template/spec/volumes/0/persistentVolumeClaim/claimName","value":"ride-archive-pvc-new"}]'
k -n ride-history scale deploy ride-archive --replicas=1
# ReadWriteOnce is per-node, but a second pod on the same node still cannot mount a claim that a
# running pod holds - hence the scale to 0 first. `cp -a /old/. /new/` copies the CONTENTS
# (including dotfiles) rather than nesting the directory inside the target.
""",
- name: old
mountPath: /old-data
- name: new
mountPath: /new-data
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: old
persistentVolumeClaim:
claimName: ride-archive-pvc
- name: new
persistentVolumeClaim:
claimName: ride-archive-pvc-new
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "old", "mountPath": "/old-data"}, {"name": "new", "mountPath": "/new-data"}]
| .spec.volumes = [{"name": "old", "persistentVolumeClaim": {"claimName": "ride-archive-pvc"}}, {"name": "new", "persistentVolumeClaim": {"claimName": "ride-archive-pvc-new"}}]' m.yaml"""),
cmd(r"""k apply -f m.yaml"""),
hand(r"""k -n ride-history logs data-migrator""",
r"""Again until it prints `migration-complete`.""",
run=r"""for i in $(seq 1 60); do kubectl -n ride-history logs data-migrator 2>/dev/null | grep -q migration-complete && break; sleep 2; done"""),
cmd(r"""k -n ride-history delete pod data-migrator --now"""),
hand(r"""k -n ride-history edit deploy ride-archive""",
r"""In the Pod template's `volumes:`, change `claimName: ride-archive-pvc` to `claimName: ride-archive-pvc-new`, then `:wq`.""",
run=r"""kubectl -n ride-history patch deploy ride-archive --type=json -p '[{"op":"replace","path":"/spec/template/spec/volumes/0/persistentVolumeClaim/claimName","value":"ride-archive-pvc-new"}]'"""),
cmd(r"""k -n ride-history scale deploy ride-archive --replicas=1
k -n ride-history get pods""")],
)
# ------------------------------------------------------------------ 73 ----
@ -445,17 +542,21 @@ for i in $(seq 1 12); do
sleep 3
done; exit 1"""),
],
solution="""
k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend # empty
k -n eta-service get deploy eta-calculator -o jsonpath='{.spec.template.metadata.labels}' # app=eta-calculator
k -n eta-service get svc eta-backend -o jsonpath='{.spec.selector}' # app=eta-service <- fault 1
k -n eta-service patch svc eta-backend -p '{"spec":{"selector":{"app":"eta-calculator"}}}'
k -n eta-service patch svc eta-backend -p '{"spec":{"ports":[{"port":80,"targetPort":80,"protocol":"TCP"}]}}'
k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend
# Empty endpoints is almost always the selector; a populated endpoint list that still refuses
# connections is almost always targetPort. Check the endpoints first - it splits the two cases
# in one command.
""",
solution=[cmd(r"""k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend
k -n eta-service get pods --show-labels
k -n eta-service describe svc eta-backend""", r"""No endpoints. The Pods are `app=eta-calculator`, the Service selects `app=eta-service`; and it sends to `TargetPort 9090`, where nginx listens on 80. Empty endpoints is almost always the selector; endpoints that refuse connections, the target port."""),
hand(r"""k -n eta-service edit svc eta-backend""",
r"""Fix both, then `:wq`:
```
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: eta-calculator
```""",
run=r"""kubectl -n eta-service patch svc eta-backend -p '{"spec":{"selector":{"app":"eta-calculator"},"ports":[{"port":80,"targetPort":80,"protocol":"TCP"}]}}'"""),
cmd(r"""k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend""")],
)
# ------------------------------------------------------------------ 74 ----
@ -521,11 +622,14 @@ sys.exit(0 if ok else 1)'"""),
C("the HTTPRoute is attached to fleet-gateway", 1, r"""
j -n fleet-ops httproute fleet-route '{.spec.parentRefs[*].name}' | grep -qw fleet-gateway"""),
],
solution="""
cat <<EOF | k apply -f -
solution=[hand(r"""vim gw.yaml""",
r"""The docs page *Gateway API* has a Gateway and an HTTPRoute to start from:
```
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: fleet-gateway, namespace: fleet-ops}
metadata:
name: fleet-gateway
namespace: fleet-ops
spec:
gatewayClassName: fleet-gateway-class
listeners:
@ -536,21 +640,73 @@ spec:
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: fleet-route, namespace: fleet-ops}
metadata:
name: fleet-route
namespace: fleet-ops
spec:
parentRefs: [{name: fleet-gateway}]
hostnames: [fleet.example.com]
parentRefs:
- name: fleet-gateway
hostnames:
- fleet.example.com
rules:
- matches:
- headers: [{type: Exact, name: X-Fleet-Version, value: canary}]
backendRefs: [{name: fleet-dashboard-v2, port: 80}]
- headers:
- name: X-Fleet-Version
value: canary
backendRefs:
- name: fleet-dashboard-v2
port: 80
- matches:
- path: {type: PathPrefix, value: /}
backendRefs: [{name: fleet-dashboard-v1, port: 80}]
EOF
# Order matters: HTTPRoute picks the most specific match, and a bare PathPrefix "/" matches
# everything - so the header rule must be able to win. Listing it first also keeps it readable.
""",
- path:
type: PathPrefix
value: /
backendRefs:
- name: fleet-dashboard-v1
port: 80
```
The more specific match wins, so the header rule takes canary requests though `/` matches everything.""",
run=r"""cat > gw.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: fleet-gateway
namespace: fleet-ops
spec:
gatewayClassName: fleet-gateway-class
listeners:
- name: http
protocol: HTTP
port: 80
hostname: fleet.example.com
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: fleet-route
namespace: fleet-ops
spec:
parentRefs:
- name: fleet-gateway
hostnames:
- fleet.example.com
rules:
- matches:
- headers:
- name: X-Fleet-Version
value: canary
backendRefs:
- name: fleet-dashboard-v2
port: 80
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: fleet-dashboard-v1
port: 80
EOF"""),
cmd(r"""k apply -f gw.yaml
k -n fleet-ops get gateway,httproute""")],
)
# ------------------------------------------------------------------ 75 ----
@ -714,29 +870,49 @@ eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \
&& $n -o jsonpath='{.spec.policyTypes}' | grep -q Egress \
&& [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ]"""),
],
solution="""
k -n kube-system get ds # kindnet -> {WS}/q77/cni.txt
# Don't answer from memory - prove it. Two pods, one deny policy, one wget:
k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv; k -n zero-trust expose pod srv --port 80
k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600
k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null && echo reachable
cat <<EOF | k apply -f -
solution=[cmd(r"""k -n kube-system get ds
echo kindnet > {WS}/q77/cni.txt""", r"""`kindnet` runs on every node: it is the CNI. `kube-proxy` is not."""),
cmd(r"""k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv
k -n zero-trust expose pod srv --port 80
k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600""", r"""Prove it rather than remember it: two Pods, a policy that denies everything, and a request between them."""),
hand(r"""k -n zero-trust get pods""",
r"""Again until both are `Running`.""",
run=r"""kubectl -n zero-trust wait --for=condition=Ready pod/srv pod/cli --timeout=180s"""),
cmd(r"""k -n zero-trust exec cli -- wget -qO- -T3 http://srv""", r"""nginx's page: no policy yet."""),
hand(r"""vim deny.yaml""",
r"""The default deny for both directions from the docs page *Network Policies*: every Pod, both policy types, no rules.
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-all, namespace: zero-trust}
metadata:
name: default-deny-all
namespace: zero-trust
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
EOF
k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null || echo BLOCKED # -> BLOCKED here
echo yes > {WS}/q77/enforced.txt
k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv # leave only the policy
# podSelector {} means every pod in the namespace; naming both policyTypes with no ingress/egress
# blocks listed means "deny everything". Recent kindnet builds (this cluster's included) embed a
# policy engine - look for `nft list table inet kindnet-network-policies` on a node. Flannel stores
# NetworkPolicies and enforces nothing; Calico and Cilium enforce. A policy that "works" on a
# non-enforcing CNI is the classic hardening-review trap.
""",
policyTypes:
- Ingress
- Egress
```""",
run=r"""cat > deny.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: zero-trust
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
EOF"""),
cmd(r"""k apply -f deny.yaml
k -n zero-trust exec cli -- wget -qO- -T3 http://srv""", r"""`download timed out`: the CNI enforces it. (Flannel would store the policy and enforce nothing; Calico and Cilium enforce. Recent kindnet has a policy engine of its own.)"""),
hand(r"""echo yes > {WS}/q77/enforced.txt""",
r"""What the test showed.""",
run=r"""sleep 6
if kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null 2>&1; then echo no; else echo yes; fi > {WS}/q77/enforced.txt"""),
cmd(r"""k -n zero-trust delete pod srv cli --now
k -n zero-trust delete svc srv""", r"""Leave only the policy.""")],
)
# ------------------------------------------------------------------ 78 ----
@ -767,23 +943,16 @@ for i in $(seq 1 20); do [ "$(j -n rider-support deploy support-portal '{.status
eq "$(j -n rider-support deploy support-portal '{.status.readyReplicas}')" 2 \
&& eq "$(j -n rider-support pod -l app=support-portal '{.items[0].spec.priority}')" 1000000"""),
],
solution="""
cat <<EOF | k apply -f -
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: {name: rider-support-priority}
value: 1000000
globalDefault: false
description: Priority for rider support services
EOF
k -n rider-support patch deploy support-portal \\
-p '{"spec":{"template":{"spec":{"priorityClassName":"rider-support-priority"}}}}'
k -n rider-support rollout status deploy support-portal
k -n rider-support get pod -l app=support-portal -o jsonpath='{.items[0].spec.priority}'
# The admission controller copies the class's value into pod.spec.priority at creation time, so
# existing pods keep their old priority until they are recreated - patching the template does
# that for you. globalDefault: true would apply the class to every pod without one, cluster-wide.
""",
solution=[cmd(r"""k create priorityclass rider-support-priority --value=1000000 --description='Priority for rider support services'""", r"""Not the global default unless `--global-default`."""),
hand(r"""k -n rider-support edit deploy support-portal""",
r"""In the Pod template's `spec:`, beside `containers:` and indented like it, then `:wq`:
```
priorityClassName: rider-support-priority
```
The value is copied into each Pod when it is created, so only new Pods carry it - the edit rolls them.""",
run=r"""kubectl -n rider-support patch deploy support-portal -p '{"spec":{"template":{"spec":{"priorityClassName":"rider-support-priority"}}}}'"""),
cmd(r"""k -n rider-support rollout status deploy support-portal
k -n rider-support get pods -o custom-columns=NAME:.metadata.name,PRIORITY:.spec.priority""")],
)
# ------------------------------------------------------------------ 79 ----
@ -892,31 +1061,35 @@ un=$(grep -n 'uncordon' "$f" | head -1 | cut -d: -f1)
grep -Eq 'kubeadm[[:space:]]+upgrade[[:space:]]+node' "$WS/q80/worker.txt" 2>/dev/null \
&& ! grep -q 'upgrade apply' "$WS/q80/worker.txt" """),
],
solution="""
k version -o json | jq -r .serverVersion.gitVersion > {WS}/q80/current.txt
solution=[cmd(r"""k version
k get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion --no-headers > {WS}/q80/kubelets.txt
cat > {WS}/q80/steps.sh <<'EOF'
sed -i 's|/v1.37/|/v1.38/|' /etc/apt/sources.list.d/kubernetes.list
cat {WS}/q80/kubelets.txt"""),
hand(r"""echo v1.37.0 > {WS}/q80/current.txt""",
r"""With the `Server Version` that `k version` printed.""",
run=r"""kubectl version -o json | jq -r .serverVersion.gitVersion > {WS}/q80/current.txt"""),
hand(r"""vim {WS}/q80/steps.sh""",
r"""The order of the docs page *Upgrading kubeadm clusters*: kubeadm first, which upgrades the control plane; then the node's kubelet and kubectl, drained; uncordon last. The next minor is 1.38 - first point `/etc/apt/sources.list.d/kubernetes.list` at the v1.38 repository, and take its newest patch (`apt-cache madison kubeadm`):
```
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.38.0-1.1 && apt-mark hold kubeadm
kubeadm upgrade plan
kubeadm upgrade apply v1.38.0 -y
kubectl drain <control-plane> --ignore-daemonsets
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
kubeadm upgrade apply v1.38.0
kubectl drain {CP} --ignore-daemonsets
apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon <control-plane>
EOF
echo 'kubeadm upgrade node' > {WS}/q80/worker.txt
# The cluster runs v1.37, so the next minor is 1.38 - and pkgs.k8s.io serves one apt repo per
# minor, so the repo line has to move to v1.38 before 1.38 packages exist for apt at all. (1.38
# is due in December 2026; until then that repo is empty, which is why this is only a plan. Take
# the newest 1.38 patch on offer then: `apt-cache madison kubeadm`.)
# kubeadm goes first because it drives the whole thing; the control plane is upgraded by
# `upgrade apply`, and only then do the node binaries (kubelet/kubectl) follow, ending with
# uncordon. Workers run `kubeadm upgrade node` - they host no control-plane static pods, so
# there is nothing for `upgrade apply` to do there. The kubelet may trail the API server by up
# to three minor versions, but must never be ahead of it.
""",
kubectl uncordon {CP}
```""",
run=r"""cat > {WS}/q80/steps.sh <<'EOF'
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.38.0-1.1 && apt-mark hold kubeadm
kubeadm upgrade plan
kubeadm upgrade apply v1.38.0
kubectl drain {CP} --ignore-daemonsets
apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon {CP}
EOF"""),
cmd(r"""echo 'kubeadm upgrade node' > {WS}/q80/worker.txt""", r"""A worker hosts no control plane, so there is nothing for `upgrade apply` to do there; `upgrade node` updates its kubelet configuration.""")],
)
# ------------------------------------------------------------------ 81 ----
@ -977,35 +1150,44 @@ done; exit 1"""),
C("fare-calculator pods are Running and Ready", 3, r"""
for i in $(seq 1 24); do [ "$(j -n fare-engine deploy fare-calculator '{.status.readyReplicas}')" -ge 1 ] 2>/dev/null && exit 0; sleep 5; done; exit 1"""),
],
solution="""
k -n fare-engine get pvc # Pending
k -n fare-engine describe pvc fare-data-pvc # storageclass.storage.k8s.io "fast-storage" not found
k get sc # on kind the class is called `standard`
# A PVC's storageClassName is immutable, so recreate it with the name your cluster really has:
k -n fare-engine delete pvc fare-data-pvc
cat <<EOF | k apply -f -
solution=[cmd(r"""k -n fare-engine get pvc,pods
k -n fare-engine describe pvc fare-data-pvc | tail -3
k get sc""", r"""`storageclass.storage.k8s.io "fast-storage" not found`: the class here is `standard`."""),
cmd(r"""k -n fare-engine delete pvc fare-data-pvc""", r"""A claim's `storageClassName` cannot change: make it again."""),
hand(r"""vim pvc.yaml""",
r"""```
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: fare-data-pvc, namespace: fare-engine}
metadata:
name: fare-data-pvc
namespace: fare-engine
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
storageClassName: standard
EOF
k -n fare-engine describe pod -l app=fare-calculator # 0/4 nodes match node affinity disk in [nvme]
k get nodes --show-labels | grep -c 'disk=nvme' # 0
# Either label a node to satisfy the rule, or drop the rule. Labelling is the smaller change:
k label node <a-worker> disk=nvme
k -n fare-engine rollout status deploy fare-calculator
# Two independent blockers stack up silently: the PVC never binds, AND nothing would schedule
# even if it did. `describe pod` reports only the scheduling failure, so check the PVC too
# whenever a pod that mounts one is stuck Pending.
# Never assume the class name - `standard` here, `local-path` on a stock local-path-provisioner
# install, something else on a cloud cluster. `k get sc` first, every time; a PVC naming a class
# that does not exist sits Pending forever with no other symptom.
# `standard` is WaitForFirstConsumer, so the PVC only binds once the pod can actually be placed -
# which is why fixing the affinity is what finally unblocks the storage too.
""",
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
```""",
run=r"""cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: fare-data-pvc
namespace: fare-engine
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
EOF"""),
cmd(r"""k apply -f pvc.yaml
k -n fare-engine describe pod -l app=fare-calculator | tail -3
k get nodes -L disk""", r"""`didn't match Pod's node affinity/selector`: it wants `disk=nvme`, which no node has. `describe pod` names only this one; the claim's fault you had to find yourself."""),
cmd(r"""k label node {W1} disk=nvme
k -n fare-engine rollout status deploy fare-calculator""", r"""Labelling a node is the smaller change than dropping the rule. `standard` binds only once the Pod is placed (`WaitForFirstConsumer`), so the claim goes `Bound` now too.""")],
)
# ------------------------------------------------------------------ 82 ----
@ -1054,30 +1236,20 @@ ok=any(p.get("type")=="Pods" and int(p.get("value",0))==2 and int(p.get("periodS
for p in b.get("policies",[]))
sys.exit(0 if ok else 1)'"""),
],
solution="""
cat <<EOF | k apply -f -
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: route-planner-hpa, namespace: route-planner}
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: route-planner}
minReplicas: 1
maxReplicas: 6
metrics:
- type: Resource
resource:
name: memory
target: {type: Utilization, averageUtilization: 70}
solution=[cmd(r"""k -n route-planner autoscale deploy route-planner --name=route-planner-hpa --min=1 --max=6 --memory=70% --dry-run=client -o yaml > hpa.yaml""", r"""`autoscale` writes an `autoscaling/v2` HPA with the memory target; it has no flag for `behavior`."""),
hand(r"""vim hpa.yaml""",
r"""Add to its `spec:`, beside `maxReplicas:` (the docs page *Horizontal Pod Autoscaling* has the block under *Configurable scaling behavior*):
```
behavior:
scaleUp:
policies:
- {type: Pods, value: 2, periodSeconds: 60}
EOF
k -n route-planner get hpa route-planner-hpa
# `kubectl autoscale` can set a memory target (--memory=70%) but has no flag for a behavior block,
# so memory + behavior means writing the v2 YAML by hand (or autoscale, then kubectl edit). Utilisation is a percentage OF THE REQUEST, so the Deployment must
# declare a memory request or the HPA reports <unknown> and never scales.
""",
- type: Pods
value: 2
periodSeconds: 60
```""",
run=r"""yq -i '.spec.behavior.scaleUp.policies = [{"type": "Pods", "value": 2, "periodSeconds": 60}]' hpa.yaml"""),
cmd(r"""k apply -f hpa.yaml
k -n route-planner get hpa route-planner-hpa""", r"""Utilisation is a percentage of the request: the Deployment has a memory request, so the target reads a number, not `<unknown>`.""")],
)
# ------------------------------------------------------------------ 83 ----
@ -1131,26 +1303,81 @@ ok=any(not r.get("to") and {p.get("protocol") for p in r.get("ports",[]) if str(
for r in eg)
sys.exit(0 if ok else 1)'"""),
],
solution="""
cat <<EOF | k apply -f -
solution=[hand(r"""vim netpol.yaml""",
r"""Paste the example from the docs page *Network Policies* and change it to:
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: safety-api-policy, namespace: safety-monitor}
metadata:
name: safety-api-policy
namespace: safety-monitor
spec:
podSelector: {matchLabels: {app: safety-api}}
policyTypes: [Ingress, Egress]
podSelector:
matchLabels:
app: safety-api
policyTypes:
- Ingress
- Egress
ingress:
- from: [{podSelector: {matchLabels: {app: safety-dashboard}}}]
ports: [{protocol: TCP, port: 8443}]
- from:
- podSelector:
matchLabels:
app: safety-dashboard
ports:
- protocol: TCP
port: 8443
egress:
- to: [{podSelector: {matchLabels: {app: safety-db}}}]
ports: [{protocol: TCP, port: 5432}]
- ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
EOF
# The DNS rule deliberately has no `to:` - CoreDNS lives in kube-system, and an egress rule with
# only `ports` allows that port to anywhere. Naming Egress in policyTypes denies all outbound
# traffic by default, so forgetting DNS is what silently breaks every name lookup in the pod.
""",
- to:
- podSelector:
matchLabels:
app: safety-db
ports:
- protocol: TCP
port: 5432
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
```
The DNS rule has no `to:` - only ports, so port 53 anywhere. With `Egress` in `policyTypes` everything else outbound is denied, and forgetting DNS breaks every name lookup in the Pod.""",
run=r"""cat > netpol.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: safety-api-policy
namespace: safety-monitor
spec:
podSelector:
matchLabels:
app: safety-api
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: safety-dashboard
ports:
- protocol: TCP
port: 8443
egress:
- to:
- podSelector:
matchLabels:
app: safety-db
ports:
- protocol: TCP
port: 5432
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
EOF"""),
cmd(r"""k apply -f netpol.yaml
k -n safety-monitor describe netpol safety-api-policy""")],
)
# ------------------------------------------------------------------ 84 ----

407
cka/bank_sim_d.py
View File

@ -16,7 +16,7 @@ task was seeded, solved and graded on an isolated kind cluster before it went in
Run with: ./ksim sim d
"""
from sim.bank import C, NS
from sim.bank import C, NS, cmd, hand
Q = []
# Tasks written for the pre-2025 exam opened by switching kubectl context. Every task now has a host
@ -73,25 +73,27 @@ if [ "$own" = ReplicaSet ]; then want="deployment $(j -n kube-system rs "$name"
else want="$(echo "$own" | tr A-Z a-z) $name"; fi
eq "$(sed -n 's/^ *dns *: *//p' "$f" | head -1 | tr -s ' ' | tr A-Z a-z | sed 's/ *$//')" "$want" """),
],
solution="""
sudo -i # the task's host is the control-plane node: no hop needed
systemctl status kubelet | head -3 # a systemd service: a plain process on the node
ls /etc/kubernetes/manifests # etcd, kube-apiserver, kube-controller-manager, kube-scheduler
exit
k -n kube-system get pod -o wide # static pods carry the node name as a suffix...
k -n kube-system get pod kube-apiserver-{CP} -o jsonpath='{.metadata.ownerReferences[0].kind}{"\\n"}' # ...and are owned by the Node
k -n kube-system get deploy,ds # coredns is a Deployment (kube-proxy, kindnet are DaemonSets)
cat > {WS}/q86/controlplane.txt <<'EOF'
solution=[cmd(r"""systemctl status kubelet | head -3
sudo ls /etc/kubernetes/manifests""", r"""The host is the control-plane node. The kubelet is a systemd service - a process; etcd and the three control-plane components are manifests in the static Pod path."""),
cmd(r"""k -n kube-system get pods -o wide
k -n kube-system get deploy,ds""", r"""Static Pods carry the node's name as a suffix. What answers DNS is the `coredns` Deployment, though its Service is called `kube-dns`; `kube-proxy` and `kindnet` are DaemonSets."""),
hand(r"""vim {WS}/q86/controlplane.txt""",
r"""```
kubelet: process
kube-apiserver: static-pod
kube-scheduler: static-pod
kube-controller-manager: static-pod
etcd: static-pod
dns: deployment coredns
EOF
# The Service is called kube-dns for historical reasons; what actually answers is the coredns
# Deployment. Follow the chain svc selector -> pod -> ownerReferences when unsure.
""",
```""",
run=r"""cat > {WS}/q86/controlplane.txt <<'EOF'
kubelet: process
kube-apiserver: static-pod
kube-scheduler: static-pod
kube-controller-manager: static-pod
etcd: static-pod
dns: deployment coredns
EOF""")],
)
# ------------------------------------------------------------------ 87 ----
@ -125,24 +127,27 @@ cfg=$(nx "$W1" 'ls /etc/cni/net.d/*.conflist /etc/cni/net.d/*.conf 2>/dev/null |
C("static Pod name suffix on {W3} (line 5)", 1, r"""
v=$(sed -n 's/^ *5 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]'); [ "$v" = "-$W3" ] || [ "$v" = "$W3" ]"""),
],
solution="""
k get nodes # ROLES shows control-plane; count the rest
k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
k -n kube-system get ds # kindnet
{SHELL_W1}
sudo -i
ls /etc/cni/net.d/ # 10-kindnet.conflist
exit; exit
cat > {WS}/q87/cluster-info.txt <<'EOF'
solution=[cmd(r"""k get nodes
k -n kube-system describe pod -l component=kube-apiserver | grep service-cluster-ip-range
k -n kube-system get ds""", r"""`ROLES control-plane` on one node, the rest workers; the API server's `--service-cluster-ip-range`; and the CNI's DaemonSet, `kindnet`."""),
hand(r"""ssh {W1} ls /etc/cni/net.d""",
r"""The CNI's configuration on a node: `10-kindnet.conflist`.""",
run=r""""""),
hand(r"""vim {WS}/q87/cluster-info.txt""",
r"""The values you read - here:
```
1: 1
2: 3
3: 10.96.0.0/16
4: kindnet, /etc/cni/net.d/10-kindnet.conflist
5: -{W3}
EOF
# A static Pod's mirror is named <pod>-<node name>, e.g. kube-apiserver-{CP}. Values come from THIS
# cluster - check each one instead of trusting the ones shown here.
""",
```
A static Pod's mirror is named `<pod>-<node>`, e.g. `kube-apiserver-{CP}`.""",
run=r"""C=$(kubectl get nodes --no-headers -l node-role.kubernetes.io/control-plane | wc -l)
W=$(kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' | wc -l)
CIDR=$(kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -o 'service-cluster-ip-range=[^ ]*' | head -1 | cut -d= -f2)
F=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
printf '1: %s\n2: %s\n3: %s\n4: kindnet, %s\n5: -{W3}\n' "$C" "$W" "$CIDR" "$F" > {WS}/q87/cluster-info.txt""")],
)
# ------------------------------------------------------------------ 88 ----
@ -218,18 +223,19 @@ grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-n|--namespa
timeout 60 bash "$f" 2>/dev/null | grep -q etcd"""),
C("no aliases in either script", 1, r"""[ -s "$WS/q89/pods-by-age.sh" ] && [ -s "$WS/q89/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q89/pods-by-age.sh" "$WS/q89/pods-by-uid.sh" """),
],
solution="""
cat > {WS}/q89/pods-by-age.sh <<'EOF'
solution=[hand(r"""vim {WS}/q89/pods-by-age.sh""",
r"""`kubectl` in full: a script runs in a fresh shell, where the `k` alias does not exist.
```
kubectl get pods -A --sort-by=.metadata.creationTimestamp
EOF
cat > {WS}/q89/pods-by-uid.sh <<'EOF'
```""",
run=r"""echo 'kubectl get pods -A --sort-by=.metadata.creationTimestamp' > {WS}/q89/pods-by-age.sh"""),
hand(r"""vim {WS}/q89/pods-by-uid.sh""",
r"""```
kubectl -n kube-system get pods --sort-by=.metadata.uid
EOF
chmod +x {WS}/q89/*.sh
bash {WS}/q89/pods-by-age.sh | head -3 # the control-plane pods come first - they are oldest
# --sort-by takes a JSONPath into each item; ascending only, so "newest first" is `| tac` or
# sorting by a field that grows the other way.
""",
```""",
run=r"""echo 'kubectl -n kube-system get pods --sort-by=.metadata.uid' > {WS}/q89/pods-by-uid.sh"""),
cmd(r"""chmod +x {WS}/q89/*.sh
{WS}/q89/pods-by-age.sh | head -3""", r"""The control-plane Pods first: they are the oldest. `--sort-by` takes a JSONPath into each item, ascending only.""")],
)
# ------------------------------------------------------------------ 90 ----
@ -273,31 +279,35 @@ for f in pod-deleted.log container-killed.log; do
done"""),
C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q90/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""),
],
solution="""
cat > {WS}/q90/cluster-events.sh <<'EOF'
solution=[hand(r"""vim {WS}/q90/cluster-events.sh""",
r"""```
kubectl get events -A --sort-by=.metadata.creationTimestamp
EOF
chmod +x {WS}/q90/cluster-events.sh
k -n kube-system get pod -l k8s-app=kube-proxy -o wide # the one on {W2}
k -n kube-system delete pod <kube-proxy-on-{W2}>
k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
# Killing (old pod), SuccessfulCreate (DaemonSet), Scheduled, Pulled, Created, Started (new pod)
{SHELL_W2}
```""",
run=r"""echo 'kubectl get events -A --sort-by=.metadata.creationTimestamp' > {WS}/q90/cluster-events.sh"""),
cmd(r"""chmod +x {WS}/q90/cluster-events.sh
k -n kube-system get pods -l k8s-app=kube-proxy -o wide""", r"""Note the name of the one on `{W2}`."""),
hand(r"""k -n kube-system delete pod kube-proxy-xxxxx""",
r"""With that name. The DaemonSet replaces it with a new Pod at once.""",
run=r"""P=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system delete pod $P
for i in $(seq 1 40); do
[ "$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].status.phase}' 2>/dev/null)" = Running ] && break; sleep 3
done
sleep 5"""),
cmd(r"""k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
cat {WS}/q90/pod-deleted.log""", r"""`Killing` the old Pod, `SuccessfulCreate` by the DaemonSet, then `Scheduled`, `Pulled`, `Created`, `Started` for the new one."""),
hand(r"""{SHELL_W2}
sudo -i
crictl ps --name kube-proxy # container id
crictl stop <id>
exit; exit
k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
# Pulled, Created, Started - same Pod, new container
k -n kube-system get pod -l k8s-app=kube-proxy -o wide # RESTARTS 1 on {W2}
echo container > {WS}/q90/restarts.txt
# Deleting the Pod makes the DaemonSet create a brand-new Pod (RESTARTS 0). Killing only the
# container makes the kubelet start a new container inside the SAME Pod - that is a restart.
# Do not `crictl rm` it straight away: on v1.35 the kubelet counts the restart from the exited
# container it sees, and if rm wins the race it starts a fresh attempt 0 and RESTARTS stays 0.
""",
crictl ps --name kube-proxy
crictl stop 1a2b3c4d5e6f
exit
exit""",
r"""With the container ID `crictl ps` printed. Only the container goes: the kubelet starts a new one in the same Pod.""",
run=r"""ssh -n {W2} 'sudo crictl stop $(sudo crictl ps --name kube-proxy -q)'
sleep 25"""),
cmd(r"""k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
k -n kube-system get pods -l k8s-app=kube-proxy -o wide
echo container > {WS}/q90/restarts.txt""", r"""`RESTARTS 1` on `{W2}`: a new container in the same Pod is a restart; a new Pod starts again at 0.""")],
)
# ------------------------------------------------------------------ 91 ----
@ -333,15 +343,15 @@ got=$(awk '{print $1}' "$f" | sed 's/\..*//' | grep . | sort -u)
best=$(for ns in census-north census-south census-east census-west; do echo "$ns $($K -n $ns get role --no-headers 2>/dev/null | wc -l)"; done | sort -k2 -n | tail -1)
eq "$(tr -s ' ' < "$WS/q91/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """),
],
solution="""
k api-resources --namespaced=true -o name > /tmp/r.txt # deployments.apps, ... (group suffix)
k api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q91/namespaced.txt
for ns in census-north census-south census-east census-west; do echo "$ns $(k -n $ns get role --no-headers | wc -l)"; done
# pick the largest, e.g. echo "census-east 8" > {WS}/q91/most-roles.txt
# --namespaced=false gives the cluster-scoped half (nodes, pv, storageclasses, clusterroles...).
# `k get role -A --no-headers | awk '{print $1}' | sort | uniq -c | sort -n` counts every
# namespace at once.
""",
solution=[cmd(r"""k api-resources --namespaced=true -o name > {WS}/q91/namespaced.txt
head -3 {WS}/q91/namespaced.txt""", r"""`-o name` prints the plural names alone (`deployments.apps`: name and group); `--namespaced=false` would give the cluster-scoped half."""),
cmd(r"""k -n census-north get role --no-headers | wc -l
k -n census-south get role --no-headers | wc -l
k -n census-east get role --no-headers | wc -l
k -n census-west get role --no-headers | wc -l""", r"""Or all at once: `k get role -A | grep census`."""),
hand(r"""echo "census-east 9" > {WS}/q91/most-roles.txt""",
r"""With the namespace that has the most, and its count.""",
run=r"""for ns in census-north census-south census-east census-west; do echo "$ns $(kubectl -n $ns get role --no-headers | wc -l)"; done | sort -k2 -n | tail -1 > {WS}/q91/most-roles.txt""")],
)
# ------------------------------------------------------------------ 92 ----
@ -480,20 +490,27 @@ ip=$(j -n default pod "ops-beacon-$CP" '{.status.podIP}'); [ -n "$ip" ] || exit
j -n default endpointslices -l kubernetes.io/service-name=ops-beacon-svc '{.items[*].endpoints[*].addresses[*]}' | tr ' ' '\n' | grep -qx "$ip" """),
C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q94/nodeport.txt" 2>/dev/null)" "$np" """),
],
solution="""
k run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon $do > beacon.yaml
# add to the container: resources: {requests: {cpu: 10m, memory: 20Mi}}
sudo -i # the task's host is the control-plane node: no hop needed
grep staticPodPath /var/lib/kubelet/config.yaml # /etc/kubernetes/manifests
vi /etc/kubernetes/manifests/ops-beacon.yaml # paste beacon.yaml
exit
k -n default get pod ops-beacon-{CP} -o wide # mirror pod, named <pod>-<node>
k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc # the mirror pod's IP
k -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt
# The control-plane taint only keeps the SCHEDULER from placing pods there; the kubelet runs its
# static pods regardless. `expose pod` copies the pod's labels into the selector.
""",
solution=[cmd(r"""k -n default run ops-beacon --image=nginx:1.27 --labels=app=ops-beacon --dry-run=client -o yaml > beacon.yaml"""),
hand(r"""vim beacon.yaml""",
r"""Replace the container's `resources: {}` with:
```
resources:
requests:
cpu: 10m
memory: 20Mi
```""",
run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "10m", "memory": "20Mi"}}' beacon.yaml"""),
cmd(r"""sudo grep staticPodPath /var/lib/kubelet/config.yaml
sudo cp beacon.yaml /etc/kubernetes/manifests/ops-beacon.yaml""", r"""The host is the control-plane node. Its taint keeps the scheduler away, not the kubelet: static Pods run regardless."""),
hand(r"""k -n default get pod ops-beacon-{CP} -o wide""",
r"""Again until the mirror Pod is `Running`.""",
run=r"""for i in $(seq 1 60); do [ "$(kubectl -n default get pod ops-beacon-{CP} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done"""),
cmd(r"""k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
k -n default get svc ops-beacon-svc
k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc""", r"""`expose pod` makes the selector from the Pod's labels: the mirror Pod's IP is the endpoint. Note the node port, after `80:`."""),
hand(r"""echo 31234 > {WS}/q94/nodeport.txt""",
r"""With the node port `get svc` showed.""",
run=r"""kubectl -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt""")],
)
# ------------------------------------------------------------------ 95 ----
@ -757,40 +774,107 @@ for i in 0 1; do eq "$($K -n lima-data exec ledger-db-$i -- cat /data/id.txt 2>/
want=$($K -n lima-data get pvc --no-headers -o custom-columns=N:.metadata.name 2>/dev/null | grep '^data-ledger-db-' | sort)
[ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q99/pvcs.txt" 2>/dev/null)" = "$want" ]"""),
],
solution="""
cat <<EOF | k apply -f -
solution=[hand(r"""vim sts.yaml""",
r"""The docs page *StatefulSets* has a headless Service and a StatefulSet with `volumeClaimTemplates` together; change them to:
```
apiVersion: v1
kind: Service
metadata: {name: ledger-db, namespace: lima-data}
spec: {clusterIP: None, selector: {app: ledger-db}, ports: [{port: 5432}]}
metadata:
name: ledger-db
namespace: lima-data
spec:
clusterIP: None
selector:
app: ledger-db
ports:
- port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: ledger-db, namespace: lima-data}
metadata:
name: ledger-db
namespace: lima-data
spec:
serviceName: ledger-db
replicas: 3
selector: {matchLabels: {app: ledger-db}}
selector:
matchLabels:
app: ledger-db
template:
metadata: {labels: {app: ledger-db}}
metadata:
labels:
app: ledger-db
spec:
containers:
- name: db
image: busybox:1.36
command: [sh, -c, "hostname > /data/id.txt && sleep 86400"]
volumeMounts: [{name: data, mountPath: /data}]
command: ["sh", "-c", "hostname > /data/id.txt && sleep 86400"]
volumeMounts:
- name: data
mountPath: /data
volumeClaimTemplates:
- metadata: {name: data}
spec: {accessModes: [ReadWriteOnce], storageClassName: standard, resources: {requests: {storage: 100Mi}}}
EOF
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: standard
resources:
requests:
storage: 100Mi
```""",
run=r"""cat > sts.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: ledger-db
namespace: lima-data
spec:
clusterIP: None
selector:
app: ledger-db
ports:
- port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: ledger-db
namespace: lima-data
spec:
serviceName: ledger-db
replicas: 3
selector:
matchLabels:
app: ledger-db
template:
metadata:
labels:
app: ledger-db
spec:
containers:
- name: db
image: busybox:1.36
command: ["sh", "-c", "hostname > /data/id.txt && sleep 86400"]
volumeMounts:
- name: data
mountPath: /data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: standard
resources:
requests:
storage: 100Mi
EOF"""),
cmd(r"""k apply -f sts.yaml
k -n lima-data rollout status sts ledger-db
k -n lima-data scale sts ledger-db --replicas=2
k -n lima-data get pvc -o custom-columns=N:.metadata.name --no-headers | grep '^data-ledger-db-' | sort > {WS}/q99/pvcs.txt
cat {WS}/q99/pvcs.txt # all THREE: data-ledger-db-0, -1, -2
# Claims are named <template>-<statefulset>-<ordinal> and outlive their pods: scaling down (or
# deleting the StatefulSet) keeps them unless spec.persistentVolumeClaimRetentionPolicy says
# otherwise, so scaling back up re-attaches ledger-db-2 to its old data.
""",
k -n lima-data scale sts ledger-db --replicas=2"""),
cmd(r"""k -n lima-data get pvc -o custom-columns=NAME:.metadata.name --no-headers | sort > {WS}/q99/pvcs.txt
cat {WS}/q99/pvcs.txt""", r"""All three, `data-ledger-db-0` to `-2`: claims are named `<template>-<statefulset>-<ordinal>` and outlive their Pods, so scaling up again gives `ledger-db-2` its old data back.""")],
)
# ------------------------------------------------------------------ 100 ----
@ -849,28 +933,63 @@ $K -n oslo-db get netpol db-ingress >/dev/null 2>&1 || exit 1
! $K -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1 \
&& ! $K -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1"""),
],
solution="""
k get ns --show-labels | grep -E 'oslo-api|lima-api' # env=prod / env=dev
cat <<EOF | k apply -f -
solution=[cmd(r"""k get ns --show-labels | grep -e oslo-api -e lima-api
k -n oslo-api get pods --show-labels""", r"""`env=prod` and `env=dev`; `api` is `role=backend`, `debug` is not."""),
hand(r"""vim netpol.yaml""",
r"""One `from` entry holding both selectors - both must match:
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: db-ingress, namespace: oslo-db}
metadata:
name: db-ingress
namespace: oslo-db
spec:
podSelector: {matchLabels: {app: db}}
policyTypes: [Ingress]
podSelector:
matchLabels:
app: db
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {matchLabels: {env: prod}}
podSelector: {matchLabels: {role: backend}} # SAME list item = AND
ports: [{protocol: TCP, port: 80}]
EOF
k -n oslo-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db | head -1 # <!DOCTYPE html>
k -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db # download timed out
k -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db # download timed out
# The trap is one dash: "- namespaceSelector: ...\\n - podSelector: ..." (two list items) means OR -
# any pod in a prod namespace, OR a backend pod in oslo-db's own namespace. One item with both
# selectors means AND.
""",
- namespaceSelector:
matchLabels:
env: prod
podSelector:
matchLabels:
role: backend
ports:
- protocol: TCP
port: 80
```
The trap is one dash: `- podSelector:` would make it a second entry, and the rule an OR - any Pod in a prod namespace, or any backend Pod in `oslo-db`.""",
run=r"""cat > netpol.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: db-ingress
namespace: oslo-db
spec:
podSelector:
matchLabels:
app: db
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
env: prod
podSelector:
matchLabels:
role: backend
ports:
- protocol: TCP
port: 80
EOF"""),
cmd(r"""k apply -f netpol.yaml
k -n oslo-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db
k -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db
k -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db""", r"""nginx's page, then `download timed out` twice.""")],
)
# ------------------------------------------------------------------ 101 ----
@ -919,18 +1038,18 @@ for i in $(seq 1 15); do
done; exit 1"""),
C("fix.txt names dnsPolicy and its new value", 1, r"""grep -Eqi '^ *dnsPolicy *= *ClusterFirst *$' "$WS/q101/fix.txt" 2>/dev/null"""),
],
solution="""
k -n lima-client logs deploy/reporter --tail=3 # wget: bad address 'ledger:8080'
k -n lima-client get endpointslices # the Service has endpoints - not the problem
k -n lima-client get deploy reporter -o yaml | grep -n dns
# dnsPolicy: Default <- "Default" means: use the NODE's resolver, NOT cluster DNS
k -n lima-client exec deploy/reporter -- cat /etc/resolv.conf # the podman/host resolver, no cluster search domains
k -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'
k -n lima-client logs deploy/reporter -f # "ledger ok"
echo dnsPolicy=ClusterFirst > {WS}/q101/fix.txt
# The trap is the name: the DEFAULT dnsPolicy is ClusterFirst, while the policy literally called
# "Default" inherits the node's /etc/resolv.conf. ClusterFirstWithHostNet is for hostNetwork pods.
""",
solution=[cmd(r"""k -n lima-client logs deploy/reporter --tail=3
k -n lima-client get endpointslices
k -n lima-client get deploy reporter -o yaml | grep dnsPolicy""", r"""`bad address 'ledger:8080'`, though the Service has endpoints. `dnsPolicy: Default` uses the *node's* resolver, not cluster DNS - the default policy is `ClusterFirst`, despite the name."""),
hand(r"""k -n lima-client edit deploy reporter""",
r"""In the Pod template's `spec:`, then `:wq`:
```
dnsPolicy: ClusterFirst
```""",
run=r"""kubectl -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'"""),
cmd(r"""echo dnsPolicy=ClusterFirst > {WS}/q101/fix.txt
k -n lima-client rollout status deploy reporter
k -n lima-client logs deploy/reporter --tail=3""", r"""`ledger ok`.""")],
)
# ------------------------------------------------------------------ 102 ----
@ -977,22 +1096,16 @@ eq "$(j -n oslo-files deploy uploader '{.status.availableReplicas}')" 1 \
&& j -n oslo-files deploy uploader '{.spec.template.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /files"""),
C("reason.txt names the unsupported access mode", 2, r"""grep -Eqi 'ReadWriteMany|RWX|access ?mode' "$WS/q102/reason.txt" 2>/dev/null"""),
],
solution="""
k -n oslo-files get pod # Pending
k -n oslo-files describe pod -l app=uploader | tail -4 # unbound immediate PersistentVolumeClaims / waiting
k -n oslo-files describe pvc shared-files | tail -4
# ProvisioningFailed ... NodePath only supports ReadWriteOnce and ReadWriteOncePod (1.22+) access modes
# accessModes is immutable on a PVC, so recreate it. The Pending pod was never scheduled, so it
# does not hold the claim (pvc-protection only counts scheduled pods) and the delete goes through.
k -n oslo-files get pvc shared-files -o yaml > pvc.yaml # change ReadWriteMany -> ReadWriteOnce, strip status/uid
k -n oslo-files delete pvc shared-files
k apply -f pvc.yaml
k -n oslo-files rollout restart deploy uploader # or just wait - the pending pod retries
k -n oslo-files get pvc,pod
echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
# RWX needs a filesystem many nodes can mount at once (NFS, CephFS, a cloud file service).
# ReadWriteOnce is per NODE, not per pod - ReadWriteOncePod is the per-pod one.
""",
solution=[cmd(r"""k -n oslo-files get pod,pvc
k -n oslo-files describe pvc shared-files | tail -4""", r"""`ProvisioningFailed ... NodePath only supports ReadWriteOnce and ReadWriteOncePod access modes`: the claim asks for `ReadWriteMany`, which local-path cannot give."""),
cmd(r"""echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
k -n oslo-files get pvc shared-files -o yaml > pvc.yaml"""),
hand(r"""vim pvc.yaml""",
r"""Under `accessModes:`, change `ReadWriteMany` to `ReadWriteOnce` - name, size and class stay.""",
run=r"""yq -i '.spec.accessModes = ["ReadWriteOnce"]' pvc.yaml"""),
cmd(r"""k replace --force -f pvc.yaml
k -n oslo-files rollout restart deploy uploader
k -n oslo-files get pvc,pod""", r"""Access modes cannot change on a claim: `--force` deletes it and creates it again. The Pending Pod was never scheduled, so it does not hold the claim. Again until it is `Bound` and the Pod `Running`.""")],
)
QUESTIONS = Q

625
cka/bank_storage.py
View File

@ -5,6 +5,8 @@ served by the local-path provisioner in Namespace local-path-storage), so a task
provisioner or change the default class. Volumes are tested for real - a claim that binds, a file
that is still on the node, a Pod that reads it.
"""
from sim.bank import cmd, hand
from cka.common import C, NS, PY, STOR, ns_pick, name_pick
# what a setup records about the objects a task must leave alone
@ -50,44 +52,85 @@ gone -n %{ns} pvc %{pvc} || exit 1
node=$(pvnode "$pv"); path=$(pvpath "$pv"); [ -n "$node" ] && [ -n "$path" ] || exit 1
eq "$(nx "$node" "cat $path/keep.txt" </dev/null 2>/dev/null | tr -d '\r')" "%{text}" """),
],
"solution": """
cat <<EOF | k apply -f -
"solution": [cmd(r"""k get sc standard -o yaml | grep provisioner""", r"""`rancher.io/local-path`."""),
hand(r"""vim sc.yaml""",
r"""The class, from the docs page *Storage Classes* - no `is-default-class` annotation, so not the default - and the claim:
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: {name: %{sc}}
provisioner: rancher.io/local-path # k get sc standard -o yaml shows it
metadata:
name: %{sc}
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
metadata:
name: %{pvc}
namespace: %{ns}
spec:
storageClassName: %{sc}
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 300Mi}}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 300Mi
```""",
run=r"""cat > sc.yaml <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: %{sc}
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
spec:
volumes: [{name: data, persistentVolumeClaim: {claimName: %{pvc}}}]
containers:
- name: writer
image: busybox:1.36
command: [sh, -c, "echo %{text} > /data/keep.txt; sleep 86400"]
volumeMounts: [{name: data, mountPath: /data}]
EOF
k -n %{ns} get pvc %{pvc} # Pending until the Pod is scheduled: WaitForFirstConsumer
k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt
k -n %{ns} delete pod %{pod}; k -n %{ns} delete pvc %{pvc}
k get pv $(cat {WS}/q114/pv.txt) # STATUS Released, RECLAIM POLICY Retain
echo Released > {WS}/q114/phase.txt
# A class's reclaimPolicy is copied into every volume it provisions. Delete (the default) removes
# the volume and its data with the claim; Retain leaves a Released volume that no new claim can
# bind until an administrator clears spec.claimRef. A StorageClass cannot be edited afterwards,
# except for a few fields - to change the policy of one volume, patch the PV.
""",
storageClassName: %{sc}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 300Mi
EOF"""),
cmd(r"""k apply -f sc.yaml
k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sh -c "echo %{text} > /data/keep.txt; sleep 86400" > pod.yaml""", r"""The claim stays `Pending` until a Pod uses it: `WaitForFirstConsumer`."""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: %{pod}`:
```
volumeMounts:
- name: data
mountPath: /data
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: data
persistentVolumeClaim:
claimName: %{pvc}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}]
| .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "%{pvc}"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod,pvc""",
r"""Again until the Pod is `Running` and the claim `Bound`; note its `VOLUME`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
hand(r"""echo pvc-0b1c2d3e-... > {WS}/q114/pv.txt""",
r"""With the volume's name.""",
run=r"""kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt"""),
cmd(r"""k -n %{ns} delete pod %{pod} --now
k -n %{ns} delete pvc %{pvc}"""),
hand(r"""k get pv
echo Released > {WS}/q114/phase.txt""",
r"""`STATUS Released`, `RECLAIM POLICY Retain`: the class's policy is copied into every volume it provisions. A Released volume binds no new claim until its `claimRef` is cleared.""",
run=r"""sleep 4
kubectl get pv $(cat {WS}/q114/pv.txt) -o jsonpath='{.status.phase}' > {WS}/q114/phase.txt""")],
},
# ------------------------------------------------------------------ 115 ----
{
@ -133,39 +176,43 @@ eq "$(one "$WS/q115/claim.txt" | sed 's#.*/##')" %{pod}-scratch || exit 1
eq "$(j -n %{ns} pvc %{pod}-scratch '{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}')" Pod/%{pod} || exit 1
eq "$(one "$WS/q115/owner.txt" | tr A-Z a-z)" pod/%{pod}"""),
],
"solution": """
cat <<EOF | k apply -f -
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
"solution": [cmd(r"""k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: %{pod}`:
```
volumeMounts:
- name: scratch
mountPath: /scratch
- name: cache
mountPath: /cache
```
and to the Pod's `spec:`, right below `spec:` (the docs page *Ephemeral Volumes* has the generic ephemeral volume):
```
volumes:
- name: scratch
ephemeral:
volumeClaimTemplate:
spec:
storageClassName: standard
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
- name: cache
emptyDir: {medium: Memory, sizeLimit: 32Mi}
containers:
- name: encoder
image: busybox:1.36
command: [sleep, "86400"]
volumeMounts:
- {name: scratch, mountPath: /scratch}
- {name: cache, mountPath: /cache}
EOF
k -n %{ns} get pvc # %{pod}-scratch: <pod name>-<volume name>
echo %{pod}-scratch > {WS}/q115/claim.txt
k -n %{ns} get pvc %{pod}-scratch -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}'
echo Pod/%{pod} > {WS}/q115/owner.txt
# A generic ephemeral volume is a claim written into the Pod: Kubernetes creates the PVC, the Pod
# owns it, and deleting the Pod deletes claim and volume. Unlike emptyDir it can come from any
# StorageClass - size, speed, snapshots - and unlike a normal claim it never outlives the Pod.
# An emptyDir with medium Memory is a tmpfs: what is written there counts as the container's memory.
""",
emptyDir:
medium: Memory
sizeLimit: 32Mi
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "scratch", "mountPath": "/scratch"}, {"name": "cache", "mountPath": "/cache"}]
| .spec.volumes = [{"name": "scratch", "ephemeral": {"volumeClaimTemplate": {"spec": {"storageClassName": "standard", "accessModes": ["ReadWriteOnce"], "resources": {"requests": {"storage": "100Mi"}}}}}}, {"name": "cache", "emptyDir": {"medium": "Memory", "sizeLimit": "32Mi"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod,pvc""",
r"""Again until the Pod is `Running`. The claim is `<pod>-<volume>`: `%{pod}-scratch`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
cmd(r"""echo %{pod}-scratch > {WS}/q115/claim.txt
k -n %{ns} get pvc %{pod}-scratch -o yaml | grep -A5 ownerReferences
echo Pod/%{pod} > {WS}/q115/owner.txt""", r"""The Pod owns it: delete the Pod, and the claim and its volume go too. An emptyDir in `Memory` is a tmpfs, counted as the container's memory.""")],
},
# ------------------------------------------------------------------ 116 ----
{
@ -234,18 +281,20 @@ gone -n %{ns} pvc data-%{sts}-1 && gone -n %{ns} pvc data-%{sts}-2 && eq "$(j -n
C("claims.txt lists what is left: data-%{sts}-0", 1, r"""
eq "$(tr -d ' \t\r' < "$WS/q116/claims.txt" 2>/dev/null | sed 's#.*/##' | grep . | sort | tr '\n' ' ')" "data-%{sts}-0 " """),
],
"solution": """
k -n %{ns} get sts %{sts} -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}' # whenDeleted Retain, whenScaled Retain
k -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'
k -n %{ns} scale sts %{sts} --replicas=1
k -n %{ns} get pvc -w # data-%{sts}-2 and data-%{sts}-1 disappear with their Pods
k -n %{ns} get pvc -o name | cut -d/ -f2 > {WS}/q116/claims.txt
# By default a StatefulSet never deletes a claim: scale down and up again and each replica finds
# its old data. persistentVolumeClaimRetentionPolicy changes that per event - whenScaled for the
# replicas a scale-down removes, whenDeleted for all of them when the StatefulSet goes. Set the
# policy BEFORE scaling: it is applied to the Pods being removed, not to claims orphaned earlier
# (those you delete by hand). It is one of the few StatefulSet fields that can be changed in place.
""",
"solution": [cmd(r"""k -n %{ns} get sts %{sts} -o yaml | grep -A2 persistentVolumeClaimRetentionPolicy""", r"""`Retain` for both: by default a StatefulSet never deletes a claim."""),
hand(r"""k -n %{ns} edit sts %{sts}""",
r"""Set it before scaling - it applies to the Pods being removed - then `:wq`:
```
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Delete
```""",
run=r"""kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'"""),
cmd(r"""k -n %{ns} scale sts %{sts} --replicas=1"""),
hand(r"""k -n %{ns} get pvc""",
r"""Again until `data-%{sts}-2` and `data-%{sts}-1` have gone with their Pods.""",
run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc --no-headers | wc -l)" = 1 ] && break; sleep 3; done"""),
cmd(r"""echo data-%{sts}-0 > {WS}/q116/claims.txt""")],
},
# ------------------------------------------------------------------ 117 ----
{
@ -292,34 +341,76 @@ eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}/{.spec.storageClassName}/{.spec.
[ -n "$(j pv %{pv} '{.metadata.name}')" ] || exit 1
eq "$(j -n %{oth} pvc any-volume '{.metadata.uid}')" "$(cat "$PRIV/q117.uid")" && eq "$(j -n %{oth} pvc any-volume '{.status.phase}')" Pending"""),
],
"solution": """
cat <<EOF | k apply -f -
"solution": [cmd(r"""k -n %{oth} get pvc any-volume""", r"""`Pending`, for any 1Gi `manual` volume: an Available one would go to it."""),
hand(r"""vim pv.yaml""",
r"""A `claimRef` on the PV reserves it for one claim, so the waiting one cannot take it:
```
apiVersion: v1
kind: PersistentVolume
metadata: {name: %{pv}}
metadata:
name: %{pv}
spec:
capacity: {storage: 1Gi}
accessModes: [ReadWriteOnce]
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath: {path: /mnt/%{pv}}
claimRef: {namespace: %{ns}, name: %{pvc}} # the reservation: no other claim can bind this volume
hostPath:
path: /mnt/%{pv}
claimRef:
namespace: %{ns}
name: %{pvc}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
metadata:
name: %{pvc}
namespace: %{ns}
spec:
storageClassName: manual
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
EOF
k get pv %{pv}; k -n %{ns} get pvc %{pvc}; k -n %{oth} get pvc any-volume # Bound, Bound, Pending
# Without claimRef the volume would be Available, and the control plane hands an Available volume
# to whichever matching claim it looks at first - here the one that has been waiting. That cannot
# be undone cleanly: a claim whose volume is taken away ends up Lost. claimRef on the PV reserves
# it; volumeName on a claim is the other half (it asks for one volume but does not keep others
# away from it). Use both when it has to be this pair and no other.
""",
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```
`volumeName` on a claim is the other half: it asks for one volume, but keeps no other claim away from it.""",
run=r"""cat > pv.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: %{pv}
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /mnt/%{pv}
claimRef:
namespace: %{ns}
name: %{pvc}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF"""),
cmd(r"""k apply -f pv.yaml
k get pv %{pv}
k -n %{ns} get pvc %{pvc}
k -n %{oth} get pvc any-volume""", r"""`Bound`, `Bound`, still `Pending`.""")],
},
# ------------------------------------------------------------------ 118 ----
{
@ -405,30 +496,53 @@ done; exit 1"""),
eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}')" %{pv} || exit 1
for i in $(seq 1 10); do gone pv "$(cat "$PRIV/q118.wrong")" && exit 0; sleep 3; done; exit 1"""),
],
"solution": """
k -n %{ns} get pvc %{pvc} # Bound to pvc-..., STORAGECLASS standard - not to %{pv}
k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt
k get pv %{pv} # Available, STORAGECLASS empty
# The manifest names no class, so the claim was given the cluster's default one (standard) and a
# brand-new, empty volume was provisioned for it. A claim's class and volume cannot be edited:
k -n %{ns} scale deploy %{dep} --replicas=0 # a claim in use is not deleted (pvc-protection)
k -n %{ns} delete pvc %{pvc} # its volume goes with it: reclaim policy Delete
cat <<EOF | k apply -f -
"solution": [cmd(r"""k -n %{ns} get pvc %{pvc}
k get pv %{pv}""", r"""Bound to a `pvc-...` volume of class `standard`, while `%{pv}` is `Available` with no class: the manifest names no class, so the claim got the default one, and a new empty volume."""),
hand(r"""echo pvc-0b1c2d3e-... > {WS}/q118/wrong-pv.txt""",
r"""With the claim's `VOLUME`.""",
run=r"""kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt"""),
cmd(r"""k -n %{ns} scale deploy %{dep} --replicas=0""", r"""A claim's class and volume cannot be edited, and a claim in use is not deleted (pvc-protection): stop the Pod first."""),
hand(r"""k -n %{ns} get pods""",
r"""Again until the Pod has gone.""",
run=r"""kubectl -n %{ns} wait --for=delete pod -l app=%{dep} --timeout=120s"""),
cmd(r"""k -n %{ns} delete pvc %{pvc}""", r"""Its volume goes with it: reclaim policy `Delete`."""),
hand(r"""vim pvc.yaml""",
r"""`storageClassName: ""` - no class: never the default, never provisioned - and the volume by name:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
metadata:
name: %{pvc}
namespace: %{ns}
spec:
storageClassName: "" # "no class": never the default, never provisioned
storageClassName: ""
volumeName: %{pv}
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 500Mi}}
EOF
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
```
`volumeName` alone would not do: the default class is still filled in, and it does not match the class-less volume.""",
run=r"""cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
spec:
storageClassName: ""
volumeName: %{pv}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
EOF"""),
cmd(r"""k apply -f pvc.yaml
k -n %{ns} scale deploy %{dep} --replicas=1
k -n %{ns} logs deploy/%{dep} # ledger %{tok} balanced
# storageClassName has three states: a name; "" = only volumes without a class, no provisioning;
# and absent = "give me the default class". volumeName alone would not have helped: the default
# class is still filled in, and it does not match the class-less volume.
""",
k -n %{ns} rollout status deploy %{dep}
k -n %{ns} logs deploy/%{dep}""", r"""`ledger %{tok} balanced`.""")],
},
# ------------------------------------------------------------------ 119 ----
{
@ -492,34 +606,120 @@ eq "$(j pv %{{p}}-{u} '{{.metadata.uid}}')" "$(sed -n {n}p "$PRIV/q119.uids")" |
eq "$(j pv %{{p}}-{u} '{{.spec.capacity.storage}}/{{.spec.accessModes[*]}}/{{.spec.storageClassName}}/{{.metadata.labels.use}}')" {size}/ReadWriteOnce/manual/{u}""")
for n, (u, size) in enumerate([("reports", "1Gi"), ("archive", "2Gi"), ("media", "1Gi")], 1)
],
"solution": """
k -n %{ns} get pvc; k get pv --show-labels
k -n %{ns} describe pvc reports archive media | grep -E '^Name:|Warning|Normal'
# media: storageclass.storage.k8s.io "manuel" not found -> class (a typo for manual)
# reports, archive: no persistent volumes available for this claim -> compare claim and volume:
k -n %{ns} get pvc -o custom-columns=NAME:.metadata.name,SIZE:.spec.resources.requests.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
k get pv -o custom-columns=NAME:.metadata.name,SIZE:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
# reports asks for 5Gi, its volume has 1Gi -> capacity
# archive asks for ReadWriteMany, its volume offers ReadWriteOnce -> accessmode
printf 'reports capacity\\narchive accessmode\\nmedia class\\n' > {WS}/q119/reasons.txt
# size (downwards), access modes and class of a claim cannot be edited: recreate the three
k -n %{ns} delete pvc reports archive media
for c in "reports 1Gi" "archive 2Gi" "media 1Gi"; do set -- $c; cat <<EOF | k apply -f -
"solution": [cmd(r"""k -n %{ns} get pvc
k -n %{ns} describe pvc reports archive media | grep -e ^Name: -e Warning -e Normal""", r"""`media`: `storageclass.storage.k8s.io "manuel" not found` - the class, a typo. The other two only say no volume fits: compare claim and volume yourself."""),
cmd(r"""k -n %{ns} get pvc -o custom-columns=NAME:.metadata.name,SIZE:.spec.resources.requests.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
k get pv -o custom-columns=NAME:.metadata.name,SIZE:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName""", r"""`reports` asks for 5Gi of a 1Gi volume; `archive` for `ReadWriteMany`, its volume offers `ReadWriteOnce`. A claim binds only when the class matches, the volume is large enough, it has every access mode asked for, and the selector matches."""),
hand(r"""vim {WS}/q119/reasons.txt""",
r"""```
reports capacity
archive accessmode
media class
```""",
run=r"""printf 'reports capacity\narchive accessmode\nmedia class\n' > {WS}/q119/reasons.txt"""),
cmd(r"""k -n %{ns} delete pvc reports archive media""", r"""None of the three fields can be edited on a claim: recreate them."""),
hand(r"""vim pvcs.yaml""",
r"""The three claims again, each keeping its selector:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: $1, namespace: %{ns}}
metadata:
name: reports
namespace: %{ns}
spec:
storageClassName: manual
accessModes: [ReadWriteOnce]
selector: {matchLabels: {use: $1}}
resources: {requests: {storage: $2}}
EOF
done
k -n %{ns} get pvc # all Bound, each to its own volume
# A claim binds a volume only if ALL of these hold: same class, the volume is at least as large,
# it offers every access mode the claim asks for, and it matches the claim's selector. The event
# only names the class problem - for the rest you compare the two objects yourself.
""",
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: reports
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archive
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: archive
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: media
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: media
resources:
requests:
storage: 1Gi
```""",
run=r"""cat > pvcs.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: reports
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: reports
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archive
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: archive
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: media
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: media
resources:
requests:
storage: 1Gi
EOF"""),
cmd(r"""k apply -f pvcs.yaml
k -n %{ns} get pvc""", r"""All three `Bound`, each to its own volume.""")],
},
# ------------------------------------------------------------------ 120 ----
{
@ -605,24 +805,20 @@ done; exit 1"""),
eq "$(j pv %{pv} '{.metadata.uid}')" "$(sed -n 1p "$PRIV/q120.uids")" && eq "$(j -n %{ns} pvc %{dep}-data '{.metadata.uid}')" "$(sed -n 2p "$PRIV/q120.uids")" \
&& eq "$(pvnode %{pv})/$(pvpath %{pv})" "$W1//mnt/%{pv}" """),
],
"solution": """
k -n %{ns} describe pod -l app=%{dep} | tail -5
# 0/3 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s)
# didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s). ...
k -n %{ns} get events --field-selector reason=FailedScheduling -o jsonpath='{.items[-1:].message}' > {WS}/q120/event.txt
k get pv %{pv} -o jsonpath='{.spec.nodeAffinity}'; echo # the volume is on {W1}
k -n %{ns} get deploy %{dep} -o jsonpath='{.spec.template.spec.nodeSelector}'; echo # disk=ssd
k get nodes -L disk # only {W2} carries disk=ssd
# The volume cannot move, so the Pod has to: drop the nodeSelector (the volume's own node
# affinity already sends the Pod to the right node) ...
k -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
# ... or, if the selector has a reason to exist, give {W1} the label: k label node {W1} disk=ssd
k -n %{ns} rollout status deploy %{dep}; k -n %{ns} logs deploy/%{dep}
# The scheduler ANDs every constraint: taints, the Pod's nodeSelector/affinity, and the node
# affinity of each volume it mounts. "didn't match PersistentVolume's node affinity" (before v1.35:
# "had volume node affinity conflict") always means: the Pod is not allowed onto the one node its
# volume lives on.
""",
"solution": [hand(r"""k -n %{ns} describe pod -l app=%{dep} | grep FailedScheduling > {WS}/q120/event.txt
cat {WS}/q120/event.txt""",
r"""`1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) didn't match Pod's node affinity/selector`: the scheduler ANDs every constraint - taints, the Pod's selector, and the node affinity of each volume it mounts.""",
run=r"""for i in $(seq 1 40); do
kubectl -n %{ns} describe pod -l app=%{dep} | grep FailedScheduling > {WS}/q120/event.txt && break; sleep 3
done"""),
cmd(r"""k describe pv %{pv} | grep -A3 'Node Affinity'
k -n %{ns} get deploy %{dep} -o yaml | grep -A1 nodeSelector
k get nodes -L disk""", r"""The volume is on `{W1}`; the Pod must go to `disk=ssd`, which only `{W2}` has. The volume cannot move, so the Pod has to."""),
hand(r"""k -n %{ns} edit deploy %{dep}""",
r"""Delete the two lines of the `nodeSelector` (`nodeSelector:` and `disk: ssd`), then `:wq` - the volume's node affinity already sends the Pod to the right node. (If the selector has a reason to exist, `k label node {W1} disk=ssd` instead.)""",
run=r"""kubectl -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'"""),
cmd(r"""k -n %{ns} rollout status deploy %{dep}
k -n %{ns} logs deploy/%{dep}""", r"""`orders %{tok} open`.""")],
},
# ------------------------------------------------------------------ 121 ----
{
@ -882,19 +1078,15 @@ running %{ns} %{pod} 60"""),
C("class.txt holds %{sc}", 1, STOR + r"""
eq "$(j -n %{ns} pvc %{pvc} '{.spec.storageClassName}')" %{sc} && eq "$(one "$WS/q123/class.txt")" %{sc}"""),
],
"solution": """
k get sc # no "(default)" behind any name
k -n %{ns} describe pvc %{pvc} | tail -2 # no persistent volumes available for this claim and no storage class is set
k patch sc %{sc} -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
k get sc # %{sc} (default)
k -n %{ns} get pvc %{pvc} -w # within seconds: STORAGECLASS %{sc}, then Bound once the Pod is scheduled
k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.storageClassName}' > {WS}/q123/class.txt
# A claim without storageClassName gets the default class when it is created. If there is none
# it stays Pending with the field unset - and is given the default as soon as one appears: the
# claim does not have to be recreated. Marking `standard` instead would have bound it as well,
# to the wrong class. With two defaults, new claims get the most recently created one - never
# leave two.
""",
"solution": [cmd(r"""k get sc
k -n %{ns} describe pvc %{pvc} | tail -2""", r"""No `(default)` anywhere, and the claim waits: `no storage class is set`."""),
cmd(r"""k annotate sc %{sc} storageclass.kubernetes.io/is-default-class=true
k get sc""", r"""`%{sc} (default)`. A claim without a class gets the default when it is created - or, if there is none, as soon as one appears: no need to recreate it."""),
hand(r"""k -n %{ns} get pvc,pod""",
r"""Again until the claim shows `STORAGECLASS %{sc}` and `Bound`, and the Pod `Running`.""",
run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.status.phase}')" = Bound ] && break; sleep 3; done
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
cmd(r"""echo %{sc} > {WS}/q123/class.txt""", r"""Marking `standard` would have bound it too - to the wrong class. With two defaults, new claims take the newest; never leave two.""")],
},
# ------------------------------------------------------------------ 124 ----
{
@ -937,53 +1129,116 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.nodeName}')" "$W2" || exit 1
[ -z "$(j -n %{ns} pod %{pod} '{.spec.nodeSelector}{.spec.affinity}')" ] || exit 1
eq "$($K -n %{ns} exec %{pod} -- cat /disk/disk.id 2>/dev/null | tr -d '\r')" "disk %{tok}" """),
],
"solution": """
cat <<EOF | k apply -f -
"solution": [hand(r"""vim local.yaml""",
r"""The docs page *Storage Classes* (under *Local*) has the class, *Persistent Volumes* a `local` volume with its required node affinity:
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: {name: %{sc}}
metadata:
name: %{sc}
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata: {name: %{pv}}
metadata:
name: %{pv}
spec:
capacity: {storage: 1Gi}
accessModes: [ReadWriteOnce]
storageClassName: %{sc}
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
local: {path: /mnt/disks/%{pv}}
nodeAffinity: # required for a local volume
storageClassName: %{sc}
local:
path: /mnt/disks/%{pv}
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions: [{key: kubernetes.io/hostname, operator: In, values: [{W2}]}]
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- {W2}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
metadata:
name: %{pvc}
namespace: %{ns}
spec:
storageClassName: %{sc}
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```""",
run=r"""cat > local.yaml <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: %{sc}
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
kind: PersistentVolume
metadata:
name: %{pv}
spec:
volumes: [{name: disk, persistentVolumeClaim: {claimName: %{pvc}}}]
containers:
- name: prom
image: busybox:1.36
command: [sleep, "86400"]
volumeMounts: [{name: disk, mountPath: /disk}]
EOF
k -n %{ns} get pod %{pod} -o wide # on {W2}
k -n %{ns} exec %{pod} -- cat /disk/disk.id
# hostPath is a directory on whatever node the Pod happens to land on; a `local` volume knows
# which node it is on, and the scheduler follows it. WaitForFirstConsumer delays binding until a
# Pod needs the claim, so that the Pod's other constraints are weighed together with the volume's.
# The page to search for: "Storage Classes" -> Local.
""",
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: %{sc}
local:
path: /mnt/disks/%{pv}
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- {W2}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
spec:
storageClassName: %{sc}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF"""),
cmd(r"""k apply -f local.yaml
k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: %{pod}`:
```
volumeMounts:
- name: disk
mountPath: /disk
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: disk
persistentVolumeClaim:
claimName: %{pvc}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "disk", "mountPath": "/disk"}]
| .spec.volumes = [{"name": "disk", "persistentVolumeClaim": {"claimName": "%{pvc}"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod %{pod} -o wide""",
r"""Again until `Running` - on `{W2}`, sent there by the volume alone. A `hostPath` is a directory on whatever node the Pod lands on; a `local` volume knows its node, and the scheduler follows it.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
cmd(r"""k -n %{ns} exec %{pod} -- cat /disk/disk.id""")],
},
]

3
cka/questions.py
View File

@ -667,7 +667,8 @@ _ids = [q["id"] for q in QUESTIONS]
assert len(_ids) == len(set(_ids)), "duplicate question ids"
from cka.solutions import S as _SOL
for _q in QUESTIONS:
_q["solution"] = (_q.get("solution") or _SOL.get(_q["id"], "")).strip()
_s = _q.get("solution") or _SOL.get(_q["id"], "")
_q["solution"] = _s if isinstance(_s, list) else _s.strip() # a list: parts (sim/bank.py cmd, hand)
_q.setdefault("set", None)
_missing = [q["id"] for q in QUESTIONS if not q["solution"]]
assert not _missing, f"tasks without a solution: {_missing}"

2905
cka/solutions.py
View File

File diff suppressed because it is too large Load Diff

502
cka/solve.py
View File

@ -8,6 +8,10 @@ substituted as in the task. A task without a script is only audited for a clean
A command that reads stdin (ssh, kubectl run -i) would swallow the rest of the script: ssh -n,
</dev/null.
A task whose reference solution is in parts (sim/bank.py: cmd, hand) has no script here: the audit
runs the parts - what is typed as it stands, and for each edit by hand its script - so the solution
the candidate reads is the one that is proven.
SOLVE: task id -> bash
"""
@ -25,142 +29,6 @@ cpwait() {
SOLVE = {
# ---- 114-124: bank_storage.py
114: r"""
cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: {name: %{sc}}
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: %{sc}
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 300Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
terminationGracePeriodSeconds: 2
volumes: [{name: data, persistentVolumeClaim: {claimName: %{pvc}}}]
containers:
- name: writer
image: busybox:1.36
command: [sh, -c, "echo %{text} > /data/keep.txt; sleep 86400"]
volumeMounts: [{name: data, mountPath: /data}]
EOF
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt
kubectl -n %{ns} delete pod %{pod}
kubectl -n %{ns} delete pvc %{pvc}
sleep 4
kubectl get pv $(cat {WS}/q114/pv.txt) -o jsonpath='{.status.phase}' > {WS}/q114/phase.txt
""",
115: r"""
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
volumes:
- name: scratch
ephemeral:
volumeClaimTemplate:
spec:
storageClassName: standard
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
- name: cache
emptyDir: {medium: Memory, sizeLimit: 32Mi}
containers:
- name: encoder
image: busybox:1.36
command: [sleep, "86400"]
volumeMounts:
- {name: scratch, mountPath: /scratch}
- {name: cache, mountPath: /cache}
EOF
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
kubectl -n %{ns} get pvc -o jsonpath='{.items[0].metadata.name}' > {WS}/q115/claim.txt
kubectl -n %{ns} get pvc %{pod}-scratch -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}' > {WS}/q115/owner.txt
""",
116: r"""
kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'
kubectl -n %{ns} scale sts %{sts} --replicas=1
for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc --no-headers | wc -l)" = 1 ] && break; sleep 3; done
kubectl -n %{ns} get pvc -o name | cut -d/ -f2 > {WS}/q116/claims.txt
""",
117: r"""
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata: {name: %{pv}}
spec:
capacity: {storage: 1Gi}
accessModes: [ReadWriteOnce]
storageClassName: manual
persistentVolumeReclaimPolicy: Retain
hostPath: {path: /mnt/%{pv}}
claimRef: {namespace: %{ns}, name: %{pvc}}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: manual
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
EOF
sleep 5
""",
118: r"""
kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt
kubectl -n %{ns} scale deploy %{dep} --replicas=0
kubectl -n %{ns} wait --for=delete pod -l app=%{dep} --timeout=120s
kubectl -n %{ns} delete pvc %{pvc}
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: ""
volumeName: %{pv}
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 500Mi}}
EOF
kubectl -n %{ns} scale deploy %{dep} --replicas=1
kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
""",
119: r"""
printf 'reports capacity\narchive accessmode\nmedia class\n' > {WS}/q119/reasons.txt
kubectl -n %{ns} delete pvc reports archive media
for c in "reports 1Gi" "archive 2Gi" "media 1Gi"; do set -- $c; cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: $1, namespace: %{ns}}
spec:
storageClassName: manual
accessModes: [ReadWriteOnce]
selector: {matchLabels: {use: $1}}
resources: {requests: {storage: $2}}
EOF
done
sleep 5
""",
120: r"""
for i in $(seq 1 40); do
M=$(kubectl -n %{ns} get events --field-selector reason=FailedScheduling -o jsonpath='{.items[-1:].message}')
[ -n "$M" ] && break; sleep 3
done
echo "$M" > {WS}/q120/event.txt
kubectl -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
""",
121: r"""
kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.metadata.finalizers[0]}' > {WS}/q121/finalizer.txt
echo deployment/%{dep} > {WS}/q121/used-by.txt
@ -176,55 +44,6 @@ for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.sta
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
echo rancher.io/local-path > {WS}/q122/provisioner.txt
""",
123: r"""
kubectl annotate sc %{sc} storageclass.kubernetes.io/is-default-class=true
for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.status.phase}')" = Bound ] && break; sleep 3; done
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.storageClassName}' > {WS}/q123/class.txt
""",
124: r"""
cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: {name: %{sc}}
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata: {name: %{pv}}
spec:
capacity: {storage: 1Gi}
accessModes: [ReadWriteOnce]
storageClassName: %{sc}
persistentVolumeReclaimPolicy: Retain
local: {path: /mnt/disks/%{pv}}
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions: [{key: kubernetes.io/hostname, operator: In, values: [{W2}]}]
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: %{sc}
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
volumes: [{name: disk, persistentVolumeClaim: {claimName: %{pvc}}}]
containers:
- name: prom
image: busybox:1.36
command: [sleep, "86400"]
volumeMounts: [{name: disk, mountPath: /disk}]
EOF
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
""",
# ---- 125-128: bank_net.py
125: r"""
kubectl -n kube-system get cm coredns -o yaml > {WS}/q125/coredns-backup.yaml
@ -233,36 +52,6 @@ kubectl -n kube-system get cm coredns -o json | jq --arg add "$ADD" '.data.Coref
kubectl -n kube-system rollout restart deploy coredns
kubectl -n kube-system rollout status deploy coredns --timeout=240s
""",
126: r"""
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: %{gw}, namespace: %{ns}}
spec:
gatewayClassName: %{gc}
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: %{host}.example.com
tls:
mode: Terminate
certificateRefs: [{kind: Secret, name: %{host}-tls}]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: %{host}, namespace: %{ns}}
spec:
parentRefs: [{name: %{gw}}]
hostnames: [%{host}.example.com]
rules:
- matches: [{path: {type: PathPrefix, value: /api}}]
backendRefs: [{name: api, port: 8080}]
- matches: [{path: {type: PathPrefix, value: /}}]
backendRefs: [{name: web, port: 80}]
EOF
kubectl -n %{ns} delete ingress %{host}
""",
127: r"""
kubectl annotate ingressclass %{old} ingressclass.kubernetes.io/is-default-class-
kubectl annotate ingressclass %{cls} ingressclass.kubernetes.io/is-default-class=true
@ -272,16 +61,6 @@ kubectl -n %{ns} patch ingress %{ing} --type=json -p '[
{"op": "replace", "path": "/spec/rules/0/http/paths/1/pathType", "value": "Prefix"}]'
kubectl apply -f {WS}/q127/blog-ingress.yaml
""",
128: r"""
kubectl -n %{ns} patch ds %{ds} --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/ports/0/hostPort", "value": %{hp}}]'
kubectl -n %{ns} rollout status ds %{ds} --timeout=240s
yq -i '.spec.dnsPolicy = "ClusterFirstWithHostNet"' {WS}/q128/collector.yaml
kubectl -n %{ns} delete pod %{pod} --grace-period=1
kubectl apply -f {WS}/q128/collector.yaml
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=180s
kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt
sleep 8
""",
# ---- 129-131: bank_tshoot.py
129: r"""
ssh -n {W1} 'sudo systemctl enable --now containerd; echo "runtime-endpoint: unix:///run/containerd/containerd.sock" | sudo tee /etc/crictl.yaml >/dev/null'
@ -321,25 +100,10 @@ sudo mkdir -p /opt/cka
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/cka/etcd-snapshot.db
""",
5: r"""
ssh -n {W3} 'sudo tee /etc/kubernetes/manifests/static-web.yaml >/dev/null <<EOF
apiVersion: v1
kind: Pod
metadata: {name: static-web, namespace: default}
spec:
containers: [{name: web, image: "nginx:1.27"}]
EOF'
for i in $(seq 1 60); do [ "$(kubectl get pod static-web-{W3} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
""",
17: r"""
ssh -n {W2} 'sudo systemctl enable --now kubelet'
for i in $(seq 1 40); do [ "$(kubectl get node {W2} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
18: r"""
kubectl -n troubleshoot set image deploy payments payments=nginx:1.27
kubectl -n troubleshoot patch svc payments -p '{"spec":{"selector":{"app":"payments"},"ports":[{"port":80,"targetPort":80}]}}'
kubectl -n troubleshoot rollout status deploy payments --timeout=240s
""",
20: CPWAIT + r"""
sudo sed -i 's/kube-schedulerr/kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
cpwait kube-scheduler
@ -359,44 +123,6 @@ kubectl -n kube-system rollout status deploy coredns --timeout=240s
sudo sed -i '/--leader-elect-lease-duration=1s/d' /etc/kubernetes/manifests/kube-controller-manager.yaml
cpwait kube-controller-manager
""",
39: r"""
ssh -n {W1} "sudo sed -i 's|ca-bundle.crt|ca.crt|' /var/lib/kubelet/config.yaml; sudo systemctl restart kubelet; sudo systemctl enable kubelet"
for i in $(seq 1 40); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
45: CPWAIT + r"""
sudo etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore
sudo sed -i 's|path: /var/lib/etcd$|path: /var/lib/etcd-restore|' /etc/kubernetes/manifests/etcd.yaml
sleep 30
for i in $(seq 1 80); do kubectl -n default get cm %{cm} >/dev/null 2>&1 && break; sleep 3; done
for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
""",
46: r"""
N=$(kubectl get nodes -l tier=edge -o jsonpath='{.items[0].metadata.name}')
D=$(ssh -n $N "sudo awk '/^staticPodPath:/ {print \$2}' /var/lib/kubelet/config.yaml")
kubectl create ns %{ns} 2>/dev/null
ssh -n $N "sudo tee $D/%{pod}.yaml >/dev/null <<EOF
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}, labels: {role: edge}}
spec:
containers:
- name: web
image: nginx:1.27
resources: {requests: {cpu: 50m, memory: 32Mi}}
EOF"
for i in $(seq 1 60); do [ "$(kubectl -n %{ns} get pod %{pod}-$N -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
""",
51: r"""
for r in $(kubectl -n %{ns} rollout history deploy %{dep} | awk '/^[0-9]/{print $1}'); do
kubectl -n %{ns} rollout history deploy %{dep} --revision=$r | grep -q 'nginx:1.25' && N=$r
done
kubectl -n %{ns} rollout undo deploy %{dep} --to-revision=$N
kubectl -n %{ns} scale deploy %{dep} --replicas=3
kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
kubectl -n %{ns} rollout pause deploy %{dep}
kubectl -n %{ns} set image deploy/%{dep} $(kubectl -n %{ns} get deploy %{dep} -o jsonpath='{.spec.template.spec.containers[0].name}')=nginx:1.27
kubectl -n %{ns} set env deploy/%{dep} RELEASE=next
""",
57: r"""
kubectl -n kube-system delete netpol maintenance-lockdown
kubectl -n kube-system scale deploy coredns --replicas=2
@ -404,11 +130,6 @@ kubectl -n kube-system get cm coredns -o json | jq '.data.Corefile |= sub("forwa
kubectl -n kube-system rollout restart deploy coredns
kubectl -n kube-system rollout status deploy coredns --timeout=240s
""",
60: r"""
ssh -n {W2} "sudo sed -i 's|ca-wrong.crt|ca.crt|' /var/lib/kubelet/config.yaml; sudo systemctl enable --now kubelet"
kubectl uncordon {W2}
for i in $(seq 1 40); do [ "$(kubectl get node {W2} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
61: CPWAIT + r"""
sudo sed -i 's/kube-schedulerr/kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
sudo sed -i 's|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)-broken|\1|' /etc/kubernetes/manifests/kube-controller-manager.yaml
@ -421,17 +142,6 @@ for i in $(seq 1 80); do kubectl get nodes >/dev/null 2>&1 && break; sleep 3; do
for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
sudo kubeadm certs check-expiration 2>/dev/null | grep '^apiserver ' > {WS}/q65/expiry.txt
""",
66: CPWAIT + r"""
sudo sed -i 's/kube-apiserverr/kube-apiserver/' /etc/kubernetes/manifests/kube-apiserver.yaml
for i in $(seq 1 80); do kubectl get nodes >/dev/null 2>&1 && break; sleep 3; done
for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
""",
67: r"""
T=$(sudo kubeadm token create --ttl 2h)
H=$(sudo cat /etc/kubernetes/pki/ca.crt | openssl x509 -pubkey | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')
E=$(kubectl get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}')
printf 'JOIN_TOKEN=%s\nCA_HASH=sha256:%s\nAPI_ENDPOINT=%s\n' "$T" "$H" "$E" > {WS}/q67/node-join.env
""",
70: r"""
kubectl -n driver-onboard set image deploy/onboard-app onboard=nginx:1.25
kubectl -n driver-onboard patch deploy onboard-app --type=json -p '[
@ -439,15 +149,6 @@ kubectl -n driver-onboard patch deploy onboard-app --type=json -p '[
{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/"}]'
kubectl -n driver-onboard rollout status deploy onboard-app --timeout=240s
""",
71: r"""
kubectl -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
kubectl -n dispatch-hub rollout status deploy dispatch-api --timeout=240s
for i in 1 2 3 4 5; do
kubectl -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt </dev/null 2>/dev/null
grep -qi nginx {WS}/q71/test-result.txt && break; kubectl -n dispatch-hub delete pod probe --ignore-not-found >/dev/null 2>&1; sleep 3
done
kubectl -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt
""",
76: r"""
kubectl -n trip-tracker get deploy trip-logger -o json | jq '
.spec.template.spec.volumes += [{"name": "config-volume", "emptyDir": {}}]
@ -467,79 +168,23 @@ kubectl -n surge-pricing get deploy surge-engine -o json | jq '
| .spec.template.spec.volumes += [{"name": "config-vol", "configMap": {"name": "surge-config"}}]' | kubectl apply -f -
kubectl -n surge-pricing rollout status deploy surge-engine --timeout=240s
""",
81: r"""
kubectl -n fare-engine delete pvc fare-data-pvc
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: fare-data-pvc, namespace: fare-engine}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
storageClassName: standard
EOF
kubectl label node {W1} disk=nvme
kubectl -n fare-engine rollout status deploy fare-calculator --timeout=240s
""",
84: r"""
kubectl -n booking-tls get cm booking-nginx-config -o json | jq '.data |= with_entries(.value |= sub("ssl_protocols TLSv1.2 TLSv1.3;"; "ssl_protocols TLSv1.3;"))' | kubectl apply -f -
kubectl -n booking-tls rollout restart deploy booking-nginx
kubectl -n booking-tls rollout status deploy booking-nginx --timeout=240s
echo 'ssl_protocols TLSv1.3;' > {WS}/q84/ssl-line.txt
""",
86: r"""
cat > {WS}/q86/controlplane.txt <<'EOF'
kubelet: process
kube-apiserver: static-pod
kube-scheduler: static-pod
kube-controller-manager: static-pod
etcd: static-pod
dns: deployment coredns
EOF
""",
87: r"""
C=$(kubectl get nodes --no-headers -l node-role.kubernetes.io/control-plane | wc -l)
W=$(kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' | wc -l)
CIDR=$(kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -o 'service-cluster-ip-range=[^ ]*' | head -1 | cut -d= -f2)
F=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
printf '1: %s\n2: %s\n3: %s\n4: kindnet, %s\n5: -{W3}\n' "$C" "$W" "$CIDR" "$F" > {WS}/q87/cluster-info.txt
""",
88: r"""
kubectl -n oslo-web run gatekeeper --image=busybox:1.36 --dry-run=client -o json -- sleep 86400 | jq '.spec.containers[0].readinessProbe = {"exec": {"command": ["sh", "-c", "wget -T2 -qO- http://ledger-api:8080"]}, "initialDelaySeconds": 5, "periodSeconds": 5}' | kubectl apply -f -
sleep 12
kubectl -n oslo-web expose deploy ledger --name=ledger-api --port=8080 --target-port=80
kubectl -n oslo-web wait --for=condition=Ready pod/gatekeeper --timeout=180s
""",
90: r"""
cat > {WS}/q90/cluster-events.sh <<'EOF'
kubectl get events -A --sort-by=.metadata.creationTimestamp
EOF
chmod +x {WS}/q90/cluster-events.sh
P=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system delete pod $P
for i in $(seq 1 40); do
[ "$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].status.phase}' 2>/dev/null)" = Running ] && break; sleep 3
done
sleep 5
kubectl -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
ssh -n {W2} 'sudo crictl stop $(sudo crictl ps --name kube-proxy -q)'
sleep 25
kubectl -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
echo container > {WS}/q90/restarts.txt
""",
92: r"""
ssh -n {W1} "sudo sed -i 's|^ExecStart=/usr/local/bin/kubelet |ExecStart=/usr/bin/kubelet |' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; sudo systemctl daemon-reload; sudo systemctl restart kubelet"
echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q92/fixed-file.txt
for i in $(seq 1 40); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
94: r"""
kubectl run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon --dry-run=client -o yaml \
| yq '.spec.containers[0].resources = {"requests": {"cpu": "10m", "memory": "20Mi"}}' | sudo tee /etc/kubernetes/manifests/ops-beacon.yaml >/dev/null
for i in $(seq 1 60); do [ "$(kubectl -n default get pod ops-beacon-{CP} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
kubectl -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
kubectl -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt
sleep 5
""",
95: r"""
CI=$(ssh -n {W2} 'sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -issuer' | sed 's/^issuer= *//')
SI=$(ssh -n {W2} 'sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -issuer' | sed 's/^issuer= *//')
@ -587,47 +232,11 @@ kubectl -n %{ns} patch deploy %{dep} --type=json -p '[
{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/%{ready}"}]'
kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
""",
107: r"""
ssh -n {W3} "printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/%{name}.conf >/dev/null
printf 'net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\n' | sudo tee /etc/sysctl.d/%{name}.conf >/dev/null"
IPF=$(ssh -n {W3} 'sudo sysctl -n net.ipv4.ip_forward')
SW=off; [ "$(ssh -n {W3} 'sed 1d /proc/swaps | grep -c .')" -gt 0 ] && SW=on
FS=$(ssh -n {W3} "sudo awk '/^failSwapOn:/ {print \$2}' /var/lib/kubelet/config.yaml"); [ -n "$FS" ] || FS=true
CG=$(ssh -n {W3} "sudo awk '/^cgroupDriver:/ {print \$2}' /var/lib/kubelet/config.yaml")
printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s\n' "$IPF" "$SW" "$FS" "$CG" > {WS}/q107/node.txt
""",
108: r"""
V=$(ssh -n {W2} 'sudo crictl version 2>/dev/null')
RN=$(echo "$V" | awk '/^RuntimeName:/ {print $2}'); RV=$(echo "$V" | awk '/^RuntimeVersion:/ {print $2}')
SOCK=$(ssh -n {W2} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
CID=$(kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#^[a-z-]*://##')
PID=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.info.pid}}' $CID 2>/dev/null")
LOG=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.status.logPath}}' $CID 2>/dev/null")
printf 'runtime=%s %s\nsocket=%s\ncontainer-id=%s\npid=%s\nlog=%s\n' "$RN" "$RV" "$SOCK" "$CID" "$PID" "$LOG" > {WS}/q108/runtime.txt
""",
110: r"""
kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/^controlPlaneEndpoint:/{print $2}' | tr -d '"' > {WS}/q110/endpoint.txt
KEY=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
sudo kubeadm token create --print-join-command --certificate-key $KEY 2>/dev/null | grep 'kubeadm join' > {WS}/q110/join-control-plane.sh
""",
111: r"""
E='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
N=$(kubectl get nodes -l node-role.kubernetes.io/control-plane --no-headers | wc -l)
M=$($E member list | wc -l)
L=$($E member list | awk -F', ' 'NR==1 {print $3}')
S=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
C=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
printf 'control-plane-nodes=%s\netcd-topology=stacked\netcd-members=%s\netcd-leader=%s\nscheduler-leader=%s\ncontroller-manager-leader=%s\n' "$N" "$M" "$L" "$S" "$C" > {WS}/q111/ha.txt
""",
112: r"""
CONF=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
PL=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[].type] | join(",")')
IPAM=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[] | select(.ipam) | .ipam.type][0]')
SOCK=$(ssh -n {W1} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
N=$(kubectl get csidrivers --no-headers 2>/dev/null | wc -l)
PROV=$(kubectl get sc -o json | jq -r '.items[] | select(.metadata.annotations["storageclass.kubernetes.io/is-default-class"] == "true") | .provisioner')
printf 'cni-config=%s\ncni-plugins=%s\ncni-ipam=%s\ncri-socket=%s\ncsi-drivers=%s\ndefault-provisioner=%s\n' "$CONF" "$PL" "$IPAM" "$SOCK" "$N" "$PROV" > {WS}/q112/interfaces.txt
""",
19: r"""
for i in $(seq 1 40); do kubectl -n troubleshoot logs crasher --previous 2>/dev/null | grep -q DB_HOST && break; sleep 3; done
kubectl -n troubleshoot logs crasher --previous > {WS}/q19/crasher.log
@ -644,71 +253,6 @@ KV=$(kubectl get node %{node} -o jsonpath='{.status.nodeInfo.kubeletVersion}')
R=$(kubectl get node %{node} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}')
printf 'apiserver=%s\netcd=%s\nkubelet=%s\nruntime=%s\n' "$A" "$E" "$KV" "$R" > {WS}/q41/versions.txt
""",
59: r"""
kubectl -n %{ns} apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: new-data, namespace: %{ns}}
spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 1Gi}}}
---
apiVersion: batch/v1
kind: Job
metadata: {name: migrate, namespace: %{ns}}
spec:
template:
spec:
restartPolicy: Never
volumes:
- {name: src, persistentVolumeClaim: {claimName: old-data}}
- {name: dst, persistentVolumeClaim: {claimName: new-data}}
containers:
- name: cp
image: busybox:1.36
command: [sh, -c, "cp -a /src/. /dst/ && ls /dst"]
volumeMounts: [{name: src, mountPath: /src}, {name: dst, mountPath: /dst}]
EOF
kubectl -n %{ns} wait --for=condition=complete job/migrate --timeout=240s
kubectl -n %{ns} get pod legacy -o json | jq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .status)
| (.spec.volumes[] | select(.persistentVolumeClaim.claimName == "old-data") | .persistentVolumeClaim.claimName) = "new-data"' > /tmp/legacy.json
kubectl -n %{ns} delete pod legacy --now
kubectl apply -f /tmp/legacy.json
kubectl -n %{ns} delete pod -l job-name=migrate
kubectl -n %{ns} delete pvc old-data
kubectl -n %{ns} wait --for=condition=Ready pod/legacy --timeout=180s
""",
77: r"""
kubectl -n kube-system get ds -o name | sed 's|.*/||' | grep -i kindnet > {WS}/q77/cni.txt
kubectl -n zero-trust run srv --image=nginx:1.27 --labels=app=srv
kubectl -n zero-trust expose pod srv --port 80
kubectl -n zero-trust run cli --image=busybox:1.36 -- sleep 3600
kubectl -n zero-trust wait --for=condition=Ready pod/srv pod/cli --timeout=180s
kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null && echo reachable
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-all, namespace: zero-trust}
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
EOF
sleep 6
if kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null 2>&1; then echo no; else echo yes; fi > {WS}/q77/enforced.txt
kubectl -n zero-trust delete pod srv cli --now
kubectl -n zero-trust delete svc srv
""",
91: r"""
kubectl api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q91/namespaced.txt
for ns in census-north census-south census-east census-west; do echo "$ns $(kubectl -n $ns get role --no-headers | wc -l)"; done | sort -k2 -n | tail -1 > {WS}/q91/most-roles.txt
""",
102: r"""
kubectl -n oslo-files get pvc shared-files -o json | jq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .metadata.annotations, .metadata.finalizers, .status)
| .spec.accessModes = ["ReadWriteOnce"]' > /tmp/pvc.json
kubectl -n oslo-files delete pvc shared-files
kubectl apply -f /tmp/pvc.json
kubectl -n oslo-files rollout restart deploy uploader
kubectl -n oslo-files rollout status deploy uploader --timeout=240s
echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
""",
93: r"""
kubectl -n oslo-secure create secret generic bundle --from-file={WS}/q93/tls-bundle.pem
kubectl -n oslo-secure create secret generic db-login --from-literal=user=svc-ledger --from-literal=pass='%{pw}'
@ -719,48 +263,14 @@ kubectl -n oslo-secure run vault-reader --image=busybox:1.36 --dry-run=client -o
{"name": "APP_PASS", "valueFrom": {"secretKeyRef": {"name": "db-login", "key": "pass"}}}]' | kubectl apply -f -
kubectl -n oslo-secure wait --for=condition=Ready pod/vault-reader --timeout=180s
""",
# ---- 132-145: bank_course.py (the others are solved by their solution as written, below)
132: r"""
apt_get() { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq -y -o Dpkg::Options::=--force-confold "$@" </dev/null >/dev/null; }
sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt_get update
sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null
sudo kubeadm upgrade apply v1.37.1 -y </dev/null | tail -2
kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl >/dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null
sudo systemctl daemon-reload && sudo systemctl restart kubelet
for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done
kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data
ssh {W1} sudo bash <<'EOF'
export DEBIAN_FRONTEND=noninteractive
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq </dev/null
apt-mark unhold kubeadm kubelet kubectl >/dev/null
apt-get install -y -qq kubeadm=1.37.1-1.1 </dev/null >/dev/null
kubeadm upgrade node </dev/null | tail -1
apt-get install -y -qq kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 </dev/null >/dev/null
apt-mark hold kubeadm kubelet kubectl >/dev/null
systemctl daemon-reload && systemctl restart kubelet
EOF
kubectl uncordon {W1}
""",
139: r"""
ssh {W1} sudo bash <<'EOF'
yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
systemctl restart kubelet
EOF
for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
}
# ---- 1-113: the banks carried over from cka-sim.
# For these tasks the reference solution is a script as it stands, so the audit runs exactly the
# text the candidate is shown - and a solution that stops working is found the next time it runs.
AS_WRITTEN = [1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
40, 42, 43, 44, 47, 48, 49, 50, 52, 53, 54, 55, 56, 58, 62, 63, 64, 68, 69, 72, 73, 74, 75, 78, 80, 82, 83,
85, 89, 98, 99, 100, 101, 103, 104, 109, 113,
AS_WRITTEN = [1, 21, 40, 55, 69, 75, 85, 98, 104, 109,
# bank_course.py: written to run as they stand
133, 134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 145, 146, 147]
133]
def _as_written():

36
ksim
View File

@ -22,7 +22,7 @@ from collections.abc import Callable
from datetime import datetime
from typing import Any
VERSION = "2026.10.26.4"
VERSION = "2026.10.27.1"
ROOT = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, ROOT)
from sim import desktop, envs # noqa: E402
@ -96,7 +96,11 @@ def load_track(track_id):
q.setdefault("set", None)
if q["id"] in workers:
q["workers"] = workers[q["id"]]
if isinstance(q.get("solution"), list): # a solution in parts: the audit runs what it shows
if q["id"] in solve:
sys.exit(f"{track_id} task {q['id']}: its solution is in parts, which the audit runs - drop its script from {track_id}/solve.py")
q["solve"] = show_script({"show": q["solution"]})
elif q["id"] in solve:
q["solve"] = solve[q["id"]]
# Every task has a cluster to itself, so nothing it breaks can reach another task: the flags
# that kept such tasks apart on a shared cluster ("breaks", "solo", "disruptive") are void.
@ -124,7 +128,7 @@ def lesson_task(l, objectives):
o = next((o for o in objectives if o["id"] == l["objective"]), None)
return dict(l, lesson=True, domain=o["domain"] if o else "", difficulty="normal", set=None,
checks=[{"name": s["title"], "pts": 1, "script": s["check"]} for s in l["steps"]], weight=len(l["steps"]),
task=l["intro"], solution="\n\n".join(show_md(s) for s in l["steps"] if show_parts(s)))
task=l["intro"], solution=[p for s in l["steps"] for p in show_parts(s)])
def show_parts(s):
@ -143,6 +147,11 @@ def show_script(s):
"""What the audit runs for a step: what you type, and for each part done by hand its script."""
return "\n".join(p["run"] for p in show_parts(s))
def solution_parts(q):
"""A task's reference solution as parts, like a lesson step's: a solution in parts (cmd, hand)
as it stands, one written as a single script as one part."""
return show_parts({"show": q.get("solution") or ""})
def set_questions(name):
return [q for q in QUESTIONS if q.get("set") == name]
@ -455,7 +464,9 @@ def task_text(q, params=None):
return prose(q["task"], params if params is not None else session_params(load_state(), q), q)
def solution_text(q, params=None):
return prose(str(q.get("solution", "")), params if params is not None else session_params(load_state(), q), q)
"""A task's reference solution as you read it (Markdown): each thing to type in a block of its
own, and beneath it what to do there by hand."""
return prose(show_md({"show": solution_parts(q)}), params if params is not None else session_params(load_state(), q), q)
def make_params(q, rng):
fn = q.get("params")
@ -765,12 +776,15 @@ def solution_allowed(st: dict[str, Any] | None, qid: int | str) -> bool:
def cmd_solution(args):
if not args:
die("usage: ./ksim solution N (drills: after grading the task; exams: after submit; --force to peek anyway)")
if args[0] == "lint": # no cluster needed, and no solution shown: only where one has you type a script
names = [a.upper() for a in args[1:]]
return print_solution_lint(solution_lint([q for q in QUESTIONS if not names or str(q["id"]) in names or q["domain"] in names]))
force = "--force" in args; args = [a for a in args if a != "--force"]
q = q_by_id(int(args[0])); st = load_state()
if not solution_allowed(st, int(q["id"])) and not force:
die("Not yet: in a drill, grade the task first (./ksim grade N); in an exam, submit first. --force overrides.")
print(bold(f"Solution - Q{q['id']} {q['title']}")); print("-" * 64)
print(solution_text(q, session_params(st, q))); print("-" * 64)
print(cli_md(solution_text(q, session_params(st, q)))); print("-" * 64)
print(dim(f"One correct way, not the only one. $do = --dry-run=client -o yaml, k = kubectl. Run it on the task's host: ssh {host_of(q)}"))
def cmd_progress(args):
@ -1897,6 +1911,19 @@ def print_lint(problems):
print(bold(f"{steps} step(s) ask you to type what you would not type in the exam.") if steps
else green("Nothing a step asks you to type is a script: every step is done as in the exam."))
def solution_lint(qs):
"""What each task's reference solution has you type, its comments aside, against TYPING_TRAPS.
Returns [(task, why, line)]."""
return [(q, why, line.strip()) for q in qs for p in solution_parts(q) for line in p["type"].splitlines()
if not line.lstrip().startswith("#") for rx, why in TYPING_TRAPS if rx.search(line)]
def print_solution_lint(problems):
for q, why, line in problems:
print(f" {yellow('Q' + str(q['id']))} {why}\n {dim(line[:150])}")
tasks = len({q["id"] for q, _, _ in problems})
print(bold(f"{tasks} solution(s) ask you to type what you would not type in the exam.") if tasks
else green("Nothing a solution asks you to type is a script: every one is done as in the exam."))
def lesson_audit(args):
"""Prove lessons on real clusters: each is seeded on a host of its own; then every step must
fail its check, have its commands run as `candidate`, and pass its check - so a lesson can be
@ -2788,6 +2815,7 @@ def cmd_help(args):
whatever else you named: ./ksim next OBS:3
progress attempted/passed per domain, normal and hard
solution N reference solution (drills: after grading N; exams: after submit)
solution lint [N|DOMAIN..] what the solutions have you type: no scripts, only what you would type in the exam
sim {'|'.join(SETS) or '<set>'}{' ' * max(1, 20 - len('|'.join(SETS)))}fixed simulator set, exam rules, {TRACK['minutes']} min
start [--hard] full exam: {sum(QUOTA.values()) or '?'} tasks sampled by domain ({sum(QUOTA_HARD.values()) or '?'} in hard mode), {TRACK['minutes']} min.
Requires every domain drilled once first (--force skips). Hard mode adds traps,

8
web/index.html
View File

@ -1030,7 +1030,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
<div class="qbody">${render(q.task)}</div>
${on && sess.mode === 'drill' ? '<div class="checkrow"><button class="btn btn-outline-primary btn-sm" id="btnCheck">Check this task</button><span class="muted">grades it per check; its reference solution opens once it is graded</span></div>' : ''}
${show ? `<div class="checks"><b>${r.earned}/${r.total} points</b>${r.checks.map(c => `<div class="${c.pass ? 'ok' : 'no'}">${esc(c.name)} <span class="pts">(${c.pts})</span></div>`).join('')}</div>` : ''}
${q.solution ? `<details class="fold"><summary>Reference solution</summary><pre>${esc(q.solution)}</pre></details>` : ''}
${q.solution ? `<details class="fold"><summary>Reference solution</summary><div class="qbody" style="margin-top:10px">${render(q.solution)}</div></details>` : ''}
${obj ? `<details class="fold"><summary>Objective ${obj.id} · ${esc(obj.text)}</summary><div class="notes">${notesHtml(obj)}</div></details>` : ''}`;
copyable($('#qContent'));
wireRebuild();
@ -1316,8 +1316,9 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
const cls = r.pass ? 't-pass' : (r.earned ? 't-partial' : 't-fail');
return `<div class="qcard ${cls}"><div class="qcard-h"><h3>Question ${i + 1} · Q${q.id} ${esc(q.title)}</h3><span class="qs">${r.earned} of ${r.total}</span></div>
<ul class="vitems">${r.checks.map(c => `<li class="vitem"><span>${esc(c.name)} <span class="muted">(${c.pts})</span></span><span class="vstat ${c.pass ? 'ok' : 'no'}">${c.pass ? '✓' : '✗'}</span></li>`).join('')}</ul>
${q.solution ? `<details class="fold"><summary>Reference solution</summary><pre>${esc(q.solution)}</pre></details>` : '<div class="pad"></div>'}</div>`;
${q.solution ? `<details class="fold"><summary>Reference solution</summary><div class="qbody" style="margin-top:10px">${render(q.solution)}</div></details>` : '<div class="pad"></div>'}</div>`;
}).join('');
copyable($('#resQuestions'));
const missed = S.questions.filter(q => s.results[q.id] && !s.results[q.id].pass).map(q => q.id);
$('#btnDrillMissed').disabled = !missed.length;
$('#btnDrillMissed').onclick = () => start('drill', missed);
@ -1325,8 +1326,9 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
function showAnswers() {
const s = S.session;
$('#answersBody').innerHTML = S.questions.filter(q => q.solution).map((q, i) =>
`<h4>Question ${S.questions.indexOf(q) + 1} · Q${q.id} ${esc(q.title)}${s && s.results[q.id] ? ` <span class="muted">(${s.results[q.id].earned}/${s.results[q.id].total})</span>` : ''}</h4><pre>${esc(q.solution)}</pre>`).join('')
`<h4>Question ${S.questions.indexOf(q) + 1} · Q${q.id} ${esc(q.title)}${s && s.results[q.id] ? ` <span class="muted">(${s.results[q.id].earned}/${s.results[q.id].total})</span>` : ''}</h4><div class="qbody">${render(q.solution)}</div>`).join('')
|| '<p class="muted">No reference solutions are open yet.</p>';
copyable($('#answersBody'));
openModal('#mAnswers');
}