vumast3r/kubestronaut-sim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: vumast3r:main
Branches Tags
vumast3r:main
vumast3r:v2026.10.27.2
vumast3r:v2026.10.27.1
vumast3r:v2026.10.26.4
vumast3r:v2026.10.26.3
vumast3r:v2026.10.26.2
..
compare: vumast3r:v2026.10.26.3
Branches Tags
vumast3r:main
vumast3r:v2026.10.27.2
vumast3r:v2026.10.27.1
vumast3r:v2026.10.26.4
vumast3r:v2026.10.26.3
vumast3r:v2026.10.26.2

No commits in common. "main" and "v2026.10.26.3" have entirely different histories.
main ... v2026.10.2

21 changed files with 3377 additions and 5427 deletions
Show Stats Expand all files Collapse all files

58
CHANGELOG.md
View File

@ -4,64 +4,6 @@ Notable changes to kubestronaut-sim, newest first. A version is the `VERSION` st
(year.month.day.n); `./ksim help` and the top bar of the web page show the one you are running.
Every entry names the commit it came in.
## 2026.10.27.2 — 2026-10-27
Everything below came in `3b72021`.
### Changed
- **The 105 CKA solutions rewritten in 2026.10.27.1 are proven**: `./ksim audit` ran each on a
host of its own on v1.37, as the solution shows it, every edit by hand done by its script. All
105 seed cleanly, score nothing untouched and reach full marks. The README's audit record says
so. No task's setup or checks have changed since the audit of 2026-10-19, nor have the other 42
solutions.
## 2026.10.27.1 — 2026-10-27
Everything below came in `e761d53`.
### Changed
- **Every CKA reference solution shows the exam way**, as the lessons do: a generator and the few
lines to add in `vim`, a manifest pasted from the docs page it names, `kubectl edit`, `ssh NODE`,
or a value you read and `echo` - no heredocs, `sed`, JSON patches, `kubectl wait` or loops. 105
of the 147 were rewritten; the other 42 were commands you type already. The page and `./ksim
solution N` show each thing to type as a block of its own, which a click copies.
- A solution in parts (`cmd`, `hand`) is its own audit script: `./ksim audit` runs what it shows,
and for each edit by hand the script beside it, so the solution you read is the one proven. Their
separate scripts in `cka/solve.py` are gone.
### Added
- `./ksim solution lint [N|DOMAIN..]`: what the reference solutions have you type, against the
list `./ksim lesson lint` uses.
## 2026.10.26.4 — 2026-10-26
Everything below came in `226ef19`.
### Changed
- **Every CKA lesson step shows the exam way** (*Show me how*, `./ksim lesson show`): `k`, a
generator and the few lines to add in `vim`, a manifest pasted from the docs page it names,
`kubectl edit`, `ssh NODE` and `sudo -i`, or a value you read and `echo` - never a script. Each
command is a block of its own, which a click copies. The scripts the audit needs (`yq`, `sed`,
waits) are kept out of sight, one per part done by hand, so `./ksim lesson audit` still proves
every step: 135 of 135 on v1.37.
### Added
- `./ksim lesson lint`: what each step has you type, against what you would not type in the exam
(`sed`, `awk`, `yq`, heredocs, loops, `kubectl wait`, `--overrides`, JSON patches, `jsonpath`
into a file, `kubectl` in full). The lesson audit runs it first.
- A task's or a lesson's host whose setup failed when the session started is marked: the page
says so above it, with **Rebuild this host**, and `./ksim lesson` names `./ksim reset N`. Before,
only the log of the preparing window said so.
### Fixed
- The page shows `*italics*` in lessons and tasks instead of the stars.
## 2026.10.26.3 — 2026-10-26
Everything below came in `95f9d1b`.

50
README.md
View File

@ -155,8 +155,7 @@ host name such as `ckad0013` only resolves in the desktop's terminal: from your
**Lessons.** A lesson teaches one objective as a guided lab on a host of its own: an introduction,
then steps you do in the desktop's terminal, each with a **Check step** button, a hint and **Show me
how** (the exam way: what you type, what you edit in vim), and a wrap-up that sends you to
the objective's timed drill. Lessons are the one
the commands**, and a wrap-up that sends you to the objective's timed drill. Lessons are the one
untimed part of the simulator — they teach, the drills test — and they write nothing to the
history, the statistics or the weak list; a lesson whose every step was checked is marked done on
its objective (`lesson ✓` in the objective list). Steps count in order. Start one from an
@ -200,7 +199,7 @@ Each track is a directory: `cka/`, `ckad/`, `cks/`.
| `bank_*.py` | more tasks; `bank_sim_*` are the fixed simulator sets |
| `curriculum.py` | the exam's objectives in order, each with study notes and the tasks that practise it |
| `solutions.py` | reference solutions (CKA keeps them apart; the others inline them) |
| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved - for the tasks whose reference solution is not in parts |
| `solve.py` | scripted solutions that `./ksim audit` runs on the task's host to prove a task can be solved |
| `lessons.py` | guided labs, one per objective (CKA: all, one `lessons_*.py` per domain · CKAD: 2.1–2.4): introduction, steps with their check, hint and commands, wrap-up |
| Track | Kubernetes | Bank today | Target (17–20 pool tasks per domain) |
@ -213,12 +212,9 @@ Each track is a directory: `cka/`, `ckad/`, `cks/`.
**CKA, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 147 tasks seed cleanly on a host of
their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when
their scripted solution is run on the host as `candidate`. Q132 really upgrades its cluster from
v1.36.4 to v1.37.1, worker included. **On 2026-10-27** the 105 solutions rewritten in parts that
day were audited again: each task seeded cleanly, scored nothing untouched, and reached full marks
from what its solution shows, every edit by hand done by its script. No task's setup or checks have
changed since 2026-10-19. The other 42 solutions are commands with a script in `solve.py`, which
for 11 of them is the solution exactly as written.
their scripted solution is run on the host as `candidate` — for 80 of them that script is the
reference solution exactly as a drill shows it once the task is graded. Q132 really upgrades its
cluster from v1.36.4 to v1.37.1, worker included.
**CKAD, as audited on 2026-10-19** (v1.37, Ubuntu hosts): all 105 tasks seed cleanly on a host of
their own, score nothing when graded untouched 45 seconds after seeding, and reach full marks when
@ -245,8 +241,7 @@ v1.36), CKAD Q1 and CKS Q60, Q61 and Q64 passed again on the rebuilt images.
"title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
"setup": "bash that seeds the scenario on the task's own cluster",
"checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...],
"solution": [cmd("k ... --dry-run=client -o yaml > f.yaml"), # or one string, a script as it stands
hand("vim f.yaml", "what to add, Markdown", run="the same, scripted: yq, a heredoc")],
"solution": "...",
"workers": 2, # optional: worker nodes of its cluster
"k8s": "v1.36"} # optional: a minor of its own (an upgrade task)
```
@ -291,35 +286,22 @@ and the untouched grade, which catches a check that only passes once the seeded
until a stopped kubelet shows as NotReady; `--keep` leaves the host up to look at
(`KUBECONFIG=.state/envs/<host>/kubeconfig`, `./ksim down` removes it).
A reference solution in parts - `cmd()` and `hand()`, as a lesson step's `show` - is its own
script: the audit runs what is typed as it stands, and for each part done by hand its `run`, so the
solution the candidate reads after a drill is the one that is proven, and one that stops working
with a new Kubernetes release fails the next audit. 105 of the 147 CKA solutions are written so
(the page and `./ksim solution N` show each thing to type as a block of its own, which a click
copies, with the edit to make in `vim` beneath it); the other 42 are commands typed as they stand,
each with its script in `solve.py`. `./ksim solution lint` reads what they have you type, against
the same list of things you would not type in the exam as `./ksim lesson lint`. In `cka/solve.py`,
`AS_WRITTEN` lists the few tasks whose reference solution is still one script, which the audit
runs as it stands.
In `cka/solve.py`, `AS_WRITTEN` lists the tasks whose reference solution is a script as it stands:
for those the audit runs exactly the text the candidate is shown, so a solution that stops working
with a new Kubernetes release fails the next audit instead of misleading you after a drill.
**Writing a lesson.** A lesson in `<track>/lessons.py` is a dict with `id` (9000 plus the objective,
`9201` for 2.1 — its host is `ckad9201`, which no task's can be), `objective`, `title`, `workers`,
`intro`, `setup` (bash, as a task's), `steps` and `outro` - and, like a task, `k8s` or `image` if
it needs a host of its own kind (lesson 4.4 runs on v1.36 to upgrade it). Each step comes from
`step(title, text, check, hint, show)` in `sim/bank.py`: `check` runs like a task's check, and
`show` is how you would do the step in the exam, as parts. `cmd(text)` is what you type as it
stands. `hand(text, how, run)` is what you do by hand: what you type to begin (`vim role.yaml`,
`k edit cm app-config`, `ssh NODE` and `sudo -i`, `echo NAME > file`), what to do there, and a script
that does the same unattended. `./ksim lesson audit` seeds each lesson on its own host and, step by
step, requires the check to fail, runs the commands (and each hand part's script) there as
`candidate`, and requires the check to pass, so a lesson can always be done as it shows, and no
step is done before you do it. `./ksim lesson lint` reads what every step has you type and refuses
scripts: no `sed`, `awk`, `yq`, heredocs, loops, `kubectl wait`, `--overrides`, JSON patches or
`jsonpath` into a file, and `k` rather than `kubectl`. In a script, a command on another node is
`ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain `ssh NODE` would read the rest of
the script as its own input. **As audited on 2026-10-26** (v1.37, Ubuntu hosts): CKA's 27 lessons,
done the exam way, 135 of 135 steps proven, and none has you type a script. CKAD 2.1–2.4 (24
steps, audited on 2026-10-19) still show their commands as scripts.
`show` is what the learner would type. `./ksim lesson audit` seeds each lesson on its own host and,
step by step, requires the check to fail, runs `show` there as `candidate`, and requires the check
to pass — so a lesson can always be done exactly as it shows, and no step is done before you do it.
Work on another node is shown as `ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain
`ssh NODE` would read the rest of the commands as its own input. **As audited on 2026-10-19**
(v1.37, Ubuntu hosts): CKA's 27 lessons, 135 of 135 steps, and CKAD 2.1–2.4, 24 of 24 steps,
proven.
## What is different from the real exam

732
cka/bank_course.py
View File

@ -12,8 +12,6 @@ operator with broken RBAC do its job, and 146-147 grow, snapshot, restore and cl
the CSI hostpath driver's class `csi-hostpath-sc`, as the real exam's clusters have it
(ensure_csi_hostpath).
"""
from sim.bank import cmd, hand
from cka.common import C, CAN, NET, NS, ONE, backup_operator, ns_pick, name_pick
_ETCD = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
@ -61,65 +59,38 @@ for n in "$CP" "$W1"; do
done
eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" v1.37.1 && eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" 2"""),
],
"solution": [hand(r"""sudo vim /etc/apt/sources.list.d/kubernetes.list""",
r"""The docs page *Upgrading kubeadm clusters* starts here: pkgs.k8s.io has one repository per minor, so 1.37 packages exist for apt only once the line says `/v1.37/` instead of `/v1.36/`.""",
run=r"""apt_get() { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq -y -o Dpkg::Options::=--force-confold "$@" </dev/null >/dev/null; }
sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list"""),
hand(r"""sudo apt-get update
"solution": """
# the control plane - this host
sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list # one repository per minor
sudo apt-get update
apt-cache madison kubeadm | head -3
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.37.1-1.1
sudo apt-mark hold kubeadm""",
r"""The control plane - this host - first, and kubeadm first: it drives the upgrade.""",
run=r"""apt_get update
sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null"""),
hand(r"""sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.1""",
r"""Answer `y`; it takes a few minutes, and the API server restarts on the new version.""",
run=r"""sudo kubeadm upgrade apply v1.37.1 -y </dev/null | tail -2"""),
cmd(r"""k drain {CP} --ignore-daemonsets --delete-emptydir-data"""),
hand(r"""sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet""",
r"""Then the node's kubelet and kubectl, held again.""",
run=r"""sudo apt-mark unhold kubelet kubectl >/dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null
sudo systemctl daemon-reload && sudo systemctl restart kubelet"""),
hand(r"""k uncordon {CP}
k get nodes""",
r"""Again until `k` answers - the kubelet is restarting. `{CP}` shows `v1.37.1`.""",
run=r"""for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""k drain {W1} --ignore-daemonsets --delete-emptydir-data"""),
hand(r"""{SHELL_W1}
sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.1 -y # a few minutes
k drain {CP} --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
k uncordon {CP}
# the worker
k drain {W1} --ignore-daemonsets --delete-emptydir-data
{SHELL_W1}
sudo -i
vim /etc/apt/sources.list.d/kubernetes.list""",
r"""`/v1.36/` to `/v1.37/` here too.""",
run=r""""""),
hand(r"""apt-get update
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-mark unhold kubeadm kubelet kubectl
apt-get install -y kubeadm=1.37.1-1.1
kubeadm upgrade node
kubeadm upgrade node # a worker: node, not apply
apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
apt-mark hold kubeadm kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
exit
exit""",
r"""A worker runs `kubeadm upgrade node`, not `apply`: it hosts no control plane, only its kubelet's configuration to update.""",
run=r"""ssh {W1} sudo bash <<'EOF'
export DEBIAN_FRONTEND=noninteractive
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq </dev/null
apt-mark unhold kubeadm kubelet kubectl >/dev/null
apt-get install -y -qq kubeadm=1.37.1-1.1 </dev/null >/dev/null
kubeadm upgrade node </dev/null | tail -1
apt-get install -y -qq kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 </dev/null >/dev/null
apt-mark hold kubeadm kubelet kubectl >/dev/null
systemctl daemon-reload && systemctl restart kubelet
EOF"""),
cmd(r"""k uncordon {W1}
k get nodes""", r"""Both `Ready`, both `v1.37.1` - `VERSION` is each kubelet's.""")],
exit; exit
k uncordon {W1}
k get nodes
# docs: "Upgrading kubeadm clusters". pkgs.k8s.io has one repository per minor, so 1.37 packages
# appear only once the line names v1.37. kubeadm first - it drives the upgrade and rewrites the
# kubelet's configuration; kubelet and kubectl after `upgrade apply`, then the restart: `k get
# nodes` shows each kubelet's running version. Workers run `kubeadm upgrade node`.
""",
},
# ------------------------------------------------------------------ 133 ----
{
@ -225,16 +196,19 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.ephemeralContainers[?(@.name=="debugger")].t
&& eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q134.uid") 0" """),
C("mode.txt holds the value the process runs with", 3, ONE + r"""eq "$(one "$WS/q134/mode.txt")" %{mode}"""),
],
"solution": [cmd(r"""k -n %{ns} exec %{pod} -- sh
k -n %{ns} get cm %{pod}-env -o yaml""", r"""No shell in the image. And the ConfigMap says `MODE=open` - now; the environment was read once, when the container started."""),
cmd(r"""k -n %{ns} debug %{pod} --image=busybox:1.36 --target=app -c debugger --profile=general -- sleep 3600""", r"""`--target` puts the debug container into `app`'s process namespace; the `general` profile gives it `SYS_PTRACE`, which reading another process's environment needs. Interactively: `k debug -it ... -- sh`."""),
hand(r"""k -n %{ns} exec %{pod} -c debugger -- ps""",
r"""Again until it answers: PID 1 is `/pause`, `app`'s process.""",
run=r"""for i in $(seq 1 30); do kubectl -n %{ns} exec %{pod} -c debugger -- true 2>/dev/null && break; sleep 2; done"""),
cmd(r"""k -n %{ns} exec %{pod} -c debugger -- cat /proc/1/environ | tr '\0' '\n'""", r"""`MODE=%{mode}`: what the process runs with."""),
hand(r"""echo %{mode} > {WS}/q134/mode.txt""",
r"""With the value after `MODE=`.""",
run=r"""kubectl -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\0" "\n" < /proc/1/environ' | sed -n 's/^MODE=//p' > {WS}/q134/mode.txt""")],
"solution": """
k -n %{ns} exec %{pod} -- sh # fails: no shell in the image
k -n %{ns} get cm %{pod}-env -o yaml # MODE=open - but that is the ConfigMap NOW
k -n %{ns} debug %{pod} --image=busybox:1.36 --target=app -c debugger --profile=general -- sleep 3600
for i in $(seq 1 30); do k -n %{ns} exec %{pod} -c debugger -- true 2>/dev/null && break; sleep 2; done
k -n %{ns} exec %{pod} -c debugger -- ps # PID 1 is /pause, the app's process
k -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\\0" "\\n" < /proc/1/environ' | grep '^MODE='
k -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\\0" "\\n" < /proc/1/environ' | sed -n 's/^MODE=//p' > {WS}/q134/mode.txt
# Environment variables from a ConfigMap are read once, when the container starts: editing the
# ConfigMap changes nothing for a running process. --target puts the debug container into app's
# process namespace, so /proc/1 is app's process; the `general` profile gives it SYS_PTRACE,
# which reading another user's /proc/PID/environ needs. Interactively: `k debug -it ... -- sh`.
""",
},
# ------------------------------------------------------------------ 135 ----
{
@ -279,19 +253,19 @@ tok = [v for v in s.get("volumes", []) if any("serviceAccountToken" in src for s
sys.exit(1 if tok else 0)' || exit 1
done"""),
],
"solution": [cmd(r"""k -n %{ns} create serviceaccount pipeline
"solution": """
k -n %{ns} create serviceaccount pipeline
k -n %{ns} create role pipeline --verb=get,list,create,update,patch --resource=deployments.apps
k -n %{ns} create rolebinding pipeline --role=pipeline --serviceaccount=%{ns}:pipeline
k auth can-i delete deployments -n %{ns} --as=system:serviceaccount:%{ns}:pipeline""", r"""`no`."""),
cmd(r"""k -n %{ns} create token pipeline --duration=%{h}h > {WS}/q135/pipeline.token""", r"""The TokenRequest API issues a short-lived, bound token; nothing is stored in a Secret."""),
hand(r"""k -n %{ns} edit deploy frontend""",
r"""In the Pod template's `spec:`, beside `containers:` and indented like it, then `:wq`:
```
automountServiceAccountToken: false
```
It leaves out the projected volume `kube-api-access-*`. (On the ServiceAccount it works too; the Pod's setting wins.)""",
run=r"""kubectl -n %{ns} patch deploy frontend -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":false}}}}'"""),
cmd(r"""k -n %{ns} rollout status deploy frontend""")],
k auth can-i delete deployments -n %{ns} --as=system:serviceaccount:%{ns}:pipeline # no
k -n %{ns} create token pipeline --duration=%{h}h > {WS}/q135/pipeline.token
k -n %{ns} patch deploy frontend -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":false}}}}'
k -n %{ns} rollout status deploy frontend
# `create token` asks the TokenRequest API for a short-lived, bound token - nothing is stored in a
# Secret. automountServiceAccountToken: false on the Pod template (or on the ServiceAccount; the
# Pod's setting wins) leaves out the projected volume kube-api-access-*. To try a token alone, use
# no kubeconfig: a client certificate in yours would be checked first.
""",
},
# ------------------------------------------------------------------ 136 ----
{
@ -324,20 +298,22 @@ eq "$(x theme)" dark && eq "$(x locale)" en_GB && eq "$(x db/password)" pw-%{mem
x() { $K -n %{ns} exec %{pod} -- cat "/etc/portal/$1" 2>/dev/null; }
x labels | grep -qx 'app="%{pod}"' && eq "$(x mem-limit)" %{mem}"""),
],
"solution": [cmd(r"""k -n %{ns} run %{pod} --image=busybox:1.36 --labels=app=%{pod} --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Replace the container's `resources: {}` with the limit and the mount:
```
"solution": """
k -n %{ns} apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: %{pod}
labels: {app: %{pod}}
spec:
containers:
- name: portal
image: busybox:1.36
command: [sleep, '86400']
resources:
limits:
memory: %{mem}Mi
limits: {memory: %{mem}Mi}
volumeMounts:
- name: portal
mountPath: /etc/portal
readOnly: true
```
and add to the Pod's `spec:`, right below `spec:` (the docs page *Projected Volumes* has an example with all three kinds):
```
- {name: portal, mountPath: /etc/portal, readOnly: true}
volumes:
- name: portal
projected:
@ -347,33 +323,20 @@ and add to the Pod's `spec:`, right below `spec:` (the docs page *Projected Volu
- secret:
name: portal-db
items:
- key: password
path: db/password
- {key: password, path: db/password}
- downwardAPI:
items:
- path: labels
fieldRef:
fieldPath: metadata.labels
fieldRef: {fieldPath: metadata.labels}
- path: mem-limit
resourceFieldRef:
containerName: %{pod}
resource: limits.memory
divisor: 1Mi
```
`items` picks keys and renames them; in a volume a `resourceFieldRef` needs the container's name, and the divisor turns bytes into MiB.""",
run=r"""yq -i '.spec.containers[0].resources = {"limits": {"memory": "%{mem}Mi"}}
| .spec.containers[0].volumeMounts = [{"name": "portal", "mountPath": "/etc/portal", "readOnly": true}]
| .spec.volumes = [{"name": "portal", "projected": {"sources": [
{"configMap": {"name": "portal-config"}},
{"secret": {"name": "portal-db", "items": [{"key": "password", "path": "db/password"}]}},
{"downwardAPI": {"items": [{"path": "labels", "fieldRef": {"fieldPath": "metadata.labels"}},
{"path": "mem-limit", "resourceFieldRef": {"containerName": "%{pod}", "resource": "limits.memory", "divisor": "1Mi"}}]}}]}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod %{pod}""",
r"""Again until it is `Running`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=120s"""),
cmd(r"""k -n %{ns} exec %{pod} -- ls -R /etc/portal
k -n %{ns} exec %{pod} -- cat /etc/portal/labels /etc/portal/mem-limit""")],
resourceFieldRef: {containerName: portal, resource: limits.memory, divisor: 1Mi}
EOF
k -n %{ns} wait --for=condition=Ready pod/%{pod}
k -n %{ns} exec %{pod} -- sh -c 'ls -R /etc/portal; cat /etc/portal/labels /etc/portal/mem-limit'
# `items` picks keys and renames them: only `password` from the Secret, as db/password. A
# resourceFieldRef needs the container's name in a volume (in an env var it is implied), and the
# divisor turns bytes into MiB.
""",
},
# ------------------------------------------------------------------ 137 ----
{
@ -441,19 +404,18 @@ eq "$(j -n %{ns} deploy critical '{.status.availableReplicas} {.status.updatedRe
eq "$(j -n %{ns} deploy critical '{.spec.template.spec.containers[0].resources.requests.cpu} {.spec.template.spec.containers[0].resources.limits.cpu} {.spec.template.spec.containers[0].resources.requests.memory} {.spec.template.spec.containers[0].resources.limits.memory}')" "200m 200m 128Mi 128Mi" || exit 1
for q in $($K -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'); do [ "$q" = Guaranteed ] || exit 1; done"""),
],
"solution": [cmd(r"""k -n %{ns} get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass""", r"""Guaranteed: every container has requests equal to limits, CPU and memory. BestEffort: no container has any. Everything else is Burstable - `delta` too, as one of its containers sets nothing."""),
hand(r"""vim {WS}/q137/qos.txt""",
r"""```
alpha Guaranteed
beta Burstable
gamma BestEffort
delta Burstable
```""",
run=r"""printf 'alpha Guaranteed\nbeta Burstable\ngamma BestEffort\ndelta Burstable\n' > {WS}/q137/qos.txt"""),
cmd(r"""echo gamma > {WS}/q137/first.txt""", r"""Under memory pressure BestEffort goes first, then Burstable Pods using more than they requested, Guaranteed last."""),
cmd(r"""k -n %{ns} set resources deploy critical --requests=cpu=200m,memory=128Mi --limits=cpu=200m,memory=128Mi
"solution": """
k -n %{ns} get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
k -n %{ns} get pods alpha beta delta gamma -o jsonpath='{range .items[*]}{.metadata.name} {.status.qosClass}{"\\n"}{end}' > {WS}/q137/qos.txt
echo gamma > {WS}/q137/first.txt
k -n %{ns} set resources deploy critical --requests=cpu=200m,memory=128Mi --limits=cpu=200m,memory=128Mi
k -n %{ns} rollout status deploy critical
k -n %{ns} get pods -l app=critical -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass""")],
k -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'; echo
# Guaranteed: every container has requests equal to limits for both CPU and memory. BestEffort: no
# container has any. Everything else is Burstable - delta too, because one container sets nothing.
# Under memory pressure the kubelet evicts BestEffort first, then Burstable Pods using more than
# they requested; Guaranteed Pods last.
""",
},
# ------------------------------------------------------------------ 138 ----
{
@ -490,21 +452,15 @@ eq "$(j -n %{ns} pod %{pod} '{.status.containerStatuses[0].resources.requests.cp
eq "$(j -n %{ns} pod %{pod} '{.spec.containers[0].resources.limits.cpu}')" %{lim} \
&& eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q138.uid") 0" """),
],
"solution": [cmd(r"""k -n %{ns} get pod %{pod} -o yaml | grep -A6 resources:"""),
hand(r"""k -n %{ns} edit pod %{pod} --subresource resize""",
r"""Through the `resize` subresource - a plain `k edit` of a Pod's resources is refused. Set the CPU, leave the memory, then `:wq`:
```
resources:
limits:
cpu: %{lim}
memory: 48Mi
requests:
cpu: %{req}
memory: 48Mi
```""",
run=r"""kubectl -n %{ns} patch pod %{pod} --subresource resize --patch '{"spec":{"containers":[{"name":"encoder","resources":{"requests":{"cpu":"%{req}"},"limits":{"cpu":"%{lim}"}}}]}}'"""),
cmd(r"""k -n %{ns} get pod %{pod} -o yaml | grep -A6 resources:
k -n %{ns} get pod %{pod}""", r"""The spec says what you asked for, `status.containerStatuses[].resources` what the kubelet applied; `RESTARTS 0` - a container's `resizePolicy` decides, and CPU needs no restart.""")],
"solution": """
k -n %{ns} get pod %{pod} -o jsonpath='{.spec.containers[0].resources}'; echo
k -n %{ns} patch pod %{pod} --subresource resize --patch '{"spec":{"containers":[{"name":"encoder","resources":{"requests":{"cpu":"%{req}"},"limits":{"cpu":"%{lim}"}}}]}}'
k -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].resources}{"\\n"}{.status.containerStatuses[0].restartCount}{"\\n"}'
# A running Pod's CPU and memory can be changed in place, through the `resize` subresource - a
# plain `k edit` or `k apply` of a Pod's resources is refused. The spec
# says what you asked for; status.containerStatuses[].resources what the kubelet has applied.
# A container's resizePolicy decides whether a resource's change needs a restart (CPU: no).
""",
},
# ------------------------------------------------------------------ 139 ----
{
@ -539,33 +495,24 @@ sys.exit(0 if s == "%{soft}Mi" and secs == 90 else 1)'"""),
nx "$W1" 'grep -q "%{soft}Mi" /var/lib/kubelet/config.yaml && grep -q "%{mem}Mi" /var/lib/kubelet/config.yaml && grep -q "^staticPodPath:" /var/lib/kubelet/config.yaml && systemctl is-active -q kubelet' </dev/null || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True"""),
],
"solution": [hand(r"""{SHELL_W1}
"solution": """
{SHELL_W1}
sudo -i
grep -n -A4 ^eviction /var/lib/kubelet/config.yaml
vim /var/lib/kubelet/config.yaml""",
r"""Under `evictionHard:` set the two hard thresholds, and add the soft one with its grace period at the top level - a soft threshold without a grace period for the same signal, and the kubelet refuses to start:
```
evictionHard:
memory.available: "%{mem}Mi"
nodefs.available: "%{fs}%"
evictionSoft:
memory.available: "%{soft}Mi"
evictionSoftGracePeriod:
memory.available: "1m30s"
```
Keep the other `evictionHard` entries as they are.""",
run=r"""ssh {W1} sudo bash <<'EOF'
yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
grep -n -A4 '^eviction' /var/lib/kubelet/config.yaml # kind sets the hard thresholds to 0%
yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%"
| .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
systemctl restart kubelet
EOF"""),
hand(r"""systemctl restart kubelet
journalctl -u kubelet -n 5 --no-pager
exit
exit""",
r"""No complaint about the configuration; then back on the host.""",
run=r"""for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done"""),
cmd(r"""k get node {W1}
k get --raw /api/v1/nodes/{W1}/proxy/configz | tr ',' '\n' | grep -i eviction""", r"""`Ready`; `configz` shows the configuration the running kubelet uses.""")],
exit; exit
k get --raw /api/v1/nodes/{W1}/proxy/configz | jq '.kubeletconfig | {evictionHard, evictionSoft, evictionSoftGracePeriod}'
# A soft threshold needs a grace period for the same signal, or the kubelet refuses to start. By
# hand: under evictionHard add `memory.available: "%{mem}Mi"` and change nodefs.available; then
# evictionSoft:
# memory.available: "%{soft}Mi"
# evictionSoftGracePeriod:
# memory.available: "1m30s"
# configz shows what the running kubelet uses.
""",
},
# ------------------------------------------------------------------ 140 ----
{
@ -601,24 +548,19 @@ ch=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A
n=$(nx "$CP" "iptables-save -t nat | grep -- '^-A $ch ' | grep -c -- '-j KUBE-SEP-'" </dev/null)
eq "$n" %{n} && eq "$(one "$WS/q140/endpoints.txt")" "$n" """),
],
"solution": [cmd(r"""k -n kube-system get cm kube-proxy -o yaml | grep mode""", r"""`mode: iptables` - an empty mode means the default, iptables too."""),
hand(r"""echo iptables > {WS}/q140/mode.txt""",
r"""The mode it printed.""",
run=r"""m=$(kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"')
echo ${m:-iptables} > {WS}/q140/mode.txt"""),
cmd(r"""k -n %{ns} get svc %{svc}""", r"""Note its `CLUSTER-IP`."""),
hand(r"""sudo iptables-save -t nat | grep 10.96.12.34""",
r"""With that IP: `KUBE-SERVICES` matches the ClusterIP and port and jumps (`-j`) to the Service's own `KUBE-SVC-...` chain.""",
run=r""""""),
hand(r"""echo KUBE-SVC-ABCDEFGHIJKLMNOP > {WS}/q140/chain.txt
sudo iptables-save -t nat | grep -- '-A KUBE-SVC-ABCDEFGHIJKLMNOP '""",
r"""With the chain's name. Its rules pick one endpoint chain at random (`statistic mode random`); each `KUBE-SEP-...` DNATs to one Pod - count them. In nftables mode: `sudo nft list ruleset | grep -A5 %{svc}`.""",
run=r"""ip=$(kubectl -n %{ns} get svc %{svc} -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/q140/chain.txt"""),
hand(r"""echo 3 > {WS}/q140/endpoints.txt""",
r"""With the number of `-j KUBE-SEP-` rules: one per ready endpoint.""",
run=r"""chain=$(cat {WS}/q140/chain.txt)
sudo iptables-save -t nat | grep -- "^-A $chain " | grep -c -- '-j KUBE-SEP-' > {WS}/q140/endpoints.txt""")],
"solution": """
k -n kube-system get cm kube-proxy -o jsonpath='{.data.config\\.conf}' | grep '^mode' # iptables
echo iptables > {WS}/q140/mode.txt
ip=$(k -n %{ns} get svc %{svc} -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32" # KUBE-SERVICES ... -j KUBE-SVC-XXXX
chain=$(sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1)
echo $chain > {WS}/q140/chain.txt
sudo iptables-save -t nat | grep -- "^-A $chain " # one -j KUBE-SEP-... per endpoint
sudo iptables-save -t nat | grep -- "^-A $chain " | grep -c -- '-j KUBE-SEP-' > {WS}/q140/endpoints.txt
# KUBE-SERVICES matches ClusterIP:port and jumps to the Service's KUBE-SVC chain; that picks an
# endpoint chain at random (statistic mode random), and each KUBE-SEP chain DNATs to one Pod.
# One KUBE-SEP per ready endpoint. In nftables mode: `sudo nft list ruleset | grep -A5 %{svc}`.
""",
},
# ------------------------------------------------------------------ 141 ----
{
@ -651,65 +593,36 @@ eq "$(j -n %{ns} endpointslice billing-1 '{.metadata.labels.kubernetes\.io/servi
C("wget billing from client returns the API's answer", 3, r"""
for i in 1 2 3 4 5; do [ "$($K -n %{ns} exec client -- wget -qO- -T 3 billing 2>/dev/null)" = billing-api-%{port} ] && exit 0; sleep 3; done; exit 1"""),
],
"solution": [cmd(r"""cat {WS}/q141/billing.txt""", r"""`ADDRESS:PORT`."""),
hand(r"""vim billing.yaml""",
r"""A Service with no selector - no controller writes its endpoints - and the EndpointSlice you maintain, tied to it by the label `kubernetes.io/service-name` (the docs page *Service* has both under *Services without selectors*). With the address and port from the file:
```
"solution": """
cat {WS}/q141/billing.txt # ADDRESS:PORT
addr=$(cut -d: -f1 {WS}/q141/billing.txt); port=$(cut -d: -f2 {WS}/q141/billing.txt)
k -n %{ns} apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: billing
namespace: %{ns}
spec:
ports:
- port: 80
protocol: TCP
- {port: 80, protocol: TCP}
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: billing-1
namespace: %{ns}
labels:
kubernetes.io/service-name: billing
addressType: IPv4
endpoints:
- addresses:
- 172.18.0.3
- addresses: ["$addr"]
ports:
- port: %{port}
protocol: TCP
```
The slice's port is where traffic goes - the Service has no `targetPort` to speak of - and its name must match the Service port's: here both unnamed.""",
run=r"""addr=$(cut -d: -f1 {WS}/q141/billing.txt); port=$(cut -d: -f2 {WS}/q141/billing.txt)
cat > billing.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
name: billing
namespace: %{ns}
spec:
ports:
- port: 80
protocol: TCP
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: billing-1
namespace: %{ns}
labels:
kubernetes.io/service-name: billing
addressType: IPv4
endpoints:
- addresses:
- "$addr"
ports:
- port: $port
protocol: TCP
EOF"""),
cmd(r"""k apply -f billing.yaml
k -n %{ns} exec client -- wget -qO- -T3 billing""", r"""`billing-api-%{port}`. (`ExternalName` would need a DNS name; an address needs this.)""")],
- {port: $port, protocol: TCP}
EOF
k -n %{ns} exec client -- wget -qO- -T3 billing
# Without a selector, no controller writes endpoints for the Service: the label
# kubernetes.io/service-name ties a slice to it. The slice's port is where traffic goes (the
# Service's targetPort plays no part), and its name must match the Service port's - here both
# unnamed. ExternalName would need a DNS name; an address needs this.
""",
},
# ------------------------------------------------------------------ 142 ----
{
@ -747,24 +660,8 @@ $K -n %{ns} create secret generic ksim-probe --from-literal=x=y >/dev/null 2>&1
nx "$CP" '""" + _ETCD + r""" get /registry/secrets/%{ns}/ksim-probe --print-value-only 2>/dev/null | head -c 64' </dev/null | grep -aq '^k8s:enc:aescbc:v1:'; r=$?
$K -n %{ns} delete secret ksim-probe --ignore-not-found >/dev/null 2>&1; exit $r"""),
],
"solution": [cmd(r"""head -c 32 /dev/urandom | base64""", r"""The host is the control-plane node. Copy the output: that is the key."""),
hand(r"""sudo mkdir -p /etc/kubernetes/enc
sudo vim /etc/kubernetes/enc/enc.yaml""",
r"""Paste the example from the docs page *Encrypting Confidential Data at Rest* (search for `aescbc`) and cut it down to:
```
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
providers:
- aescbc:
keys:
- name: key1
secret: <your key>
- identity: {}
```
The first provider encrypts; `identity` after it still reads what was stored before.""",
run=r"""sudo mkdir -p /etc/kubernetes/enc
"solution": """
sudo mkdir -p /etc/kubernetes/enc
sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
@ -776,31 +673,23 @@ resources:
- name: key1
secret: $(head -c 32 /dev/urandom | base64)
- identity: {}
EOF"""),
cmd(r"""sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/""", r"""Edit a copy: the kubelet acts on every save in `manifests/`, half-done edits too."""),
hand(r"""sudo vim ~/kube-apiserver.yaml""",
r"""Add three lines, each beside its kind (`yy` and `p` on an existing `k8s-certs` mount or volume, then change it):
```
- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
```
in the `command:` list,
```
- {name: enc, mountPath: /etc/kubernetes/enc, readOnly: true}
```
under `volumeMounts:`, and
```
- {name: enc, hostPath: {path: /etc/kubernetes/enc, type: DirectoryOrCreate}}
```
under `volumes:` - the API server runs in a container, and sees only what is mounted.""",
run=r"""sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
EOF
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak # a backup OUTSIDE manifests/
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
| .spec.containers[0].volumeMounts += [{"name": "enc", "mountPath": "/etc/kubernetes/enc", "readOnly": true}]
| .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' ~/kube-apiserver.yaml"""),
cmd(r"""sudo cp ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
hand(r"""k -n kube-system get pods""",
r"""Again until it answers: the API server is back in a minute or so.""",
run=r"""sleep 20; for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""k -n %{ns} get secret %{sec} -o yaml | k replace -f -""", r"""Written again, the Secret is stored encrypted; every Secret at once: `k get secrets -A -o json | k replace -f -`."""),
cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/%{ns}/%{sec} | head -c 80""", r"""`k8s:enc:aescbc:v1:key1:` and no plain text.""")],
| .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' /tmp/kube-apiserver.yaml
sudo cp /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
sleep 20; for i in $(seq 1 40); do k get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
k -n %{ns} get secret %{sec} -o json | k replace -f - # stored again: encrypted now
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \\
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \\
get /registry/secrets/%{ns}/%{sec} | od -c | head -3 # k 8 s : e n c : a e s c b c : v 1 : k e y 1
# By hand: in kube-apiserver.yaml add the flag to the command, a volumeMount for /etc/kubernetes/enc
# and a hostPath volume of the same path - the API server runs in a container and sees only what is
# mounted. The FIRST provider encrypts; `identity` after it reads what was stored before. Every
# Secret: `k get secrets -A -o json | k replace -f -`.
""",
},
# ------------------------------------------------------------------ 143 ----
{
@ -841,39 +730,20 @@ sys.exit(0 if ok else 1)'"""),
eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas} {.status.updatedReplicas} {.status.replicas}')" "3 3 3" || exit 1
[ "$($K -n %{ns} get pods -l app=%{dep} --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 3 ]"""),
],
"solution": [cmd(r"""k label node {W2} {W3} tier=edge"""),
hand(r"""k -n %{ns} edit deploy %{dep}""",
r"""In the Pod template's `spec:`, above `containers:` (the docs page *Assigning Pods to Nodes* has both kinds):
```
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: %{dep}
topologyKey: kubernetes.io/hostname
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: tier
operator: In
values:
- edge
```
and in the same edit, under `strategy:`, let the rollout replace one Pod at a time:
```
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
```
With 3 workers and one Pod each, a surge Pod has nowhere to go, and 25% of 3 unavailable rounds down to 0: the default rollout would wait for ever. Then `:wq`.""",
run=r"""kubectl -n %{ns} patch deploy %{dep} --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}},"template":{"spec":{"affinity":{
"solution": """
k label node {W2} {W3} tier=edge
k -n %{ns} patch deploy %{dep} --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}},"template":{"spec":{"affinity":{
"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app":"%{dep}"}},"topologyKey":"kubernetes.io/hostname"}]},
"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"tier","operator":"In","values":["edge"]}]}}]}}}}}}'"""),
cmd(r"""k -n %{ns} rollout status deploy %{dep}
k -n %{ns} get pods -o wide""", r"""Three nodes, one Pod each. The anti-affinity's selector names the Pods to keep away from - its own - and `topologyKey` per what: one per hostname, one per node.""")],
"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"tier","operator":"In","values":["edge"]}]}}]}}}}}}'
k -n %{ns} rollout status deploy %{dep}
k -n %{ns} get pods -o wide
# The anti-affinity's labelSelector picks the Pods to stay away from - its own, by its template's
# labels - and topologyKey says per what: one per hostname = one per node. With 3 workers a 4th
# replica would stay Pending - and so does the rolling update's surge Pod, since every node already
# has one of the old Pods (same labels). The default maxUnavailable of 25% is 0 Pods for 3
# replicas, so the rollout would wait for ever: let it replace one Pod at a time instead
# (maxSurge 0, maxUnavailable 1), in the same change.
""",
},
# ------------------------------------------------------------------ 144 ----
{
@ -917,18 +787,15 @@ eq "$(j -n %{ns} pod %{sts}-0 '{.spec.containers[0].image}') $(j -n %{ns} pod %{
eq "$(for i in 0 1; do j -n %{ns} pod %{sts}-$i '{.metadata.uid}'; echo; done)" "$(cat "$PRIV/q144.uids")" \
&& eq "$(j -n %{ns} sts %{sts} '{.spec.template.spec.containers[0].image}')" nginx:1.27"""),
],
"solution": [hand(r"""k -n %{ns} edit sts %{sts}""",
r"""The partition first - a StatefulSet updates from the highest ordinal down, and without one the new image would reach all three within moments. Under `updateStrategy:`, then `:wq`:
```
updateStrategy:
rollingUpdate:
partition: 2
type: RollingUpdate
```""",
run=r"""kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'"""),
cmd(r"""k -n %{ns} set image sts %{sts} nginx=nginx:1.27
"solution": """
k -n %{ns} patch sts %{sts} -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'
k -n %{ns} set image sts/%{sts} nginx=nginx:1.27
k -n %{ns} rollout status sts %{sts}
k -n %{ns} get pods -l app=%{sts} -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image""", r"""Ordinals at or above the partition get the new template: `%{sts}-2` alone. The next steps of the canary: partition 1, then 0.""")],
k -n %{ns} get pods -l app=%{sts} -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image
# The partition FIRST: a StatefulSet updates from the highest ordinal down, and with no partition
# the new image would reach -1 and -0 within moments. Pods with an ordinal >= partition get the new
# template; the others keep theirs. Next step of the canary: partition 1, then 0.
""",
},
# ------------------------------------------------------------------ 145 ----
{
@ -958,53 +825,32 @@ eq "$(j crd backups.data.example.com '{.status.conditions[?(@.type=="Established
C("Backup %{name} is Scheduled: the controller may update its status", 3, r"""
eq "$(j -n %{ns} backup %{name} '{.status.phase}')" Scheduled && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller"""),
],
"solution": [cmd(r"""grep -e ^kind: -e '^ name:' {WS}/q145/operator.yaml
k apply -f {WS}/q145/operator.yaml"""),
hand(r"""k get crd backups.data.example.com
k -n backup-operator get pods""",
r"""Again until the controller is `Running`.""",
run=r"""kubectl wait --for condition=established crd/backups.data.example.com --timeout=60s
kubectl -n backup-operator rollout status deploy controller --timeout=180s"""),
cmd(r"""k create ns %{ns}"""),
hand(r"""vim backup.yaml""",
r"""```
"solution": """
grep -E '^kind:|^ name:' {WS}/q145/operator.yaml
k apply -f {WS}/q145/operator.yaml
k wait --for condition=established crd/backups.data.example.com
k -n backup-operator rollout status deploy controller
k create ns %{ns}
k -n %{ns} apply -f - <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: %{name}
namespace: %{ns}
spec:
schedule: "30 1 * * *"
```""",
run=r"""cat > backup.yaml <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: %{name}
namespace: %{ns}
spec:
schedule: "30 1 * * *"
EOF"""),
cmd(r"""k apply -f backup.yaml"""),
hand(r"""k -n %{ns} get cm backup-%{name}
k -n %{ns} get backup %{name} -o yaml | grep -A1 status
k -n backup-operator logs deploy/controller | tail -3""",
r"""The ConfigMap is made, but no `phase`: the log says `cannot patch resource "backups/status"`. The CRD has a status subresource, and for RBAC a subresource is a resource of its own.""",
run=r"""sleep 10"""),
cmd(r"""k auth can-i patch backups.data.example.com --subresource=status --as=system:serviceaccount:backup-operator:controller -A""", r"""`no`."""),
hand(r"""k edit clusterrole backup-operator""",
r"""Add `backups/status` to the first rule's resources, then `:wq` - the one thing wrong:
```
- apiGroups:
- data.example.com
resources:
- backups
- backups/status
```""",
run=r"""kubectl patch clusterrole backup-operator --type json -p '[{"op":"add","path":"/rules/0/resources/-","value":"backups/status"}]'"""),
hand(r"""k -n %{ns} get backup %{name} -o yaml | grep -A1 status""",
r"""Again until `phase: Scheduled` - the controller tries every 5 seconds.""",
run=r"""for i in $(seq 1 20); do [ "$(kubectl -n %{ns} get backup %{name} -o jsonpath='{.status.phase}')" = Scheduled ] && break; sleep 3; done""")],
EOF
sleep 10
k -n %{ns} get cm backup-%{name} # made - but:
k -n %{ns} get backup %{name} -o jsonpath='{.status}'; echo # no phase
k -n backup-operator logs deploy/controller | tail -3 # ... cannot patch resource "backups/status" ...
k auth can-i patch backups.data.example.com --subresource=status --as=system:serviceaccount:backup-operator:controller -A
k patch clusterrole backup-operator --type json -p '[{"op":"add","path":"/rules/0/resources/-","value":"backups/status"}]'
sleep 10
k -n %{ns} get backup %{name} -o jsonpath='{.status.phase}'; echo
# The CRD has a status subresource, and a subresource is a resource of its own for RBAC:
# `backups/status`. The controller could read Backups and write ConfigMaps, but not set a phase.
# The log of an operator's controller is where such a gap shows - here as `forbidden`.
""",
},
# ------------------------------------------------------------------ 146 ----
{
@ -1035,66 +881,41 @@ eq "$(j -n %{ns} pod web-server "{.spec.volumes[?(@.name==\"$v\")].persistentVol
eq "$(j -n %{ns} pvc pv-volume '{.spec.resources.requests.storage} {.status.capacity.storage}')" "%{size} %{size}" || exit 1
eq "$(j -n %{ns} pod web-server '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """),
],
"solution": [cmd(r"""k get sc csi-hostpath-sc""", r"""`ALLOWVOLUMEEXPANSION true`: a claim of this class may grow."""),
hand(r"""vim pvc.yaml""",
r"""```
"solution": """
k get sc csi-hostpath-sc # ALLOWVOLUMEEXPANSION: true - a claim of this class may grow
k -n %{ns} apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pv-volume
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
accessModes:
- ReadWriteOnce
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 10Mi
```""",
run=r"""cat > pvc.yaml <<'EOF'
requests: {storage: 10Mi}
---
apiVersion: v1
kind: PersistentVolumeClaim
kind: Pod
metadata:
name: pv-volume
namespace: %{ns}
name: web-server
spec:
storageClassName: csi-hostpath-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Mi
EOF"""),
cmd(r"""k apply -f pvc.yaml
k -n %{ns} run web-server --image=nginx:1.27 --dry-run=client -o yaml > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: web-server`:
```
containers:
- name: web-server
image: nginx:1.27
volumeMounts:
- name: html
mountPath: /usr/share/nginx/html
```
and to the Pod's `spec:`, right below `spec:`:
```
- {name: html, mountPath: /usr/share/nginx/html}
volumes:
- name: html
persistentVolumeClaim:
claimName: pv-volume
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "html", "mountPath": "/usr/share/nginx/html"}]
| .spec.volumes = [{"name": "html", "persistentVolumeClaim": {"claimName": "pv-volume"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod web-server""",
r"""Again until it is `Running`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/web-server --timeout=180s"""),
hand(r"""k -n %{ns} edit pvc pv-volume""",
r"""Under `spec.resources.requests`, then `:wq` - a claim only grows:
```
storage: %{size}
```""",
run=r"""kubectl -n %{ns} patch pvc pv-volume -p '{"spec":{"resources":{"requests":{"storage":"%{size}"}}}}'"""),
cmd(r"""k -n %{ns} get pvc pv-volume
k -n %{ns} get pod web-server""", r"""Again until `CAPACITY %{size}`: `spec.resources.requests` is what you asked for, `status.capacity` what the driver has done. This driver grows filesystems online - the Pod keeps running.""")],
persistentVolumeClaim: {claimName: pv-volume}
EOF
k -n %{ns} wait --for=condition=Ready pod/web-server
k -n %{ns} patch pvc pv-volume -p '{"spec":{"resources":{"requests":{"storage":"%{size}"}}}}'
k -n %{ns} get pvc pv-volume # CAPACITY follows once the driver has resized it
# `k edit pvc pv-volume` works just as well. A claim only grows, and only when its class has
# allowVolumeExpansion: true and its driver can expand; spec.resources.requests is what you ask
# for, status.capacity what you have. Filesystem volumes of this driver grow online - the Pod
# keeps running.
""",
},
# ------------------------------------------------------------------ 147 ----
{
@ -1142,132 +963,71 @@ x() { $K -n %{ns} exec auditor -- cat "$1/ledger.txt" 2>/dev/null; }
eq "$(x /restore)" entries-%{n} && eq "$(x /copy)" entries-%{n} \
&& eq "$(j -n %{ns} pod auditor '{.spec.volumes[*].persistentVolumeClaim.claimName}' | tr ' ' '\n' | sort | tr '\n' ' ')" "ledger-copy ledger-restore " """),
],
"solution": [cmd(r"""k get volumesnapshotclass""", r"""`csi-hostpath-snapclass`, driver `hostpath.csi.k8s.io`."""),
hand(r"""vim snap.yaml""",
r"""The docs page *Volume Snapshots* has the VolumeSnapshot:
```
"solution": """
k get volumesnapshotclass # csi-hostpath-snapclass, driver hostpath.csi.k8s.io
k -n %{ns} apply -f - <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: ledger-snap
namespace: %{ns}
spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: ledger-data
```""",
run=r"""cat > snap.yaml <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: ledger-snap
namespace: %{ns}
spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: ledger-data
EOF"""),
cmd(r"""k apply -f snap.yaml"""),
hand(r"""k -n %{ns} get volumesnapshot ledger-snap""",
r"""Again until `READYTOUSE true`.""",
run=r"""kubectl -n %{ns} wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/ledger-snap --timeout=120s"""),
hand(r"""vim claims.yaml""",
r"""`dataSource` makes a claim from a snapshot (*Volume Snapshots*) or from another claim of the same class and namespace - a clone (*CSI Volume Cloning*; no `apiGroup`, claims are core). Both at least as big as the source:
```
EOF
k -n %{ns} wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/ledger-snap --timeout=120s
k -n %{ns} apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger-restore
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: ledger-snap
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger-copy
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
dataSource:
kind: PersistentVolumeClaim
name: ledger-data
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
```""",
run=r"""cat > claims.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger-restore
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: ledger-snap
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
kind: Pod
metadata:
name: ledger-copy
namespace: %{ns}
name: auditor
spec:
storageClassName: csi-hostpath-sc
dataSource:
kind: PersistentVolumeClaim
name: ledger-data
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
EOF"""),
cmd(r"""k apply -f claims.yaml
k -n %{ns} run auditor --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: auditor`:
```
containers:
- name: auditor
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- name: restore
mountPath: /restore
- name: copy
mountPath: /copy
```
and to the Pod's `spec:`, right below `spec:`:
```
- {name: restore, mountPath: /restore}
- {name: copy, mountPath: /copy}
volumes:
- name: restore
persistentVolumeClaim:
claimName: ledger-restore
persistentVolumeClaim: {claimName: ledger-restore}
- name: copy
persistentVolumeClaim:
claimName: ledger-copy
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "restore", "mountPath": "/restore"}, {"name": "copy", "mountPath": "/copy"}]
| .spec.volumes = [{"name": "restore", "persistentVolumeClaim": {"claimName": "ledger-restore"}}, {"name": "copy", "persistentVolumeClaim": {"claimName": "ledger-copy"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pvc,pod""",
r"""Again until both claims are `Bound` and `auditor` `Running`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/auditor --timeout=180s"""),
cmd(r"""k -n %{ns} exec auditor -- cat /restore/ledger.txt /copy/ledger.txt""", r"""`entries-%{n}`, twice.""")],
persistentVolumeClaim: {claimName: ledger-copy}
EOF
k -n %{ns} wait --for=condition=Ready pod/auditor --timeout=180s
k -n %{ns} exec auditor -- cat /restore/ledger.txt /copy/ledger.txt
# A snapshot is taken by the CSI driver (VolumeSnapshotClass -> driver); `dataSource` makes a new
# claim from it - apiGroup snapshot.storage.k8s.io - or from another claim of the same class and
# Namespace, a clone (no apiGroup: claims are core). Both need a CSI driver that can do it, and
# the new claim must be at least as big as the source.
""",
},
]

4
cka/bank_hard.py
View File

@ -186,9 +186,7 @@ $K delete ns %{ns} --ignore-not-found --wait=false >/dev/null
},
{
"id": 49, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults",
# low, short: the plural's stem and the shortName, which the solution spells out
"params": lambda r: (lambda p: dict(p, low=p["kind"].lower(), short=p["kind"][:2].lower()))(
{"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])}),
"params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])},
"task": CTX + """
Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the kind in lowercase plus
`s`, shortName = the first two letters of the kind in lowercase) with schema:

163
cka/bank_net.py
View File

@ -5,8 +5,6 @@ Written for the per-task hosts: a task may rewrite the cluster's CoreDNS configu
resolution and hostPorts are tested for real; Ingress and Gateway objects are graded as objects,
as no controller for them runs on the hosts.
"""
from sim.bank import cmd, hand
from cka.common import C, DNS, NS, ONE, PODS, PY, ns_pick, name_pick
QUESTIONS = [
@ -155,102 +153,43 @@ parents == [("%{gw}", "%{ns}")] and s.get("hostnames") == ["%{host}.example.com"
out=$($K -n %{ns} get ingress %{host} 2>&1); case "$out" in *NotFound*) ;; *) exit 1 ;; esac
eq "$(j -n %{ns} secret %{host}-tls '{.metadata.uid}')" "$(cat "$PRIV/q126.uid")" """),
],
"solution": [cmd(r"""k -n %{ns} get ingress %{host} -o yaml""", r"""The host, the TLS Secret `%{host}-tls`, `/api` to `api:8080` and `/` to `web:80`."""),
hand(r"""vim gw.yaml""",
r"""What the Ingress keeps in one object, the Gateway API splits: the listener - port, hostname, certificate - goes in the Gateway, the routing in the HTTPRoute. The docs page *Gateway API* has both; `k explain gateway.spec.listeners.tls` the TLS fields:
```
"solution": """
k -n %{ns} get ingress %{host} -o yaml # host, the TLS Secret (%{host}-tls), /api -> api:8080, / -> web:80
cat <<EOF | k apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: %{gw}
namespace: %{ns}
metadata: {name: %{gw}, namespace: %{ns}}
spec:
gatewayClassName: %{gc}
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: %{host}.example.com
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: %{host}-tls
- name: https
protocol: HTTPS
port: 443
hostname: %{host}.example.com
tls:
mode: Terminate
certificateRefs: [{kind: Secret, name: %{host}-tls}]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: %{host}
namespace: %{ns}
metadata: {name: %{host}, namespace: %{ns}}
spec:
parentRefs:
- name: %{gw}
hostnames:
- %{host}.example.com
parentRefs: [{name: %{gw}}]
hostnames: [%{host}.example.com]
rules:
- matches:
- path:
type: PathPrefix
value: /api
backendRefs:
- name: api
port: 8080
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: web
port: 80
```""",
run=r"""cat > gw.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: %{gw}
namespace: %{ns}
spec:
gatewayClassName: %{gc}
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: %{host}.example.com
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: %{host}-tls
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: %{host}
namespace: %{ns}
spec:
parentRefs:
- name: %{gw}
hostnames:
- %{host}.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /api
backendRefs:
- name: api
port: 8080
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: web
port: 80
EOF"""),
cmd(r"""k apply -f gw.yaml
- matches: [{path: {type: PathPrefix, value: /api}}]
backendRefs: [{name: api, port: 8080}]
- matches: [{path: {type: PathPrefix, value: /}}]
backendRefs: [{name: web, port: 80}]
EOF
k -n %{ns} get gateway,httproute
k -n %{ns} delete ingress %{host}""")],
k -n %{ns} delete ingress %{host}
# What an Ingress keeps in one object, the Gateway API splits by role: the Gateway is the
# listener - port, hostname, certificate - and belongs to whoever runs the edge; the HTTPRoute is
# the routing and belongs to the application. spec.tls of the Ingress becomes tls.certificateRefs
# on the listener, each path becomes a rule with a PathPrefix match, and a Service backend becomes
# a backendRef. `k explain gateway.spec.listeners.tls` and the Gateway API docs are allowed.
""",
},
# ------------------------------------------------------------------ 127 ----
{
@ -460,37 +399,23 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.hostNetwork}')" true || exit 1
pip=$(j -n %{ns} pod %{pod} '{.status.podIP}'); hip=$(j -n %{ns} pod %{pod} '{.status.hostIP}')
[ -n "$pip" ] && eq "$pip" "$hip" && eq "$(tr -s ' \t\r\n' ' ' < "$WS/q128/ips.txt" 2>/dev/null | sed 's/^ //; s/ $//')" "$pip $hip" """),
],
"solution": [hand(r"""k -n %{ns} edit ds %{ds}""",
r"""Give the container's port a `hostPort`, then `:wq`:
```
ports:
- containerPort: 8080
hostPort: %{hp}
protocol: TCP
```
`hostPort` maps one port of the node to the Pod on it (the CNI's `portmap` plugin writes the rule); `hostNetwork` would put the whole Pod in the node's network.""",
run=r"""kubectl -n %{ns} patch ds %{ds} --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/ports/0/hostPort", "value": %{hp}}]'"""),
cmd(r"""k -n %{ns} rollout status ds %{ds}
k get nodes -o wide""", r"""Note a worker's `INTERNAL-IP`."""),
hand(r"""curl -s http://172.18.0.3:%{hp}/""",
r"""With that IP: `agent on <that node>`.""",
run=r""""""),
cmd(r"""k -n %{ns} logs %{pod} --tail=2
k -n %{ns} get pod %{pod} -o yaml | grep -e hostNetwork -e dnsPolicy""", r"""`bad address`: a `hostNetwork` Pod uses the node's `/etc/resolv.conf`, which knows nothing of `cluster.local` - unless its `dnsPolicy` is `ClusterFirstWithHostNet`."""),
hand(r"""vim {WS}/q128/collector.yaml""",
r"""In the Pod's `spec:`, below `hostNetwork: true`:
```
dnsPolicy: ClusterFirstWithHostNet
```""",
run=r"""yq -i '.spec.dnsPolicy = "ClusterFirstWithHostNet"' {WS}/q128/collector.yaml"""),
cmd(r"""k replace --force -f {WS}/q128/collector.yaml""", r"""A running Pod's `dnsPolicy` cannot change: `--force` deletes it and creates it again."""),
hand(r"""k -n %{ns} logs %{pod} --tail=2
k -n %{ns} get pod %{pod} -o wide""",
r"""Again until `resolved %{svc}`. Its IP is its node's: note it.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=180s
sleep 8"""),
hand(r"""echo "172.18.0.3 172.18.0.3" > {WS}/q128/ips.txt""",
r"""The Pod's IP and its node's - the same address.""",
run=r"""kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt""")],
"solution": """
k -n %{ns} edit ds %{ds} # under the container:
# ports: [{containerPort: 8080, hostPort: %{hp}}]
k -n %{ns} rollout status ds %{ds}
k get nodes -o wide # INTERNAL-IP
curl -s http://<a worker's INTERNAL-IP>:%{hp}/ # agent on <that node>
k -n %{ns} logs %{pod} --tail=2 # bad address %{svc}
k -n %{ns} get pod %{pod} -o jsonpath='{.spec.dnsPolicy}'; echo # ClusterFirst - which a hostNetwork Pod does not get
vi {WS}/q128/collector.yaml # spec.dnsPolicy: ClusterFirstWithHostNet
k -n %{ns} delete pod %{pod}; k apply -f {WS}/q128/collector.yaml
k -n %{ns} logs %{pod} --tail=2 # resolved %{svc}
k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt
# hostPort maps one port of the node to one Pod on it (the CNI's portmap plugin writes the rule);
# hostNetwork puts the whole Pod into the node's network namespace - its IP IS the node's IP, and
# it uses the node's /etc/resolv.conf, which knows nothing of cluster.local, unless dnsPolicy is
# ClusterFirstWithHostNet. dnsPolicy cannot be changed on a running Pod: recreate it.
""",
},
]

222
cka/bank_objectives.py
View File

@ -20,7 +20,7 @@ kubeadm, etcd, the runtime. All normal difficulty, all in the random pool; nothi
node or a control-plane component (109 cordons the control-plane node while you work on it and has
you uncordon it; the session baseline uncordons it if you do not).
"""
from sim.bank import C, NS, ns_pick, name_pick, cmd, hand
from sim.bank import C, NS, ns_pick, name_pick
from ckad import bank_extra as _ckad_extra, bank_more as _ckad_more
@ -30,34 +30,18 @@ def _port(bank, src_id, new_id, domain, **over):
old, new = f"q{src_id}", f"q{new_id}"
q = dict(src, id=new_id, domain=domain, **over)
for k in ("task", "setup", "solution"):
if k not in over: # a solution of its own (in parts) is the CKA one already
q[k] = src[k].replace(old, new)
q[k] = src[k].replace(old, new)
q["checks"] = [dict(c, script=c["script"].replace(old, new), name=c["name"].replace(old, new)) for c in src["checks"]]
return q
# 103's own solution: the CKAD task's, done as in the exam
_SOL103 = [hand(r"""k -n %{ns} top pod""",
r"""Again until it shows all three: metrics-server needs about 30 seconds after the Pods start. Without it, `Metrics API not available`.""",
run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} top pod --no-headers 2>/dev/null | wc -l)" -ge 3 ] && break; sleep 5; done"""),
cmd(r"""k -n %{ns} top pod --sort-by=cpu
k -n %{ns} top pod --sort-by=memory""", r"""The first line of each is the heaviest."""),
hand(r"""echo %{cpu} > {WS}/q103/cpu.txt
echo %{mem} > {WS}/q103/memory.txt""",
r"""With the names on those first lines.""",
run=r"""kubectl -n %{ns} top pod --sort-by=cpu --no-headers | head -1 | awk '{print $1}' > {WS}/q103/cpu.txt
kubectl -n %{ns} top pod --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/q103/memory.txt"""),
cmd(r"""echo 'kubectl top node' > {WS}/q103/nodes.sh
echo 'kubectl top pod -A --sort-by=memory' > {WS}/q103/pods.sh""", r"""`k top` reads the metrics API (`metrics.k8s.io`) that metrics-server serves; `--containers` breaks a Pod down per container.""")]
ETCDCTL = ("etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
"--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key")
QUESTIONS = [
# ---------------------------------------------------------------- TSHOOT ----
# needs a working metrics API, which Q23 breaks on purpose: never in one session with it
_port(_ckad_extra, 23, 103, "TSHOOT", breaks="metrics-server", solution=_SOL103),
_port(_ckad_extra, 23, 103, "TSHOOT", breaks="metrics-server"),
_port(_ckad_more, 64, 104, "TSHOOT"),
# ------------------------------------------------------------------ WORK ----
{
@ -199,44 +183,32 @@ cg=$(nx "$W3" "awk '/^cgroupDriver:/{print \$2}' /var/lib/kubelet/config.yaml" <
want=$(printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s' "$fwd" "$swap" "$fso" "$cg")
eq "$(tr -d ' \t\r' < "$WS/q107/node.txt" 2>/dev/null | grep .)" "$want" """),
],
"solution": [hand(r"""{SHELL_W3}
"solution": """
{SHELL_W3}
sudo -i
vim /etc/modules-load.d/%{name}.conf""",
r"""The docs page *Container Runtimes* has both files under *Install and configure prerequisites*. One module per line:
```
overlay
br_netfilter
```""",
run=r"""ssh -n {W3} 'printf "overlay\nbr_netfilter\n" | sudo tee /etc/modules-load.d/%{name}.conf >/dev/null'"""),
hand(r"""vim /etc/sysctl.d/%{name}.conf""",
r"""```
printf 'overlay\\nbr_netfilter\\n' > /etc/modules-load.d/%{name}.conf
cat > /etc/sysctl.d/%{name}.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
```
On a real machine they would now be loaded and applied without a reboot: `modprobe overlay`, `modprobe br_netfilter`, `sysctl --system`. `sysctl -w` and `modprobe` alone are gone after a reboot; the files make them stick.""",
run=r"""ssh -n {W3} 'printf "net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\n" | sudo tee /etc/sysctl.d/%{name}.conf >/dev/null'"""),
hand(r"""sysctl -n net.ipv4.ip_forward
cat /proc/swaps
grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml
exit
exit""",
r"""Any line below the header of `/proc/swaps` means swap is on. No `failSwapOn` line means the default, `true`.""",
run=r""""""),
hand(r"""vim {WS}/q107/node.txt""",
r"""What the node printed:
```
ip_forward=<0 or 1>
swap=<on or off>
fail-swap-on=<true or false>
cgroup-driver=<systemd or cgroupfs>
```
kubeadm's preflight refuses a node with swap on unless the kubelet tolerates it (`failSwapOn: false`); and the kubelet's cgroup driver has to match the runtime's (`SystemdCgroup = true` in containerd goes with `cgroupDriver: systemd`).""",
run=r"""IPF=$(ssh -n {W3} 'sudo sysctl -n net.ipv4.ip_forward')
SW=off; [ "$(ssh -n {W3} 'sed 1d /proc/swaps | grep -c .')" -gt 0 ] && SW=on
FS=$(ssh -n {W3} "sudo awk '/^failSwapOn:/ {print \$2}' /var/lib/kubelet/config.yaml"); [ -n "$FS" ] || FS=true
CG=$(ssh -n {W3} "sudo awk '/^cgroupDriver:/ {print \$2}' /var/lib/kubelet/config.yaml")
printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s\n' "$IPF" "$SW" "$FS" "$CG" > {WS}/q107/node.txt""")],
EOF
# on a real machine you now load and apply them without a reboot:
# modprobe overlay && modprobe br_netfilter && sysctl --system
sysctl -n net.ipv4.ip_forward # 1
cat /proc/swaps # any line under the header = swap is on (swapon --show)
grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml
exit; exit
cat > {WS}/q107/node.txt <<EOF
ip_forward=1
swap=<on|off>
fail-swap-on=<true|false>
cgroup-driver=<systemd|cgroupfs>
EOF
# `sysctl -w` and `modprobe` alone are gone after a reboot - the files are what makes them stick.
# kubeadm's preflight refuses a node with swap on unless the kubelet is told to tolerate it
# (failSwapOn: false), and the kubelet's cgroup driver has to match the container runtime's
# (SystemdCgroup = true in /etc/containerd/config.toml goes with cgroupDriver: systemd).
""",
},
{
"id": 108, "domain": "ARCH", "title": "The container runtime behind a Pod",
@ -291,35 +263,29 @@ cid=$($K -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].cont
log=$(nx "$W2" "crictl inspect -o go-template --template '{{.status.logPath}}' $cid 2>/dev/null" </dev/null | tr -d '[:space:]')
[ -n "$cid" ] && [ -n "$log" ] && tr -d ' \t\r' < "$WS/q108/runtime.txt" 2>/dev/null | grep -qx "log=$log" """),
],
"solution": [cmd(r"""k -n %{ns} get pod %{pod} -o wide""", r"""On `{W2}`."""),
hand(r"""{SHELL_W2}
"solution": """
k -n %{ns} get pod %{pod} -o wide # on {W2}
{SHELL_W2}
sudo -i
crictl version
grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
crictl ps --no-trunc --name app""",
r"""`RuntimeName` and `RuntimeVersion`; the CRI socket the kubelet uses - crictl speaks the same protocol, which is why it works even when the API server does not; and the container's full ID (`--no-trunc`), the line whose `POD` is `%{pod}`.""",
run=r""""""),
hand(r"""crictl inspect 0123abcd... | grep -e '"pid"' -e logPath
exit
exit""",
r"""With that ID: `.info.pid` and `.status.logPath`. The log is the file `kubectl logs` reads, under `/var/log/pods`.""",
run=r""""""),
hand(r"""vim {WS}/q108/runtime.txt""",
r"""The values you read:
```
crictl version # RuntimeName: containerd RuntimeVersion: v2.x.y
grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml # unix:///run/containerd/containerd.sock
crictl pods --name %{pod} # the Pod sandbox
crictl ps --no-trunc --name app --pod $(crictl pods --name %{pod} -q) # full container ID
crictl inspect <id> | grep -E '"pid"|logPath' # .info.pid and .status.logPath
# or: crictl inspect -o go-template --template '{{.info.pid}} {{.status.logPath}}' <id>
exit; exit
cat > {WS}/q108/runtime.txt <<EOF
runtime=containerd <version>
socket=unix:///run/containerd/containerd.sock
container-id=<the 64 hex characters>
pid=<the number>
container-id=<64 hex characters>
pid=<number>
log=/var/log/pods/%{ns}_%{pod}_<pod uid>/app/0.log
```""",
run=r"""V=$(ssh -n {W2} 'sudo crictl version 2>/dev/null')
RN=$(echo "$V" | awk '/^RuntimeName:/ {print $2}'); RV=$(echo "$V" | awk '/^RuntimeVersion:/ {print $2}')
SOCK=$(ssh -n {W2} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
CID=$(kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#^[a-z-]*://##')
PID=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.info.pid}}' $CID 2>/dev/null")
LOG=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.status.logPath}}' $CID 2>/dev/null")
printf 'runtime=%s %s\nsocket=%s\ncontainer-id=%s\npid=%s\nlog=%s\n' "$RN" "$RV" "$SOCK" "$CID" "$PID" "$LOG" > {WS}/q108/runtime.txt""")],
EOF
# The kubelet drives the runtime over the CRI socket, and crictl speaks the same protocol - which
# is why it still works when the API server does not. `kubectl get pod -o
# jsonpath='{.status.containerStatuses[0].containerID}'` shows the same ID with a containerd://
# prefix; /var/log/pods/... is the file `kubectl logs` reads.
""",
},
{
# drains the control-plane node: CoreDNS moves off it, so never beside the tasks that break DNS
@ -502,33 +468,29 @@ c=$(j -n kube-system lease kube-controller-manager '{.spec.holderIdentity}' | cu
f=$(tr -d ' \t\r' < "$WS/q111/ha.txt" 2>/dev/null)
[ -n "$s" ] && [ -n "$c" ] && echo "$f" | grep -qx "scheduler-leader=$s" && echo "$f" | grep -qx "controller-manager-leader=$c" """),
],
"solution": [cmd(r"""k get nodes -l node-role.kubernetes.io/control-plane
k -n kube-system get pods -l component=etcd -o wide
k -n kube-system get lease kube-scheduler kube-controller-manager""", r"""An etcd static Pod on the control-plane node: stacked. A Lease's `HOLDER` is `<node>_<uuid>`."""),
hand(r"""sudo -i
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl member list -w table
etcdctl endpoint status -w table
exit""",
r"""The host is the control-plane node. One line per member; `IS LEADER` in the status (`--cluster` asks every member).""",
run=r""""""),
hand(r"""vim {WS}/q111/ha.txt""",
r"""```
"solution": """
k get nodes -l node-role.kubernetes.io/control-plane # how many
k -n kube-system get pods -l component=etcd -o wide # an etcd static Pod on the control-plane node = stacked
k -n kube-system get lease kube-scheduler kube-controller-manager # HOLDER is <node>_<uuid>
sudo -i # the task's host is the control-plane node: no hop needed
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \\
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl member list -w table # one line per member: ID, NAME, peer and client URLs
etcdctl endpoint status -w table # IS LEADER (add --cluster to ask every member)
exit
cat > {WS}/q111/ha.txt <<EOF
control-plane-nodes=1
etcd-topology=stacked
etcd-members=1
etcd-leader={CP}
scheduler-leader={CP}
controller-manager-leader={CP}
```
In an HA control plane every API server serves at once, but one scheduler and one controller manager act, through the leader Lease. etcd elects its own leader and needs a majority to stay writable: three members survive the loss of one, five of two.""",
run=r"""E='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
N=$(kubectl get nodes -l node-role.kubernetes.io/control-plane --no-headers | wc -l)
M=$($E member list | wc -l)
L=$($E member list | awk -F', ' 'NR==1 {print $3}')
S=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
C=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
printf 'control-plane-nodes=%s\netcd-topology=stacked\netcd-members=%s\netcd-leader=%s\nscheduler-leader=%s\ncontroller-manager-leader=%s\n' "$N" "$M" "$L" "$S" "$C" > {WS}/q111/ha.txt""")],
EOF
# In an HA control plane every kube-apiserver serves at once, but only one kube-scheduler and one
# kube-controller-manager act: they elect a leader through a Lease in kube-system, and the others
# wait. etcd elects its own leader and needs a majority of members to stay writable - three
# members survive the loss of one, five of two, and an even number adds nothing.
""",
},
{
"id": 112, "domain": "ARCH", "title": "CNI, CSI and CRI: what this cluster plugs in",
@ -570,35 +532,30 @@ prov=$($K get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclas
f=$(tr -d ' \t\r' < "$WS/q112/interfaces.txt" 2>/dev/null)
echo "$f" | grep -qx "csi-drivers=$n" && echo "$f" | grep -qx "default-provisioner=$prov" """),
],
"solution": [hand(r"""{SHELL_W1}
"solution": """
{SHELL_W1}
sudo -i
ls /etc/cni/net.d
cat /etc/cni/net.d/10-kindnet.conflist
ls /etc/cni/net.d/ # 10-kindnet.conflist - the kubelet's runtime takes the first file
cat /etc/cni/net.d/10-kindnet.conflist # "plugins": [{"type": "ptp", "ipam": {"type": "host-local"}}, {"type": "portmap"}]
ls /opt/cni/bin # the plugin binaries those types name
grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
exit
exit""",
r"""The runtime takes the first file of `/etc/cni/net.d`. Its `plugins` chain, in order - each `type` names a binary in `/opt/cni/bin` - and the `ipam` type inside one of them.""",
run=r""""""),
cmd(r"""k get csidrivers
k get sc""", r"""`No resources found`: no CSI driver - the local-path provisioner is not one. The `(default)` class's `PROVISIONER`."""),
hand(r"""vim {WS}/q112/interfaces.txt""",
r"""The values you read - here:
```
exit; exit
k get csidrivers # No resources found = 0
k get sc # the (default) class and its PROVISIONER
cat > {WS}/q112/interfaces.txt <<EOF
cni-config=/etc/cni/net.d/10-kindnet.conflist
cni-plugins=ptp,portmap
cni-ipam=host-local
cri-socket=unix:///run/containerd/containerd.sock
csi-drivers=0
default-provisioner=rancher.io/local-path
```
`ptp` wires each Pod's veth pair, `host-local` hands out addresses from the node's Pod CIDR, `portmap` serves hostPorts.""",
run=r"""CONF=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
PL=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[].type] | join(",")')
IPAM=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[] | select(.ipam) | .ipam.type][0]')
SOCK=$(ssh -n {W1} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
N=$(kubectl get csidrivers --no-headers 2>/dev/null | wc -l)
PROV=$(kubectl get sc -o json | jq -r '.items[] | select(.metadata.annotations["storageclass.kubernetes.io/is-default-class"] == "true") | .provisioner')
printf 'cni-config=%s\ncni-plugins=%s\ncni-ipam=%s\ncri-socket=%s\ncsi-drivers=%s\ndefault-provisioner=%s\n' "$CONF" "$PL" "$IPAM" "$SOCK" "$N" "$PROV" > {WS}/q112/interfaces.txt""")],
EOF
# CRI: the kubelet talks to the runtime over that socket (crictl uses the same one). CNI: the
# runtime calls the plugins of the chain in order for every Pod sandbox - here ptp wires the veth
# pair, host-local hands out an address from the node's Pod CIDR, portmap serves hostPorts.
# CSI: a storage driver registers a CSIDriver object and a StorageClass names it as provisioner;
# this cluster's local-path provisioner is not a CSI driver, hence none.
""",
},
# ------------------------------------------------------------------- NET ----
{
@ -666,18 +623,19 @@ eq "$(tr -s ' \t' ' ' < "$WS/q113/pod-cidrs.txt" 2>/dev/null | tr -d '\r' | sed
C("outside.txt names %{edge}, why.txt names hostNetwork", 2, r"""
eq "$(tr -d '[:space:]' < "$WS/q113/outside.txt" 2>/dev/null | sed 's#^pod/##')" %{edge} && grep -qi 'hostNetwork' "$WS/q113/why.txt" 2>/dev/null"""),
],
"solution": [cmd(r"""k -n %{ns} get pods -o wide
k get nodes -o wide""", r"""The `IP` and `NODE` of every Pod. `%{edge}`'s IP is its node's own `INTERNAL-IP`, not one from a Pod CIDR."""),
hand(r"""echo "10.244.1.5 {W1}" > {WS}/q113/web.txt""",
r"""With `%{web}`'s IP and node.""",
run=r"""kubectl -n %{ns} get pod %{web} -o jsonpath='{.status.podIP} {.spec.nodeName}' > {WS}/q113/web.txt"""),
hand(r"""k -n %{ns} exec %{client} -- wget -qO- -T3 http://10.244.1.5 > {WS}/q113/direct.html""",
r"""With that IP: Pod to Pod, across nodes, no Service and no NAT - the network model the CNI makes true.""",
run=r"""kubectl -n %{ns} exec %{client} -- wget -qO- -T3 http://$(kubectl -n %{ns} get pod %{web} -o jsonpath='{.status.podIP}') > {WS}/q113/direct.html"""),
cmd(r"""k get nodes -o custom-columns=NAME:.metadata.name,CIDR:.spec.podCIDR --no-headers > {WS}/q113/pod-cidrs.txt
cat {WS}/q113/pod-cidrs.txt
k -n %{ns} get pod %{edge} -o yaml | grep hostNetwork
"solution": """
k -n %{ns} get pods -o wide # IP and NODE of every Pod
k -n %{ns} get pod %{web} -o jsonpath='{.status.podIP} {.spec.nodeName}' > {WS}/q113/web.txt
k -n %{ns} exec %{client} -- wget -qO- -T3 http://$(cut -d' ' -f1 {WS}/q113/web.txt) > {WS}/q113/direct.html
k get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\\n"}{end}' > {WS}/q113/pod-cidrs.txt
# %{edge}'s IP is its node's own address (compare with `k get nodes -o wide`), not one from 10.244.x.0/24
echo %{edge} > {WS}/q113/outside.txt
echo hostNetwork > {WS}/q113/why.txt""", r"""A `hostNetwork` Pod lives in its node's network namespace and shares the node's address and ports.""")],
k -n %{ns} get pod %{edge} -o yaml | grep hostNetwork
echo hostNetwork > {WS}/q113/why.txt
# Every Pod gets its own address from its node's Pod CIDR, and any Pod reaches any other Pod on
# any node at that address without NAT - that is the whole network model, and the CNI plugin is
# what makes it true. A hostNetwork Pod opts out: it lives in the node's network namespace and
# shares its address and ports. Pod addresses change with every new Pod, which is why Services exist.
""",
},
]

653
cka/bank_sim_c.py
View File

@ -12,7 +12,7 @@ Two scenarios are deliberately adapted so they cannot wreck the practice cluster
Run with: ./ksim sim c
"""
from sim.bank import C, NS, cmd, hand
from sim.bank import C, NS
Q = []
@ -62,87 +62,42 @@ eq "$(j -n geo-fence pod geofence-reader '{.status.phase}')" Running \
&& eq "$(j -n geo-fence pod geofence-reader '{.spec.volumes[0].persistentVolumeClaim.claimName}')" geofence-pvc \
&& j -n geo-fence pod geofence-reader '{.spec.containers[0].volumeMounts[*].mountPath}' | grep -q /geo-data"""),
],
solution=[hand(r"""vim pv.yaml""",
r"""The PersistentVolume and PersistentVolumeClaim from the docs page *Configure a Pod to Use a PersistentVolume for Storage*, `---` between them. `volumeName` pins the claim to this PV: without it the binder takes any PV whose size, access mode and class fit.
```
solution="""
cat <<EOF | k apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
name: geofence-pv
metadata: {name: geofence-pv}
spec:
capacity: {storage: 250Mi}
accessModes: [ReadWriteOnce]
hostPath: {path: /data/geofence}
storageClassName: manual
capacity:
storage: 250Mi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/geofence
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: geofence-pvc
namespace: geo-fence
metadata: {name: geofence-pvc, namespace: geo-fence}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 250Mi}}
storageClassName: manual
volumeName: geofence-pv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 250Mi
```""",
run=r"""cat > pv.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: geofence-pv
spec:
storageClassName: manual
capacity:
storage: 250Mi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/geofence
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: geofence-pvc
namespace: geo-fence
kind: Pod
metadata: {name: geofence-reader, namespace: geo-fence}
spec:
storageClassName: manual
volumeName: geofence-pv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 250Mi
EOF"""),
cmd(r"""k apply -f pv.yaml
volumes: [{name: geo, persistentVolumeClaim: {claimName: geofence-pvc}}]
containers:
- name: reader
image: busybox:1.36
command: [sh, -c, "ls /geo-data && sleep 3600"]
volumeMounts: [{name: geo, mountPath: /geo-data}]
EOF
k -n geo-fence get pvc
k -n geo-fence run geofence-reader --image=busybox:1.36 --dry-run=client -o yaml --command -- sh -c "ls /geo-data && sleep 3600" > pod.yaml""", r"""`Bound` to `geofence-pv`."""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: geofence-reader`:
```
volumeMounts:
- name: geo
mountPath: /geo-data
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: geo
persistentVolumeClaim:
claimName: geofence-pvc
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "geo", "mountPath": "/geo-data"}]
| .spec.volumes = [{"name": "geo", "persistentVolumeClaim": {"claimName": "geofence-pvc"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml
k -n geo-fence get pod geofence-reader""")],
# volumeName pins the claim to one PV by name. Without it the binder picks ANY PV whose capacity,
# accessModes and storageClassName match - which on a busy cluster is rarely the one you meant.
""",
)
# ------------------------------------------------------------------ 69 ----
@ -305,19 +260,15 @@ want=$(j -n dispatch-hub svc dispatch-service '{.spec.clusterIP}')
got=$(tr -d '[:space:]' < "$WS/q71/service-ip.txt" 2>/dev/null)
[ -n "$want" ] && eq "$got" "$want" """),
],
solution=[cmd(r"""k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
k -n dispatch-hub get svc,endpointslices""", r"""`expose` copies the Deployment's selector, which is what makes the endpoints appear. Note the Service's `CLUSTER-IP`."""),
hand(r"""k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt
cat {WS}/q71/test-result.txt""",
r"""nginx's welcome page, fetched from a Pod in the cluster.""",
run=r"""kubectl -n dispatch-hub rollout status deploy dispatch-api --timeout=240s
for i in 1 2 3 4 5; do
kubectl -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt </dev/null 2>/dev/null
grep -qi nginx {WS}/q71/test-result.txt && break; kubectl -n dispatch-hub delete pod probe --ignore-not-found >/dev/null 2>&1; sleep 3
done"""),
hand(r"""echo 10.96.123.45 > {WS}/q71/service-ip.txt""",
r"""With the `CLUSTER-IP` that `get svc` showed - the address alone.""",
run=r"""kubectl -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt""")],
solution="""
k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
k -n dispatch-hub get endpointslices -l kubernetes.io/service-name=dispatch-service # must not be empty
k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- \\
wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt
k -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt
# `expose` copies the Deployment's selector for you, which is what makes the endpoints appear.
# Write the IP with jsonpath, not by eyeballing `get svc` - the grader compares it literally.
""",
)
# ------------------------------------------------------------------ 72 ----
@ -407,102 +358,54 @@ j -n ride-history deploy ride-archive '{.spec.template.spec.volumes[*].persisten
j -n ride-history deploy ride-archive '{.spec.template.spec.volumes[*].persistentVolumeClaim.claimName}' | grep -qw ride-archive-pvc-new || exit 1
for i in $(seq 1 20); do [ "$(j -n ride-history deploy ride-archive '{.status.availableReplicas}')" -ge 1 ] 2>/dev/null && exit 0; sleep 5; done; exit 1"""),
],
solution=[cmd(r"""k -n ride-history get pvc
k get pv ride-archive-pv""", r"""`RECLAIM POLICY Delete` on the old PV: losing the claim loses the data."""),
hand(r"""vim new.yaml""",
r"""The new PV and its claim, pinned to it by name (the docs page *Configure a Pod to Use a PersistentVolume for Storage* has both):
```
solution="""
k -n ride-history get pvc,pv # note reclaimPolicy Delete on the old PV
cat <<EOF | k apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
name: ride-archive-pv-new
metadata: {name: ride-archive-pv-new}
spec:
capacity: {storage: 500Mi}
accessModes: [ReadWriteOnce]
hostPath: {path: /data/ride-archive-new}
storageClassName: manual
capacity:
storage: 500Mi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/ride-archive-new
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ride-archive-pvc-new
namespace: ride-history
metadata: {name: ride-archive-pvc-new, namespace: ride-history}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 500Mi}}
storageClassName: manual
volumeName: ride-archive-pv-new
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
```""",
run=r"""cat > new.yaml <<'EOF'
EOF
k -n ride-history scale deploy ride-archive --replicas=0 # release the RWO claim first
cat <<EOF | k apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
name: ride-archive-pv-new
kind: Pod
metadata: {name: data-migrator, namespace: ride-history}
spec:
storageClassName: manual
capacity:
storage: 500Mi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /data/ride-archive-new
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ride-archive-pvc-new
namespace: ride-history
spec:
storageClassName: manual
volumeName: ride-archive-pv-new
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
EOF"""),
cmd(r"""k apply -f new.yaml
k -n ride-history scale deploy ride-archive --replicas=0
k -n ride-history run data-migrator --image=busybox:1.36 --restart=Never --dry-run=client -o yaml -- sh -c "cp -a /old-data/. /new-data/ && echo migration-complete && sleep 3600" > m.yaml""", r"""Scale to 0 first: the running Pod holds the old claim. `cp -a /old-data/. /new-data/` copies the contents, dotfiles too, rather than the directory itself."""),
hand(r"""vim m.yaml""",
r"""Both claims: add to the container, below `name: data-migrator`,
```
volumeMounts:
- name: old
mountPath: /old-data
- name: new
mountPath: /new-data
```
and to the Pod's `spec:`, right below `spec:`:
```
restartPolicy: Never
volumes:
- name: old
persistentVolumeClaim:
claimName: ride-archive-pvc
- name: new
persistentVolumeClaim:
claimName: ride-archive-pvc-new
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "old", "mountPath": "/old-data"}, {"name": "new", "mountPath": "/new-data"}]
| .spec.volumes = [{"name": "old", "persistentVolumeClaim": {"claimName": "ride-archive-pvc"}}, {"name": "new", "persistentVolumeClaim": {"claimName": "ride-archive-pvc-new"}}]' m.yaml"""),
cmd(r"""k apply -f m.yaml"""),
hand(r"""k -n ride-history logs data-migrator""",
r"""Again until it prints `migration-complete`.""",
run=r"""for i in $(seq 1 60); do kubectl -n ride-history logs data-migrator 2>/dev/null | grep -q migration-complete && break; sleep 2; done"""),
cmd(r"""k -n ride-history delete pod data-migrator --now"""),
hand(r"""k -n ride-history edit deploy ride-archive""",
r"""In the Pod template's `volumes:`, change `claimName: ride-archive-pvc` to `claimName: ride-archive-pvc-new`, then `:wq`.""",
run=r"""kubectl -n ride-history patch deploy ride-archive --type=json -p '[{"op":"replace","path":"/spec/template/spec/volumes/0/persistentVolumeClaim/claimName","value":"ride-archive-pvc-new"}]'"""),
cmd(r"""k -n ride-history scale deploy ride-archive --replicas=1
k -n ride-history get pods""")],
- {name: old, persistentVolumeClaim: {claimName: ride-archive-pvc}}
- {name: new, persistentVolumeClaim: {claimName: ride-archive-pvc-new}}
containers:
- name: migrator
image: busybox:1.36
command: [sh, -c, "cp -a /old-data/. /new-data/ && echo migration-complete && sleep 3600"]
volumeMounts:
- {name: old, mountPath: /old-data}
- {name: new, mountPath: /new-data}
EOF
k -n ride-history logs data-migrator # migration-complete
k -n ride-history patch deploy ride-archive --type=json \\
-p '[{"op":"replace","path":"/spec/template/spec/volumes/0/persistentVolumeClaim/claimName","value":"ride-archive-pvc-new"}]'
k -n ride-history scale deploy ride-archive --replicas=1
# ReadWriteOnce is per-node, but a second pod on the same node still cannot mount a claim that a
# running pod holds - hence the scale to 0 first. `cp -a /old/. /new/` copies the CONTENTS
# (including dotfiles) rather than nesting the directory inside the target.
""",
)
# ------------------------------------------------------------------ 73 ----
@ -542,21 +445,17 @@ for i in $(seq 1 12); do
sleep 3
done; exit 1"""),
],
solution=[cmd(r"""k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend
k -n eta-service get pods --show-labels
k -n eta-service describe svc eta-backend""", r"""No endpoints. The Pods are `app=eta-calculator`, the Service selects `app=eta-service`; and it sends to `TargetPort 9090`, where nginx listens on 80. Empty endpoints is almost always the selector; endpoints that refuse connections, the target port."""),
hand(r"""k -n eta-service edit svc eta-backend""",
r"""Fix both, then `:wq`:
```
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: eta-calculator
```""",
run=r"""kubectl -n eta-service patch svc eta-backend -p '{"spec":{"selector":{"app":"eta-calculator"},"ports":[{"port":80,"targetPort":80,"protocol":"TCP"}]}}'"""),
cmd(r"""k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend""")],
solution="""
k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend # empty
k -n eta-service get deploy eta-calculator -o jsonpath='{.spec.template.metadata.labels}' # app=eta-calculator
k -n eta-service get svc eta-backend -o jsonpath='{.spec.selector}' # app=eta-service <- fault 1
k -n eta-service patch svc eta-backend -p '{"spec":{"selector":{"app":"eta-calculator"}}}'
k -n eta-service patch svc eta-backend -p '{"spec":{"ports":[{"port":80,"targetPort":80,"protocol":"TCP"}]}}'
k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend
# Empty endpoints is almost always the selector; a populated endpoint list that still refuses
# connections is almost always targetPort. Check the endpoints first - it splits the two cases
# in one command.
""",
)
# ------------------------------------------------------------------ 74 ----
@ -622,91 +521,36 @@ sys.exit(0 if ok else 1)'"""),
C("the HTTPRoute is attached to fleet-gateway", 1, r"""
j -n fleet-ops httproute fleet-route '{.spec.parentRefs[*].name}' | grep -qw fleet-gateway"""),
],
solution=[hand(r"""vim gw.yaml""",
r"""The docs page *Gateway API* has a Gateway and an HTTPRoute to start from:
```
solution="""
cat <<EOF | k apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: fleet-gateway
namespace: fleet-ops
metadata: {name: fleet-gateway, namespace: fleet-ops}
spec:
gatewayClassName: fleet-gateway-class
listeners:
- name: http
protocol: HTTP
port: 80
hostname: fleet.example.com
- name: http
protocol: HTTP
port: 80
hostname: fleet.example.com
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: fleet-route
namespace: fleet-ops
metadata: {name: fleet-route, namespace: fleet-ops}
spec:
parentRefs:
- name: fleet-gateway
hostnames:
- fleet.example.com
parentRefs: [{name: fleet-gateway}]
hostnames: [fleet.example.com]
rules:
- matches:
- headers:
- name: X-Fleet-Version
value: canary
backendRefs:
- name: fleet-dashboard-v2
port: 80
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: fleet-dashboard-v1
port: 80
```
The more specific match wins, so the header rule takes canary requests though `/` matches everything.""",
run=r"""cat > gw.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: fleet-gateway
namespace: fleet-ops
spec:
gatewayClassName: fleet-gateway-class
listeners:
- name: http
protocol: HTTP
port: 80
hostname: fleet.example.com
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: fleet-route
namespace: fleet-ops
spec:
parentRefs:
- name: fleet-gateway
hostnames:
- fleet.example.com
rules:
- matches:
- headers:
- name: X-Fleet-Version
value: canary
backendRefs:
- name: fleet-dashboard-v2
port: 80
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: fleet-dashboard-v1
port: 80
EOF"""),
cmd(r"""k apply -f gw.yaml
k -n fleet-ops get gateway,httproute""")],
- matches:
- headers: [{type: Exact, name: X-Fleet-Version, value: canary}]
backendRefs: [{name: fleet-dashboard-v2, port: 80}]
- matches:
- path: {type: PathPrefix, value: /}
backendRefs: [{name: fleet-dashboard-v1, port: 80}]
EOF
# Order matters: HTTPRoute picks the most specific match, and a bare PathPrefix "/" matches
# everything - so the header rule must be able to win. Listing it first also keeps it readable.
""",
)
# ------------------------------------------------------------------ 75 ----
@ -870,49 +714,29 @@ eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \
&& $n -o jsonpath='{.spec.policyTypes}' | grep -q Egress \
&& [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ]"""),
],
solution=[cmd(r"""k -n kube-system get ds
echo kindnet > {WS}/q77/cni.txt""", r"""`kindnet` runs on every node: it is the CNI. `kube-proxy` is not."""),
cmd(r"""k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv
k -n zero-trust expose pod srv --port 80
k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600""", r"""Prove it rather than remember it: two Pods, a policy that denies everything, and a request between them."""),
hand(r"""k -n zero-trust get pods""",
r"""Again until both are `Running`.""",
run=r"""kubectl -n zero-trust wait --for=condition=Ready pod/srv pod/cli --timeout=180s"""),
cmd(r"""k -n zero-trust exec cli -- wget -qO- -T3 http://srv""", r"""nginx's page: no policy yet."""),
hand(r"""vim deny.yaml""",
r"""The default deny for both directions from the docs page *Network Policies*: every Pod, both policy types, no rules.
```
solution="""
k -n kube-system get ds # kindnet -> {WS}/q77/cni.txt
# Don't answer from memory - prove it. Two pods, one deny policy, one wget:
k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv; k -n zero-trust expose pod srv --port 80
k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600
k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null && echo reachable
cat <<EOF | k apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: zero-trust
metadata: {name: default-deny-all, namespace: zero-trust}
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
```""",
run=r"""cat > deny.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: zero-trust
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
EOF"""),
cmd(r"""k apply -f deny.yaml
k -n zero-trust exec cli -- wget -qO- -T3 http://srv""", r"""`download timed out`: the CNI enforces it. (Flannel would store the policy and enforce nothing; Calico and Cilium enforce. Recent kindnet has a policy engine of its own.)"""),
hand(r"""echo yes > {WS}/q77/enforced.txt""",
r"""What the test showed.""",
run=r"""sleep 6
if kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null 2>&1; then echo no; else echo yes; fi > {WS}/q77/enforced.txt"""),
cmd(r"""k -n zero-trust delete pod srv cli --now
k -n zero-trust delete svc srv""", r"""Leave only the policy.""")],
policyTypes: [Ingress, Egress]
EOF
k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null || echo BLOCKED # -> BLOCKED here
echo yes > {WS}/q77/enforced.txt
k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv # leave only the policy
# podSelector {} means every pod in the namespace; naming both policyTypes with no ingress/egress
# blocks listed means "deny everything". Recent kindnet builds (this cluster's included) embed a
# policy engine - look for `nft list table inet kindnet-network-policies` on a node. Flannel stores
# NetworkPolicies and enforces nothing; Calico and Cilium enforce. A policy that "works" on a
# non-enforcing CNI is the classic hardening-review trap.
""",
)
# ------------------------------------------------------------------ 78 ----
@ -943,16 +767,23 @@ for i in $(seq 1 20); do [ "$(j -n rider-support deploy support-portal '{.status
eq "$(j -n rider-support deploy support-portal '{.status.readyReplicas}')" 2 \
&& eq "$(j -n rider-support pod -l app=support-portal '{.items[0].spec.priority}')" 1000000"""),
],
solution=[cmd(r"""k create priorityclass rider-support-priority --value=1000000 --description='Priority for rider support services'""", r"""Not the global default unless `--global-default`."""),
hand(r"""k -n rider-support edit deploy support-portal""",
r"""In the Pod template's `spec:`, beside `containers:` and indented like it, then `:wq`:
```
priorityClassName: rider-support-priority
```
The value is copied into each Pod when it is created, so only new Pods carry it - the edit rolls them.""",
run=r"""kubectl -n rider-support patch deploy support-portal -p '{"spec":{"template":{"spec":{"priorityClassName":"rider-support-priority"}}}}'"""),
cmd(r"""k -n rider-support rollout status deploy support-portal
k -n rider-support get pods -o custom-columns=NAME:.metadata.name,PRIORITY:.spec.priority""")],
solution="""
cat <<EOF | k apply -f -
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: {name: rider-support-priority}
value: 1000000
globalDefault: false
description: Priority for rider support services
EOF
k -n rider-support patch deploy support-portal \\
-p '{"spec":{"template":{"spec":{"priorityClassName":"rider-support-priority"}}}}'
k -n rider-support rollout status deploy support-portal
k -n rider-support get pod -l app=support-portal -o jsonpath='{.items[0].spec.priority}'
# The admission controller copies the class's value into pod.spec.priority at creation time, so
# existing pods keep their old priority until they are recreated - patching the template does
# that for you. globalDefault: true would apply the class to every pod without one, cluster-wide.
""",
)
# ------------------------------------------------------------------ 79 ----
@ -1061,35 +892,31 @@ un=$(grep -n 'uncordon' "$f" | head -1 | cut -d: -f1)
grep -Eq 'kubeadm[[:space:]]+upgrade[[:space:]]+node' "$WS/q80/worker.txt" 2>/dev/null \
&& ! grep -q 'upgrade apply' "$WS/q80/worker.txt" """),
],
solution=[cmd(r"""k version
solution="""
k version -o json | jq -r .serverVersion.gitVersion > {WS}/q80/current.txt
k get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion --no-headers > {WS}/q80/kubelets.txt
cat {WS}/q80/kubelets.txt"""),
hand(r"""echo v1.37.0 > {WS}/q80/current.txt""",
r"""With the `Server Version` that `k version` printed.""",
run=r"""kubectl version -o json | jq -r .serverVersion.gitVersion > {WS}/q80/current.txt"""),
hand(r"""vim {WS}/q80/steps.sh""",
r"""The order of the docs page *Upgrading kubeadm clusters*: kubeadm first, which upgrades the control plane; then the node's kubelet and kubectl, drained; uncordon last. The next minor is 1.38 - first point `/etc/apt/sources.list.d/kubernetes.list` at the v1.38 repository, and take its newest patch (`apt-cache madison kubeadm`):
```
cat > {WS}/q80/steps.sh <<'EOF'
sed -i 's|/v1.37/|/v1.38/|' /etc/apt/sources.list.d/kubernetes.list
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.38.0-1.1 && apt-mark hold kubeadm
kubeadm upgrade plan
kubeadm upgrade apply v1.38.0
kubectl drain {CP} --ignore-daemonsets
apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
kubeadm upgrade apply v1.38.0 -y
kubectl drain <control-plane> --ignore-daemonsets
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon {CP}
```""",
run=r"""cat > {WS}/q80/steps.sh <<'EOF'
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.38.0-1.1 && apt-mark hold kubeadm
kubeadm upgrade plan
kubeadm upgrade apply v1.38.0
kubectl drain {CP} --ignore-daemonsets
apt-mark unhold kubelet kubectl && apt-get update && apt-get install -y kubelet=1.38.0-1.1 kubectl=1.38.0-1.1 && apt-mark hold kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon {CP}
EOF"""),
cmd(r"""echo 'kubeadm upgrade node' > {WS}/q80/worker.txt""", r"""A worker hosts no control plane, so there is nothing for `upgrade apply` to do there; `upgrade node` updates its kubelet configuration.""")],
kubectl uncordon <control-plane>
EOF
echo 'kubeadm upgrade node' > {WS}/q80/worker.txt
# The cluster runs v1.37, so the next minor is 1.38 - and pkgs.k8s.io serves one apt repo per
# minor, so the repo line has to move to v1.38 before 1.38 packages exist for apt at all. (1.38
# is due in December 2026; until then that repo is empty, which is why this is only a plan. Take
# the newest 1.38 patch on offer then: `apt-cache madison kubeadm`.)
# kubeadm goes first because it drives the whole thing; the control plane is upgraded by
# `upgrade apply`, and only then do the node binaries (kubelet/kubectl) follow, ending with
# uncordon. Workers run `kubeadm upgrade node` - they host no control-plane static pods, so
# there is nothing for `upgrade apply` to do there. The kubelet may trail the API server by up
# to three minor versions, but must never be ahead of it.
""",
)
# ------------------------------------------------------------------ 81 ----
@ -1150,44 +977,35 @@ done; exit 1"""),
C("fare-calculator pods are Running and Ready", 3, r"""
for i in $(seq 1 24); do [ "$(j -n fare-engine deploy fare-calculator '{.status.readyReplicas}')" -ge 1 ] 2>/dev/null && exit 0; sleep 5; done; exit 1"""),
],
solution=[cmd(r"""k -n fare-engine get pvc,pods
k -n fare-engine describe pvc fare-data-pvc | tail -3
k get sc""", r"""`storageclass.storage.k8s.io "fast-storage" not found`: the class here is `standard`."""),
cmd(r"""k -n fare-engine delete pvc fare-data-pvc""", r"""A claim's `storageClassName` cannot change: make it again."""),
hand(r"""vim pvc.yaml""",
r"""```
solution="""
k -n fare-engine get pvc # Pending
k -n fare-engine describe pvc fare-data-pvc # storageclass.storage.k8s.io "fast-storage" not found
k get sc # on kind the class is called `standard`
# A PVC's storageClassName is immutable, so recreate it with the name your cluster really has:
k -n fare-engine delete pvc fare-data-pvc
cat <<EOF | k apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: fare-data-pvc
namespace: fare-engine
metadata: {name: fare-data-pvc, namespace: fare-engine}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
```""",
run=r"""cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: fare-data-pvc
namespace: fare-engine
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
EOF"""),
cmd(r"""k apply -f pvc.yaml
k -n fare-engine describe pod -l app=fare-calculator | tail -3
k get nodes -L disk""", r"""`didn't match Pod's node affinity/selector`: it wants `disk=nvme`, which no node has. `describe pod` names only this one; the claim's fault you had to find yourself."""),
cmd(r"""k label node {W1} disk=nvme
k -n fare-engine rollout status deploy fare-calculator""", r"""Labelling a node is the smaller change than dropping the rule. `standard` binds only once the Pod is placed (`WaitForFirstConsumer`), so the claim goes `Bound` now too.""")],
EOF
k -n fare-engine describe pod -l app=fare-calculator # 0/4 nodes match node affinity disk in [nvme]
k get nodes --show-labels | grep -c 'disk=nvme' # 0
# Either label a node to satisfy the rule, or drop the rule. Labelling is the smaller change:
k label node <a-worker> disk=nvme
k -n fare-engine rollout status deploy fare-calculator
# Two independent blockers stack up silently: the PVC never binds, AND nothing would schedule
# even if it did. `describe pod` reports only the scheduling failure, so check the PVC too
# whenever a pod that mounts one is stuck Pending.
# Never assume the class name - `standard` here, `local-path` on a stock local-path-provisioner
# install, something else on a cloud cluster. `k get sc` first, every time; a PVC naming a class
# that does not exist sits Pending forever with no other symptom.
# `standard` is WaitForFirstConsumer, so the PVC only binds once the pod can actually be placed -
# which is why fixing the affinity is what finally unblocks the storage too.
""",
)
# ------------------------------------------------------------------ 82 ----
@ -1236,20 +1054,30 @@ ok=any(p.get("type")=="Pods" and int(p.get("value",0))==2 and int(p.get("periodS
for p in b.get("policies",[]))
sys.exit(0 if ok else 1)'"""),
],
solution=[cmd(r"""k -n route-planner autoscale deploy route-planner --name=route-planner-hpa --min=1 --max=6 --memory=70% --dry-run=client -o yaml > hpa.yaml""", r"""`autoscale` writes an `autoscaling/v2` HPA with the memory target; it has no flag for `behavior`."""),
hand(r"""vim hpa.yaml""",
r"""Add to its `spec:`, beside `maxReplicas:` (the docs page *Horizontal Pod Autoscaling* has the block under *Configurable scaling behavior*):
```
solution="""
cat <<EOF | k apply -f -
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: route-planner-hpa, namespace: route-planner}
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: route-planner}
minReplicas: 1
maxReplicas: 6
metrics:
- type: Resource
resource:
name: memory
target: {type: Utilization, averageUtilization: 70}
behavior:
scaleUp:
policies:
- type: Pods
value: 2
periodSeconds: 60
```""",
run=r"""yq -i '.spec.behavior.scaleUp.policies = [{"type": "Pods", "value": 2, "periodSeconds": 60}]' hpa.yaml"""),
cmd(r"""k apply -f hpa.yaml
k -n route-planner get hpa route-planner-hpa""", r"""Utilisation is a percentage of the request: the Deployment has a memory request, so the target reads a number, not `<unknown>`.""")],
- {type: Pods, value: 2, periodSeconds: 60}
EOF
k -n route-planner get hpa route-planner-hpa
# `kubectl autoscale` can set a memory target (--memory=70%) but has no flag for a behavior block,
# so memory + behavior means writing the v2 YAML by hand (or autoscale, then kubectl edit). Utilisation is a percentage OF THE REQUEST, so the Deployment must
# declare a memory request or the HPA reports <unknown> and never scales.
""",
)
# ------------------------------------------------------------------ 83 ----
@ -1303,81 +1131,26 @@ ok=any(not r.get("to") and {p.get("protocol") for p in r.get("ports",[]) if str(
for r in eg)
sys.exit(0 if ok else 1)'"""),
],
solution=[hand(r"""vim netpol.yaml""",
r"""Paste the example from the docs page *Network Policies* and change it to:
```
solution="""
cat <<EOF | k apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: safety-api-policy
namespace: safety-monitor
metadata: {name: safety-api-policy, namespace: safety-monitor}
spec:
podSelector:
matchLabels:
app: safety-api
policyTypes:
- Ingress
- Egress
podSelector: {matchLabels: {app: safety-api}}
policyTypes: [Ingress, Egress]
ingress:
- from:
- podSelector:
matchLabels:
app: safety-dashboard
ports:
- protocol: TCP
port: 8443
- from: [{podSelector: {matchLabels: {app: safety-dashboard}}}]
ports: [{protocol: TCP, port: 8443}]
egress:
- to:
- podSelector:
matchLabels:
app: safety-db
ports:
- protocol: TCP
port: 5432
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
```
The DNS rule has no `to:` - only ports, so port 53 anywhere. With `Egress` in `policyTypes` everything else outbound is denied, and forgetting DNS breaks every name lookup in the Pod.""",
run=r"""cat > netpol.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: safety-api-policy
namespace: safety-monitor
spec:
podSelector:
matchLabels:
app: safety-api
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: safety-dashboard
ports:
- protocol: TCP
port: 8443
egress:
- to:
- podSelector:
matchLabels:
app: safety-db
ports:
- protocol: TCP
port: 5432
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
EOF"""),
cmd(r"""k apply -f netpol.yaml
k -n safety-monitor describe netpol safety-api-policy""")],
- to: [{podSelector: {matchLabels: {app: safety-db}}}]
ports: [{protocol: TCP, port: 5432}]
- ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
EOF
# The DNS rule deliberately has no `to:` - CoreDNS lives in kube-system, and an egress rule with
# only `ports` allows that port to anywhere. Naming Egress in policyTypes denies all outbound
# traffic by default, so forgetting DNS is what silently breaks every name lookup in the pod.
""",
)
# ------------------------------------------------------------------ 84 ----

413
cka/bank_sim_d.py
View File

@ -16,7 +16,7 @@ task was seeded, solved and graded on an isolated kind cluster before it went in
Run with: ./ksim sim d
"""
from sim.bank import C, NS, cmd, hand
from sim.bank import C, NS
Q = []
# Tasks written for the pre-2025 exam opened by switching kubectl context. Every task now has a host
@ -73,27 +73,25 @@ if [ "$own" = ReplicaSet ]; then want="deployment $(j -n kube-system rs "$name"
else want="$(echo "$own" | tr A-Z a-z) $name"; fi
eq "$(sed -n 's/^ *dns *: *//p' "$f" | head -1 | tr -s ' ' | tr A-Z a-z | sed 's/ *$//')" "$want" """),
],
solution=[cmd(r"""systemctl status kubelet | head -3
sudo ls /etc/kubernetes/manifests""", r"""The host is the control-plane node. The kubelet is a systemd service - a process; etcd and the three control-plane components are manifests in the static Pod path."""),
cmd(r"""k -n kube-system get pods -o wide
k -n kube-system get deploy,ds""", r"""Static Pods carry the node's name as a suffix. What answers DNS is the `coredns` Deployment, though its Service is called `kube-dns`; `kube-proxy` and `kindnet` are DaemonSets."""),
hand(r"""vim {WS}/q86/controlplane.txt""",
r"""```
solution="""
sudo -i # the task's host is the control-plane node: no hop needed
systemctl status kubelet | head -3 # a systemd service: a plain process on the node
ls /etc/kubernetes/manifests # etcd, kube-apiserver, kube-controller-manager, kube-scheduler
exit
k -n kube-system get pod -o wide # static pods carry the node name as a suffix...
k -n kube-system get pod kube-apiserver-{CP} -o jsonpath='{.metadata.ownerReferences[0].kind}{"\\n"}' # ...and are owned by the Node
k -n kube-system get deploy,ds # coredns is a Deployment (kube-proxy, kindnet are DaemonSets)
cat > {WS}/q86/controlplane.txt <<'EOF'
kubelet: process
kube-apiserver: static-pod
kube-scheduler: static-pod
kube-controller-manager: static-pod
etcd: static-pod
dns: deployment coredns
```""",
run=r"""cat > {WS}/q86/controlplane.txt <<'EOF'
kubelet: process
kube-apiserver: static-pod
kube-scheduler: static-pod
kube-controller-manager: static-pod
etcd: static-pod
dns: deployment coredns
EOF""")],
EOF
# The Service is called kube-dns for historical reasons; what actually answers is the coredns
# Deployment. Follow the chain svc selector -> pod -> ownerReferences when unsure.
""",
)
# ------------------------------------------------------------------ 87 ----
@ -127,27 +125,24 @@ cfg=$(nx "$W1" 'ls /etc/cni/net.d/*.conflist /etc/cni/net.d/*.conf 2>/dev/null |
C("static Pod name suffix on {W3} (line 5)", 1, r"""
v=$(sed -n 's/^ *5 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]'); [ "$v" = "-$W3" ] || [ "$v" = "$W3" ]"""),
],
solution=[cmd(r"""k get nodes
k -n kube-system describe pod -l component=kube-apiserver | grep service-cluster-ip-range
k -n kube-system get ds""", r"""`ROLES control-plane` on one node, the rest workers; the API server's `--service-cluster-ip-range`; and the CNI's DaemonSet, `kindnet`."""),
hand(r"""ssh {W1} ls /etc/cni/net.d""",
r"""The CNI's configuration on a node: `10-kindnet.conflist`.""",
run=r""""""),
hand(r"""vim {WS}/q87/cluster-info.txt""",
r"""The values you read - here:
```
solution="""
k get nodes # ROLES shows control-plane; count the rest
k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range
k -n kube-system get ds # kindnet
{SHELL_W1}
sudo -i
ls /etc/cni/net.d/ # 10-kindnet.conflist
exit; exit
cat > {WS}/q87/cluster-info.txt <<'EOF'
1: 1
2: 3
3: 10.96.0.0/16
4: kindnet, /etc/cni/net.d/10-kindnet.conflist
5: -{W3}
```
A static Pod's mirror is named `<pod>-<node>`, e.g. `kube-apiserver-{CP}`.""",
run=r"""C=$(kubectl get nodes --no-headers -l node-role.kubernetes.io/control-plane | wc -l)
W=$(kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' | wc -l)
CIDR=$(kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -o 'service-cluster-ip-range=[^ ]*' | head -1 | cut -d= -f2)
F=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
printf '1: %s\n2: %s\n3: %s\n4: kindnet, %s\n5: -{W3}\n' "$C" "$W" "$CIDR" "$F" > {WS}/q87/cluster-info.txt""")],
EOF
# A static Pod's mirror is named <pod>-<node name>, e.g. kube-apiserver-{CP}. Values come from THIS
# cluster - check each one instead of trusting the ones shown here.
""",
)
# ------------------------------------------------------------------ 88 ----
@ -223,19 +218,18 @@ grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-n|--namespa
timeout 60 bash "$f" 2>/dev/null | grep -q etcd"""),
C("no aliases in either script", 1, r"""[ -s "$WS/q89/pods-by-age.sh" ] && [ -s "$WS/q89/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q89/pods-by-age.sh" "$WS/q89/pods-by-uid.sh" """),
],
solution=[hand(r"""vim {WS}/q89/pods-by-age.sh""",
r"""`kubectl` in full: a script runs in a fresh shell, where the `k` alias does not exist.
```
solution="""
cat > {WS}/q89/pods-by-age.sh <<'EOF'
kubectl get pods -A --sort-by=.metadata.creationTimestamp
```""",
run=r"""echo 'kubectl get pods -A --sort-by=.metadata.creationTimestamp' > {WS}/q89/pods-by-age.sh"""),
hand(r"""vim {WS}/q89/pods-by-uid.sh""",
r"""```
EOF
cat > {WS}/q89/pods-by-uid.sh <<'EOF'
kubectl -n kube-system get pods --sort-by=.metadata.uid
```""",
run=r"""echo 'kubectl -n kube-system get pods --sort-by=.metadata.uid' > {WS}/q89/pods-by-uid.sh"""),
cmd(r"""chmod +x {WS}/q89/*.sh
{WS}/q89/pods-by-age.sh | head -3""", r"""The control-plane Pods first: they are the oldest. `--sort-by` takes a JSONPath into each item, ascending only.""")],
EOF
chmod +x {WS}/q89/*.sh
bash {WS}/q89/pods-by-age.sh | head -3 # the control-plane pods come first - they are oldest
# --sort-by takes a JSONPath into each item; ascending only, so "newest first" is `| tac` or
# sorting by a field that grows the other way.
""",
)
# ------------------------------------------------------------------ 90 ----
@ -279,35 +273,31 @@ for f in pod-deleted.log container-killed.log; do
done"""),
C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q90/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""),
],
solution=[hand(r"""vim {WS}/q90/cluster-events.sh""",
r"""```
solution="""
cat > {WS}/q90/cluster-events.sh <<'EOF'
kubectl get events -A --sort-by=.metadata.creationTimestamp
```""",
run=r"""echo 'kubectl get events -A --sort-by=.metadata.creationTimestamp' > {WS}/q90/cluster-events.sh"""),
cmd(r"""chmod +x {WS}/q90/cluster-events.sh
k -n kube-system get pods -l k8s-app=kube-proxy -o wide""", r"""Note the name of the one on `{W2}`."""),
hand(r"""k -n kube-system delete pod kube-proxy-xxxxx""",
r"""With that name. The DaemonSet replaces it with a new Pod at once.""",
run=r"""P=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system delete pod $P
for i in $(seq 1 40); do
[ "$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].status.phase}' 2>/dev/null)" = Running ] && break; sleep 3
done
sleep 5"""),
cmd(r"""k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
cat {WS}/q90/pod-deleted.log""", r"""`Killing` the old Pod, `SuccessfulCreate` by the DaemonSet, then `Scheduled`, `Pulled`, `Created`, `Started` for the new one."""),
hand(r"""{SHELL_W2}
EOF
chmod +x {WS}/q90/cluster-events.sh
k -n kube-system get pod -l k8s-app=kube-proxy -o wide # the one on {W2}
k -n kube-system delete pod <kube-proxy-on-{W2}>
k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
# Killing (old pod), SuccessfulCreate (DaemonSet), Scheduled, Pulled, Created, Started (new pod)
{SHELL_W2}
sudo -i
crictl ps --name kube-proxy
crictl stop 1a2b3c4d5e6f
exit
exit""",
r"""With the container ID `crictl ps` printed. Only the container goes: the kubelet starts a new one in the same Pod.""",
run=r"""ssh -n {W2} 'sudo crictl stop $(sudo crictl ps --name kube-proxy -q)'
sleep 25"""),
cmd(r"""k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
k -n kube-system get pods -l k8s-app=kube-proxy -o wide
echo container > {WS}/q90/restarts.txt""", r"""`RESTARTS 1` on `{W2}`: a new container in the same Pod is a restart; a new Pod starts again at 0.""")],
crictl ps --name kube-proxy # container id
crictl stop <id>
exit; exit
k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
# Pulled, Created, Started - same Pod, new container
k -n kube-system get pod -l k8s-app=kube-proxy -o wide # RESTARTS 1 on {W2}
echo container > {WS}/q90/restarts.txt
# Deleting the Pod makes the DaemonSet create a brand-new Pod (RESTARTS 0). Killing only the
# container makes the kubelet start a new container inside the SAME Pod - that is a restart.
# Do not `crictl rm` it straight away: on v1.35 the kubelet counts the restart from the exited
# container it sees, and if rm wins the race it starts a fresh attempt 0 and RESTARTS stays 0.
""",
)
# ------------------------------------------------------------------ 91 ----
@ -343,15 +333,15 @@ got=$(awk '{print $1}' "$f" | sed 's/\..*//' | grep . | sort -u)
best=$(for ns in census-north census-south census-east census-west; do echo "$ns $($K -n $ns get role --no-headers 2>/dev/null | wc -l)"; done | sort -k2 -n | tail -1)
eq "$(tr -s ' ' < "$WS/q91/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """),
],
solution=[cmd(r"""k api-resources --namespaced=true -o name > {WS}/q91/namespaced.txt
head -3 {WS}/q91/namespaced.txt""", r"""`-o name` prints the plural names alone (`deployments.apps`: name and group); `--namespaced=false` would give the cluster-scoped half."""),
cmd(r"""k -n census-north get role --no-headers | wc -l
k -n census-south get role --no-headers | wc -l
k -n census-east get role --no-headers | wc -l
k -n census-west get role --no-headers | wc -l""", r"""Or all at once: `k get role -A | grep census`."""),
hand(r"""echo "census-east 9" > {WS}/q91/most-roles.txt""",
r"""With the namespace that has the most, and its count.""",
run=r"""for ns in census-north census-south census-east census-west; do echo "$ns $(kubectl -n $ns get role --no-headers | wc -l)"; done | sort -k2 -n | tail -1 > {WS}/q91/most-roles.txt""")],
solution="""
k api-resources --namespaced=true -o name > /tmp/r.txt # deployments.apps, ... (group suffix)
k api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q91/namespaced.txt
for ns in census-north census-south census-east census-west; do echo "$ns $(k -n $ns get role --no-headers | wc -l)"; done
# pick the largest, e.g. echo "census-east 8" > {WS}/q91/most-roles.txt
# --namespaced=false gives the cluster-scoped half (nodes, pv, storageclasses, clusterroles...).
# `k get role -A --no-headers | awk '{print $1}' | sort | uniq -c | sort -n` counts every
# namespace at once.
""",
)
# ------------------------------------------------------------------ 92 ----
@ -490,27 +480,20 @@ ip=$(j -n default pod "ops-beacon-$CP" '{.status.podIP}'); [ -n "$ip" ] || exit
j -n default endpointslices -l kubernetes.io/service-name=ops-beacon-svc '{.items[*].endpoints[*].addresses[*]}' | tr ' ' '\n' | grep -qx "$ip" """),
C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q94/nodeport.txt" 2>/dev/null)" "$np" """),
],
solution=[cmd(r"""k -n default run ops-beacon --image=nginx:1.27 --labels=app=ops-beacon --dry-run=client -o yaml > beacon.yaml"""),
hand(r"""vim beacon.yaml""",
r"""Replace the container's `resources: {}` with:
```
resources:
requests:
cpu: 10m
memory: 20Mi
```""",
run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "10m", "memory": "20Mi"}}' beacon.yaml"""),
cmd(r"""sudo grep staticPodPath /var/lib/kubelet/config.yaml
sudo cp beacon.yaml /etc/kubernetes/manifests/ops-beacon.yaml""", r"""The host is the control-plane node. Its taint keeps the scheduler away, not the kubelet: static Pods run regardless."""),
hand(r"""k -n default get pod ops-beacon-{CP} -o wide""",
r"""Again until the mirror Pod is `Running`.""",
run=r"""for i in $(seq 1 60); do [ "$(kubectl -n default get pod ops-beacon-{CP} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done"""),
cmd(r"""k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
k -n default get svc ops-beacon-svc
k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc""", r"""`expose pod` makes the selector from the Pod's labels: the mirror Pod's IP is the endpoint. Note the node port, after `80:`."""),
hand(r"""echo 31234 > {WS}/q94/nodeport.txt""",
r"""With the node port `get svc` showed.""",
run=r"""kubectl -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt""")],
solution="""
k run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon $do > beacon.yaml
# add to the container: resources: {requests: {cpu: 10m, memory: 20Mi}}
sudo -i # the task's host is the control-plane node: no hop needed
grep staticPodPath /var/lib/kubelet/config.yaml # /etc/kubernetes/manifests
vi /etc/kubernetes/manifests/ops-beacon.yaml # paste beacon.yaml
exit
k -n default get pod ops-beacon-{CP} -o wide # mirror pod, named <pod>-<node>
k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc # the mirror pod's IP
k -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt
# The control-plane taint only keeps the SCHEDULER from placing pods there; the kubelet runs its
# static pods regardless. `expose pod` copies the pod's labels into the selector.
""",
)
# ------------------------------------------------------------------ 95 ----
@ -774,107 +757,40 @@ for i in 0 1; do eq "$($K -n lima-data exec ledger-db-$i -- cat /data/id.txt 2>/
want=$($K -n lima-data get pvc --no-headers -o custom-columns=N:.metadata.name 2>/dev/null | grep '^data-ledger-db-' | sort)
[ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q99/pvcs.txt" 2>/dev/null)" = "$want" ]"""),
],
solution=[hand(r"""vim sts.yaml""",
r"""The docs page *StatefulSets* has a headless Service and a StatefulSet with `volumeClaimTemplates` together; change them to:
```
solution="""
cat <<EOF | k apply -f -
apiVersion: v1
kind: Service
metadata:
name: ledger-db
namespace: lima-data
spec:
clusterIP: None
selector:
app: ledger-db
ports:
- port: 5432
metadata: {name: ledger-db, namespace: lima-data}
spec: {clusterIP: None, selector: {app: ledger-db}, ports: [{port: 5432}]}
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: ledger-db
namespace: lima-data
metadata: {name: ledger-db, namespace: lima-data}
spec:
serviceName: ledger-db
replicas: 3
selector:
matchLabels:
app: ledger-db
selector: {matchLabels: {app: ledger-db}}
template:
metadata:
labels:
app: ledger-db
metadata: {labels: {app: ledger-db}}
spec:
containers:
- name: db
image: busybox:1.36
command: ["sh", "-c", "hostname > /data/id.txt && sleep 86400"]
volumeMounts:
- name: data
mountPath: /data
- name: db
image: busybox:1.36
command: [sh, -c, "hostname > /data/id.txt && sleep 86400"]
volumeMounts: [{name: data, mountPath: /data}]
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: standard
resources:
requests:
storage: 100Mi
```""",
run=r"""cat > sts.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: ledger-db
namespace: lima-data
spec:
clusterIP: None
selector:
app: ledger-db
ports:
- port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: ledger-db
namespace: lima-data
spec:
serviceName: ledger-db
replicas: 3
selector:
matchLabels:
app: ledger-db
template:
metadata:
labels:
app: ledger-db
spec:
containers:
- name: db
image: busybox:1.36
command: ["sh", "-c", "hostname > /data/id.txt && sleep 86400"]
volumeMounts:
- name: data
mountPath: /data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: standard
resources:
requests:
storage: 100Mi
EOF"""),
cmd(r"""k apply -f sts.yaml
- metadata: {name: data}
spec: {accessModes: [ReadWriteOnce], storageClassName: standard, resources: {requests: {storage: 100Mi}}}
EOF
k -n lima-data rollout status sts ledger-db
k -n lima-data scale sts ledger-db --replicas=2"""),
cmd(r"""k -n lima-data get pvc -o custom-columns=NAME:.metadata.name --no-headers | sort > {WS}/q99/pvcs.txt
cat {WS}/q99/pvcs.txt""", r"""All three, `data-ledger-db-0` to `-2`: claims are named `<template>-<statefulset>-<ordinal>` and outlive their Pods, so scaling up again gives `ledger-db-2` its old data back.""")],
k -n lima-data scale sts ledger-db --replicas=2
k -n lima-data get pvc -o custom-columns=N:.metadata.name --no-headers | grep '^data-ledger-db-' | sort > {WS}/q99/pvcs.txt
cat {WS}/q99/pvcs.txt # all THREE: data-ledger-db-0, -1, -2
# Claims are named <template>-<statefulset>-<ordinal> and outlive their pods: scaling down (or
# deleting the StatefulSet) keeps them unless spec.persistentVolumeClaimRetentionPolicy says
# otherwise, so scaling back up re-attaches ledger-db-2 to its old data.
""",
)
# ------------------------------------------------------------------ 100 ----
@ -933,63 +849,28 @@ $K -n oslo-db get netpol db-ingress >/dev/null 2>&1 || exit 1
! $K -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1 \
&& ! $K -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1"""),
],
solution=[cmd(r"""k get ns --show-labels | grep -e oslo-api -e lima-api
k -n oslo-api get pods --show-labels""", r"""`env=prod` and `env=dev`; `api` is `role=backend`, `debug` is not."""),
hand(r"""vim netpol.yaml""",
r"""One `from` entry holding both selectors - both must match:
```
solution="""
k get ns --show-labels | grep -E 'oslo-api|lima-api' # env=prod / env=dev
cat <<EOF | k apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: db-ingress
namespace: oslo-db
metadata: {name: db-ingress, namespace: oslo-db}
spec:
podSelector:
matchLabels:
app: db
policyTypes:
- Ingress
podSelector: {matchLabels: {app: db}}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels:
env: prod
podSelector:
matchLabels:
role: backend
ports:
- protocol: TCP
port: 80
```
The trap is one dash: `- podSelector:` would make it a second entry, and the rule an OR - any Pod in a prod namespace, or any backend Pod in `oslo-db`.""",
run=r"""cat > netpol.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: db-ingress
namespace: oslo-db
spec:
podSelector:
matchLabels:
app: db
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
env: prod
podSelector:
matchLabels:
role: backend
ports:
- protocol: TCP
port: 80
EOF"""),
cmd(r"""k apply -f netpol.yaml
k -n oslo-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db
k -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db
k -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db""", r"""nginx's page, then `download timed out` twice.""")],
- from:
- namespaceSelector: {matchLabels: {env: prod}}
podSelector: {matchLabels: {role: backend}} # SAME list item = AND
ports: [{protocol: TCP, port: 80}]
EOF
k -n oslo-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db | head -1 # <!DOCTYPE html>
k -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db # download timed out
k -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db # download timed out
# The trap is one dash: "- namespaceSelector: ...\\n - podSelector: ..." (two list items) means OR -
# any pod in a prod namespace, OR a backend pod in oslo-db's own namespace. One item with both
# selectors means AND.
""",
)
# ------------------------------------------------------------------ 101 ----
@ -1038,18 +919,18 @@ for i in $(seq 1 15); do
done; exit 1"""),
C("fix.txt names dnsPolicy and its new value", 1, r"""grep -Eqi '^ *dnsPolicy *= *ClusterFirst *$' "$WS/q101/fix.txt" 2>/dev/null"""),
],
solution=[cmd(r"""k -n lima-client logs deploy/reporter --tail=3
k -n lima-client get endpointslices
k -n lima-client get deploy reporter -o yaml | grep dnsPolicy""", r"""`bad address 'ledger:8080'`, though the Service has endpoints. `dnsPolicy: Default` uses the *node's* resolver, not cluster DNS - the default policy is `ClusterFirst`, despite the name."""),
hand(r"""k -n lima-client edit deploy reporter""",
r"""In the Pod template's `spec:`, then `:wq`:
```
dnsPolicy: ClusterFirst
```""",
run=r"""kubectl -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'"""),
cmd(r"""echo dnsPolicy=ClusterFirst > {WS}/q101/fix.txt
k -n lima-client rollout status deploy reporter
k -n lima-client logs deploy/reporter --tail=3""", r"""`ledger ok`.""")],
solution="""
k -n lima-client logs deploy/reporter --tail=3 # wget: bad address 'ledger:8080'
k -n lima-client get endpointslices # the Service has endpoints - not the problem
k -n lima-client get deploy reporter -o yaml | grep -n dns
# dnsPolicy: Default <- "Default" means: use the NODE's resolver, NOT cluster DNS
k -n lima-client exec deploy/reporter -- cat /etc/resolv.conf # the podman/host resolver, no cluster search domains
k -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'
k -n lima-client logs deploy/reporter -f # "ledger ok"
echo dnsPolicy=ClusterFirst > {WS}/q101/fix.txt
# The trap is the name: the DEFAULT dnsPolicy is ClusterFirst, while the policy literally called
# "Default" inherits the node's /etc/resolv.conf. ClusterFirstWithHostNet is for hostNetwork pods.
""",
)
# ------------------------------------------------------------------ 102 ----
@ -1096,16 +977,22 @@ eq "$(j -n oslo-files deploy uploader '{.status.availableReplicas}')" 1 \
&& j -n oslo-files deploy uploader '{.spec.template.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /files"""),
C("reason.txt names the unsupported access mode", 2, r"""grep -Eqi 'ReadWriteMany|RWX|access ?mode' "$WS/q102/reason.txt" 2>/dev/null"""),
],
solution=[cmd(r"""k -n oslo-files get pod,pvc
k -n oslo-files describe pvc shared-files | tail -4""", r"""`ProvisioningFailed ... NodePath only supports ReadWriteOnce and ReadWriteOncePod access modes`: the claim asks for `ReadWriteMany`, which local-path cannot give."""),
cmd(r"""echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
k -n oslo-files get pvc shared-files -o yaml > pvc.yaml"""),
hand(r"""vim pvc.yaml""",
r"""Under `accessModes:`, change `ReadWriteMany` to `ReadWriteOnce` - name, size and class stay.""",
run=r"""yq -i '.spec.accessModes = ["ReadWriteOnce"]' pvc.yaml"""),
cmd(r"""k replace --force -f pvc.yaml
k -n oslo-files rollout restart deploy uploader
k -n oslo-files get pvc,pod""", r"""Access modes cannot change on a claim: `--force` deletes it and creates it again. The Pending Pod was never scheduled, so it does not hold the claim. Again until it is `Bound` and the Pod `Running`.""")],
solution="""
k -n oslo-files get pod # Pending
k -n oslo-files describe pod -l app=uploader | tail -4 # unbound immediate PersistentVolumeClaims / waiting
k -n oslo-files describe pvc shared-files | tail -4
# ProvisioningFailed ... NodePath only supports ReadWriteOnce and ReadWriteOncePod (1.22+) access modes
# accessModes is immutable on a PVC, so recreate it. The Pending pod was never scheduled, so it
# does not hold the claim (pvc-protection only counts scheduled pods) and the delete goes through.
k -n oslo-files get pvc shared-files -o yaml > pvc.yaml # change ReadWriteMany -> ReadWriteOnce, strip status/uid
k -n oslo-files delete pvc shared-files
k apply -f pvc.yaml
k -n oslo-files rollout restart deploy uploader # or just wait - the pending pod retries
k -n oslo-files get pvc,pod
echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
# RWX needs a filesystem many nodes can mount at once (NFS, CephFS, a cloud file service).
# ReadWriteOnce is per NODE, not per pod - ReadWriteOncePod is the per-pod one.
""",
)
QUESTIONS = Q

637
cka/bank_storage.py
View File

@ -5,8 +5,6 @@ served by the local-path provisioner in Namespace local-path-storage), so a task
provisioner or change the default class. Volumes are tested for real - a claim that binds, a file
that is still on the node, a Pod that reads it.
"""
from sim.bank import cmd, hand
from cka.common import C, NS, PY, STOR, ns_pick, name_pick
# what a setup records about the objects a task must leave alone
@ -52,85 +50,44 @@ gone -n %{ns} pvc %{pvc} || exit 1
node=$(pvnode "$pv"); path=$(pvpath "$pv"); [ -n "$node" ] && [ -n "$path" ] || exit 1
eq "$(nx "$node" "cat $path/keep.txt" </dev/null 2>/dev/null | tr -d '\r')" "%{text}" """),
],
"solution": [cmd(r"""k get sc standard -o yaml | grep provisioner""", r"""`rancher.io/local-path`."""),
hand(r"""vim sc.yaml""",
r"""The class, from the docs page *Storage Classes* - no `is-default-class` annotation, so not the default - and the claim:
```
"solution": """
cat <<EOF | k apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: %{sc}
provisioner: rancher.io/local-path
metadata: {name: %{sc}}
provisioner: rancher.io/local-path # k get sc standard -o yaml shows it
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: %{sc}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 300Mi
```""",
run=r"""cat > sc.yaml <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: %{sc}
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 300Mi}}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
storageClassName: %{sc}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 300Mi
EOF"""),
cmd(r"""k apply -f sc.yaml
k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sh -c "echo %{text} > /data/keep.txt; sleep 86400" > pod.yaml""", r"""The claim stays `Pending` until a Pod uses it: `WaitForFirstConsumer`."""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: %{pod}`:
```
volumeMounts:
- name: data
mountPath: /data
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: data
persistentVolumeClaim:
claimName: %{pvc}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}]
| .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "%{pvc}"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod,pvc""",
r"""Again until the Pod is `Running` and the claim `Bound`; note its `VOLUME`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
hand(r"""echo pvc-0b1c2d3e-... > {WS}/q114/pv.txt""",
r"""With the volume's name.""",
run=r"""kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt"""),
cmd(r"""k -n %{ns} delete pod %{pod} --now
k -n %{ns} delete pvc %{pvc}"""),
hand(r"""k get pv
echo Released > {WS}/q114/phase.txt""",
r"""`STATUS Released`, `RECLAIM POLICY Retain`: the class's policy is copied into every volume it provisions. A Released volume binds no new claim until its `claimRef` is cleared.""",
run=r"""sleep 4
kubectl get pv $(cat {WS}/q114/pv.txt) -o jsonpath='{.status.phase}' > {WS}/q114/phase.txt""")],
volumes: [{name: data, persistentVolumeClaim: {claimName: %{pvc}}}]
containers:
- name: writer
image: busybox:1.36
command: [sh, -c, "echo %{text} > /data/keep.txt; sleep 86400"]
volumeMounts: [{name: data, mountPath: /data}]
EOF
k -n %{ns} get pvc %{pvc} # Pending until the Pod is scheduled: WaitForFirstConsumer
k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt
k -n %{ns} delete pod %{pod}; k -n %{ns} delete pvc %{pvc}
k get pv $(cat {WS}/q114/pv.txt) # STATUS Released, RECLAIM POLICY Retain
echo Released > {WS}/q114/phase.txt
# A class's reclaimPolicy is copied into every volume it provisions. Delete (the default) removes
# the volume and its data with the claim; Retain leaves a Released volume that no new claim can
# bind until an administrator clears spec.claimRef. A StorageClass cannot be edited afterwards,
# except for a few fields - to change the policy of one volume, patch the PV.
""",
},
# ------------------------------------------------------------------ 115 ----
{
@ -176,43 +133,39 @@ eq "$(one "$WS/q115/claim.txt" | sed 's#.*/##')" %{pod}-scratch || exit 1
eq "$(j -n %{ns} pvc %{pod}-scratch '{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}')" Pod/%{pod} || exit 1
eq "$(one "$WS/q115/owner.txt" | tr A-Z a-z)" pod/%{pod}"""),
],
"solution": [cmd(r"""k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: %{pod}`:
```
volumeMounts:
- name: scratch
mountPath: /scratch
- name: cache
mountPath: /cache
```
and to the Pod's `spec:`, right below `spec:` (the docs page *Ephemeral Volumes* has the generic ephemeral volume):
```
"solution": """
cat <<EOF | k apply -f -
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
volumes:
- name: scratch
ephemeral:
volumeClaimTemplate:
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
- name: cache
emptyDir:
medium: Memory
sizeLimit: 32Mi
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "scratch", "mountPath": "/scratch"}, {"name": "cache", "mountPath": "/cache"}]
| .spec.volumes = [{"name": "scratch", "ephemeral": {"volumeClaimTemplate": {"spec": {"storageClassName": "standard", "accessModes": ["ReadWriteOnce"], "resources": {"requests": {"storage": "100Mi"}}}}}}, {"name": "cache", "emptyDir": {"medium": "Memory", "sizeLimit": "32Mi"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod,pvc""",
r"""Again until the Pod is `Running`. The claim is `<pod>-<volume>`: `%{pod}-scratch`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
cmd(r"""echo %{pod}-scratch > {WS}/q115/claim.txt
k -n %{ns} get pvc %{pod}-scratch -o yaml | grep -A5 ownerReferences
echo Pod/%{pod} > {WS}/q115/owner.txt""", r"""The Pod owns it: delete the Pod, and the claim and its volume go too. An emptyDir in `Memory` is a tmpfs, counted as the container's memory.""")],
- name: scratch
ephemeral:
volumeClaimTemplate:
spec:
storageClassName: standard
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
- name: cache
emptyDir: {medium: Memory, sizeLimit: 32Mi}
containers:
- name: encoder
image: busybox:1.36
command: [sleep, "86400"]
volumeMounts:
- {name: scratch, mountPath: /scratch}
- {name: cache, mountPath: /cache}
EOF
k -n %{ns} get pvc # %{pod}-scratch: <pod name>-<volume name>
echo %{pod}-scratch > {WS}/q115/claim.txt
k -n %{ns} get pvc %{pod}-scratch -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}'
echo Pod/%{pod} > {WS}/q115/owner.txt
# A generic ephemeral volume is a claim written into the Pod: Kubernetes creates the PVC, the Pod
# owns it, and deleting the Pod deletes claim and volume. Unlike emptyDir it can come from any
# StorageClass - size, speed, snapshots - and unlike a normal claim it never outlives the Pod.
# An emptyDir with medium Memory is a tmpfs: what is written there counts as the container's memory.
""",
},
# ------------------------------------------------------------------ 116 ----
{
@ -281,20 +234,18 @@ gone -n %{ns} pvc data-%{sts}-1 && gone -n %{ns} pvc data-%{sts}-2 && eq "$(j -n
C("claims.txt lists what is left: data-%{sts}-0", 1, r"""
eq "$(tr -d ' \t\r' < "$WS/q116/claims.txt" 2>/dev/null | sed 's#.*/##' | grep . | sort | tr '\n' ' ')" "data-%{sts}-0 " """),
],
"solution": [cmd(r"""k -n %{ns} get sts %{sts} -o yaml | grep -A2 persistentVolumeClaimRetentionPolicy""", r"""`Retain` for both: by default a StatefulSet never deletes a claim."""),
hand(r"""k -n %{ns} edit sts %{sts}""",
r"""Set it before scaling - it applies to the Pods being removed - then `:wq`:
```
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Delete
```""",
run=r"""kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'"""),
cmd(r"""k -n %{ns} scale sts %{sts} --replicas=1"""),
hand(r"""k -n %{ns} get pvc""",
r"""Again until `data-%{sts}-2` and `data-%{sts}-1` have gone with their Pods.""",
run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc --no-headers | wc -l)" = 1 ] && break; sleep 3; done"""),
cmd(r"""echo data-%{sts}-0 > {WS}/q116/claims.txt""")],
"solution": """
k -n %{ns} get sts %{sts} -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}' # whenDeleted Retain, whenScaled Retain
k -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'
k -n %{ns} scale sts %{sts} --replicas=1
k -n %{ns} get pvc -w # data-%{sts}-2 and data-%{sts}-1 disappear with their Pods
k -n %{ns} get pvc -o name | cut -d/ -f2 > {WS}/q116/claims.txt
# By default a StatefulSet never deletes a claim: scale down and up again and each replica finds
# its old data. persistentVolumeClaimRetentionPolicy changes that per event - whenScaled for the
# replicas a scale-down removes, whenDeleted for all of them when the StatefulSet goes. Set the
# policy BEFORE scaling: it is applied to the Pods being removed, not to claims orphaned earlier
# (those you delete by hand). It is one of the few StatefulSet fields that can be changed in place.
""",
},
# ------------------------------------------------------------------ 117 ----
{
@ -341,76 +292,34 @@ eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}/{.spec.storageClassName}/{.spec.
[ -n "$(j pv %{pv} '{.metadata.name}')" ] || exit 1
eq "$(j -n %{oth} pvc any-volume '{.metadata.uid}')" "$(cat "$PRIV/q117.uid")" && eq "$(j -n %{oth} pvc any-volume '{.status.phase}')" Pending"""),
],
"solution": [cmd(r"""k -n %{oth} get pvc any-volume""", r"""`Pending`, for any 1Gi `manual` volume: an Available one would go to it."""),
hand(r"""vim pv.yaml""",
r"""A `claimRef` on the PV reserves it for one claim, so the waiting one cannot take it:
```
"solution": """
cat <<EOF | k apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
name: %{pv}
metadata: {name: %{pv}}
spec:
capacity: {storage: 1Gi}
accessModes: [ReadWriteOnce]
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /mnt/%{pv}
claimRef:
namespace: %{ns}
name: %{pvc}
hostPath: {path: /mnt/%{pv}}
claimRef: {namespace: %{ns}, name: %{pvc}} # the reservation: no other claim can bind this volume
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```
`volumeName` on a claim is the other half: it asks for one volume, but keeps no other claim away from it.""",
run=r"""cat > pv.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: %{pv}
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /mnt/%{pv}
claimRef:
namespace: %{ns}
name: %{pvc}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF"""),
cmd(r"""k apply -f pv.yaml
k get pv %{pv}
k -n %{ns} get pvc %{pvc}
k -n %{oth} get pvc any-volume""", r"""`Bound`, `Bound`, still `Pending`.""")],
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
EOF
k get pv %{pv}; k -n %{ns} get pvc %{pvc}; k -n %{oth} get pvc any-volume # Bound, Bound, Pending
# Without claimRef the volume would be Available, and the control plane hands an Available volume
# to whichever matching claim it looks at first - here the one that has been waiting. That cannot
# be undone cleanly: a claim whose volume is taken away ends up Lost. claimRef on the PV reserves
# it; volumeName on a claim is the other half (it asks for one volume but does not keep others
# away from it). Use both when it has to be this pair and no other.
""",
},
# ------------------------------------------------------------------ 118 ----
{
@ -496,53 +405,30 @@ done; exit 1"""),
eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}')" %{pv} || exit 1
for i in $(seq 1 10); do gone pv "$(cat "$PRIV/q118.wrong")" && exit 0; sleep 3; done; exit 1"""),
],
"solution": [cmd(r"""k -n %{ns} get pvc %{pvc}
k get pv %{pv}""", r"""Bound to a `pvc-...` volume of class `standard`, while `%{pv}` is `Available` with no class: the manifest names no class, so the claim got the default one, and a new empty volume."""),
hand(r"""echo pvc-0b1c2d3e-... > {WS}/q118/wrong-pv.txt""",
r"""With the claim's `VOLUME`.""",
run=r"""kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt"""),
cmd(r"""k -n %{ns} scale deploy %{dep} --replicas=0""", r"""A claim's class and volume cannot be edited, and a claim in use is not deleted (pvc-protection): stop the Pod first."""),
hand(r"""k -n %{ns} get pods""",
r"""Again until the Pod has gone.""",
run=r"""kubectl -n %{ns} wait --for=delete pod -l app=%{dep} --timeout=120s"""),
cmd(r"""k -n %{ns} delete pvc %{pvc}""", r"""Its volume goes with it: reclaim policy `Delete`."""),
hand(r"""vim pvc.yaml""",
r"""`storageClassName: ""` - no class: never the default, never provisioned - and the volume by name:
```
"solution": """
k -n %{ns} get pvc %{pvc} # Bound to pvc-..., STORAGECLASS standard - not to %{pv}
k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt
k get pv %{pv} # Available, STORAGECLASS empty
# The manifest names no class, so the claim was given the cluster's default one (standard) and a
# brand-new, empty volume was provisioned for it. A claim's class and volume cannot be edited:
k -n %{ns} scale deploy %{dep} --replicas=0 # a claim in use is not deleted (pvc-protection)
k -n %{ns} delete pvc %{pvc} # its volume goes with it: reclaim policy Delete
cat <<EOF | k apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: ""
storageClassName: "" # "no class": never the default, never provisioned
volumeName: %{pv}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
```
`volumeName` alone would not do: the default class is still filled in, and it does not match the class-less volume.""",
run=r"""cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
spec:
storageClassName: ""
volumeName: %{pv}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
EOF"""),
cmd(r"""k apply -f pvc.yaml
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 500Mi}}
EOF
k -n %{ns} scale deploy %{dep} --replicas=1
k -n %{ns} rollout status deploy %{dep}
k -n %{ns} logs deploy/%{dep}""", r"""`ledger %{tok} balanced`.""")],
k -n %{ns} logs deploy/%{dep} # ledger %{tok} balanced
# storageClassName has three states: a name; "" = only volumes without a class, no provisioning;
# and absent = "give me the default class". volumeName alone would not have helped: the default
# class is still filled in, and it does not match the class-less volume.
""",
},
# ------------------------------------------------------------------ 119 ----
{
@ -606,120 +492,34 @@ eq "$(j pv %{{p}}-{u} '{{.metadata.uid}}')" "$(sed -n {n}p "$PRIV/q119.uids")" |
eq "$(j pv %{{p}}-{u} '{{.spec.capacity.storage}}/{{.spec.accessModes[*]}}/{{.spec.storageClassName}}/{{.metadata.labels.use}}')" {size}/ReadWriteOnce/manual/{u}""")
for n, (u, size) in enumerate([("reports", "1Gi"), ("archive", "2Gi"), ("media", "1Gi")], 1)
],
"solution": [cmd(r"""k -n %{ns} get pvc
k -n %{ns} describe pvc reports archive media | grep -e ^Name: -e Warning -e Normal""", r"""`media`: `storageclass.storage.k8s.io "manuel" not found` - the class, a typo. The other two only say no volume fits: compare claim and volume yourself."""),
cmd(r"""k -n %{ns} get pvc -o custom-columns=NAME:.metadata.name,SIZE:.spec.resources.requests.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
k get pv -o custom-columns=NAME:.metadata.name,SIZE:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName""", r"""`reports` asks for 5Gi of a 1Gi volume; `archive` for `ReadWriteMany`, its volume offers `ReadWriteOnce`. A claim binds only when the class matches, the volume is large enough, it has every access mode asked for, and the selector matches."""),
hand(r"""vim {WS}/q119/reasons.txt""",
r"""```
reports capacity
archive accessmode
media class
```""",
run=r"""printf 'reports capacity\narchive accessmode\nmedia class\n' > {WS}/q119/reasons.txt"""),
cmd(r"""k -n %{ns} delete pvc reports archive media""", r"""None of the three fields can be edited on a claim: recreate them."""),
hand(r"""vim pvcs.yaml""",
r"""The three claims again, each keeping its selector:
```
"solution": """
k -n %{ns} get pvc; k get pv --show-labels
k -n %{ns} describe pvc reports archive media | grep -E '^Name:|Warning|Normal'
# media: storageclass.storage.k8s.io "manuel" not found -> class (a typo for manual)
# reports, archive: no persistent volumes available for this claim -> compare claim and volume:
k -n %{ns} get pvc -o custom-columns=NAME:.metadata.name,SIZE:.spec.resources.requests.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
k get pv -o custom-columns=NAME:.metadata.name,SIZE:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName
# reports asks for 5Gi, its volume has 1Gi -> capacity
# archive asks for ReadWriteMany, its volume offers ReadWriteOnce -> accessmode
printf 'reports capacity\\narchive accessmode\\nmedia class\\n' > {WS}/q119/reasons.txt
# size (downwards), access modes and class of a claim cannot be edited: recreate the three
k -n %{ns} delete pvc reports archive media
for c in "reports 1Gi" "archive 2Gi" "media 1Gi"; do set -- $c; cat <<EOF | k apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: reports
namespace: %{ns}
metadata: {name: $1, namespace: %{ns}}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: reports
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archive
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: archive
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: media
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: media
resources:
requests:
storage: 1Gi
```""",
run=r"""cat > pvcs.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: reports
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: reports
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archive
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: archive
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: media
namespace: %{ns}
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
selector:
matchLabels:
use: media
resources:
requests:
storage: 1Gi
EOF"""),
cmd(r"""k apply -f pvcs.yaml
k -n %{ns} get pvc""", r"""All three `Bound`, each to its own volume.""")],
accessModes: [ReadWriteOnce]
selector: {matchLabels: {use: $1}}
resources: {requests: {storage: $2}}
EOF
done
k -n %{ns} get pvc # all Bound, each to its own volume
# A claim binds a volume only if ALL of these hold: same class, the volume is at least as large,
# it offers every access mode the claim asks for, and it matches the claim's selector. The event
# only names the class problem - for the rest you compare the two objects yourself.
""",
},
# ------------------------------------------------------------------ 120 ----
{
@ -805,20 +605,24 @@ done; exit 1"""),
eq "$(j pv %{pv} '{.metadata.uid}')" "$(sed -n 1p "$PRIV/q120.uids")" && eq "$(j -n %{ns} pvc %{dep}-data '{.metadata.uid}')" "$(sed -n 2p "$PRIV/q120.uids")" \
&& eq "$(pvnode %{pv})/$(pvpath %{pv})" "$W1//mnt/%{pv}" """),
],
"solution": [hand(r"""k -n %{ns} describe pod -l app=%{dep} | grep FailedScheduling > {WS}/q120/event.txt
cat {WS}/q120/event.txt""",
r"""`1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) didn't match Pod's node affinity/selector`: the scheduler ANDs every constraint - taints, the Pod's selector, and the node affinity of each volume it mounts.""",
run=r"""for i in $(seq 1 40); do
kubectl -n %{ns} describe pod -l app=%{dep} | grep FailedScheduling > {WS}/q120/event.txt && break; sleep 3
done"""),
cmd(r"""k describe pv %{pv} | grep -A3 'Node Affinity'
k -n %{ns} get deploy %{dep} -o yaml | grep -A1 nodeSelector
k get nodes -L disk""", r"""The volume is on `{W1}`; the Pod must go to `disk=ssd`, which only `{W2}` has. The volume cannot move, so the Pod has to."""),
hand(r"""k -n %{ns} edit deploy %{dep}""",
r"""Delete the two lines of the `nodeSelector` (`nodeSelector:` and `disk: ssd`), then `:wq` - the volume's node affinity already sends the Pod to the right node. (If the selector has a reason to exist, `k label node {W1} disk=ssd` instead.)""",
run=r"""kubectl -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'"""),
cmd(r"""k -n %{ns} rollout status deploy %{dep}
k -n %{ns} logs deploy/%{dep}""", r"""`orders %{tok} open`.""")],
"solution": """
k -n %{ns} describe pod -l app=%{dep} | tail -5
# 0/3 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s)
# didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s). ...
k -n %{ns} get events --field-selector reason=FailedScheduling -o jsonpath='{.items[-1:].message}' > {WS}/q120/event.txt
k get pv %{pv} -o jsonpath='{.spec.nodeAffinity}'; echo # the volume is on {W1}
k -n %{ns} get deploy %{dep} -o jsonpath='{.spec.template.spec.nodeSelector}'; echo # disk=ssd
k get nodes -L disk # only {W2} carries disk=ssd
# The volume cannot move, so the Pod has to: drop the nodeSelector (the volume's own node
# affinity already sends the Pod to the right node) ...
k -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
# ... or, if the selector has a reason to exist, give {W1} the label: k label node {W1} disk=ssd
k -n %{ns} rollout status deploy %{dep}; k -n %{ns} logs deploy/%{dep}
# The scheduler ANDs every constraint: taints, the Pod's nodeSelector/affinity, and the node
# affinity of each volume it mounts. "didn't match PersistentVolume's node affinity" (before v1.35:
# "had volume node affinity conflict") always means: the Pod is not allowed onto the one node its
# volume lives on.
""",
},
# ------------------------------------------------------------------ 121 ----
{
@ -1078,15 +882,19 @@ running %{ns} %{pod} 60"""),
C("class.txt holds %{sc}", 1, STOR + r"""
eq "$(j -n %{ns} pvc %{pvc} '{.spec.storageClassName}')" %{sc} && eq "$(one "$WS/q123/class.txt")" %{sc}"""),
],
"solution": [cmd(r"""k get sc
k -n %{ns} describe pvc %{pvc} | tail -2""", r"""No `(default)` anywhere, and the claim waits: `no storage class is set`."""),
cmd(r"""k annotate sc %{sc} storageclass.kubernetes.io/is-default-class=true
k get sc""", r"""`%{sc} (default)`. A claim without a class gets the default when it is created - or, if there is none, as soon as one appears: no need to recreate it."""),
hand(r"""k -n %{ns} get pvc,pod""",
r"""Again until the claim shows `STORAGECLASS %{sc}` and `Bound`, and the Pod `Running`.""",
run=r"""for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.status.phase}')" = Bound ] && break; sleep 3; done
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
cmd(r"""echo %{sc} > {WS}/q123/class.txt""", r"""Marking `standard` would have bound it too - to the wrong class. With two defaults, new claims take the newest; never leave two.""")],
"solution": """
k get sc # no "(default)" behind any name
k -n %{ns} describe pvc %{pvc} | tail -2 # no persistent volumes available for this claim and no storage class is set
k patch sc %{sc} -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
k get sc # %{sc} (default)
k -n %{ns} get pvc %{pvc} -w # within seconds: STORAGECLASS %{sc}, then Bound once the Pod is scheduled
k -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.storageClassName}' > {WS}/q123/class.txt
# A claim without storageClassName gets the default class when it is created. If there is none
# it stays Pending with the field unset - and is given the default as soon as one appears: the
# claim does not have to be recreated. Marking `standard` instead would have bound it as well,
# to the wrong class. With two defaults, new claims get the most recently created one - never
# leave two.
""",
},
# ------------------------------------------------------------------ 124 ----
{
@ -1129,116 +937,53 @@ eq "$(j -n %{ns} pod %{pod} '{.spec.nodeName}')" "$W2" || exit 1
[ -z "$(j -n %{ns} pod %{pod} '{.spec.nodeSelector}{.spec.affinity}')" ] || exit 1
eq "$($K -n %{ns} exec %{pod} -- cat /disk/disk.id 2>/dev/null | tr -d '\r')" "disk %{tok}" """),
],
"solution": [hand(r"""vim local.yaml""",
r"""The docs page *Storage Classes* (under *Local*) has the class, *Persistent Volumes* a `local` volume with its required node affinity:
```
"solution": """
cat <<EOF | k apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: %{sc}
metadata: {name: %{sc}}
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: %{pv}
metadata: {name: %{pv}}
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
capacity: {storage: 1Gi}
accessModes: [ReadWriteOnce]
storageClassName: %{sc}
local:
path: /mnt/disks/%{pv}
nodeAffinity:
persistentVolumeReclaimPolicy: Retain
local: {path: /mnt/disks/%{pv}}
nodeAffinity: # required for a local volume
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- {W2}
- matchExpressions: [{key: kubernetes.io/hostname, operator: In, values: [{W2}]}]
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: %{sc}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```""",
run=r"""cat > local.yaml <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: %{sc}
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: %{pv}
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: %{sc}
local:
path: /mnt/disks/%{pv}
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- {W2}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: %{pvc}
namespace: %{ns}
spec:
storageClassName: %{sc}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF"""),
cmd(r"""k apply -f local.yaml
k -n %{ns} run %{pod} --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: %{pod}`:
```
volumeMounts:
- name: disk
mountPath: /disk
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: disk
persistentVolumeClaim:
claimName: %{pvc}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "disk", "mountPath": "/disk"}]
| .spec.volumes = [{"name": "disk", "persistentVolumeClaim": {"claimName": "%{pvc}"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod %{pod} -o wide""",
r"""Again until `Running` - on `{W2}`, sent there by the volume alone. A `hostPath` is a directory on whatever node the Pod lands on; a `local` volume knows its node, and the scheduler follows it.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s"""),
cmd(r"""k -n %{ns} exec %{pod} -- cat /disk/disk.id""")],
volumes: [{name: disk, persistentVolumeClaim: {claimName: %{pvc}}}]
containers:
- name: prom
image: busybox:1.36
command: [sleep, "86400"]
volumeMounts: [{name: disk, mountPath: /disk}]
EOF
k -n %{ns} get pod %{pod} -o wide # on {W2}
k -n %{ns} exec %{pod} -- cat /disk/disk.id
# hostPath is a directory on whatever node the Pod happens to land on; a `local` volume knows
# which node it is on, and the scheduler follows it. WaitForFirstConsumer delays binding until a
# Pod needs the claim, so that the Pod's other constraints are weighed together with the volume's.
# The page to search for: "Storage Classes" -> Local.
""",
},
]

13
cka/lessons.py
View File

@ -10,14 +10,11 @@ Lessons write nothing to the history or the scores - only a "lesson done" mark o
./ksim lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after
A lesson's host is named after its id - lesson 9101 runs on `cka9101` - which never clashes with a
task's. Its files go under {WS}/lesson-<objective>/. A step's `show` is how you would do it in the
exam: cmd() for what you type as it stands, hand() for what you do by hand - `vim` a generated
skeleton or a manifest pasted from the docs, `kubectl edit`, `ssh NODE` and `sudo -i`, a value you
read and `echo` - each with a script that does the same unattended. The audit runs the commands
as `candidate` on the host and, for each hand() part, its script; `./ksim lesson lint` keeps
scripts (sed, awk, yq, heredocs, loops, JSON patches) out of what you type. In a script, a command
on another node is `ssh NODE sudo bash <<'EOF' ... EOF` or `ssh -n NODE 'COMMAND'`: without -n,
ssh would read the rest of the script as its input.
task's. Its files go under {WS}/lesson-<objective>/. A step's `show` commands are what the learner
types; the audit runs exactly them, as `candidate` on the host, so what a lesson shows always
works. Work on another node is shown as `ssh NODE sudo bash <<'EOF' ... EOF`: the same commands
you would type after `ssh NODE` and `sudo -i`, in a form that runs unattended. A single command
there is `ssh -n NODE 'COMMAND'`: without -n, ssh would read the rest of the commands as its input.
One file per domain, as with the task banks. Besides the curriculum's competencies, the lessons
cover what a full CKA course teaches under each of them (2026-10-19: the topics of KubeDojo's CKA

541
cka/lessons_arch.py
View File

@ -1,7 +1,7 @@
"""CKA lessons of the Cluster Architecture, Installation and Configuration domain (4.1-4.8). See
cka/lessons.py for what a lesson is."""
from cka.common import backup_operator
from sim.bank import CAN, NS, PY, step, cmd, hand
from sim.bank import CAN, NS, PY, step
_ETCDCTL = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
"--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key")
@ -38,20 +38,23 @@ S=system:serviceaccount:team:deployer
can --as=$S -- create deployments.apps -n team && can --as=$S -- list pods -n team && cannot --as=$S -- delete pods -n team \
&& cannot --as=$S -- create deployments.apps -n default""",
hint=r"""`kubectl create role deploy-edit --verb=... --resource=deployments.apps` - a Role with two resources and different verbs needs two rules, so write YAML or create it and `kubectl edit`. `kubectl create rolebinding NAME --role=ROLE --serviceaccount=team:deployer`.""",
show=[cmd(r"""k -n team create sa deployer
k -n team create role deploy-edit --verb=get,list,watch,create,update,patch,delete --resource=deployments.apps --dry-run=client -o yaml > role.yaml"""),
hand(r"""vim role.yaml""",
r"""`kubectl create role` gives every resource the same verbs: add the second rule, for Pods, at the end of `rules:`:
```
show=r"""kubectl -n team create serviceaccount deployer
kubectl -n team apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deploy-edit
rules:
- apiGroups: [apps]
resources: [deployments]
verbs: [get, list, watch, create, update, patch, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get, list]
```""",
run=r"""yq -i '.rules += [{"apiGroups": [""], "resources": ["pods"], "verbs": ["get", "list"]}]' role.yaml"""),
cmd(r"""k apply -f role.yaml
k -n team create rolebinding deployer --role=deploy-edit --serviceaccount=team:deployer
k auth can-i create deploy -n team --as=system:serviceaccount:team:deployer
k auth can-i delete po -n team --as=system:serviceaccount:team:deployer""", r"""`yes`, then `no`.""")]),
EOF
kubectl -n team create rolebinding deployer --role=deploy-edit --serviceaccount=team:deployer
kubectl auth can-i create deployments -n team --as=system:serviceaccount:team:deployer
kubectl auth can-i delete pods -n team --as=system:serviceaccount:team:deployer"""),
step("One ClusterRole, one Namespace",
r"""Create ClusterRole `pod-reader` (`get`, `list`, `watch` on pods). Give it to the user **`jane`** in Namespace `team` only - with a **RoleBinding** `jane-reads-pods`, not a ClusterRoleBinding.
@ -61,10 +64,10 @@ eq "$(j clusterrole pod-reader '{.rules[0].verbs}')" '["get","list","watch"]' ||
eq "$(j -n team rolebinding jane-reads-pods '{.roleRef.kind}/{.roleRef.name}')" ClusterRole/pod-reader || exit 1
can --as=jane -- list pods -n team && cannot --as=jane -- list pods -n default && cannot --as=jane -- list pods -A""",
hint=r"""`kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods`, then `kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane`. Users exist only in certificates and tokens: nothing to create for `jane`.""",
show=[cmd(r"""k create clusterrole pod-reader --verb=get,list,watch --resource=pods
k -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane
k auth can-i list po -n team --as=jane
k auth can-i list po -n default --as=jane""", r"""`yes` in `team`, `no` elsewhere.""")]),
show=r"""kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods
kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane
kubectl auth can-i list pods -n team --as=jane
kubectl auth can-i list pods -n default --as=jane"""),
step("A token for the ServiceAccount",
r"""Issue a token for `deployer` that is valid for **one hour**, and save it in `{WS}/lesson-4.1/deployer.token`. Then use it - as the **only** credential: your kubeconfig carries the admin's client certificate, and the API server checks a certificate before a token, so `kubectl --token=...` alone would still be you. With no kubeconfig, just the server, the CA and the token, `kubectl auth whoami` says `system:serviceaccount:team:deployer`, `get deployments` works in `team` and `get secrets` is forbidden.""",
check=r"""
@ -76,41 +79,44 @@ import base64, json, sys
s = sys.stdin.read().strip(); c = json.loads(base64.b64decode(s + "=" * (-len(s) % 4)))
sys.exit(0 if c["exp"] - c["iat"] == 3600 else 1)'""",
hint=r"""`kubectl -n team create token deployer --duration=1h`. Then `kubectl --kubeconfig=/dev/null --server=URL --certificate-authority=/etc/kubernetes/pki/ca.crt --token=TOKEN ...`; the URL is in your kubeconfig. The token is a JWT: its middle part, base64-decoded, holds `sub`, `iat` and `exp`.""",
show=[cmd(r"""k -n team create token deployer --duration=1h > {WS}/lesson-4.1/deployer.token
k config view --minify | grep server"""),
hand(r"""k --kubeconfig=/dev/null --server=SERVER --certificate-authority=/etc/kubernetes/pki/ca.crt --token=TOKEN auth whoami""",
r"""`SERVER`: the address above; `TOKEN`: the file's content (`cat` it, copy it). With `-n team get deploy` instead of `auth whoami` it works; `get secrets` is forbidden.""",
run=r"""server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
kubectl --kubeconfig=/dev/null --server=$server --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$(cat {WS}/lesson-4.1/deployer.token) auth whoami""")]),
show=r"""kubectl -n team create token deployer --duration=1h > {WS}/lesson-4.1/deployer.token
server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
as_deployer() { kubectl --kubeconfig=/dev/null --server=$server --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$(cat {WS}/lesson-4.1/deployer.token) "$@"; }
as_deployer auth whoami
as_deployer -n team get deployments
as_deployer -n team get secrets || true"""),
step("A Pod without a token",
r"""Most Pods never talk to the API server, and do not need a token in their filesystem. Create ServiceAccount `quiet` in `team` with `automountServiceAccountToken: false`, and run Pod `notoken` (`busybox:1.36`, `sleep 86400`) as that ServiceAccount. Inside it, `/var/run/secrets/kubernetes.io/serviceaccount` does not exist.""",
check=r"""
eq "$(j -n team sa quiet '{.automountServiceAccountToken}')" false && eq "$(j -n team pod notoken '{.spec.serviceAccountName} {.status.phase}')" "quiet Running" || exit 1
! $K -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount >/dev/null 2>&1""",
hint=r"""The field sits at the top level of the ServiceAccount (and can be set on a Pod's `spec` too, which wins). The Pod names its ServiceAccount with `spec.serviceAccountName`.""",
show=[cmd(r"""k -n team create sa quiet --dry-run=client -o yaml > quiet.yaml"""),
hand(r"""vim quiet.yaml""",
r"""Add, at the top level (no indent):
```
show=r"""kubectl -n team apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: quiet
automountServiceAccountToken: false
```""",
run=r"""yq -i '.automountServiceAccountToken = false' quiet.yaml"""),
cmd(r"""k apply -f quiet.yaml
k -n team run notoken --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > notoken.yaml"""),
hand(r"""vim notoken.yaml""",
r"""Add to the Pod's `spec:`, above `containers:`:
```
---
apiVersion: v1
kind: Pod
metadata:
name: notoken
spec:
serviceAccountName: quiet
```""",
run=r"""yq -i '.spec.serviceAccountName = "quiet"' notoken.yaml"""),
cmd(r"""k -n team apply -f notoken.yaml
k -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount""", r"""Once it runs: "No such file or directory" - no token.""")]),
containers:
- name: app
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n team wait --for=condition=Ready pod/notoken --timeout=120s
kubectl -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount || echo 'no token mounted'"""),
step("Everything one subject may do",
r"""Write the complete list of what `deployer` may do in `team` to `{WS}/lesson-4.1/deployer-can.txt` - `kubectl auth can-i` prints it with `--list`. It includes what every authenticated subject gets from the built-in ClusterRoles, such as creating `selfsubjectreviews`.""",
check=r"""f="$WS/lesson-4.1/deployer-can.txt"; grep -q '^deployments.apps' "$f" 2>/dev/null && grep -q '^pods ' "$f" && grep -qi 'verbs' "$f" """,
hint=r"""`kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer`.""",
show=[cmd(r"""k auth can-i --list -n team --as=system:serviceaccount:team:deployer > {WS}/lesson-4.1/deployer-can.txt
cat {WS}/lesson-4.1/deployer-can.txt""")]),
show=r"""kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer > {WS}/lesson-4.1/deployer-can.txt
cat {WS}/lesson-4.1/deployer-can.txt"""),
],
"outro": r"""You gave a ServiceAccount a Role, reused a ClusterRole in one Namespace, issued and used a token, ran a Pod without one, and listed a subject's permissions.
@ -152,18 +158,10 @@ nx "$W1" 'rm -f /etc/modules-load.d/k8s.conf /etc/sysctl.d/k8s.conf' </dev/null
r"""On `{W1}`, make `overlay` and `br_netfilter` load at every boot: list them, one per line, in `/etc/modules-load.d/k8s.conf`.""",
check=r"""eq "$(nx "$W1" 'grep -v "^#" /etc/modules-load.d/k8s.conf 2>/dev/null | tr -d " " | grep . | sort | tr "\n" " "' </dev/null)" "br_netfilter overlay " """,
hint=r"""`printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/k8s.conf`. On a real machine you would also `modprobe` both now, and `lsmod | grep br_netfilter` would show it.""",
show=[hand(r"""ssh {W1}
sudo -i
vim /etc/modules-load.d/k8s.conf""",
r"""Two lines (the docs page *Container Runtimes* has them, under *Install and configure prerequisites*):
```
overlay
br_netfilter
```
then `exit` twice, back to the control-plane node.""",
run=r"""ssh {W1} sudo bash <<'EOF'
show=r"""ssh {W1} sudo bash <<'EOF'
printf 'overlay\nbr_netfilter\n' > /etc/modules-load.d/k8s.conf
EOF""")]),
cat /etc/modules-load.d/k8s.conf
EOF"""),
step("Kernel parameters, for good",
r"""On `{W1}`, put the three parameters Kubernetes needs in `/etc/sysctl.d/k8s.conf` - `net.bridge.bridge-nf-call-iptables = 1`, `net.bridge.bridge-nf-call-ip6tables = 1`, `net.ipv4.ip_forward = 1` - and apply every sysctl file now with `sysctl --system`.""",
check=r"""
@ -171,29 +169,15 @@ f=$(nx "$W1" 'cat /etc/sysctl.d/k8s.conf 2>/dev/null' </dev/null | tr -d ' ')
for p in net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1; do echo "$f" | grep -qx "$p" || exit 1; done
eq "$(nx "$W1" 'sysctl -n net.ipv4.ip_forward' </dev/null)" 1""",
hint=r"""A here-document into `sudo tee`, then `sudo sysctl --system` - it prints every file it reads, yours among them. (The bridge parameters only exist once `br_netfilter` is loaded.)""",
show=[hand(r"""ssh {W1}
sudo -i
vim /etc/sysctl.d/k8s.conf""",
r"""Three lines (the same docs page has them):
```
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
```
then apply every sysctl file and look, still on `{W1}`:
```
sysctl --system
sysctl net.ipv4.ip_forward
```
and `exit` twice.""",
run=r"""ssh {W1} sudo bash <<'EOF'
show=r"""ssh {W1} sudo bash <<'EOF'
cat > /etc/sysctl.d/k8s.conf <<'CONF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
CONF
sysctl --system >/dev/null 2>&1
EOF""")]),
sysctl --system 2>/dev/null | grep -A3 k8s.conf
sysctl net.ipv4.ip_forward
EOF"""),
step("Swap and the cgroup driver",
r"""Two more things kubeadm's preflight checks. Write to `{WS}/lesson-4.2/node.txt`, in this format:
@ -212,30 +196,17 @@ sc=$(nx "$W1" 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml |
want=$(printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s' "$swap" "${fso% *}" "${fso#* }" "$sc")
eq "$(tr -d ' \r' < "$WS/lesson-4.2/node.txt" 2>/dev/null | grep .)" "$want" """,
hint=r"""`cat /proc/swaps` (or `swapon --show`): any line below the header is swap. `grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml` - a field that is not there has its default (`failSwapOn` is `true`). `grep SystemdCgroup /etc/containerd/config.toml`; no such line means `false`.""",
show=[hand(r"""ssh {W1}
sudo -i
show=r"""ssh {W1} sudo bash <<'EOF'
cat /proc/swaps
grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml
grep SystemdCgroup /etc/containerd/config.toml""",
r"""- swap: `on` if `/proc/swaps` lists a device below its header line, `off` if it lists none;
- `failSwapOn`: `true` when the file does not set it - that is its default;
- `cgroupDriver`, and `SystemdCgroup` (`false` when it is not set).
Then `exit` twice.""",
run=r"""ssh -n {W1} 'sudo cat /proc/swaps; sudo grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml; sudo grep SystemdCgroup /etc/containerd/config.toml' """),
hand(r"""vim {WS}/lesson-4.2/node.txt""",
r"""The four lines, each `VALUE` what you found (`on` or `off`, `true` or `false`, `systemd` or `cgroupfs`):
```
swap=VALUE
failSwapOn=VALUE
kubelet-cgroup=VALUE
containerd-systemd-cgroup=VALUE
```""",
run=r"""swap=$(ssh -n {W1} '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off')
grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml
grep -n SystemdCgroup /etc/containerd/config.toml || echo 'SystemdCgroup not set'
EOF
swap=$(ssh -n {W1} '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off')
fso=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.failSwapOn)
cg=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver)
sc=$(ssh -n {W1} 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml | head -1 | tr -d " " | cut -d= -f2')
printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s\n' "$swap" "$fso" "$cg" "${sc:-false}" > {WS}/lesson-4.2/node.txt""")]),
printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s\n' "$swap" "$fso" "$cg" "${sc:-false}" > {WS}/lesson-4.2/node.txt
cat {WS}/lesson-4.2/node.txt"""),
step("The packages, as kubeadm wants them",
r"""On an exam node, kubeadm, kubelet and kubectl come from the repository of **one** minor at `pkgs.k8s.io`, and are **held** so that an `apt upgrade` never moves them by accident.
@ -244,10 +215,9 @@ Write to `{WS}/lesson-4.2/packages.txt`: the line of `{W1}`'s Kubernetes reposit
want=$(nx "$W1" 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' </dev/null | grep .)
[ -n "$want" ] && eq "$(grep . "$WS/lesson-4.2/packages.txt" 2>/dev/null | tr -d '\r')" "$want" """,
hint=r"""`cat /etc/apt/sources.list.d/kubernetes.list` and `apt-mark showhold`. `apt-cache madison kubeadm` lists the versions that repository offers; `dpkg -l | grep kube` what is installed.""",
show=[hand(r"""ssh {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt""",
r"""The repository line, then the three held packages.""",
run=r"""ssh -n {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt"""),
cmd(r"""cat {WS}/lesson-4.2/packages.txt""")]),
show=r"""ssh -n {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt
cat {WS}/lesson-4.2/packages.txt
ssh -n {W1} 'dpkg -l | grep -E "kube(adm|let|ctl)"'"""),
],
"outro": r"""You set up modules and sysctls that survive a reboot, checked swap and the cgroup drivers, and read how Kubernetes is installed on a node.
@ -289,10 +259,9 @@ nx "$W1" 'rm -f /etc/kubernetes/manifests/node-web.yaml' </dev/null
want=$($K -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}')
eq "$(tr -s ' \t' ' ' < "$WS/lesson-4.3/networks.txt" 2>/dev/null | tr -d '\r\n')" "$want" """,
hint=r"""`kubectl -n kube-system get cm kubeadm-config -o yaml` - look under `networking:`. `kubeadm config print init-defaults` shows every field such a file can have.""",
show=[cmd(r"""k -n kube-system get cm kubeadm-config -o yaml | grep -i subnet"""),
hand(r"""echo 'PODSUBNET SERVICESUBNET' > {WS}/lesson-4.3/networks.txt""",
r"""The two values, `podSubnet` first.""",
run=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}' > {WS}/lesson-4.3/networks.txt""")]),
show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep -A4 networking
kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}' > {WS}/lesson-4.3/networks.txt
cat {WS}/lesson-4.3/networks.txt"""),
step("A join command for a new worker",
r"""Create a new bootstrap token that expires in **2 hours**, and save the complete `kubeadm join` command for a worker in `{WS}/lesson-4.3/join.sh`, on one line. Run it on the control-plane node.""",
check=r"""
@ -301,9 +270,9 @@ id=$(grep -o -- '--token [a-z0-9]*\.[a-z0-9]*' "$f" | awk '{print $2}' | cut -d.
exp=$(j -n kube-system secret "bootstrap-token-$id" '{.data.expiration}' | base64 -d); [ -n "$exp" ] || exit 1
left=$(( $(date -d "$exp" +%s) - $(date +%s) )); [ "$left" -gt 6600 ] && [ "$left" -le 7300 ]""",
hint=r"""`sudo kubeadm token create --ttl 2h --print-join-command`. `kubeadm token list` shows the tokens and when they expire; each is a Secret `bootstrap-token-ID` in `kube-system`.""",
show=[cmd(r"""sudo kubeadm token create --ttl 2h --print-join-command > {WS}/lesson-4.3/join.sh
show=r"""sudo kubeadm token create --ttl 2h --print-join-command > {WS}/lesson-4.3/join.sh
cat {WS}/lesson-4.3/join.sh
sudo kubeadm token list""")]),
sudo kubeadm token list"""),
step("Renew the API server's certificate",
r"""Check the certificates' expiry dates, then **renew** the API server's serving certificate - only that one - and make the API server use it. A running process keeps the certificate it read at start, so the static Pod has to be restarted: move its manifest out of `/etc/kubernetes/manifests`, wait until the container is gone, and move it back.
@ -316,32 +285,24 @@ file=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -seri
served=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial' </dev/null)
[ -n "$file" ] && eq "$file" "$served" """,
hint=r"""`sudo kubeadm certs check-expiration`, `sudo kubeadm certs renew apiserver`. Restart: `sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/`, wait for `sudo crictl ps --name kube-apiserver` to be empty, move it back. Kubernetes renews the certificates itself on every `kubeadm upgrade`.""",
show=[cmd(r"""sudo kubeadm certs check-expiration
show=r"""sudo kubeadm certs check-expiration | head -8
sudo kubeadm certs renew apiserver
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml ~/"""),
hand(r"""sudo crictl ps --name kube-apiserver""",
r"""Again until it lists nothing: the old API server is gone.""",
run=r"""for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name kube-apiserver -q)" ] && break; sleep 2; done"""),
cmd(r"""sudo mv ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
hand(r"""k get no""",
r"""Again until it answers: the API server is back with the new certificate.""",
run=r"""for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""sudo kubeadm certs check-expiration | grep apiserver""")]),
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name kube-apiserver -q)" ] && break; sleep 2; done
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
sudo kubeadm certs check-expiration | grep '^apiserver '"""),
step("A static Pod on the worker",
r"""Every kubelet runs static Pods, not only the control plane's. On `{W1}`, find the kubelet's `staticPodPath` and put a manifest there for Pod `node-web` (`nginx:1.27`).
The API server shows it as a **mirror Pod** named after the node: `node-web-{W1}`. Deleting the mirror Pod with kubectl does nothing lasting - the kubelet puts it back; only removing the file does.""",
check=r"""eq "$(j -n default pod "node-web-$W1" '{.status.phase} {.metadata.annotations.kubernetes\.io/config\.source}')" "Running file" """,
hint=r"""`grep staticPodPath /var/lib/kubelet/config.yaml` on the node. `kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml` gives you the manifest; write it with `sudo tee`.""",
show=[cmd(r"""k run node-web --image=nginx:1.27 --dry-run=client -o yaml > node-web.yaml"""),
hand(r"""scp node-web.yaml {W1}:
ssh {W1}
sudo -i
grep staticPodPath /var/lib/kubelet/config.yaml
cp ~candidate/node-web.yaml /etc/kubernetes/manifests/""",
r"""The manifest goes into the `staticPodPath` directory; then `exit` twice.""",
run=r"""ssh {W1} 'sudo tee /etc/kubernetes/manifests/node-web.yaml >/dev/null' < node-web.yaml"""),
cmd(r"""k get po -o wide""", r"""Again in a few seconds: `node-web-{W1}`, the mirror Pod.""")]),
show=r"""kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml > {WS}/lesson-4.3/node-web.yaml
ssh -n {W1} 'grep staticPodPath /var/lib/kubelet/config.yaml'
ssh {W1} 'sudo tee /etc/kubernetes/manifests/node-web.yaml >/dev/null' < {WS}/lesson-4.3/node-web.yaml
sleep 15
kubectl get pods -o wide"""),
],
"outro": r"""You read the cluster's kubeadm configuration, made a join command, renewed a certificate and made the API server load it, and ran a static Pod on a worker.
@ -397,14 +358,15 @@ etcd only accepts clients with a certificate: give `etcdctl` the endpoint and th
nx "$CP" 'etcdutl snapshot status /opt/backup/etcd.db >/dev/null 2>&1' </dev/null && grep -qi 'hash' "$WS/lesson-4.4/snapshot.txt" 2>/dev/null \
&& grep -qi 'total key' "$WS/lesson-4.4/snapshot.txt" """,
hint=r"""`etcdctl --endpoints=https://127.0.0.1:2379 --cacert=.../ca.crt --cert=.../server.crt --key=.../server.key snapshot save FILE`, as root. Since etcd 3.6, `status` and `restore` are `etcdutl` only.""",
show=[cmd(r"""sudo grep -e listen-client-urls -e cert-file -e key-file -e trusted-ca-file /etc/kubernetes/manifests/etcd.yaml""", r"""The endpoint, and the three files etcdctl needs: `--trusted-ca-file` is its `--cacert`, `--cert-file` its `--cert`, `--key-file` its `--key`."""),
cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup/etcd.db
sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/snapshot.txt""")]),
show=r"""sudo grep -E -- '--(listen-client-urls|cert-file|key-file|trusted-ca-file)' /etc/kubernetes/manifests/etcd.yaml
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup/etcd.db
sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/snapshot.txt"""),
step("Lose something",
r"""Now the mistake a backup is for: delete ConfigMap `marker` in `life`.""",
check=r"""[ -z "$(j -n life cm marker '{.metadata.name}')" ] && nx "$CP" 'test -s /opt/backup/etcd.db' </dev/null""",
hint=r"""`kubectl -n life delete configmap marker`.""",
show=[cmd(r"""k -n life delete cm marker""")]),
show=r"""kubectl -n life delete configmap marker"""),
step("Restore the snapshot",
r"""Restore `/opt/backup/etcd.db` so that `marker` is back. Do it in place, so that etcd keeps its data directory `/var/lib/etcd`:
@ -415,16 +377,13 @@ sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/
Restoring into a **new** directory and pointing the manifest's `hostPath` at it works too, and is what many tasks ask for - but then the `dataDir` in kubeadm's configuration still names the old one, and the next `kubeadm upgrade` would write it back into the manifest.""",
check=r"""eq "$(j -n life cm marker '{.data.state}')" before-backup && nx "$CP" 'test -d /var/lib/etcd/member' </dev/null""",
hint=r"""`sudo etcdutl snapshot restore /opt/backup/etcd.db --data-dir /var/lib/etcd` - the target directory must not exist yet. `sudo crictl ps --name etcd` shows whether etcd is running.""",
show=[cmd(r"""sudo mv /etc/kubernetes/manifests/etcd.yaml ~/"""),
hand(r"""sudo crictl ps --name etcd""",
r"""Again until it lists nothing: etcd has stopped. (kubectl fails from now on until etcd is back.)""",
run=r"""for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name '^etcd$' -q)" ] && break; sleep 2; done"""),
cmd(r"""sudo mv /var/lib/etcd /var/lib/etcd.bak
show=r"""sudo mv /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml
for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name '^etcd$' -q)" ] && break; sleep 2; done
sudo mv /var/lib/etcd /var/lib/etcd.before-restore
sudo etcdutl snapshot restore /opt/backup/etcd.db --data-dir /var/lib/etcd
sudo mv ~/etcd.yaml /etc/kubernetes/manifests/"""),
hand(r"""k -n life get cm marker""",
r"""Again until it answers - a minute or two: etcd, then the API server come back, and `marker` with them.""",
run=r"""for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done""")]),
sudo mv /root/etcd.yaml /etc/kubernetes/manifests/etcd.yaml
for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
kubectl -n life get configmap marker -o yaml | grep state"""),
step("Plan the upgrade",
r"""On the control-plane node, point the Kubernetes repository at **v1.37**, install the **newest 1.37 kubeadm** the repository offers (unhold it first, hold it again after), and save the output of `kubeadm upgrade plan` in `{WS}/lesson-4.4/plan.txt`.
@ -433,19 +392,13 @@ The plan shows two choices: the newest 1.36 patch, and the 1.37 release your kub
t=$(newest); [ -n "$t" ] || exit 1
eq "$(ver "$CP" kubeadm)" "$t" && held "$CP" && grep -q "kubeadm upgrade apply v${t%%-*}" "$WS/lesson-4.4/plan.txt" 2>/dev/null""",
hint=r"""`sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list`, `sudo apt-get update`, `apt-cache madison kubeadm` (newest first), `sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.X-1.1 && sudo apt-mark hold kubeadm`.""",
show=[hand(r"""sudo vim /etc/apt/sources.list.d/kubernetes.list""",
r"""In the URL, change `v1.36` to `v1.37`.""",
run=r"""sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list"""),
cmd(r"""sudo apt-get update
apt-cache madison kubeadm""", r"""The newest `1.37.x-1.1` (at the top) is what to install. The docs page *Upgrading kubeadm clusters* has the commands that follow."""),
hand(r"""sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=VERSION
sudo apt-mark hold kubeadm""",
r"""`VERSION`: that one, for example `1.37.1-1.1`.""",
run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo apt-mark unhold kubeadm && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$v && sudo apt-mark hold kubeadm"""),
cmd(r"""kubeadm version
sudo kubeadm upgrade plan | tee {WS}/lesson-4.4/plan.txt""")]),
show=r"""sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -qq
apt-cache madison kubeadm | head -3
v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo apt-mark unhold kubeadm && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$v && sudo apt-mark hold kubeadm
kubeadm version -o short
sudo kubeadm upgrade plan | tee {WS}/lesson-4.4/plan.txt | tail -12"""),
step("Upgrade the control plane",
r"""Apply the upgrade to that 1.37 release (`kubeadm upgrade apply`, a few minutes - it upgrades etcd, the static Pods, CoreDNS and kube-proxy). Then drain the node, upgrade `kubelet` and `kubectl` to the same version, reload systemd, restart the kubelet, and uncordon.
@ -456,23 +409,13 @@ eq "$($K version -o json 2>/dev/null | python3 -c 'import json,sys; print(json.l
eq "$(j node "$CP" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$CP")" "$t $t $t " && held "$CP" \
&& [ "$(j node "$CP" '{.spec.unschedulable}')" != true ]""",
hint=r"""`sudo kubeadm upgrade apply v1.37.X -y`; `kubectl drain {CP} --ignore-daemonsets`; `sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=... kubectl=... && sudo apt-mark hold kubelet kubectl`; `sudo systemctl daemon-reload && sudo systemctl restart kubelet`; `kubectl uncordon {CP}`.""",
show=[hand(r"""sudo kubeadm upgrade apply vVERSION""",
r"""`vVERSION`: the one the plan names, for example `v1.37.1`. Answer `y`; it takes a few minutes.""",
run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo kubeadm upgrade apply v${v%%-*} -y | tail -3"""),
cmd(r"""k drain {CP} --ignore-daemonsets --delete-emptydir-data"""),
hand(r"""sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=VERSION kubectl=VERSION
sudo apt-mark hold kubelet kubectl""",
r"""`VERSION` as for kubeadm, for example `1.37.1-1.1`.""",
run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo apt-mark unhold kubelet kubectl && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$v kubectl=$v && sudo apt-mark hold kubelet kubectl"""),
cmd(r"""sudo systemctl daemon-reload
sudo systemctl restart kubelet"""),
hand(r"""k uncordon {CP}""",
r"""Again if it fails at first: the kubelet takes a moment to come back.""",
run=r"""for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""k get no""")]),
show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo kubeadm upgrade apply v${v%%-*} -y | tail -3
kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$v kubectl=$v && sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done
kubectl get nodes"""),
step("Upgrade the worker",
r"""Now `{W1}`: drain it from the control-plane node; on `{W1}`, the repository line, kubeadm, **`kubeadm upgrade node`** (a worker has no control-plane Pods to replace - it only refreshes the kubelet's configuration), kubelet and kubectl, daemon-reload, restart; then uncordon it.
@ -482,20 +425,8 @@ t=$(newest); v="v${t%%-*}"
eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$W1")" "$t $t $t " && held "$W1" \
&& [ "$(j node "$W1" '{.spec.unschedulable}')" != true ] && eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""The same packages, from the same repository line, as on the control plane - only `kubeadm upgrade node` instead of `apply`. Drain and uncordon are kubectl commands: run them on the control-plane node.""",
show=[cmd(r"""k drain {W1} --ignore-daemonsets --delete-emptydir-data"""),
hand(r"""ssh {W1}
sudo -i
vim /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-mark unhold kubeadm kubelet kubectl
apt-get install -y kubeadm=VERSION
kubeadm upgrade node
apt-get install -y kubelet=VERSION kubectl=VERSION
apt-mark hold kubeadm kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet""",
r"""In the repository line `v1.36` becomes `v1.37`; `VERSION` as on the control-plane node. Then `exit` twice.""",
run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data
ssh {W1} sudo bash -s $v <<'EOF'
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq
@ -505,9 +436,9 @@ kubeadm upgrade node | tail -2
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$1 kubectl=$1
apt-mark hold kubeadm kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet
EOF"""),
cmd(r"""k uncordon {W1}
k get no""")]),
EOF
kubectl uncordon {W1}
kubectl get nodes"""),
],
"outro": r"""You backed etcd up, restored it after a loss, and took a whole cluster from v1.36 to v1.37: control plane first, then the worker, with the packages held at every step.
@ -549,11 +480,9 @@ $K -n kube-system delete secret kubeadm-certs --ignore-not-found >/dev/null 2>&1
want=$($K -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}')
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-4.5/endpoint.txt" 2>/dev/null)" "$want" """,
hint=r"""`controlPlaneEndpoint:` in the `ClusterConfiguration` of ConfigMap `kubeadm-config`.""",
show=[cmd(r"""k -n kube-system get cm kubeadm-config -o yaml | grep controlPlaneEndpoint"""),
hand(r"""echo ENDPOINT > {WS}/lesson-4.5/endpoint.txt""",
r"""`ENDPOINT`: that value, `host:port`.""",
run=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}' > {WS}/lesson-4.5/endpoint.txt"""),
cmd(r"""sudo grep server: /etc/kubernetes/kubelet.conf""")]),
show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}' > {WS}/lesson-4.5/endpoint.txt
cat {WS}/lesson-4.5/endpoint.txt
sudo grep server: /etc/kubernetes/kubelet.conf"""),
step("etcd's members and its leader",
r"""Ask etcd itself. Write to `{WS}/lesson-4.5/etcd.txt`:
@ -569,14 +498,13 @@ import json, sys; m = json.load(sys.stdin)
st = json.loads(sys.argv[1])[0]["Status"]; print(next(x["name"] for x in m["members"] if x["ID"] == st["leader"]))' "$out")
eq "$(tr -d ' \r' < "$WS/lesson-4.5/etcd.txt" 2>/dev/null | grep .)" "$(printf 'members=%s\nleader=%s' "$n" "$lead")" """,
hint=r"""`etcdctl member list -w table` and `etcdctl endpoint status -w table` (the IS LEADER column), with the endpoint and certificates as for a snapshot. `export ETCDCTL_ENDPOINTS=... ETCDCTL_CACERT=...` saves typing.""",
show=[cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list -w table""", r"""For the next one, press ↑ and change the last words."""),
cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table""", r"""`IS LEADER` says which member leads."""),
hand(r"""vim {WS}/lesson-4.5/etcd.txt""",
r"""```
members=1
leader={CP}
```""",
run=r"""printf 'members=1\nleader={CP}\n' > {WS}/lesson-4.5/etcd.txt""")]),
show=r"""sudo bash <<'EOF'
export ETCDCTL_API=3 ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl member list -w table
etcdctl endpoint status -w table
EOF
printf 'members=1\nleader={CP}\n' > {WS}/lesson-4.5/etcd.txt"""),
step("Who leads the scheduler and the controller manager",
r"""Only one kube-scheduler and one kube-controller-manager act at a time; the others wait for the **Lease** in `kube-system` to expire. Write the node of each holder to `{WS}/lesson-4.5/leaders.txt`:
@ -588,28 +516,20 @@ controller-manager=<node>
s=$(j -n kube-system lease kube-scheduler '{.spec.holderIdentity}'); c=$(j -n kube-system lease kube-controller-manager '{.spec.holderIdentity}')
eq "$(tr -d ' \r' < "$WS/lesson-4.5/leaders.txt" 2>/dev/null | grep .)" "$(printf 'scheduler=%s\ncontroller-manager=%s' "${s%%_*}" "${c%%_*}")" """,
hint=r"""`kubectl -n kube-system get lease kube-scheduler kube-controller-manager` - HOLDER is `NODE_UUID`.""",
show=[cmd(r"""k -n kube-system get lease kube-scheduler kube-controller-manager""", r"""`HOLDER` is the node's name, then `_` and an ID: keep only the node's name."""),
hand(r"""vim {WS}/lesson-4.5/leaders.txt""",
r"""```
scheduler=NODE
controller-manager=NODE
```""",
run=r"""s=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}')
show=r"""kubectl -n kube-system get lease kube-scheduler kube-controller-manager
s=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}')
c=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}')
printf 'scheduler=%s\ncontroller-manager=%s\n' "${s%%_*}" "${c%%_*}" > {WS}/lesson-4.5/leaders.txt""")]),
printf 'scheduler=%s\ncontroller-manager=%s\n' "${s%%_*}" "${c%%_*}" > {WS}/lesson-4.5/leaders.txt"""),
step("What a second control-plane node needs",
r"""Upload the control-plane certificates for a joining node, and save the complete `kubeadm join` command a new **control-plane** node would run - new token, CA hash, `--control-plane` and the certificate key - to `{WS}/lesson-4.5/join-cp.sh`, on one line. Do not run it anywhere.""",
check=r"""
f="$WS/lesson-4.5/join-cp.sh"; grep -q '^kubeadm join .*--token .*--discovery-token-ca-cert-hash sha256:.*--control-plane' "$f" 2>/dev/null || exit 1
key=$(grep -o -- '--certificate-key [0-9a-f]\{64\}' "$f" | awk '{print $2}'); [ -n "$key" ] && [ -n "$(j -n kube-system secret kubeadm-certs '{.metadata.name}')" ]""",
hint=r"""`sudo kubeadm init phase upload-certs --upload-certs` prints the key as its last line; `sudo kubeadm token create --print-join-command --certificate-key KEY` adds `--control-plane --certificate-key` itself.""",
show=[cmd(r"""sudo kubeadm init phase upload-certs --upload-certs""", r"""The last line is the certificate key."""),
hand(r"""sudo kubeadm token create --print-join-command --certificate-key KEY > {WS}/lesson-4.5/join-cp.sh""",
r"""`KEY`: that line.""",
run=r"""key=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
sudo kubeadm token create --print-join-command --certificate-key $key > {WS}/lesson-4.5/join-cp.sh"""),
cmd(r"""cat {WS}/lesson-4.5/join-cp.sh
k -n kube-system get secret kubeadm-certs""")]),
show=r"""key=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
sudo kubeadm token create --print-join-command --certificate-key $key > {WS}/lesson-4.5/join-cp.sh
cat {WS}/lesson-4.5/join-cp.sh
kubectl -n kube-system get secret kubeadm-certs"""),
],
"outro": r"""You read the endpoint that makes HA possible, asked etcd for its members and leader, found the leaders of the scheduler and controller manager, and prepared a control-plane join.
@ -717,23 +637,23 @@ A chart is at `{WS}/lesson-4.6/chart` (values `replicaCount`, `image.tag`, `serv
r"""Render the chart as release `web` with `replicaCount=2`, without installing anything, and save the manifests in `{WS}/lesson-4.6/rendered.yaml`. That is how you check what a set of values does.""",
check=r"""f="$WS/lesson-4.6/rendered.yaml"; grep -q 'name: web-shop' "$f" 2>/dev/null && grep -q 'replicas: 2' "$f" && [ -z "$(j -n apps deploy web-shop '{.metadata.name}')" ]""",
hint=r"""`helm template RELEASE CHART --set KEY=VALUE`.""",
show=[cmd(r"""helm template web {WS}/lesson-4.6/chart --set replicaCount=2 > {WS}/lesson-4.6/rendered.yaml
grep -e kind: -e replicas: -e image: {WS}/lesson-4.6/rendered.yaml""")]),
show=r"""helm template web {WS}/lesson-4.6/chart --set replicaCount=2 > {WS}/lesson-4.6/rendered.yaml
grep -E 'kind:|replicas:|image:' {WS}/lesson-4.6/rendered.yaml"""),
step("Install a release",
r"""Install the chart as release `web` in Namespace `apps` (create it), with `replicaCount=2`.""",
check=r"""eq "$(helm -n apps status web -o json 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["info"]["status"], d["version"])')" "deployed 1" && eq "$(j -n apps deploy web-shop '{.status.readyReplicas}')" 2""",
hint=r"""`helm install web CHART -n apps --create-namespace --set replicaCount=2`; `helm list -n apps`.""",
show=[cmd(r"""helm install web {WS}/lesson-4.6/chart -n apps --create-namespace --set replicaCount=2
show=r"""helm install web {WS}/lesson-4.6/chart -n apps --create-namespace --set replicaCount=2
helm list -n apps
k -n apps rollout status deploy web-shop""")]),
kubectl -n apps rollout status deploy web-shop --timeout=120s"""),
step("Upgrade it, keeping what you set",
r"""Upgrade release `web` so that its Service becomes a `NodePort` - and keep `replicaCount=2`. A plain `helm upgrade --set service.type=NodePort` would quietly put the replica count back to the chart's default.""",
check=r"""eq "$(j -n apps svc web-shop '{.spec.type}')" NodePort && eq "$(j -n apps deploy web-shop '{.spec.replicas}')" 2 \
&& [ "$(helm -n apps history web -o json 2>/dev/null | python3 -c 'import json,sys; print(len(json.load(sys.stdin)))')" -ge 2 ]""",
hint=r"""`--reuse-values` keeps the values of the last release and adds yours; `helm get values web -n apps` shows what is set.""",
show=[cmd(r"""helm upgrade web {WS}/lesson-4.6/chart -n apps --reuse-values --set service.type=NodePort
show=r"""helm upgrade web {WS}/lesson-4.6/chart -n apps --reuse-values --set service.type=NodePort
helm get values web -n apps
k -n apps get svc web-shop""")]),
kubectl -n apps get svc web-shop"""),
step("Roll the release back",
r"""The NodePort was a mistake: roll release `web` back to **revision 1**. A rollback is itself a new revision.""",
check=r"""
@ -742,18 +662,16 @@ helm -n apps history web -o json 2>/dev/null | python3 -c '
import json, sys; h = json.load(sys.stdin)
sys.exit(0 if len(h) >= 3 and h[-1]["status"] == "deployed" and "Rollback to 1" in h[-1]["description"] else 1)'""",
hint=r"""`helm rollback web 1 -n apps`, then `helm history web -n apps`.""",
show=[cmd(r"""helm rollback web 1 -n apps
show=r"""helm rollback web 1 -n apps
helm history web -n apps
k -n apps get svc web-shop""")]),
kubectl -n apps get svc web-shop"""),
step("An overlay for production",
r"""Create a Kustomize overlay at `{WS}/lesson-4.6/kustomize/prod` on top of the base, which: puts everything in Namespace `prod`, prefixes names with `prod-`, runs **3** replicas, and uses image `nginx:1.27`. Apply it. Leave the base as it is.""",
check=r"""eq "$(j -n prod deploy prod-api '{.spec.replicas} {.spec.template.spec.containers[0].image} {.status.readyReplicas}')" "3 nginx:1.27 3" \
&& grep -q 'image: nginx:1.26' "$WS/lesson-4.6/kustomize/base/deployment.yaml" """,
hint=r"""A `kustomization.yaml` with `resources: [../base]`, `namespace: prod`, `namePrefix: prod-`, `replicas: [{name: api, count: 3}]`, `images: [{name: nginx, newTag: "1.27"}]`. The Namespace itself must exist (or be one of the resources). `kubectl kustomize DIR` shows the result first.""",
show=[cmd(r"""mkdir -p {WS}/lesson-4.6/kustomize/prod"""),
hand(r"""vim {WS}/lesson-4.6/kustomize/prod/kustomization.yaml""",
r"""(the docs page *Declarative Management of Kubernetes Objects Using Kustomize* has each field)
```
show=r"""mkdir -p {WS}/lesson-4.6/kustomize/prod
cat > {WS}/lesson-4.6/kustomize/prod/kustomization.yaml <<'EOF'
resources:
- ../base
namespace: prod
@ -764,23 +682,11 @@ replicas:
images:
- name: nginx
newTag: "1.27"
```""",
run=r"""cat > {WS}/lesson-4.6/kustomize/prod/kustomization.yaml <<'EOF'
resources:
- ../base
namespace: prod
namePrefix: prod-
replicas:
- name: api
count: 3
images:
- name: nginx
newTag: "1.27"
EOF"""),
cmd(r"""k kustomize {WS}/lesson-4.6/kustomize/prod
k create ns prod
k apply -k {WS}/lesson-4.6/kustomize/prod
k -n prod rollout status deploy prod-api""")]),
EOF
kubectl kustomize {WS}/lesson-4.6/kustomize/prod
kubectl create namespace prod
kubectl apply -k {WS}/lesson-4.6/kustomize/prod
kubectl -n prod rollout status deploy prod-api --timeout=120s"""),
],
"outro": r"""You rendered a chart, installed, upgraded and rolled back a release, and built a production overlay with Kustomize without touching its base.
@ -828,18 +734,11 @@ f=$(nx "$W1" 'ls /etc/cni/net.d/*.conf* 2>/dev/null | sort | head -1' </dev/null
p=$(nx "$W1" "cat $f" </dev/null | python3 -c 'import json,sys; c=json.load(sys.stdin); print(",".join(x["type"] for x in c.get("plugins", [c])))')
eq "$(tr -d ' \r' < "$WS/lesson-4.7/cni.txt" 2>/dev/null | grep .)" "$(printf 'config=%s\nplugins=%s' "$f" "$p")" """,
hint=r"""`ls /etc/cni/net.d/` and `cat` the file: a `.conflist` has a `"plugins"` list, each with a `"type"` - the binary under `/opt/cni/bin` that is run.""",
show=[hand(r"""ssh {W1} ls /etc/cni/net.d/
ssh {W1} cat /etc/cni/net.d/10-kindnet.conflist""",
r"""The runtime uses the first file by name; the `type` of each entry of `plugins`, in order, is the chain.""",
run=r"""ssh -n {W1} 'ls /etc/cni/net.d/; cat /etc/cni/net.d/10-kindnet.conflist' """),
hand(r"""vim {WS}/lesson-4.7/cni.txt""",
r"""```
config=/etc/cni/net.d/10-kindnet.conflist
plugins=TYPE,TYPE,...
```""",
run=r"""f=$(ssh -n {W1} 'ls /etc/cni/net.d/*.conf* | sort | head -1')
show=r"""ssh -n {W1} 'ls /etc/cni/net.d/; cat /etc/cni/net.d/10-kindnet.conflist; ls /opt/cni/bin'
f=$(ssh -n {W1} 'ls /etc/cni/net.d/*.conf* | sort | head -1')
p=$(ssh -n {W1} "cat $f" | jq -r '[.plugins[].type] | join(",")')
printf 'config=%s\nplugins=%s\n' "$f" "$p" > {WS}/lesson-4.7/cni.txt""")]),
printf 'config=%s\nplugins=%s\n' "$f" "$p" > {WS}/lesson-4.7/cni.txt
cat {WS}/lesson-4.7/cni.txt"""),
step("The runtime, through CRI",
r"""Write to `{WS}/lesson-4.7/cri.txt`:
@ -852,63 +751,33 @@ e=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" | python3 -c 'import json,sys
r=$(j node "$W1" '{.status.nodeInfo.containerRuntimeVersion}')
eq "$(tr -d ' \r' < "$WS/lesson-4.7/cri.txt" 2>/dev/null | grep .)" "$(printf 'endpoint=%s\nruntime=%s' "$e" "$r")" """,
hint=r"""`containerRuntimeEndpoint` in the kubelet's configuration (`/var/lib/kubelet/config.yaml`, or `configz`); `kubectl get node -o wide` shows CONTAINER-RUNTIME.""",
show=[hand(r"""ssh {W1} grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml""",
r"""The endpoint the kubelet talks CRI to.""",
run=r"""ssh -n {W1} grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml"""),
cmd(r"""k get no -o wide""", r"""`CONTAINER-RUNTIME` is the runtime and its version."""),
hand(r"""vim {WS}/lesson-4.7/cri.txt""",
r"""```
endpoint=ENDPOINT
runtime=RUNTIME
```""",
run=r"""e=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.containerRuntimeEndpoint)
show=r"""e=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.containerRuntimeEndpoint)
r=$(kubectl get node {W1} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}')
printf 'endpoint=%s\nruntime=%s\n' "$e" "$r" > {WS}/lesson-4.7/cri.txt""")]),
printf 'endpoint=%s\nruntime=%s\n' "$e" "$r" > {WS}/lesson-4.7/cri.txt
cat {WS}/lesson-4.7/cri.txt"""),
step("A Pod's container, from the runtime's side",
r"""Pod `probe` in `plug` runs on `{W1}`. Ask the runtime about it there, with `crictl`: find its container, and write the container's **process ID on the node** to `{WS}/lesson-4.7/pid.txt`.""",
check=r"""
want=$(nx "$W1" 'crictl inspect $(crictl ps --name probe -q | head -1) 2>/dev/null' </dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["info"]["pid"])')
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-4.7/pid.txt" 2>/dev/null)" "$want" """,
hint=r"""`sudo crictl pods --name probe`, `sudo crictl ps --name probe`, `sudo crictl inspect ID` - the PID is under `info.pid`. `crictl` talks to the same socket as the kubelet.""",
show=[hand(r"""ssh {W1}
sudo -i
crictl ps --name probe
crictl inspect CONTAINER | grep -m1 '"pid"' """,
r"""`CONTAINER`: the ID `crictl ps` printed. Then `exit` twice.""",
run=r"""ssh -n {W1} 'sudo crictl pods --name probe; sudo crictl ps --name probe' """),
hand(r"""echo PID > {WS}/lesson-4.7/pid.txt""",
r"""`PID`: that number.""",
run=r"""ssh -n {W1} 'sudo crictl inspect $(sudo crictl ps --name probe -q | head -1)' | jq .info.pid > {WS}/lesson-4.7/pid.txt""")]),
show=r"""ssh -n {W1} 'sudo crictl pods --name probe; sudo crictl ps --name probe'
ssh -n {W1} 'sudo crictl inspect $(sudo crictl ps --name probe -q | head -1)' | jq .info.pid > {WS}/lesson-4.7/pid.txt
cat {WS}/lesson-4.7/pid.txt"""),
step("A RuntimeClass",
r"""Create RuntimeClass `plain` for containerd's handler `runc`, and run Pod `classy` in `plug` (`busybox:1.36`, `sleep 86400`) with that runtime class. A Pod that names a handler the node does not have stays in `ContainerCreating`.""",
check=r"""eq "$(j runtimeclass plain '{.handler}')" runc && eq "$(j -n plug pod classy '{.spec.runtimeClassName} {.status.phase}')" "plain Running" """,
hint=r"""A RuntimeClass is `node.k8s.io/v1` with a top-level `handler:` and no `spec`. The Pod: `spec.runtimeClassName: plain`.""",
show=[hand(r"""vim rc.yaml""",
r"""No `kubectl create` for a RuntimeClass: paste the example from the docs page *Runtime Class*, and make it:
```
show=r"""kubectl apply -f - <<'EOF'
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: plain
handler: runc
```""",
run=r"""cat > rc.yaml <<'EOF'
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: plain
handler: runc
EOF"""),
cmd(r"""k apply -f rc.yaml
k -n plug run classy --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > classy.yaml"""),
hand(r"""vim classy.yaml""",
r"""Add to the Pod's `spec:`, above `containers:`:
```
runtimeClassName: plain
```""",
run=r"""yq -i '.spec.runtimeClassName = "plain"' classy.yaml"""),
cmd(r"""k -n plug apply -f classy.yaml
k -n plug get po classy""")]),
EOF
kubectl -n plug run classy --image=busybox:1.36 --overrides='{"spec":{"runtimeClassName":"plain"}}' -- sleep 86400
kubectl -n plug wait --for=condition=Ready pod/classy --timeout=120s
kubectl get runtimeclass"""),
step("Storage drivers",
r"""Write to `{WS}/lesson-4.7/csi.txt`:
@ -923,16 +792,11 @@ n=$($K get csidrivers --no-headers 2>/dev/null | grep -c .)
p=$($K get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.provisioner}{end}')
eq "$(tr -d ' \r' < "$WS/lesson-4.7/csi.txt" 2>/dev/null | grep .)" "$(printf 'csidrivers=%s\nprovisioner=%s' "$n" "$p")" """,
hint=r"""`kubectl get csidrivers`, `kubectl get csinodes`, `kubectl get sc`.""",
show=[cmd(r"""k get csidrivers
k get csinodes
k get sc"""),
hand(r"""vim {WS}/lesson-4.7/csi.txt""",
r"""```
csidrivers=N
provisioner=PROVISIONER
```
`N`: how many CSIDrivers `get csidrivers` listed (`0` for "No resources found"); `PROVISIONER`: the `(default)` class's.""",
run=r"""printf 'csidrivers=%s\nprovisioner=%s\n' "$(kubectl get csidrivers --no-headers 2>/dev/null | grep -c .)" "$(kubectl get sc standard -o jsonpath='{.provisioner}')" > {WS}/lesson-4.7/csi.txt""")]),
show=r"""kubectl get csidrivers
kubectl get csinodes
kubectl get sc
printf 'csidrivers=%s\nprovisioner=%s\n' "$(kubectl get csidrivers --no-headers 2>/dev/null | grep -c .)" "$(kubectl get sc standard -o jsonpath='{.provisioner}')" > {WS}/lesson-4.7/csi.txt
cat {WS}/lesson-4.7/csi.txt"""),
],
"outro": r"""You read the CNI chain on a node, the runtime the kubelet talks to, a container as the runtime sees it, ran a Pod with a RuntimeClass, and counted the storage drivers.
@ -986,9 +850,7 @@ py crd $c 's = o["spec"]["versions"][0]; p = s["schema"]["openAPIV3Schema"]["pro
s["served"] and s["storage"] and "status" in s.get("subresources", {}) and "schedule" in p.get("required", []) and p["properties"]["retention"].get("default") == 7 and p["properties"]["retention"].get("minimum") == 1' \
&& eq "$(j crd $c '{.status.conditions[?(@.type=="Established")].status}')" True""",
hint=r"""The docs page "Extend the Kubernetes API with CustomResourceDefinitions" has the skeleton. `required:` sits beside `properties:`, at the level of `spec`'s schema; `default:` and `minimum:` inside `retention`. `subresources: {status: {}}` goes under the version.""",
show=[hand(r"""vim crd.yaml""",
r"""Paste the example from the docs page *Extend the Kubernetes API with CustomResourceDefinitions*, and make it:
```
show=r"""kubectl apply -f - <<'EOF'
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
@ -1017,101 +879,54 @@ spec:
type: object
properties:
phase: {type: string}
```""",
run=r"""cat > crd.yaml <<'EOF'
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: backups.data.example.com
spec:
group: data.example.com
scope: Namespaced
names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]}
versions:
- name: v1
served: true
storage: true
subresources:
status: {}
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [schedule]
properties:
schedule: {type: string}
retention: {type: integer, minimum: 1, default: 7}
status:
type: object
properties:
phase: {type: string}
EOF"""),
cmd(r"""k apply -f crd.yaml
k api-resources --api-group=data.example.com""")]),
EOF
kubectl wait --for condition=established crd/backups.data.example.com
kubectl api-resources --api-group=data.example.com"""),
step("Defaults and validation",
r"""In Namespace `data`, create Backup `nightly` with `schedule: "0 2 * * *"` and no retention: the API server fills in `7`. Then try a Backup `broken` with `retention: 0` - it is refused. Save the refusal (kubectl writes it to **stderr**) in `{WS}/lesson-4.8/refused.txt`.""",
check=r"""
eq "$(j -n data bk nightly '{.spec.schedule} {.spec.retention}')" "0 2 * * * 7" && [ -z "$(j -n data bk broken '{.metadata.name}')" ] \
&& grep -q 'retention' "$WS/lesson-4.8/refused.txt" 2>/dev/null && grep -qi 'greater than or equal to 1' "$WS/lesson-4.8/refused.txt" """,
hint=r"""`apiVersion: data.example.com/v1`, `kind: Backup`. Redirect with `2>` - with `>` the file stays empty.""",
show=[hand(r"""vim nightly.yaml""",
r"""```
show=r"""kubectl -n data apply -f - <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: nightly
spec:
schedule: "0 2 * * *"
```""",
run=r"""cat > nightly.yaml <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: nightly
spec:
schedule: "0 2 * * *"
EOF"""),
cmd(r"""k -n data apply -f nightly.yaml
k -n data get bk nightly -o yaml | grep -A2 '^spec:' """, r"""`retention: 7`, filled in by the API server."""),
hand(r"""cp nightly.yaml broken.yaml
vim broken.yaml""",
r"""Name it `broken`, and add `retention: 0` under `spec:`.""",
run=r"""printf 'apiVersion: data.example.com/v1\nkind: Backup\nmetadata: {name: broken}\nspec: {schedule: "@hourly", retention: 0}\n' > broken.yaml"""),
cmd(r"""k -n data apply -f broken.yaml 2> {WS}/lesson-4.8/refused.txt
cat {WS}/lesson-4.8/refused.txt""")]),
EOF
kubectl -n data get bk nightly -o jsonpath='{.spec}'; echo
printf 'apiVersion: data.example.com/v1\nkind: Backup\nmetadata: {name: broken}\nspec: {schedule: "@hourly", retention: 0}\n' | kubectl -n data apply -f - 2> {WS}/lesson-4.8/refused.txt
cat {WS}/lesson-4.8/refused.txt"""),
step("What the schema says",
r"""`kubectl explain` reads the CRD's schema like any built-in kind's. Save the output of `kubectl explain backup.spec` in `{WS}/lesson-4.8/explain.txt`.""",
check=r"""f="$WS/lesson-4.8/explain.txt"; grep -q 'retention' "$f" 2>/dev/null && grep -q 'schedule' "$f" && grep -qi 'required' "$f" """,
hint=r"""`kubectl explain backup.spec` - or `--recursive` for the whole tree.""",
show=[cmd(r"""k explain backup.spec | tee {WS}/lesson-4.8/explain.txt""")]),
show=r"""kubectl explain backup.spec | tee {WS}/lesson-4.8/explain.txt"""),
step("Install the operator",
r"""Install the operator from `{WS}/lesson-4.8/operator/` and wait until its controller runs. Read the manifests first: a Namespace, a ServiceAccount, a ClusterRole for `backups` and ConfigMaps, the binding, and the controller's Deployment.""",
check=r"""eq "$(j -n backup-operator deploy controller '{.status.readyReplicas}')" 1 && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller""",
hint=r"""`kubectl apply -f DIR` applies every file in it; `kubectl -n backup-operator rollout status deploy controller`.""",
show=[cmd(r"""grep -e ^kind: -e '^ name:' {WS}/lesson-4.8/operator/operator.yaml
k apply -f {WS}/lesson-4.8/operator/
k -n backup-operator rollout status deploy controller""")]),
show=r"""grep -E '^kind:|^ name:' {WS}/lesson-4.8/operator/operator.yaml
kubectl apply -f {WS}/lesson-4.8/operator/
kubectl -n backup-operator rollout status deploy controller --timeout=120s"""),
step("Let it work",
r"""Create Backup `weekly` in `data` with `schedule: "0 3 * * 0"`. Within a few seconds the controller has acted on it - and on `nightly`: each has `status.phase: Scheduled`, and a ConfigMap `backup-NAME` holds its schedule. `kubectl -n backup-operator logs deploy/controller` is where you would look if it did not.""",
check=r"""
eq "$(j -n data bk weekly '{.status.phase}')" Scheduled && eq "$(j -n data cm backup-weekly '{.data.schedule}')" "0 3 * * 0" && eq "$(j -n data bk nightly '{.status.phase}')" Scheduled""",
hint=r"""A Backup like `nightly`. `kubectl -n data get bk` - add `additionalPrinterColumns` to the CRD and the phase would show there too.""",
show=[hand(r"""cp nightly.yaml weekly.yaml
vim weekly.yaml""",
r"""Name it `weekly`, with `schedule: "0 3 * * 0"`.""",
run=r"""cat > weekly.yaml <<'EOF'
show=r"""kubectl -n data apply -f - <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: weekly
spec:
schedule: "0 3 * * 0"
EOF"""),
cmd(r"""k -n data apply -f weekly.yaml
k -n data get bk -o yaml | grep phase
k -n data get cm""", r"""Again in a few seconds: both Backups `phase: Scheduled`, and a ConfigMap `backup-NAME` for each.""")]),
EOF
sleep 10
kubectl -n data get bk -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PHASE:.status.phase
kubectl -n data get cm"""),
],
"outro": r"""You defined a kind with validation and defaults, saw the schema refuse bad objects, installed an operator with its RBAC, and watched it act on your custom resources.

479
cka/lessons_net.py
View File

@ -1,6 +1,6 @@
"""CKA lessons of the Services and Networking domain (5.1-5.6). See cka/lessons.py for what a lesson is."""
from cka.common import DNS
from sim.bank import NET, NS, PODS, step, cmd, hand
from sim.bank import NET, NS, PODS, step
# ---- 5.1 connectivity between Pods ------------------------------------------------------------
L51 = {
@ -35,55 +35,35 @@ ready net || exit 1
want=$($K get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' | sort)
eq "$(tr -s ' \t' ' ' < "$WS/lesson-5.1/cidrs.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """,
hint=r"""`.spec.podCIDR` of each node: `-o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}'`, or `custom-columns`. The cluster-wide range is the `podSubnet` in kubeadm's configuration.""",
show=[cmd(r"""k get no -o custom-columns=NODE:.metadata.name,CIDR:.spec.podCIDR --no-headers > {WS}/lesson-5.1/cidrs.txt
show=r"""kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' > {WS}/lesson-5.1/cidrs.txt
cat {WS}/lesson-5.1/cidrs.txt
k -n net get po -o wide""", r"""Every Pod's IP is from the range of the node it runs on.""")]),
kubectl -n net get pods -o wide"""),
step("Across nodes, Pod to Pod",
r"""From Pod `a`, send a request straight to Pod `b`'s **IP address** on port 80 - no Service - and save the answer in `{WS}/lesson-5.1/answer.txt`. It crossed from `{W1}` to `{W2}`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/answer.txt" 2>/dev/null)" b""",
hint=r"""`kubectl -n net get pod b -o jsonpath='{.status.podIP}'`, then `kubectl -n net exec a -- wget -qO- -T3 http://IP`.""",
show=[cmd(r"""k -n net get po -o wide"""),
hand(r"""k -n net exec a -- wget -qO- -T3 IP > {WS}/lesson-5.1/answer.txt""",
r"""`IP`: Pod `b`'s address, from the list.""",
run=r"""ip=$(kubectl -n net get pod b -o jsonpath='{.status.podIP}')
kubectl -n net exec a -- wget -qO- -T3 http://$ip > {WS}/lesson-5.1/answer.txt"""),
cmd(r"""cat {WS}/lesson-5.1/answer.txt""")]),
show=r"""ip=$(kubectl -n net get pod b -o jsonpath='{.status.podIP}')
kubectl -n net exec a -- wget -qO- -T3 http://$ip | tee {WS}/lesson-5.1/answer.txt"""),
step("The route that carries it",
r"""On `{W1}`, look at the routing table: there is a route for `{W2}`'s Pod CIDR, via `{W2}`'s node address. Write that route's destination and gateway to `{WS}/lesson-5.1/route.txt`, as `CIDR via ADDRESS`.""",
check=r"""
c=$(j node "$W2" '{.spec.podCIDR}'); gw=$(j node "$W2" '{.status.addresses[?(@.type=="InternalIP")].address}')
nx "$W1" "ip route | grep -q '^$c via $gw '" </dev/null && eq "$(tr -s ' ' < "$WS/lesson-5.1/route.txt" 2>/dev/null | tr -d '\r\n' | sed 's/ *$//')" "$c via $gw" """,
hint=r"""`ip route` on the node. The node's address is its `InternalIP` (`kubectl get nodes -o wide`).""",
show=[cmd(r"""k get no -o wide
k get no -o custom-columns=NODE:.metadata.name,CIDR:.spec.podCIDR""", r"""`{W2}`'s address (`INTERNAL-IP`) and its Pod CIDR."""),
hand(r"""ssh {W1} ip route""",
r"""The line for `{W2}`'s Pod CIDR reads `CIDR via ADDRESS dev ...` - the route to the other node's Pods.""",
run=r"""ssh -n {W1} ip route"""),
hand(r"""echo 'CIDR via ADDRESS' > {WS}/lesson-5.1/route.txt""",
r"""The first three words of that line.""",
run=r"""c=$(kubectl get node {W2} -o jsonpath='{.spec.podCIDR}')
ssh -n {W1} ip route | grep "^$c " | awk '{print $1, $2, $3}' > {WS}/lesson-5.1/route.txt""")]),
show=r"""ssh -n {W1} ip route
c=$(kubectl get node {W2} -o jsonpath='{.spec.podCIDR}')
ssh -n {W1} ip route | grep "^$c " | awk '{print $1, $2, $3}' > {WS}/lesson-5.1/route.txt
cat {WS}/lesson-5.1/route.txt"""),
step("A Pod on the host's network",
r"""Run Pod `hostnet` in `net` (`busybox:1.36`, `sleep 86400`) with `hostNetwork: true`, on `{W1}`. Its IP is not from the Pod CIDR: it is `{W1}`'s own address, and it sees the node's interfaces. Write its IP to `{WS}/lesson-5.1/hostnet.txt`.""",
check=r"""
want=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}')
eq "$(j -n net pod hostnet '{.spec.hostNetwork} {.status.podIP}')" "true $want" && eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/hostnet.txt" 2>/dev/null)" "$want" """,
hint=r"""`spec.hostNetwork: true` and `spec.nodeName: {W1}` (or a nodeSelector). `kubectl run ... --overrides='{"spec":{...}}'` adds both.""",
show=[cmd(r"""k -n net run hostnet --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > hostnet.yaml"""),
hand(r"""vim hostnet.yaml""",
r"""Add to the Pod's `spec:`, above `containers:`:
```
hostNetwork: true
nodeName: {W1}
```""",
run=r"""yq -i '.spec.hostNetwork = true | .spec.nodeName = "{W1}"' hostnet.yaml"""),
cmd(r"""k -n net apply -f hostnet.yaml
k -n net get po hostnet -o wide""", r"""Its IP is `{W1}`'s own address, not one from the Pod CIDR."""),
hand(r"""echo IP > {WS}/lesson-5.1/hostnet.txt""",
r"""`IP`: that address.""",
run=r"""kubectl -n net wait --for=condition=Ready pod/hostnet --timeout=120s
kubectl -n net get pod hostnet -o jsonpath='{.status.podIP}' > {WS}/lesson-5.1/hostnet.txt"""),
cmd(r"""k -n net exec hostnet -- ip -4 addr show eth0""")]),
show=r"""kubectl -n net run hostnet --image=busybox:1.36 --overrides='{"spec":{"hostNetwork":true,"nodeName":"{W1}"}}' -- sleep 86400
kubectl -n net wait --for=condition=Ready pod/hostnet --timeout=120s
kubectl -n net get pod hostnet -o jsonpath='{.status.podIP}' > {WS}/lesson-5.1/hostnet.txt
kubectl -n net exec hostnet -- ip -4 addr show eth0"""),
],
"outro": r"""You read how the Pod network is divided among nodes, sent a request from one node's Pod to another's, found the route that carried it, and ran a Pod on the host network.
@ -128,9 +108,7 @@ ready prod && ready dev || exit 1
eq "$(j -n prod netpol default-deny '{.spec.policyTypes[0]}')" Ingress && [ -z "$(j -n prod netpol default-deny '{.spec.ingress}')" ] || exit 1
blocked prod api http://db && blocked dev tester http://web.prod""",
hint=r"""`podSelector: {}` and `policyTypes: [Ingress]`, with no `ingress:` list. Test with `kubectl -n prod exec api -- wget -qO- -T3 db`.""",
show=[hand(r"""vim deny.yaml""",
r"""No `kubectl create` for a NetworkPolicy: paste *Default deny all ingress traffic* from the docs page *Network Policies*, and name it:
```
show=r"""kubectl -n prod apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@ -138,25 +116,13 @@ metadata:
spec:
podSelector: {}
policyTypes: [Ingress]
```""",
run=r"""cat > deny.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes: [Ingress]
EOF"""),
cmd(r"""k -n prod apply -f deny.yaml
k -n prod exec api -- wget -qO- -T3 db""", r"""It times out: blocked.""")]),
EOF
kubectl -n prod exec api -- wget -qO- -T3 db || echo blocked"""),
step("Let the API reach the database",
r"""Create NetworkPolicy `db-from-api` in `prod`: Pods `app=db` accept TCP 80 from Pods `app=api` **of `prod`**. `web` still cannot reach `db`, and neither can `dev`'s `api` - a `podSelector` alone means "in this Namespace".""",
check=NET + r"""okr prod api http://db && blocked prod web http://db && blocked dev api http://db.prod""",
hint=r"""`podSelector: {matchLabels: {app: db}}`, `ingress: [{from: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}]`.""",
show=[hand(r"""vim db-from-api.yaml""",
r"""Paste the example from the docs page *Network Policies* and cut it down to:
```
show=r"""kubectl -n prod apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@ -171,36 +137,16 @@ spec:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
```""",
run=r"""cat > db-from-api.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: db-from-api
spec:
podSelector:
matchLabels: {app: db}
policyTypes: [Ingress]
ingress:
- from:
- podSelector:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
EOF"""),
cmd(r"""k -n prod apply -f db-from-api.yaml
k -n prod exec api -- wget -qO- -T3 db
k -n prod exec web -- wget -qO- -T3 db""", r"""`api` gets the page; `web` times out.""")]),
EOF
kubectl -n prod exec api -- wget -qO- -T3 db
kubectl -n prod exec web -- wget -qO- -T3 db || echo blocked"""),
step("Those Pods in that Namespace",
r"""Create NetworkPolicy `web-from-dev-api` in `prod`: Pods `app=web` accept TCP 80 from Pods `app=api` **in** Namespaces labelled `team=dev` - both conditions in **one** element of `from`.
`dev`'s `api` gets in; `dev`'s `tester` does not, and neither does `prod`'s own `api`. Written as two elements, the rule would let every Pod of `dev` in, and every `api` Pod of `prod` too.""",
check=NET + r"""okr dev api http://web.prod && blocked dev tester http://web.prod && blocked prod api http://web""",
hint=r"""`from: [{namespaceSelector: {matchLabels: {team: dev}}, podSelector: {matchLabels: {app: api}}}]` - mind the single `-`.""",
show=[hand(r"""cp db-from-api.yaml web-from-dev-api.yaml
vim web-from-dev-api.yaml""",
r"""Make it - `namespaceSelector` and `podSelector` in **one** element of `from` (no `-` before `podSelector`):
```
show=r"""kubectl -n prod apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@ -217,28 +163,9 @@ spec:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
```""",
run=r"""cat > web-from-dev-api.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-from-dev-api
spec:
podSelector:
matchLabels: {app: web}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels: {team: dev}
podSelector:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
EOF"""),
cmd(r"""k -n prod apply -f web-from-dev-api.yaml
k -n dev exec api -- wget -qO- -T3 web.prod
k -n dev exec tester -- wget -qO- -T3 web.prod""", r"""`dev`'s `api` gets in; `tester` times out.""")]),
EOF
kubectl -n dev exec api -- wget -qO- -T3 web.prod
kubectl -n dev exec tester -- wget -qO- -T3 web.prod || echo blocked"""),
step("Egress, and DNS",
r"""Now limit what `tester` in `dev` may **send**: NetworkPolicy `tester-egress` in `dev` selects `app=tester` and allows egress only to `dev`'s `app=api` Pods on TCP 80 - and DNS, to `kube-dns` in `kube-system` on UDP and TCP 53.
@ -247,9 +174,7 @@ k -n dev exec tester -- wget -qO- -T3 web.prod""", r"""`dev`'s `api` gets in; `t
eq "$(j -n dev netpol tester-egress '{.spec.policyTypes[0]}')" Egress || exit 1
okr dev tester http://api && blocked dev tester http://echo""",
hint=r"""Two egress rules: `{to: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}` and `{to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}], ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]}`. Every Namespace carries the label `kubernetes.io/metadata.name`.""",
show=[hand(r"""vim tester-egress.yaml""",
r"""An egress policy - two elements of `egress`, one for `api`, one for DNS:
```
show=r"""kubectl -n dev apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@ -272,34 +197,9 @@ spec:
ports:
- {protocol: UDP, port: 53}
- {protocol: TCP, port: 53}
```""",
run=r"""cat > tester-egress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: tester-egress
spec:
podSelector:
matchLabels: {app: tester}
policyTypes: [Egress]
egress:
- to:
- podSelector:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: kube-system}
podSelector:
matchLabels: {k8s-app: kube-dns}
ports:
- {protocol: UDP, port: 53}
- {protocol: TCP, port: 53}
EOF"""),
cmd(r"""k -n dev apply -f tester-egress.yaml
k -n dev exec tester -- wget -qO- -T3 api
k -n dev exec tester -- wget -qO- -T3 echo""", r"""`api` answers; `echo` times out.""")]),
EOF
kubectl -n dev exec tester -- wget -qO- -T3 api
kubectl -n dev exec tester -- wget -qO- -T3 echo || echo blocked"""),
],
"outro": r"""You closed a Namespace, opened single paths into it, used the AND form of a rule, and limited a Pod's egress without breaking its DNS.
@ -343,28 +243,20 @@ $K -n svc rollout status deploy web --timeout=180s >/dev/null && ready svc
r"""Expose Deployment `web` as Service `web`, port 80. From `client`, `wget -qO- web` returns nginx's page.""",
check=NET + r"""eq "$(j -n svc svc web '{.spec.type} {.spec.ports[0].port}')" "ClusterIP 80" && okr svc client http://web""",
hint=r"""`kubectl expose deploy web --port=80` - it copies the Deployment's selector.""",
show=[cmd(r"""k -n svc expose deploy web --port=80
k -n svc get svc,endpointslices
k -n svc exec client -- wget -qO- -T3 web | head -4""")]),
show=r"""kubectl -n svc expose deploy web --port=80
kubectl -n svc get svc,endpointslices
kubectl -n svc exec client -- wget -qO- -T3 web | head -4"""),
step("NodePort",
r"""Create Service `web-np` of type **NodePort** for the same Pods, with node port **30080**. Then reach it from the control-plane node: `curl http://{W1}-ADDRESS:30080` - every node answers on that port, whether or not it runs one of the Pods.""",
check=r"""
eq "$(j -n svc svc web-np '{.spec.type} {.spec.ports[0].nodePort}')" "NodePort 30080" || exit 1
ip=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}'); nx "$CP" "curl -s -m 3 http://$ip:30080 | grep -q nginx" </dev/null""",
hint=r"""`kubectl create service nodeport web-np --tcp=80:80 --node-port=30080` makes a selector `app=web-np` - change it to `app=web`; or write the YAML, or `kubectl expose deploy web --name=web-np --type=NodePort` and then set the `nodePort`.""",
show=[cmd(r"""k -n svc expose deploy web --name=web-np --type=NodePort --port=80 --dry-run=client -o yaml > web-np.yaml"""),
hand(r"""vim web-np.yaml""",
r"""Add to the port:
```
nodePort: 30080
```""",
run=r"""yq -i '.spec.ports[0].nodePort = 30080' web-np.yaml"""),
cmd(r"""k -n svc apply -f web-np.yaml
k get no -o wide""", r"""`{W1}`'s `INTERNAL-IP`:"""),
hand(r"""curl IP:30080""",
r"""nginx's page, from the node port.""",
run=r"""ip=$(kubectl get node {W1} -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
curl -s -m 3 http://$ip:30080 | head -4""")]),
show=r"""kubectl -n svc expose deploy web --name=web-np --type=NodePort --port=80 --dry-run=client -o yaml > {WS}/lesson-5.3/web-np.yaml
yq -i '.spec.ports[0].nodePort = 30080' {WS}/lesson-5.3/web-np.yaml
kubectl -n svc apply -f {WS}/lesson-5.3/web-np.yaml
ip=$(kubectl get node {W1} -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
curl -s -m 3 http://$ip:30080 | head -4"""),
step("Headless",
r"""Create Service `web-headless` with `clusterIP: None` for the `web` Pods. Look it up from `client` with `nslookup web-headless`: you get the two Pod IPs, not a Service IP. Write them, one per line, to `{WS}/lesson-5.3/headless.txt`.""",
check=DNS + r"""
@ -372,19 +264,18 @@ eq "$(j -n svc svc web-headless '{.spec.clusterIP}')" None || exit 1
want=$($K -n svc get pods -l app=web -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' | sort)
eq "$(tr -d ' \r' < "$WS/lesson-5.3/headless.txt" 2>/dev/null | grep . | sort)" "$want" """,
hint=r"""`kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80`. In busybox's `nslookup` output the addresses follow the `Name:` line.""",
show=[cmd(r"""k -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80
k -n svc exec client -- nslookup web-headless""", r"""Two addresses: the Pods' own. Again in a few seconds if it does not resolve yet."""),
hand(r"""vim {WS}/lesson-5.3/headless.txt""",
r"""The two addresses, one per line.""",
run=r"""sleep 5
kubectl -n svc exec client -- nslookup web-headless | awk '/^Name:/ {n=1} n && /^Address/ {print $NF}' > {WS}/lesson-5.3/headless.txt""")]),
show=r"""kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80
sleep 5
kubectl -n svc exec client -- nslookup web-headless
kubectl -n svc exec client -- nslookup web-headless | awk '/^Name:/ {n=1} n && /^Address/ {print $NF}' > {WS}/lesson-5.3/headless.txt
cat {WS}/lesson-5.3/headless.txt"""),
step("ExternalName",
r"""Create Service `docs` of type **ExternalName** pointing at `kubernetes.io`. Inside the cluster, `docs.svc.svc.cluster.local` is now a CNAME for that name - DNS only: nothing is proxied, and the port does not matter.""",
check=r"""eq "$(j -n svc svc docs '{.spec.type} {.spec.externalName}')" "ExternalName kubernetes.io" """,
hint=r"""`kubectl -n svc create service externalname docs --external-name=kubernetes.io`.""",
show=[cmd(r"""k -n svc create svc externalname docs --external-name=kubernetes.io
k -n svc get svc docs
k -n svc exec client -- nslookup docs | head -6""")]),
show=r"""kubectl -n svc create service externalname docs --external-name=kubernetes.io
kubectl -n svc get svc docs
kubectl -n svc exec client -- nslookup docs | head -6"""),
step("A Service without a selector",
r"""Create Service `legacy` (port 80) with **no selector**, and an EndpointSlice for it by hand that holds `legacy-box`'s Pod IP on port 80. From `client`, `wget -qO- legacy` then answers `legacy`.
@ -393,32 +284,8 @@ An EndpointSlice belongs to a Service by the label `kubernetes.io/service-name`;
[ -z "$(j -n svc svc legacy '{.spec.selector}')" ] || exit 1
for i in 1 2 3 4 5; do [ "$($K -n svc exec client -- wget -qO- -T3 legacy 2>/dev/null)" = legacy ] && exit 0; sleep 3; done; exit 1""",
hint=r"""Service: `spec.ports: [{port: 80, targetPort: 80}]` and no `selector`. EndpointSlice (`discovery.k8s.io/v1`): `addressType: IPv4`, `endpoints: [{addresses: [IP]}]`, `ports: [{port: 80, protocol: TCP}]`, and the label.""",
show=[cmd(r"""k -n svc get po legacy-box -o wide""", r"""Its IP goes into the EndpointSlice."""),
hand(r"""vim legacy.yaml""",
r"""No `kubectl create` makes a Service without a selector, nor an EndpointSlice: paste them from the docs page *Service* (*Services without selectors*), and make them, with that IP:
```
apiVersion: v1
kind: Service
metadata:
name: legacy
spec:
ports:
- {port: 80, targetPort: 80, protocol: TCP}
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: legacy-1
labels:
kubernetes.io/service-name: legacy
addressType: IPv4
endpoints:
- addresses: ["IP"]
ports:
- {port: 80, protocol: TCP}
```""",
run=r"""ip=$(kubectl -n svc get pod legacy-box -o jsonpath='{.status.podIP}')
cat > legacy.yaml <<EOF
show=r"""ip=$(kubectl -n svc get pod legacy-box -o jsonpath='{.status.podIP}')
kubectl -n svc apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
@ -438,9 +305,8 @@ endpoints:
- addresses: ["$ip"]
ports:
- {port: 80, protocol: TCP}
EOF"""),
cmd(r"""k -n svc apply -f legacy.yaml
k -n svc exec client -- wget -qO- -T3 legacy""")]),
EOF
kubectl -n svc exec client -- wget -qO- -T3 legacy"""),
],
"outro": r"""You exposed the same Pods four ways, saw what headless DNS returns, aliased an outside name, and backed a Service with endpoints you wrote yourself.
@ -482,9 +348,7 @@ mkdir -p "$WS/lesson-5.4"
eq "$(j gatewayclass example '{.spec.controllerName}')" example.com/gateway-controller || exit 1
eq "$(j -n gw gateway shop '{.spec.gatewayClassName} {.spec.listeners[0].name} {.spec.listeners[0].protocol} {.spec.listeners[0].port} {.spec.listeners[0].hostname} {.spec.listeners[0].allowedRoutes.namespaces.from}')" "example http HTTP 80 shop.example.com Same" """,
hint=r"""Both are `gateway.networking.k8s.io/v1`. `kubectl explain gateway.spec.listeners` once the CRDs are in; the docs at gateway-api.sigs.k8s.io have every example.""",
show=[hand(r"""vim gw.yaml""",
r"""No `kubectl create` for these: paste a GatewayClass and a Gateway from gateway-api.sigs.k8s.io (*API overview*), and make them:
```
show=r"""kubectl apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
@ -507,33 +371,8 @@ spec:
allowedRoutes:
namespaces:
from: Same
```""",
run=r"""cat > gw.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: example
spec:
controllerName: example.com/gateway-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: shop
namespace: gw
spec:
gatewayClassName: example
listeners:
- name: http
protocol: HTTP
port: 80
hostname: shop.example.com
allowedRoutes:
namespaces:
from: Same
EOF"""),
cmd(r"""k apply -f gw.yaml
k -n gw get gateway shop""")]),
EOF
kubectl -n gw get gateway shop"""),
step("Route by path",
r"""Create HTTPRoute `shop` in `gw`, attached to Gateway `shop`: requests for paths under `/api` go to Service `api` port 8080, everything else to `web` port 80.""",
check=r"""
@ -547,9 +386,7 @@ def to(path):
return [(b["name"], b.get("port")) for b in x["backendRefs"]]
sys.exit(0 if p == "shop" and to("/api") == [("api", 8080)] and to("/") == [("web", 80)] else 1)'""",
hint=r"""`parentRefs: [{name: shop}]`, then two `rules`, each with `matches: [{path: {type: PathPrefix, value: ...}}]` and `backendRefs: [{name: ..., port: ...}]`.""",
show=[hand(r"""vim route.yaml""",
r"""Paste an HTTPRoute from gateway-api.sigs.k8s.io (*HTTP routing*), and make it:
```
show=r"""kubectl -n gw apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
@ -567,28 +404,8 @@ spec:
- path: {type: PathPrefix, value: /}
backendRefs:
- {name: web, port: 80}
```""",
run=r"""cat > route.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop
spec:
parentRefs:
- name: shop
hostnames: [shop.example.com]
rules:
- matches:
- path: {type: PathPrefix, value: /api}
backendRefs:
- {name: api, port: 8080}
- matches:
- path: {type: PathPrefix, value: /}
backendRefs:
- {name: web, port: 80}
EOF"""),
cmd(r"""k -n gw apply -f route.yaml
k -n gw describe httproute shop""")]),
EOF
kubectl -n gw describe httproute shop | grep -A3 'Backend Refs' | head -8"""),
step("A canary by weight",
r"""Change the `/` rule of route `shop` so that **90%** of its requests go to `web` and **10%** to `web-v2` (both port 80). The `/api` rule stays as it is.""",
check=r"""
@ -599,20 +416,10 @@ root, api = by("/"), by("/api")
w = sorted((b["name"], b.get("weight", 1)) for b in root["backendRefs"]) if root else []
sys.exit(0 if w == [("web", 90), ("web-v2", 10)] and api and [b["name"] for b in api["backendRefs"]] == ["api"] else 1)'""",
hint=r"""Two `backendRefs` in the same rule, each with a `weight`. The weights are relative: 9 and 1 would do the same.""",
show=[hand(r"""k -n gw edit httproute shop""",
r"""In the rule whose path is `/`, give `web` a `weight: 90` and add `web-v2` beside it:
```
- name: web
port: 80
weight: 90
- name: web-v2
port: 80
weight: 10
```""",
run=r"""kubectl -n gw get httproute shop -o yaml > {WS}/lesson-5.4/shop.yaml
show=r"""kubectl -n gw get httproute shop -o yaml > {WS}/lesson-5.4/shop.yaml
yq -i '(.spec.rules[] | select(.matches[0].path.value == "/")).backendRefs = [{"name": "web", "port": 80, "weight": 90}, {"name": "web-v2", "port": 80, "weight": 10}]' {WS}/lesson-5.4/shop.yaml
kubectl apply -f {WS}/lesson-5.4/shop.yaml"""),
cmd(r"""k -n gw describe httproute shop""")]),
kubectl apply -f {WS}/lesson-5.4/shop.yaml
kubectl -n gw get httproute shop -o jsonpath='{.spec.rules[1].backendRefs}'; echo"""),
step("Beta testers by header",
r"""Add HTTPRoute `shop-beta` in `gw`, attached to `shop`: requests carrying the header `x-env: beta` go **all** to `web-v2`. A more specific match wins, so these requests no longer take part in the 90/10 split.""",
check=r"""
@ -621,9 +428,7 @@ import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0]
h = r["matches"][0].get("headers", [{}])[0]
sys.exit(0 if s["parentRefs"][0]["name"] == "shop" and h.get("name", "").lower() == "x-env" and h.get("value") == "beta" and [b["name"] for b in r["backendRefs"]] == ["web-v2"] else 1)'""",
hint=r"""`matches: [{headers: [{name: x-env, value: beta}]}]` (type `Exact` by default).""",
show=[hand(r"""vim beta.yaml""",
r"""Paste an HTTPRoute with a header match from gateway-api.sigs.k8s.io (*HTTP header matching*), and make it:
```
show=r"""kubectl -n gw apply -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
@ -638,25 +443,8 @@ spec:
- {name: x-env, value: beta}
backendRefs:
- {name: web-v2, port: 80}
```""",
run=r"""cat > beta.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop-beta
spec:
parentRefs:
- name: shop
hostnames: [shop.example.com]
rules:
- matches:
- headers:
- {name: x-env, value: beta}
backendRefs:
- {name: web-v2, port: 80}
EOF"""),
cmd(r"""k -n gw apply -f beta.yaml
k -n gw get httproute""")]),
EOF
kubectl -n gw get httproute"""),
step("HTTPS on the Gateway",
r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `gw`, and add a second listener to Gateway `shop`: `https`, protocol HTTPS, port 443, the same hostname, TLS mode `Terminate` with that Secret as its certificate.""",
check=r"""
@ -664,27 +452,12 @@ eq "$(j -n gw secret shop-tls '{.type}')" kubernetes.io/tls || exit 1
eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="https")].protocol} {.spec.listeners[?(@.name=="https")].port} {.spec.listeners[?(@.name=="https")].tls.mode} {.spec.listeners[?(@.name=="https")].tls.certificateRefs[0].name}')" "HTTPS 443 Terminate shop-tls" \
&& eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="http")].port}')" 80""",
hint=r"""`openssl req -x509 ...`, `kubectl create secret tls`, then a listener with `tls: {mode: Terminate, certificateRefs: [{name: shop-tls}]}`. Keep the `http` listener.""",
show=[cmd(r"""cd {WS}/lesson-5.4
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com
k -n gw create secret tls shop-tls --cert=tls.crt --key=tls.key
show=r"""cd {WS}/lesson-5.4
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
kubectl -n gw create secret tls shop-tls --cert=tls.crt --key=tls.key
kubectl -n gw patch gateway shop --type json -p '[{"op":"add","path":"/spec/listeners/-","value":{"name":"https","protocol":"HTTPS","port":443,"hostname":"shop.example.com","tls":{"mode":"Terminate","certificateRefs":[{"name":"shop-tls"}]},"allowedRoutes":{"namespaces":{"from":"Same"}}}}]'
kubectl -n gw get gateway shop -o jsonpath='{range .spec.listeners[*]}{.name} {.port}{"\n"}{end}'
cd"""),
hand(r"""k -n gw edit gateway shop""",
r"""Add a second listener below `http`, in `listeners:`:
```
- name: https
protocol: HTTPS
port: 443
hostname: shop.example.com
tls:
mode: Terminate
certificateRefs:
- name: shop-tls
allowedRoutes:
namespaces:
from: Same
```""",
run=r"""kubectl -n gw patch gateway shop --type json -p '[{"op":"add","path":"/spec/listeners/-","value":{"name":"https","protocol":"HTTPS","port":443,"hostname":"shop.example.com","tls":{"mode":"Terminate","certificateRefs":[{"name":"shop-tls"}]},"allowedRoutes":{"namespaces":{"from":"Same"}}}}]' """),
cmd(r"""k -n gw describe gateway shop | grep -A3 -i listener""")]),
],
"outro": r"""You built a Gateway with HTTP and HTTPS listeners, routed by path, split traffic by weight, and sent one header's requests elsewhere.
@ -725,9 +498,7 @@ mkdir -p "$WS/lesson-5.5"
r"""Create IngressClass `nginx` for controller `k8s.io/ingress-nginx`, and make it the cluster's **default** class: an Ingress without `ingressClassName` then belongs to it.""",
check=r"""eq "$(j ingressclass nginx '{.spec.controller} {.metadata.annotations.ingressclass\.kubernetes\.io/is-default-class}')" "k8s.io/ingress-nginx true" """,
hint=r"""`networking.k8s.io/v1`, kind `IngressClass`, `spec.controller`, and the annotation `ingressclass.kubernetes.io/is-default-class: "true"`. Installing ingress-nginx makes this object for you.""",
show=[hand(r"""vim ic.yaml""",
r"""No `kubectl create` for an IngressClass: paste the example from the docs page *Ingress* (*Default IngressClass*), and make it:
```
show=r"""kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
@ -736,19 +507,8 @@ metadata:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: k8s.io/ingress-nginx
```""",
run=r"""cat > ic.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: nginx
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: k8s.io/ingress-nginx
EOF"""),
cmd(r"""k apply -f ic.yaml
k get ingressclass""")]),
EOF
kubectl get ingressclass"""),
step("Host and paths, from the command line",
r"""Create Ingress `shop` in `ing` with class `nginx` and host `shop.example.com`: paths under `/api` go to `api:8080`, everything else to `web:80`. `kubectl create ingress` can do it in one line.""",
check=r"""
@ -757,8 +517,8 @@ import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0]
p = {x["path"]: (x["pathType"], x["backend"]["service"]["name"], x["backend"]["service"]["port"]["number"]) for x in r["http"]["paths"]}
sys.exit(0 if s.get("ingressClassName") == "nginx" and r["host"] == "shop.example.com" and p.get("/api") == ("Prefix", "api", 8080) and p.get("/") == ("Prefix", "web", 80) else 1)'""",
hint=r"""`kubectl create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'` - a trailing `*` means `Prefix`, without it `Exact`.""",
show=[cmd(r"""k -n ing create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'
k -n ing describe ingress shop""")]),
show=r"""kubectl -n ing create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'
kubectl -n ing describe ingress shop | sed -n '/Rules/,/Annotations/p'"""),
step("An exact path",
r"""Add a rule to Ingress `shop`: exactly `/healthz` (pathType `Exact`) goes to `health:8081`. `/healthz/deep` would not match it - it would fall through to the `/` prefix.""",
check=r"""
@ -767,53 +527,26 @@ import json, sys; ps = json.load(sys.stdin)["spec"]["rules"][0]["http"]["paths"]
h = [x for x in ps if x["path"] == "/healthz"]
sys.exit(0 if h and h[0]["pathType"] == "Exact" and h[0]["backend"]["service"]["name"] == "health" and h[0]["backend"]["service"]["port"]["number"] == 8081 and len(ps) == 3 else 1)'""",
hint=r"""`kubectl edit ingress shop` and add a third entry under `paths:`, or a JSON patch that appends to `/spec/rules/0/http/paths`.""",
show=[hand(r"""k -n ing edit ingress shop""",
r"""Add a third path to `paths:`, indented like the other two:
```
- path: /healthz
pathType: Exact
backend:
service:
name: health
port:
number: 8081
```""",
run=r"""kubectl -n ing patch ingress shop --type json -p '[{"op":"add","path":"/spec/rules/0/http/paths/-","value":{"path":"/healthz","pathType":"Exact","backend":{"service":{"name":"health","port":{"number":8081}}}}}]' """),
cmd(r"""k -n ing describe ingress shop""")]),
show=r"""kubectl -n ing patch ingress shop --type json -p '[{"op":"add","path":"/spec/rules/0/http/paths/-","value":{"path":"/healthz","pathType":"Exact","backend":{"service":{"name":"health","port":{"number":8081}}}}}]'
kubectl -n ing get ingress shop -o jsonpath='{range .spec.rules[0].http.paths[*]}{.pathType} {.path} -> {.backend.service.name}{"\n"}{end}'"""),
step("TLS",
r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `ing`, and add a `tls` section to Ingress `shop` for that host with that Secret. The controller would then serve HTTPS for it and, by default, redirect HTTP there.""",
check=r"""
eq "$(j -n ing secret shop-tls '{.type}')" kubernetes.io/tls && eq "$(j -n ing ingress shop '{.spec.tls[0].hosts[0]} {.spec.tls[0].secretName}')" "shop.example.com shop-tls" \
&& j -n ing secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q shop.example.com""",
hint=r"""`spec.tls: [{hosts: [shop.example.com], secretName: shop-tls}]`. The Secret must be in the Ingress's Namespace.""",
show=[cmd(r"""cd {WS}/lesson-5.5
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com
k -n ing create secret tls shop-tls --cert=tls.crt --key=tls.key
show=r"""cd {WS}/lesson-5.5
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
kubectl -n ing create secret tls shop-tls --cert=tls.crt --key=tls.key
kubectl -n ing patch ingress shop --type merge -p '{"spec":{"tls":[{"hosts":["shop.example.com"],"secretName":"shop-tls"}]}}'
kubectl -n ing get ingress shop
cd"""),
hand(r"""k -n ing edit ingress shop""",
r"""Add to `spec:`, beside `rules:`:
```
tls:
- hosts: [shop.example.com]
secretName: shop-tls
```""",
run=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"tls":[{"hosts":["shop.example.com"],"secretName":"shop-tls"}]}}' """),
cmd(r"""k -n ing get ingress shop""")]),
step("What matches nothing",
r"""Give Ingress `shop` a **default backend**: `web:80`, for requests whose host or path no rule matches.""",
check=r"""eq "$(j -n ing ingress shop '{.spec.defaultBackend.service.name} {.spec.defaultBackend.service.port.number}')" "web 80" """,
hint=r"""`spec.defaultBackend: {service: {name: web, port: {number: 80}}}`; `kubectl create ingress --default-backend=web:80` does it at creation.""",
show=[hand(r"""k -n ing edit ingress shop""",
r"""Add to `spec:`, beside `rules:`:
```
defaultBackend:
service:
name: web
port:
number: 80
```""",
run=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"defaultBackend":{"service":{"name":"web","port":{"number":80}}}}}' """),
cmd(r"""k -n ing describe ingress shop""")]),
show=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"defaultBackend":{"service":{"name":"web","port":{"number":80}}}}}'
kubectl -n ing describe ingress shop | head -12"""),
],
"outro": r"""You made a default IngressClass and an Ingress with host, prefix and exact paths, TLS and a default backend.
@ -875,18 +608,16 @@ ready dns-lab && $K -n dns-lab rollout status sts db --timeout=180s >/dev/null
r"""From `client`, look up `web` by its **fully qualified** name, and write the address you get to `{WS}/lesson-5.6/web.txt`. It is the Service's ClusterIP.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/web.txt" 2>/dev/null)" "$(j -n dns-lab svc web '{.spec.clusterIP}')" """,
hint=r"""`kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local`.""",
show=[cmd(r"""k -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local""", r"""The `Address` below `Name:` is the Service's ClusterIP."""),
hand(r"""echo IP > {WS}/lesson-5.6/web.txt""",
r"""`IP`: that address.""",
run=r"""kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local | awk '/^Name:/ {n=1} n && /^Address/ {print $NF; exit}' > {WS}/lesson-5.6/web.txt""")]),
show=r"""kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local
kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local | awk '/^Name:/ {n=1} n && /^Address/ {print $NF; exit}' > {WS}/lesson-5.6/web.txt"""),
step("One Pod of a StatefulSet",
r"""A StatefulSet's Pods get stable names through their headless Service. Write the fully qualified DNS name of Pod `db-0` to `{WS}/lesson-5.6/db0.txt`, and check from `client` that it resolves to `db-0`'s IP.""",
check=DNS + r"""
eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/db0.txt" 2>/dev/null | sed 's/\.$//')" db-0.db.dns-lab.svc.cluster.local && resolves dns-lab client db-0.db.dns-lab.svc.cluster.local "$(j -n dns-lab pod db-0 '{.status.podIP}')" """,
hint=r"""`POD.SERVICE.NAMESPACE.svc.cluster.local` - the Service is the StatefulSet's `serviceName`.""",
show=[cmd(r"""echo db-0.db.dns-lab.svc.cluster.local > {WS}/lesson-5.6/db0.txt
k -n dns-lab exec client -- nslookup db-0.db.dns-lab.svc.cluster.local
k -n dns-lab get po db-0 -o wide""")]),
show=r"""echo db-0.db.dns-lab.svc.cluster.local > {WS}/lesson-5.6/db0.txt
kubectl -n dns-lab exec client -- nslookup db-0.db.dns-lab.svc.cluster.local
kubectl -n dns-lab get pod db-0 -o wide"""),
step("Names of your own",
r"""Add a `hosts` block to CoreDNS's Corefile so that every Pod resolves **`legacy.internal`** to `10.96.0.250`, and every other name still resolves as before (`fallthrough`). Put it inside the `.:53 { ... }` server block, before `forward`.
@ -895,19 +626,12 @@ CoreDNS reloads the changed ConfigMap by itself within a minute; `client` then r
$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep -q 'kubernetes cluster.local' || exit 1
resolves dns-lab client legacy.internal 10.96.0.250 && resolves dns-lab client web.dns-lab.svc.cluster.local "$(j -n dns-lab svc web '{.spec.clusterIP}')" """,
hint=r"""`kubectl -n kube-system edit cm coredns`. The block: `hosts {` / `10.96.0.250 legacy.internal` / `fallthrough` / `}`. A syntax error leaves CoreDNS on its old configuration: `kubectl -n kube-system logs -l k8s-app=kube-dns` says so.""",
show=[hand(r"""k -n kube-system edit cm coredns""",
r"""In `Corefile`, add above the `forward` line, indented like it:
```
hosts {
10.96.0.250 legacy.internal
fallthrough
}
```""",
run=r"""kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > {WS}/lesson-5.6/Corefile
show=r"""kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > {WS}/lesson-5.6/Corefile
sed -i '0,/^ forward/s// hosts {\n 10.96.0.250 legacy.internal\n fallthrough\n }\n forward/' {WS}/lesson-5.6/Corefile
cat {WS}/lesson-5.6/Corefile
kubectl -n kube-system create configmap coredns --from-file=Corefile={WS}/lesson-5.6/Corefile --dry-run=client -o yaml | kubectl apply -f -
sleep 40"""),
cmd(r"""k -n dns-lab exec client -- nslookup legacy.internal""", r"""Again until it answers `10.96.0.250`: CoreDNS reloads the ConfigMap by itself within a minute or two.""")]),
sleep 40
kubectl -n dns-lab exec client -- nslookup legacy.internal"""),
step("A Pod's own resolver settings",
r"""Run Pod `tuned` in `dns-lab` (`busybox:1.36`, `sleep 86400`) whose resolver you set yourself: `dnsPolicy: None`, with the `kube-dns` Service's IP as its only nameserver, the search domains `dns-lab.svc.cluster.local` and `svc.cluster.local`, and the option `ndots: 2`. Its `/etc/resolv.conf` shows exactly that.""",
check=r"""
@ -916,21 +640,26 @@ r=$($K -n dns-lab exec tuned -- cat /etc/resolv.conf 2>/dev/null)
echo "$r" | grep -qx "nameserver $ip" && echo "$r" | grep -qx 'search dns-lab.svc.cluster.local svc.cluster.local' && echo "$r" | grep -q 'ndots:2' \
&& eq "$(j -n dns-lab pod tuned '{.spec.dnsPolicy}')" None""",
hint=r"""`spec.dnsPolicy: None` and `spec.dnsConfig: {nameservers: [IP], searches: [...], options: [{name: ndots, value: "2"}]}`. The other policies: `ClusterFirst` (the default), `Default` (the node's resolver), `ClusterFirstWithHostNet`.""",
show=[cmd(r"""k -n kube-system get svc kube-dns""", r"""Its `CLUSTER-IP` is the nameserver."""),
cmd(r"""k -n dns-lab run tuned --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > tuned.yaml"""),
hand(r"""vim tuned.yaml""",
r"""Change `dnsPolicy: ClusterFirst` to `dnsPolicy: None`, and add beside it (the docs page *DNS for Services and Pods* has it, under *Pod's DNS Config*):
```
show=r"""ip=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}')
kubectl -n dns-lab apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: tuned
spec:
dnsPolicy: None
dnsConfig:
nameservers: [IP]
nameservers: [$ip]
searches: [dns-lab.svc.cluster.local, svc.cluster.local]
options:
- {name: ndots, value: "2"}
```""",
run=r"""export IP=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}')
yq -i '.spec.dnsPolicy = "None" | .spec.dnsConfig = {"nameservers": [strenv(IP)], "searches": ["dns-lab.svc.cluster.local", "svc.cluster.local"], "options": [{"name": "ndots", "value": "2"}]}' tuned.yaml"""),
cmd(r"""k -n dns-lab apply -f tuned.yaml
k -n dns-lab exec tuned -- cat /etc/resolv.conf""")]),
containers:
- name: tuned
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n dns-lab wait --for=condition=Ready pod/tuned --timeout=120s
kubectl -n dns-lab exec tuned -- cat /etc/resolv.conf"""),
],
"outro": r"""You resolved a Service and a StatefulSet Pod by their full names, taught CoreDNS a name of your own, and set a Pod's resolver by hand.

431
cka/lessons_storage.py
View File

@ -1,5 +1,5 @@
"""CKA lessons of the Storage domain (1.1-1.3). See cka/lessons.py for what a lesson is."""
from sim.bank import NS, step, cmd, hand
from sim.bank import NS, step
# ---- 1.1 StorageClasses and dynamic provisioning ---------------------------------------------
L11 = {
@ -29,10 +29,9 @@ mkdir -p "$WS/lesson-1.1"
want="$(j sc standard '{.metadata.name} {.provisioner} {.volumeBindingMode}')"
[ -n "$want" ] && eq "$(tr -s ' \t' ' ' < "$WS/lesson-1.1/default.txt" 2>/dev/null | tr -d '\r\n' | sed 's/ *$//')" "$want" """,
hint=r"""`kubectl get sc` shows all three columns. For exactly the values: `-o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}'`.""",
show=[cmd(r"""k get sc"""),
hand(r"""echo 'NAME PROVISIONER MODE' > {WS}/lesson-1.1/default.txt""",
r"""From the `(default)` line: its name, `PROVISIONER` and `VOLUMEBINDINGMODE`.""",
run=r"""kubectl get sc standard -o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}' > {WS}/lesson-1.1/default.txt""")]),
show=r"""kubectl get sc
kubectl get sc standard -o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}' > {WS}/lesson-1.1/default.txt
cat {WS}/lesson-1.1/default.txt"""),
step("A class that keeps its volumes",
r"""Create StorageClass `keep`: the same provisioner as `standard` (`rancher.io/local-path`), binding mode `WaitForFirstConsumer`, and reclaim policy **`Retain`**, so a volume outlives its claim.
@ -41,9 +40,7 @@ There is no `kubectl create storageclass`: write the manifest. `kubectl get sc s
eq "$(j sc keep '{.provisioner} {.reclaimPolicy} {.volumeBindingMode}')" "rancher.io/local-path Retain WaitForFirstConsumer" \
&& [ "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""",
hint=r"""A StorageClass has no `spec`: `provisioner`, `reclaimPolicy` and `volumeBindingMode` sit at the top level, beside `metadata`. Its `apiVersion` is `storage.k8s.io/v1`.""",
show=[hand(r"""vim keep.yaml""",
r"""No `kubectl create` for a StorageClass: paste the example from the docs page *Storage Classes*, and make it:
```
show=r"""kubectl apply -f - <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
@ -51,18 +48,8 @@ metadata:
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
```""",
run=r"""cat > keep.yaml <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: keep
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
EOF"""),
cmd(r"""k apply -f keep.yaml
k get sc""")]),
EOF
kubectl get sc"""),
step("A claim that waits",
r"""In Namespace `store`, create PersistentVolumeClaim `ledger`: class `keep`, `ReadWriteOnce`, `100Mi`.
@ -71,9 +58,7 @@ It stays `Pending`, and that is correct. `kubectl -n store describe pvc ledger`
eq "$(j -n store pvc ledger '{.spec.storageClassName}')" keep && eq "$(j -n store pvc ledger '{.spec.resources.requests.storage}')" 100Mi \
&& grep -qx 'WaitForFirstConsumer' <(tr -d ' \r' < "$WS/lesson-1.1/why.txt" 2>/dev/null)""",
hint=r"""The events at the bottom of `describe` have a Reason column. A claim is namespaced; the class is not.""",
show=[hand(r"""vim ledger.yaml""",
r"""No `kubectl create` for a claim: paste the example from the docs page *Persistent Volumes*, and make it:
```
show=r"""kubectl -n store apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@ -84,22 +69,9 @@ spec:
resources:
requests:
storage: 100Mi
```""",
run=r"""cat > ledger.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger
spec:
storageClassName: keep
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 100Mi
EOF"""),
cmd(r"""k -n store apply -f ledger.yaml
k -n store describe pvc ledger | tail -4""", r"""The event's reason: `WaitForFirstConsumer`."""),
cmd(r"""echo WaitForFirstConsumer > {WS}/lesson-1.1/why.txt""")]),
EOF
kubectl -n store describe pvc ledger | tail -4
echo WaitForFirstConsumer > {WS}/lesson-1.1/why.txt"""),
step("Give it a consumer",
r"""Run Pod `bookkeeper` in `store` (image `busybox:1.36`) that mounts claim `ledger` at `/data` and runs `sh -c 'echo first > /data/entry.txt && sleep 86400'`.
@ -109,22 +81,27 @@ eq "$(j -n store pvc ledger '{.status.phase}')" Bound || exit 1
pv=$(j -n store pvc ledger '{.spec.volumeName}'); eq "$(j pv "$pv" '{.spec.persistentVolumeReclaimPolicy}')" Retain || exit 1
eq "$($K -n store exec bookkeeper -- cat /data/entry.txt 2>/dev/null)" first""",
hint=r"""`kubectl run bookkeeper --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'` gives you the Pod; add `volumes:` (with `persistentVolumeClaim.claimName`) at Pod level and `volumeMounts:` in the container.""",
show=[cmd(r"""k -n store run bookkeeper --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo first > /data/entry.txt && sleep 86400' > bk.yaml"""),
hand(r"""vim bk.yaml""",
r"""Add the mount to the container, below `resources: {}`:
```
show=r"""kubectl -n store apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: bookkeeper
spec:
containers:
- name: bookkeeper
image: busybox:1.36
command: [sh, -c, 'echo first > /data/entry.txt && sleep 86400']
volumeMounts:
- {name: data, mountPath: /data}
```
and the volume to the Pod's `spec:`, beside `restartPolicy:`:
```
- name: data
mountPath: /data
volumes:
- {name: data, persistentVolumeClaim: {claimName: ledger}}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}] | .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "ledger"}}]' bk.yaml"""),
cmd(r"""k -n store apply -f bk.yaml
k -n store get pvc ledger
k get pv""", r"""Again once the Pod runs: the claim is `Bound`, and its volume has `RECLAIM POLICY` `Retain`.""")]),
- name: data
persistentVolumeClaim:
claimName: ledger
EOF
kubectl -n store wait --for=condition=Ready pod/bookkeeper --timeout=120s
kubectl -n store get pvc ledger
kubectl get pv"""),
step("Delete the claim, keep the volume",
r"""Delete Pod `bookkeeper` and claim `ledger`. Because of `Retain`, the volume stays, with status `Released`: its data is still on disk, and no new claim can bind it until an admin says so.
@ -134,22 +111,19 @@ Write the PersistentVolume's name to `{WS}/lesson-1.1/released.txt`.""",
pv=$(tr -d ' \r\n' < "$WS/lesson-1.1/released.txt" 2>/dev/null); [ -n "$pv" ] || exit 1
eq "$(j pv "$pv" '{.status.phase} {.spec.claimRef.namespace}/{.spec.claimRef.name}')" "Released store/ledger" """,
hint=r"""Read the volume name off the claim before you delete it (`kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}'`), or find it afterwards in `kubectl get pv`: the CLAIM column still names `store/ledger`.""",
show=[cmd(r"""k -n store get pvc ledger""", r"""`VOLUME` is the PersistentVolume's name."""),
hand(r"""echo PV > {WS}/lesson-1.1/released.txt""",
r"""`PV`: that name.""",
run=r"""kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}' > {WS}/lesson-1.1/released.txt"""),
cmd(r"""k -n store delete po bookkeeper --now
k -n store delete pvc ledger
k get pv""", r"""The volume stays, `Released`.""")]),
show=r"""kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}' > {WS}/lesson-1.1/released.txt
kubectl -n store delete pod bookkeeper --grace-period=1
kubectl -n store delete pvc ledger
kubectl get pv"""),
step("Change the default class",
r"""Make `keep` the cluster's default class, and `standard` no longer the default. Two defaults would be ambiguous: set the annotation to `"true"` on one and `"false"` on the other.""",
check=r"""
eq "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" true \
&& [ "$(j sc standard '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""",
hint=r"""`kubectl annotate sc NAME storageclass.kubernetes.io/is-default-class=true --overwrite`, or `kubectl patch` with a `metadata.annotations` map.""",
show=[cmd(r"""k annotate sc standard storageclass.kubernetes.io/is-default-class=false --overwrite
k annotate sc keep storageclass.kubernetes.io/is-default-class=true --overwrite
k get sc""")]),
show=r"""kubectl annotate sc standard storageclass.kubernetes.io/is-default-class=false --overwrite
kubectl annotate sc keep storageclass.kubernetes.io/is-default-class=true --overwrite
kubectl get sc"""),
],
"outro": r"""You read a class, made one that keeps its volumes, watched a claim wait for its Pod, kept a volume past its claim, and moved the default.
@ -198,29 +172,29 @@ The reader sees what the writer wrote: an emptyDir is shared by every container
check=r"""eq "$($K -n vols exec pair -c reader -- cat /shared/msg 2>/dev/null)" hello \
&& [ -n "$(j -n vols pod pair '{.spec.volumes[?(@.name=="shared")].emptyDir}')" ]""",
hint=r"""Both containers list the same `volumeMounts` entry (`name: shared`); the Pod lists the volume once, as `emptyDir: {}`.""",
show=[cmd(r"""k -n vols run pair --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo hello > /shared/msg && sleep 86400' > pair.yaml"""),
hand(r"""vim pair.yaml""",
r"""Name the container `writer` and give it the mount; add `reader` below it; add the volume:
```
show=r"""kubectl -n vols apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: pair
spec:
containers:
- name: writer
image: busybox:1.36
args: [sh, -c, 'echo hello > /shared/msg && sleep 86400']
command: [sh, -c, 'echo hello > /shared/msg && sleep 86400']
volumeMounts:
- {name: shared, mountPath: /shared}
- name: reader
image: busybox:1.36
args: [sleep, '86400']
command: [sleep, '86400']
volumeMounts:
- {name: shared, mountPath: /shared}
volumes:
- {name: shared, emptyDir: {}}
```""",
run=r"""yq -i '.spec.containers[0].name = "writer" | .spec.containers[0].volumeMounts = [{"name": "shared", "mountPath": "/shared"}]
| .spec.containers += [{"name": "reader", "image": "busybox:1.36", "args": ["sleep", "86400"], "volumeMounts": [{"name": "shared", "mountPath": "/shared"}]}]
| .spec.volumes = [{"name": "shared", "emptyDir": {}}]' pair.yaml"""),
cmd(r"""k -n vols apply -f pair.yaml
k -n vols exec pair -c reader -- cat /shared/msg""", r"""Once it runs: `hello`.""")]),
- name: shared
emptyDir: {}
EOF
kubectl -n vols wait --for=condition=Ready pod/pair --timeout=120s
kubectl -n vols exec pair -c reader -- cat /shared/msg"""),
step("Scratch space in memory",
r"""Create Pod `scratch` in `vols` (`busybox:1.36`, `sleep 86400`) with an emptyDir at `/scratch` that is held in **memory**, at most `64Mi`.
@ -229,21 +203,24 @@ Inside the Pod, `/proc/mounts` shows the difference: `/scratch` is a `tmpfs`. Wh
eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.medium}')" Memory && eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.sizeLimit}')" 64Mi \
&& $K -n vols exec scratch -- grep -q ' /scratch tmpfs ' /proc/mounts 2>/dev/null""",
hint=r"""`emptyDir: {medium: Memory, sizeLimit: 64Mi}`.""",
show=[cmd(r"""k -n vols run scratch --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > scratch.yaml"""),
hand(r"""vim scratch.yaml""",
r"""Add the mount to the container, below `resources: {}`:
```
show=r"""kubectl -n vols apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: scratch
spec:
containers:
- name: scratch
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- {name: tmp, mountPath: /scratch}
```
and the volume to the Pod's `spec:`, beside `restartPolicy:`:
```
volumes:
- {name: tmp, emptyDir: {medium: Memory, sizeLimit: 64Mi}}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "tmp", "mountPath": "/scratch"}] | .spec.volumes = [{"name": "tmp", "emptyDir": {"medium": "Memory", "sizeLimit": "64Mi"}}]' scratch.yaml"""),
cmd(r"""k -n vols apply -f scratch.yaml
k -n vols exec scratch -- grep /scratch /proc/mounts""", r"""Once it runs: a `tmpfs`.""")]),
- name: tmp
emptyDir: {medium: Memory, sizeLimit: 64Mi}
EOF
kubectl -n vols wait --for=condition=Ready pod/scratch --timeout=120s
kubectl -n vols exec scratch -- grep /scratch /proc/mounts"""),
step("One directory from three sources",
r"""Create Pod `bundle` in `vols` (`busybox:1.36`, `sleep 86400`) with **one** `projected` volume mounted at `/etc/bundle`, made of:
@ -256,15 +233,18 @@ Separate volumes would need three mount points; a projected volume puts them sid
x() { $K -n vols exec bundle -- cat "/etc/bundle/$1" 2>/dev/null; }
[ -n "$(j -n vols pod bundle '{.spec.volumes[0].projected}')" ] && eq "$(x mode)" prod && eq "$(x password)" s3cr3t && eq "$(x podname)" bundle""",
hint=r"""`projected: {sources: [...]}` - each source is `configMap: {name: ...}`, `secret: {name: ...}` or `downwardAPI: {items: [{path: podname, fieldRef: {fieldPath: metadata.name}}]}`. `kubectl explain pod.spec.volumes.projected.sources` lists them.""",
show=[cmd(r"""k -n vols run bundle --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > bundle.yaml"""),
hand(r"""vim bundle.yaml""",
r"""Add the mount to the container, below `resources: {}`:
```
show=r"""kubectl -n vols apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: bundle
spec:
containers:
- name: app
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- {name: bundle, mountPath: /etc/bundle, readOnly: true}
```
and the volume to the Pod's `spec:`, beside `restartPolicy:` (the docs page *Projected Volumes* has an example):
```
volumes:
- name: bundle
projected:
@ -275,11 +255,9 @@ and the volume to the Pod's `spec:`, beside `restartPolicy:` (the docs page *Pro
items:
- path: podname
fieldRef: {fieldPath: metadata.name}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "bundle", "mountPath": "/etc/bundle", "readOnly": true}]
| .spec.volumes = [{"name": "bundle", "projected": {"sources": [{"configMap": {"name": "app-config"}}, {"secret": {"name": "app-secret"}}, {"downwardAPI": {"items": [{"path": "podname", "fieldRef": {"fieldPath": "metadata.name"}}]}}]}}]' bundle.yaml"""),
cmd(r"""k -n vols apply -f bundle.yaml
k -n vols exec bundle -- ls /etc/bundle""", r"""Once it runs: `mode`, `password` and `podname`.""")]),
EOF
kubectl -n vols wait --for=condition=Ready pod/bundle --timeout=120s
kubectl -n vols exec bundle -- ls /etc/bundle"""),
step("A volume for exactly one Pod",
r"""Create PersistentVolume `archive`: `hostPath` `/srv/archive` (type `DirectoryOrCreate`), `200Mi`, access mode **`ReadWriteOncePod`**, reclaim policy `Retain`, and `storageClassName: ""`.
@ -290,9 +268,7 @@ The empty class matters on both sides. A claim with no class at all gets the def
eq "$(j pv archive '{.spec.accessModes[0]} {.spec.persistentVolumeReclaimPolicy} {.spec.hostPath.path}')" "ReadWriteOncePod Retain /srv/archive" \
&& eq "$(j -n vols pvc archive-claim '{.status.phase} {.spec.volumeName}')" "Bound archive" """,
hint=r"""`storageClassName: ""` is spelled out, with the quotes, under `spec` of both objects. A PersistentVolume is cluster-wide: no namespace.""",
show=[hand(r"""vim archive.yaml""",
r"""No `kubectl create` for these: paste the PersistentVolume and the claim from the docs page *Configure a Pod to Use a PersistentVolume for Storage*, and make them:
```
show=r"""kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
@ -314,33 +290,9 @@ spec:
accessModes: [ReadWriteOncePod]
resources:
requests: {storage: 200Mi}
```""",
run=r"""cat > archive.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: archive
spec:
capacity: {storage: 200Mi}
accessModes: [ReadWriteOncePod]
persistentVolumeReclaimPolicy: Retain
storageClassName: ""
hostPath: {path: /srv/archive, type: DirectoryOrCreate}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archive-claim
namespace: vols
spec:
storageClassName: ""
accessModes: [ReadWriteOncePod]
resources:
requests: {storage: 200Mi}
EOF"""),
cmd(r"""k apply -f archive.yaml
k get pv archive
k -n vols get pvc archive-claim""", r"""Both `Bound`, to each other.""")]),
EOF
kubectl get pv archive
kubectl -n vols get pvc archive-claim"""),
step("Hand a Released volume to a new claim",
r"""Delete claim `archive-claim`. The volume becomes `Released`: it still remembers its old claim in `spec.claimRef`, and nothing binds it while it does.
@ -348,14 +300,10 @@ Make it `Available` again by removing `spec.claimRef` from the PV, then create c
check=r"""
[ -z "$(j -n vols pvc archive-claim '{.metadata.name}')" ] && eq "$(j -n vols pvc archive-claim-2 '{.status.phase} {.spec.volumeName}')" "Bound archive" """,
hint=r"""`kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'` - or `kubectl edit pv archive` and delete the `claimRef:` block.""",
show=[cmd(r"""k -n vols delete pvc archive-claim
k get pv archive""", r"""`Released`."""),
hand(r"""k edit pv archive""",
r"""Delete the whole `claimRef:` block, then `:wq`. The volume is `Available` again.""",
run=r"""kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' """),
hand(r"""vim claim2.yaml""",
r"""The claim of `archive.yaml`, named `archive-claim-2`:
```
show=r"""kubectl -n vols delete pvc archive-claim
kubectl get pv archive
kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@ -366,21 +314,8 @@ spec:
accessModes: [ReadWriteOncePod]
resources:
requests: {storage: 200Mi}
```""",
run=r"""cat > claim2.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archive-claim-2
namespace: vols
spec:
storageClassName: ""
accessModes: [ReadWriteOncePod]
resources:
requests: {storage: 200Mi}
EOF"""),
cmd(r"""k apply -f claim2.yaml
k get pv archive""")]),
EOF
kubectl get pv archive"""),
],
"outro": r"""You shared an emptyDir between containers, put one in memory, combined three sources in one projected volume, and took a `Retain` volume through `Bound`, `Released` and back.
@ -466,9 +401,7 @@ $K -n claims wait --for=condition=Ready pod/grower --timeout=180s >/dev/null
Without the selector it would bind `pv-small`, the smallest volume that fits. With it, it binds `pv-large`.""",
check=r"""eq "$(j -n claims pvc gold-claim '{.status.phase} {.spec.volumeName}')" "Bound pv-large" """,
hint=r"""`spec.selector.matchLabels: {tier: gold}` on the claim.""",
show=[hand(r"""vim gold.yaml""",
r"""A claim with a `selector` (the docs page *Persistent Volumes* has one):
```
show=r"""kubectl -n claims apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@ -480,30 +413,14 @@ spec:
requests: {storage: 100Mi}
selector:
matchLabels: {tier: gold}
```""",
run=r"""cat > gold.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gold-claim
spec:
storageClassName: manual
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 100Mi}
selector:
matchLabels: {tier: gold}
EOF"""),
cmd(r"""k -n claims apply -f gold.yaml
k get pv""", r"""`gold-claim` bound to `pv-large`.""")]),
EOF
kubectl get pv"""),
step("Bind by name, and use it",
r"""Create claim `pinned` in `claims` that binds `pv-small` **by name** (class `manual`, `ReadWriteOnce`, `100Mi`), and Pod `user` (`busybox:1.36`, `sleep 86400`) that mounts it at `/data`.""",
check=r"""eq "$(j -n claims pvc pinned '{.status.phase} {.spec.volumeName}')" "Bound pv-small" && eq "$(j -n claims pod user '{.status.phase}')" Running \
&& eq "$(j -n claims pod user '{.spec.volumes[0].persistentVolumeClaim.claimName}')" pinned""",
hint=r"""`spec.volumeName: pv-small` on the claim. The volume's class and modes must still match the claim's.""",
show=[hand(r"""vim pinned.yaml""",
r"""A claim that names its volume:
```
show=r"""kubectl -n claims apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@ -514,35 +431,24 @@ spec:
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 100Mi}
```""",
run=r"""cat > pinned.yaml <<'EOF'
---
apiVersion: v1
kind: PersistentVolumeClaim
kind: Pod
metadata:
name: pinned
name: user
spec:
storageClassName: manual
volumeName: pv-small
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 100Mi}
EOF"""),
cmd(r"""k -n claims apply -f pinned.yaml
k -n claims run user --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > user.yaml"""),
hand(r"""vim user.yaml""",
r"""Add the mount to the container, below `resources: {}`:
```
containers:
- name: user
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- {name: data, mountPath: /data}
```
and the volume to the Pod's `spec:`, beside `restartPolicy:`:
```
volumes:
- {name: data, persistentVolumeClaim: {claimName: pinned}}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}] | .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "pinned"}}]' user.yaml"""),
cmd(r"""k -n claims apply -f user.yaml
k -n claims get pvc,po""")]),
- name: data
persistentVolumeClaim: {claimName: pinned}
EOF
kubectl -n claims wait --for=condition=Ready pod/user --timeout=120s
kubectl -n claims get pvc"""),
step("A claim in use does not go",
r"""Delete claim `pinned` without waiting (`--wait=false`). It does not disappear: `kubectl -n claims get pvc` shows it `Terminating`, held by a **finalizer**.
@ -551,19 +457,17 @@ Write that finalizer's name to `{WS}/lesson-1.3/finalizer.txt`.""",
[ -n "$(j -n claims pvc pinned '{.metadata.deletionTimestamp}')" ] \
&& eq "$(tr -d ' \r\n' < "$WS/lesson-1.3/finalizer.txt" 2>/dev/null)" kubernetes.io/pvc-protection""",
hint=r"""`kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers}'`. Without `--wait=false`, `kubectl delete` would sit there until the claim is really gone.""",
show=[cmd(r"""k -n claims delete pvc pinned --wait=false
k -n claims get pvc pinned
k -n claims get pvc pinned -o yaml | grep -A1 finalizers"""),
hand(r"""echo FINALIZER > {WS}/lesson-1.3/finalizer.txt""",
r"""`FINALIZER`: the name under `finalizers:`.""",
run=r"""kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers[0]}' > {WS}/lesson-1.3/finalizer.txt""")]),
show=r"""kubectl -n claims delete pvc pinned --wait=false
kubectl -n claims get pvc pinned
kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers[0]}' > {WS}/lesson-1.3/finalizer.txt
cat {WS}/lesson-1.3/finalizer.txt"""),
step("Let it go",
r"""Delete Pod `user`. With no Pod using it any more, the protection lifts and claim `pinned` is removed. `pv-small` (reclaim policy `Retain`) is now `Released`.""",
check=r"""[ -z "$(j -n claims pod user '{.metadata.name}')" ] && [ -z "$(j -n claims pvc pinned '{.metadata.name}')" ] && eq "$(j pv pv-small '{.status.phase}')" Released""",
hint=r"""`kubectl -n claims delete pod user`, then `kubectl -n claims get pvc` and `kubectl get pv`.""",
show=[cmd(r"""k -n claims delete po user --now
k -n claims get pvc
k get pv pv-small""", r"""The claim is gone; `pv-small` is `Released`.""")]),
show=r"""kubectl -n claims delete pod user --grace-period=1
kubectl -n claims get pvc
kubectl get pv pv-small"""),
step("Claims made by a StatefulSet",
r"""Create a StatefulSet `db` in `claims` with 2 replicas (`busybox:1.36`, `sleep 86400`) and its headless Service `db`. Give it a `volumeClaimTemplates` entry `data` (`ReadWriteOnce`, `50Mi`, class `standard`) mounted at `/var/lib/db`, and a `persistentVolumeClaimRetentionPolicy` that **deletes** a replica's claim when the StatefulSet is scaled down and **retains** the claims when the StatefulSet is deleted.
@ -572,9 +476,7 @@ Each replica gets a claim of its own: `data-db-0`, `data-db-1`.""",
eq "$(j -n claims sts db '{.spec.persistentVolumeClaimRetentionPolicy.whenScaled} {.spec.persistentVolumeClaimRetentionPolicy.whenDeleted}')" "Delete Retain" || exit 1
eq "$(j -n claims sts db '{.status.readyReplicas}')" 2 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound && eq "$(j -n claims pvc data-db-1 '{.status.phase}')" Bound""",
hint=r"""`kubectl create deploy db --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400` is a start: change the kind, add `serviceName: db`, drop `strategy`, add `volumeClaimTemplates` beside `template`. The Service: `clusterIP: None`. `kubectl explain sts.spec.persistentVolumeClaimRetentionPolicy`.""",
show=[hand(r"""vim db.yaml""",
r"""No `kubectl create` for a StatefulSet: paste the example from the docs page *StatefulSets* (it comes with its headless Service), and make it:
```
show=r"""kubectl -n claims apply -f - <<'EOF'
apiVersion: v1
kind: Service
metadata:
@ -615,52 +517,9 @@ spec:
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
```""",
run=r"""cat > db.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: db
spec:
clusterIP: None
selector: {app: db}
ports:
- port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
serviceName: db
replicas: 2
selector:
matchLabels: {app: db}
persistentVolumeClaimRetentionPolicy:
whenScaled: Delete
whenDeleted: Retain
template:
metadata:
labels: {app: db}
spec:
containers:
- name: db
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- {name: data, mountPath: /var/lib/db}
volumeClaimTemplates:
- metadata:
name: data
spec:
storageClassName: standard
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
EOF"""),
cmd(r"""k -n claims apply -f db.yaml
k -n claims rollout status sts db
k -n claims get pvc""", r"""`data-db-0` and `data-db-1`.""")]),
EOF
kubectl -n claims rollout status sts db --timeout=180s
kubectl -n claims get pvc"""),
step("Scale down, and the claim goes with the replica",
r"""Scale `db` to 1 replica. Pod `db-1` goes, and because `whenScaled` is `Delete`, so does `data-db-1`. `data-db-0` stays.
@ -669,26 +528,23 @@ The default for both fields is `Retain`: without the policy, `data-db-1` would w
eq "$(j -n claims sts db '{.spec.replicas}')" 1 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound || exit 1
for i in $(seq 1 10); do [ -z "$(j -n claims pvc data-db-1 '{.metadata.name}')" ] && exit 0; sleep 3; done; exit 1""",
hint=r"""`kubectl -n claims scale sts db --replicas=1`, then `kubectl -n claims get pvc -w`.""",
show=[cmd(r"""k -n claims scale sts db --replicas=1
k -n claims rollout status sts db
k -n claims get pvc""", r"""Again in a moment: `data-db-1` is gone, `data-db-0` stays.""")]),
show=r"""kubectl -n claims scale sts db --replicas=1
kubectl -n claims rollout status sts db --timeout=120s
kubectl -n claims get pvc"""),
step("Grow a claim in use",
r"""Claim `growing` is full. Its class allows expansion (`kubectl get sc csi-hostpath-sc`: `ALLOWVOLUMEEXPANSION true`), so make it **50Mi** - while Pod `grower` keeps running. Raise the request; the driver does the rest, and `status.capacity` follows.""",
check=r"""eq "$(j -n claims pvc growing '{.spec.resources.requests.storage} {.status.capacity.storage}')" "50Mi 50Mi" && eq "$(j -n claims pod grower '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """,
hint=r"""`kubectl -n claims edit pvc growing`, or `kubectl patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}'`. A claim never shrinks; with `standard` (local-path) the same patch is refused.""",
show=[cmd(r"""k get sc csi-hostpath-sc"""),
hand(r"""k -n claims edit pvc growing""",
r"""Make `storage:` under `spec.resources.requests` read `50Mi`, then `:wq`.""",
run=r"""kubectl -n claims patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}' """),
cmd(r"""k -n claims get pvc growing""", r"""Again until `CAPACITY` shows `50Mi`.""")]),
show=r"""kubectl get sc csi-hostpath-sc
kubectl -n claims patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}'
sleep 10
kubectl -n claims get pvc growing"""),
step("Snapshot it, and restore the snapshot",
r"""Take VolumeSnapshot `growing-snap` of claim `growing` (snapshot class `csi-hostpath-snapclass`), wait until it is ready to use, then create claim `restored` (`csi-hostpath-sc`, 50Mi) **from** the snapshot: its `dataSource` names the VolumeSnapshot. The new volume holds what `growing` held when the snapshot was taken.""",
check=r"""
eq "$(j -n claims volumesnapshot growing-snap '{.spec.source.persistentVolumeClaimName} {.status.readyToUse}')" "growing true" && eq "$(j -n claims pvc restored '{.spec.dataSource.kind}/{.spec.dataSource.name} {.status.phase}')" "VolumeSnapshot/growing-snap Bound" """,
hint=r"""`apiVersion: snapshot.storage.k8s.io/v1`, `kind: VolumeSnapshot`, `spec: {volumeSnapshotClassName: ..., source: {persistentVolumeClaimName: growing}}`. The claim: `dataSource: {apiGroup: snapshot.storage.k8s.io, kind: VolumeSnapshot, name: growing-snap}`. A clone is the same with `dataSource: {kind: PersistentVolumeClaim, name: growing}`.""",
show=[hand(r"""vim snap.yaml""",
r"""Paste a VolumeSnapshot from the docs page *Volume Snapshots*, and make it:
```
show=r"""kubectl -n claims apply -f - <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
@ -697,24 +553,9 @@ spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: growing
```""",
run=r"""cat > snap.yaml <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: growing-snap
spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: growing
EOF"""),
cmd(r"""k -n claims apply -f snap.yaml"""),
hand(r"""k -n claims get volumesnapshot""",
r"""Again until `READYTOUSE` is `true`.""",
run=r"""kubectl -n claims wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/growing-snap --timeout=120s"""),
hand(r"""vim restored.yaml""",
r"""A claim whose `dataSource` is the snapshot (the same docs page, *Provisioning Volumes from Snapshots*):
```
EOF
kubectl -n claims wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/growing-snap --timeout=120s
kubectl -n claims apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@ -728,24 +569,8 @@ spec:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: growing-snap
```""",
run=r"""cat > restored.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: restored
spec:
storageClassName: csi-hostpath-sc
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: growing-snap
EOF"""),
cmd(r"""k -n claims apply -f restored.yaml
k -n claims get volumesnapshot,pvc""")]),
EOF
kubectl -n claims get volumesnapshot,pvc"""),
],
"outro": r"""You steered binding with a selector and with a volume's name, watched claim protection hold a claim until its Pod went, let a StatefulSet's claims follow its replicas, and grew, snapshotted and restored a claim on a CSI class.

240
cka/lessons_tshoot.py
View File

@ -1,5 +1,5 @@
"""CKA lessons of the Troubleshooting domain (2.1-2.5). See cka/lessons.py for what a lesson is."""
from sim.bank import NS, step, cmd, hand
from sim.bank import NS, step
# ---- 2.1 clusters and nodes -------------------------------------------------------------------
L21 = {
@ -35,11 +35,9 @@ echo "$W1 is still Ready a minute after its kubelet was stopped" >&2; exit 1
Then look at `kubectl describe node` for it: under **Conditions** every row says `Kubelet stopped posting node status` - the control plane has not heard from the kubelet, so it knows nothing more. The answer is on the node.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/node.txt" 2>/dev/null)" "$W1" """,
hint=r"""`kubectl get nodes`. `--no-headers` and `awk '$2 != "Ready" {print $1}'` give just the name.""",
show=[cmd(r"""k get no
k describe no {W1} | grep -A8 Conditions"""),
hand(r"""echo NODE > {WS}/lesson-2.1/node.txt""",
r"""`NODE`: the one that is not `Ready`.""",
run=r"""kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt""")]),
show=r"""kubectl get nodes
kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt
kubectl describe node {W1} | grep -A8 Conditions"""),
step("Bring the kubelet back - for good",
r"""On `{W1}` (`ssh {W1}`, then `sudo -i`), look at the kubelet: `systemctl status kubelet` says `inactive (dead)` and `disabled`. Start it, and enable it so that it survives a reboot - a fix that is gone after the next reboot is not a fix.
@ -48,15 +46,12 @@ Within a few seconds the node is `Ready` again.""",
nx "$W1" 'systemctl is-active -q kubelet && systemctl is-enabled -q kubelet' </dev/null || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""`systemctl enable --now kubelet` does both at once. `journalctl -u kubelet -n 20` tells you when it is running happily.""",
show=[hand(r"""ssh {W1}
sudo -i
systemctl status kubelet
systemctl enable --now kubelet""",
r"""Then `exit` twice.""",
run=r"""ssh {W1} sudo bash <<'EOF'
show=r"""ssh {W1} sudo bash <<'EOF'
systemctl status kubelet --no-pager | head -4
systemctl enable --now kubelet
EOF"""),
cmd(r"""k get no""", r"""Again in a few seconds: `Ready`.""")]),
systemctl is-active kubelet; systemctl is-enabled kubelet
EOF
kubectl get nodes"""),
step("Where the kubelet's configuration is",
r"""A kubelet reads its settings from the file named by its `--config` flag, and the kubeadm flags from `/var/lib/kubelet/kubeadm-flags.env`. On `{W1}`, find the file (`ps -o args= -C kubelet`, or the drop-in under `/etc/systemd/system/kubelet.service.d/`).
@ -65,16 +60,9 @@ The kubelet also tells the API server what it actually runs with: `kubectl get -
want=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["cgroupDriver"])')
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/cgroup.txt" 2>/dev/null)" "$want" """,
hint=r"""`configz` returns JSON: `| jq -r .kubeletconfig.cgroupDriver`. It must match containerd's `SystemdCgroup` setting, or Pods fail to start.""",
show=[hand(r"""ssh {W1}
sudo -i
ps -o args= -C kubelet
grep cgroupDriver /var/lib/kubelet/config.yaml""",
r"""`--config=` names the kubelet's file. Then `exit` twice.""",
run=r"""ssh -n {W1} 'ps -o args= -C kubelet; sudo grep cgroupDriver /var/lib/kubelet/config.yaml' """),
cmd(r"""k get --raw /api/v1/nodes/{W1}/proxy/configz | tr , '\n' | grep cgroupDriver""", r"""The live value."""),
hand(r"""echo VALUE > {WS}/lesson-2.1/cgroup.txt""",
r"""`VALUE`: `systemd` or `cgroupfs`.""",
run=r"""kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt""")]),
show=r"""ssh -n {W1} 'ps -o args= -C kubelet | tr " " "\n" | grep -- --config; sudo grep -n cgroupDriver /var/lib/kubelet/config.yaml'
kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt
cat {WS}/lesson-2.1/cgroup.txt"""),
step("Set the eviction thresholds",
r"""Look at `evictionHard` in `/var/lib/kubelet/config.yaml` on `{W1}`: kind sets every threshold to `0%`, which means "never evict" - fine for a test cluster, not for a real one.
@ -86,26 +74,15 @@ e = json.load(sys.stdin)["kubeletconfig"].get("evictionHard") or {}
sys.exit(0 if e.get("memory.available") == "100Mi" and e.get("nodefs.available") == "10%" else 1)' || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""Edit the file with `vim`, or `sed`: change the `nodefs.available` line and add a `memory.available` line inside the `evictionHard:` block (two spaces of indent). A kubelet that does not come back after the restart: `journalctl -u kubelet -n 20` shows the line it could not parse.""",
show=[hand(r"""ssh {W1}
sudo -i
vim /var/lib/kubelet/config.yaml""",
r"""Under `evictionHard:`, set `nodefs.available` to `10%` and add `memory.available`, indented like the others:
```
evictionHard:
memory.available: 100Mi
nodefs.available: 10%
```
then restart the kubelet, and `exit` twice:
```
systemctl restart kubelet
```""",
run=r"""ssh {W1} sudo bash <<'EOF'
show=r"""ssh {W1} sudo bash <<'EOF'
f=/var/lib/kubelet/config.yaml
grep -n -A4 '^evictionHard:' $f
sed -i 's/^ nodefs.available: .*/ nodefs.available: 10%/' $f
sed -i '/^evictionHard:/a\ memory.available: 100Mi' $f
grep -n -A5 '^evictionHard:' $f
systemctl restart kubelet
EOF"""),
cmd(r"""k get --raw /api/v1/nodes/{W1}/proxy/configz | tr , '\n' | grep -A4 evictionHard""", r"""Again if the kubelet is not back yet.""")]),
EOF
kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq .kubeletconfig.evictionHard"""),
step("Read the pressure conditions",
r"""The kubelet reports the result of those thresholds as node conditions. Write the status of `{W1}`'s three pressure conditions to `{WS}/lesson-2.1/pressure.txt`, one per line, as `TYPE STATUS` - for example `MemoryPressure False`.
@ -114,15 +91,8 @@ EOF"""),
want=$(for c in MemoryPressure DiskPressure PIDPressure; do echo "$c $(j node "$W1" "{.status.conditions[?(@.type==\"$c\")].status}")"; done | sort)
eq "$(tr -s ' \t' ' ' < "$WS/lesson-2.1/pressure.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """,
hint=r"""`kubectl describe node {W1}` lists them under Conditions. As a list: `-o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}'`, then keep the three `*Pressure` lines.""",
show=[cmd(r"""k describe no {W1} | grep Pressure"""),
hand(r"""vim {WS}/lesson-2.1/pressure.txt""",
r"""Three lines, `TYPE STATUS`, with the statuses you read:
```
MemoryPressure False
DiskPressure False
PIDPressure False
```""",
run=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt""")]),
show=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt
cat {WS}/lesson-2.1/pressure.txt"""),
],
"outro": r"""You found a `NotReady` node, brought its kubelet back for good, read its configuration from the file and from the running kubelet, set eviction thresholds and read what the kubelet reports about them.
@ -174,12 +144,10 @@ Two things in Namespace `ops` do not work: Deployment `queue` has no Pods, and P
`kubectl -n kube-system get pods` shows them; on the control-plane node, `sudo crictl ps -a` shows their containers, and `ls /etc/kubernetes/manifests` what the kubelet is told to run.""",
check=r"""eq "$(tr -d ' \r' < "$WS/lesson-2.2/down.txt" 2>/dev/null | grep . | sort | tr '\n' ' ')" "kube-controller-manager kube-scheduler " """,
hint=r"""A Pod that is not `Running` with all its containers ready. The Pod names end in the node's name; the component is the part before it.""",
show=[cmd(r"""k -n kube-system get po
show=r"""kubectl -n kube-system get pods
ls /etc/kubernetes/manifests
sudo crictl ps -a | grep kube-"""),
hand(r"""vim {WS}/lesson-2.2/down.txt""",
r"""The components that do not run, one per line.""",
run=r"""printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt""")]),
sudo crictl ps -a | grep kube-
printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt"""),
step("Repair the scheduler",
r"""Find out why `kube-scheduler` does not start, and fix its manifest. `kubectl -n kube-system describe pod kube-scheduler-{CP}` shows the error in its events, and so do `sudo crictl ps -a --name kube-scheduler` and `sudo journalctl -u kubelet`.
@ -188,33 +156,30 @@ The kubelet notices the saved file on its own: no restart, no `kubectl apply`. O
nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml && [ -n "$(crictl ps --name kube-scheduler -q)" ]' </dev/null || exit 1
[ -n "$(j -n ops pod waiting '{.spec.nodeName}')" ]""",
hint=r"""Look at the first entry of the container's `command:` - the program the container runs. A static Pod's manifest is edited in place (`sudo vim`), never with `kubectl edit`.""",
show=[cmd(r"""k -n kube-system describe po kube-scheduler-{CP} | tail -5""", r"""The command it cannot find: a typo."""),
hand(r"""sudo vim /etc/kubernetes/manifests/kube-scheduler.yaml""",
r"""Under `command:`, make `- kube-schedulerr` read `- kube-scheduler`, then `:wq`.""",
run=r"""sudo sed -i 's/- kube-schedulerr$/- kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml"""),
cmd(r"""sudo crictl ps --name kube-scheduler
k -n ops get po waiting -o wide""", r"""Again in a few seconds: the scheduler runs, and `waiting` has a node.""")]),
show=r"""kubectl -n kube-system describe pod kube-scheduler-{CP} | tail -5
sudo grep -n -A2 'command:' /etc/kubernetes/manifests/kube-scheduler.yaml
sudo sed -i 's/- kube-schedulerr$/- kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
sudo crictl ps --name kube-scheduler
kubectl -n ops get pod waiting -o wide"""),
step("Repair the controller manager",
r"""Now `kube-controller-manager`. Its Pod is not crashing - its container never starts. `describe` on it tells you which step fails. Fix the manifest; once it runs, Deployment `queue` gets its ReplicaSet and both Pods.""",
check=r"""
nx "$CP" '! grep -q -- "-broken" /etc/kubernetes/manifests/kube-controller-manager.yaml && [ -n "$(crictl ps --name kube-controller-manager -q)" ]' </dev/null || exit 1
eq "$(j -n ops deploy queue '{.status.availableReplicas}')" 2""",
hint=r"""`ErrImagePull` / `ImagePullBackOff`: the image tag does not exist. Every control-plane image runs the cluster's version - compare with `kube-apiserver.yaml`.""",
show=[cmd(r"""k -n kube-system describe po kube-controller-manager-{CP} | tail -5
sudo grep image: /etc/kubernetes/manifests/*.yaml""", r"""Its image tag ends in `-broken`; the other components show the right tag."""),
hand(r"""sudo vim /etc/kubernetes/manifests/kube-controller-manager.yaml""",
r"""Remove `-broken` from the end of the `image:` line, then `:wq`.""",
run=r"""sudo sed -i 's/-broken$//' /etc/kubernetes/manifests/kube-controller-manager.yaml"""),
cmd(r"""sudo crictl ps --name kube-controller-manager
k -n ops get deploy queue""", r"""Again until `queue` shows `2/2`.""")]),
show=r"""kubectl -n kube-system describe pod kube-controller-manager-{CP} | tail -5
sudo grep -n 'image:' /etc/kubernetes/manifests/*.yaml
sudo sed -i 's/-broken$//' /etc/kubernetes/manifests/kube-controller-manager.yaml
sudo crictl ps --name kube-controller-manager
kubectl -n ops get deploy queue"""),
step("Ask the API server how it feels",
r"""The API server has health endpoints of its own: `/livez` (restart me if this fails) and `/readyz` (send me traffic). With `?verbose` they list every check.
Save the output of `kubectl get --raw '/readyz?verbose'` in `{WS}/lesson-2.2/readyz.txt`.""",
check=r"""grep -q '^\[+\]etcd ok' "$WS/lesson-2.2/readyz.txt" 2>/dev/null && grep -q 'readyz check passed' "$WS/lesson-2.2/readyz.txt" """,
hint=r"""`kubectl get --raw` sends a GET to any API path, with your credentials. Quote the path: `?` means something to the shell.""",
show=[cmd(r"""k get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt
k get --raw /livez""")]),
show=r"""kubectl get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt | tail -3
kubectl get --raw '/livez'; echo"""),
],
"outro": r"""You read the control plane from both sides - `kubectl` and the node's container runtime - and repaired two static Pods by editing their manifests in place.
@ -296,46 +261,38 @@ echo "kubectl top shows no metrics for the lesson's Pods" >&2; exit 1
r"""Write the name of the Pod in `metrics` that uses the most memory to `{WS}/lesson-2.3/memory.txt`. `kubectl top` can sort for you.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/memory.txt" 2>/dev/null)" hoarder""",
hint=r"""`kubectl top pod -n metrics --sort-by=memory`. `--no-headers | head -1 | awk '{print $1}'` keeps just the name.""",
show=[cmd(r"""k top po -n metrics --sort-by=memory"""),
hand(r"""echo POD > {WS}/lesson-2.3/memory.txt""",
r"""`POD`: the first one.""",
run=r"""kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt""")]),
show=r"""kubectl top pod -n metrics --sort-by=memory
kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt"""),
step("The busiest Pod in the cluster",
r"""Across **all** Namespaces, which Pod uses the most CPU? Write it as `NAMESPACE/NAME` to `{WS}/lesson-2.3/cpu.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/cpu.txt" 2>/dev/null)" metrics/burner""",
hint=r"""`kubectl top pod -A --sort-by=cpu`; with `-A` the first column is the Namespace.""",
show=[cmd(r"""k top po -A --sort-by=cpu | head -5"""),
hand(r"""echo NAMESPACE/POD > {WS}/lesson-2.3/cpu.txt""",
r"""From the first line.""",
run=r"""kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt""")]),
show=r"""kubectl top pod -A --sort-by=cpu | head -5
kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt"""),
step("Inside one Pod",
r"""`kubectl top pod` adds up a Pod's containers. Which container of Pod `duo` uses the memory? Write its name to `{WS}/lesson-2.3/container.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/container.txt" 2>/dev/null)" cache""",
hint=r"""`--containers` splits the Pod into its containers.""",
show=[cmd(r"""k top po duo -n metrics --containers"""),
hand(r"""echo CONTAINER > {WS}/lesson-2.3/container.txt""",
r"""The container with the most `MEMORY(bytes)`.""",
run=r"""kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt""")]),
show=r"""kubectl top pod duo -n metrics --containers
kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt"""),
step("Usage against what is reserved",
r"""`kubectl top node` shows use; `kubectl describe node {W1}`, under **Allocated resources**, shows what Pods have **requested** - which is what decides whether another Pod fits.
Write the memory that `{W1}` can give to Pods - its **allocatable** memory, as the API reports it (for example `8012345Ki`) - to `{WS}/lesson-2.3/allocatable.txt`.""",
check=r"""want=$(j node "$W1" '{.status.allocatable.memory}'); [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/allocatable.txt" 2>/dev/null)" "$want" """,
hint=r"""Allocatable is capacity minus what the node keeps for itself and for the eviction threshold: `.status.allocatable.memory`.""",
show=[cmd(r"""k top no
k describe no {W1} | grep -A6 -e Allocatable: -e 'Allocated resources' """, r"""`memory:` under `Allocatable:` is what `{W1}` can give to Pods, for example `8012345Ki`."""),
hand(r"""echo MEMORY > {WS}/lesson-2.3/allocatable.txt""",
r"""`MEMORY`: that value, as it stands.""",
run=r"""kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt""")]),
show=r"""kubectl top node
kubectl describe node {W1} | grep -A9 'Allocated resources'
kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt"""),
step("The metrics API itself",
r"""`kubectl top` only reads an API. Read it yourself: save what `/apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder` returns in `{WS}/lesson-2.3/raw.json`.
If this path answers `NotFound` or `ServiceUnavailable` on some cluster, `kubectl top` fails there too: look at the `metrics-server` Deployment in `kube-system`, and at `kubectl get apiservice v1beta1.metrics.k8s.io`.""",
check=r"""python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); sys.exit(0 if d.get("kind")=="PodMetrics" and d["metadata"]["name"]=="hoarder" and d.get("containers") else 1)' "$WS/lesson-2.3/raw.json" 2>/dev/null""",
hint=r"""`kubectl get --raw PATH`.""",
show=[cmd(r"""k get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json
cat {WS}/lesson-2.3/raw.json
k get apiservice v1beta1.metrics.k8s.io""")]),
show=r"""kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json
jq '.containers' {WS}/lesson-2.3/raw.json
kubectl get apiservice v1beta1.metrics.k8s.io"""),
],
"outro": r"""You ranked Pods by memory and CPU, looked inside a Pod, compared use with what a node can give, and read the metrics API that `kubectl top` uses.
@ -404,17 +361,16 @@ echo "flaky did not restart" >&2; exit 1
Save those in `{WS}/lesson-2.4/previous.log`.""",
check=r"""grep -q 'FATAL config /etc/flaky.conf not found' "$WS/lesson-2.4/previous.log" 2>/dev/null && ! grep -q 'second attempt' "$WS/lesson-2.4/previous.log" """,
hint=r"""`kubectl logs --previous` (or `-p`).""",
show=[cmd(r"""k -n logs logs flaky
k -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log
cat {WS}/lesson-2.4/previous.log""")]),
show=r"""kubectl -n logs logs flaky
kubectl -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log
cat {WS}/lesson-2.4/previous.log"""),
step("The one replica that complains",
r"""Pods `api-1`, `api-2` and `api-3` all carry `app=api`. One of them logs errors. Read all three at once, find it, and write its name to `{WS}/lesson-2.4/erroring.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/erroring.txt" 2>/dev/null)" api-2""",
hint=r"""`kubectl logs -l app=api --prefix` puts `[pod/NAME/CONTAINER]` before each line. With a selector, kubectl shows only the last 10 lines of each Pod unless you say `--tail`.""",
show=[cmd(r"""k -n logs logs -l app=api --prefix --tail=20 | grep ERROR""", r"""`[pod/NAME/...]` at the start of the line names the Pod."""),
hand(r"""echo POD > {WS}/lesson-2.4/erroring.txt""",
r"""`POD`: that name.""",
run=r"""kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt""")]),
show=r"""kubectl -n logs logs -l app=api --prefix --tail=2
kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt
cat {WS}/lesson-2.4/erroring.txt"""),
step("The log file on the node",
r"""The node keeps one file per container instance. Write the full path of the file that holds the output of `flaky`'s **current** instance to `{WS}/lesson-2.4/logfile.txt`.
@ -424,15 +380,11 @@ p=$(tr -d ' \r\n' < "$WS/lesson-2.4/logfile.txt" 2>/dev/null)
uid=$(j -n logs pod flaky '{.metadata.uid}'); n=$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}'); node=$(j -n logs pod flaky '{.spec.nodeName}')
eq "$p" "/var/log/pods/logs_flaky_$uid/flaky/$n.log" && nx "$node" "grep -q 'second attempt' '$p'" </dev/null""",
hint=r"""`/var/log/pods/NAMESPACE_POD_UID/CONTAINER/RESTARTS.log`. `sudo ls -R /var/log/pods/ | grep flaky` on the node gets you close.""",
show=[cmd(r"""k -n logs get po flaky -o wide
k -n logs get po flaky -o yaml | grep uid:""", r"""Its node, and its UID."""),
hand(r"""ssh {W1} sudo ls /var/log/pods/""",
r"""`logs_flaky_UID` holds `flaky/`, one file per instance: `0.log` the first, `1.log` the current one.""",
run=r"""ssh -n {W1} sudo ls /var/log/pods/"""),
hand(r"""echo /var/log/pods/logs_flaky_UID/flaky/1.log > {WS}/lesson-2.4/logfile.txt""",
r"""With the Pod's UID.""",
run=r"""uid=$(kubectl -n logs get pod flaky -o jsonpath='{.metadata.uid}')
echo "/var/log/pods/logs_flaky_$uid/flaky/1.log" > {WS}/lesson-2.4/logfile.txt""")]),
show=r"""kubectl -n logs get pod flaky -o wide
uid=$(kubectl -n logs get pod flaky -o jsonpath='{.metadata.uid}')
ssh -n {W1} "sudo ls /var/log/pods/logs_flaky_$uid/flaky/"
echo "/var/log/pods/logs_flaky_$uid/flaky/1.log" > {WS}/lesson-2.4/logfile.txt
ssh -n {W1} "sudo cat $(cat {WS}/lesson-2.4/logfile.txt)"""),
step("Debug a container that has no shell",
r"""`kubectl -n logs exec quiet -- sh` fails: the `pause` image has nothing but its one program. Add an ephemeral container to the running Pod instead: name `dbg`, image `busybox:1.36`, **targeting** container `quiet` so that it sees its processes, running `sleep 3600`.
@ -441,13 +393,10 @@ Then run `ps` in `dbg` and write the command of the target's process - the first
eq "$(j -n logs pod quiet '{.spec.ephemeralContainers[0].name} {.spec.ephemeralContainers[0].targetContainerName}')" "dbg quiet" || exit 1
grep -qx '/pause' <(tr -d ' \r' < "$WS/lesson-2.4/target.txt" 2>/dev/null)""",
hint=r"""`kubectl debug POD --image=IMAGE --target=CONTAINER -c NAME -- COMMAND`. Interactively you would add `-it` and use the shell straight away; without it, `kubectl exec POD -c dbg -- ps` reaches it once it runs. Ephemeral containers cannot be removed - the Pod keeps them until it goes.""",
show=[cmd(r"""k -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600"""),
hand(r"""k -n logs exec quiet -c dbg -- ps""",
r"""Again if `dbg` is not up yet. PID 1 is the target's process.""",
run=r"""for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done"""),
hand(r"""echo COMMAND > {WS}/lesson-2.4/target.txt""",
r"""`COMMAND`: PID 1's command, as `ps` shows it.""",
run=r"""kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt""")]),
show=r"""kubectl -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600
for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done
kubectl -n logs exec quiet -c dbg -- ps
kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt"""),
step("Debug the node itself",
r"""`kubectl debug node/{W1} --image=busybox:1.36` starts a Pod on that node with the node's whole filesystem mounted at `/host` - useful when you cannot `ssh` to a node, but can reach the API.
@ -457,16 +406,11 @@ want=$(nx "$W1" '. /etc/os-release; echo "$PRETTY_NAME"' </dev/null); [ -n "$wan
eq "$(tr -d '\r"' < "$WS/lesson-2.4/os.txt" 2>/dev/null | sed 's/^PRETTY_NAME=//; s/ *$//')" "$want" \
&& [ -n "$($K get pods -A -o name 2>/dev/null | grep node-debugger)" ]""",
hint=r"""Without `-it`, give it a command and read its output with `kubectl logs`: the Pod is called `node-debugger-NODE-xxxxx`, in your current Namespace. Delete it afterwards.""",
show=[cmd(r"""k debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release
k get po""", r"""The `node-debugger-...` Pod: once it is `Completed`, its log is the file."""),
hand(r"""k logs node-debugger-NAME""",
r"""`node-debugger-NAME`: its full name, from `get po`.""",
run=r"""pod=$(kubectl get pods -o name | grep node-debugger | head -1)
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s"""),
hand(r"""echo 'VALUE' > {WS}/lesson-2.4/os.txt""",
r"""`VALUE`: what `PRETTY_NAME=` says, without the double quotes.""",
run=r"""pod=$(kubectl get pods -o name | grep node-debugger | head -1)
kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt""")]),
show=r"""kubectl debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release
pod=$(kubectl get pods -o name | grep node-debugger | head -1)
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s
kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt
cat {WS}/lesson-2.4/os.txt"""),
],
"outro": r"""You read a crashed instance's logs, searched several Pods at once, found the log file on the node, and debugged both a Pod without a shell and a node.
@ -533,20 +477,19 @@ eq "$(j -n shop svc web '{.spec.selector.app}')" web || exit 1
n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' | grep -c .)
[ "$n" -eq 2 ]""",
hint=r"""Compare `kubectl -n shop get svc web -o wide` (SELECTOR) with `kubectl -n shop get pods --show-labels`. `kubectl set selector` or `kubectl edit svc`.""",
show=[cmd(r"""k -n shop get endpointslices -l kubernetes.io/service-name=web
k -n shop get svc web -o wide
k -n shop get po --show-labels
k -n shop set selector svc web app=web
k -n shop get endpointslices -l kubernetes.io/service-name=web""")]),
show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
kubectl -n shop get svc web -o wide
kubectl -n shop get pods --show-labels
kubectl -n shop set selector svc web app=web
kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""),
step("The wrong port",
r"""Now the Service has endpoints, but they point at port `8080` - and nginx listens on `80`. Make the Service send traffic to the port the container serves.""",
check=r"""
eq "$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{.items[0].ports[0].port}')" 80 && eq "$(j -n shop svc web '{.spec.ports[0].port}')" 80""",
hint=r"""`targetPort` is the container's port; `port` is the Service's own. `kubectl -n shop edit svc web`, or a JSON patch on `/spec/ports/0/targetPort`.""",
show=[hand(r"""k -n shop edit svc web""",
r"""Make `targetPort: 8080` read `targetPort: 80`, then `:wq`.""",
run=r"""kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]' """),
cmd(r"""k -n shop get endpointslices -l kubernetes.io/service-name=web""")]),
show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]'
kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""),
step("Let the right client in",
r"""Still nothing: NetworkPolicy `lockdown` selects the `web` Pods and allows no ingress at all.
@ -557,17 +500,7 @@ ok() { $K -n shop exec "$1" -- wget -qO- -T 3 web >/dev/null 2>&1; }
for i in 1 2 3 4 5; do ok client && break; sleep 3; done; ok client || exit 1
! ok stranger""",
hint=r"""Add an `ingress` rule to `lockdown`: `from: [{podSelector: {matchLabels: {app: client}}}]` with `ports: [{protocol: TCP, port: 80}]`. A NetworkPolicy's ports are the Pod's, not the Service's.""",
show=[hand(r"""k -n shop edit netpol lockdown""",
r"""Add the rule to its `spec:`, beside `policyTypes:`:
```
ingress:
- from:
- podSelector:
matchLabels: {app: client}
ports:
- {protocol: TCP, port: 80}
```""",
run=r"""kubectl -n shop apply -f - <<'EOF'
show=r"""kubectl -n shop apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@ -582,9 +515,9 @@ spec:
matchLabels: {app: client}
ports:
- {protocol: TCP, port: 80}
EOF"""),
cmd(r"""k -n shop exec client -- wget -qO- -T3 web | head -4
k -n shop exec stranger -- wget -qO- -T3 web""", r"""`client` gets the page; `stranger` times out.""")]),
EOF
kubectl -n shop exec client -- wget -qO- -T3 web | head -4
kubectl -n shop exec stranger -- wget -qO- -T3 web || echo refused"""),
step("How kube-proxy routes it",
r"""kube-proxy turned Service `web` into rules on every node. Find out how: its mode is in ConfigMap `kube-proxy` in `kube-system`. Write the mode to `{WS}/lesson-2.5/mode.txt`.
@ -596,18 +529,11 @@ ip=$(j -n shop svc web '{.spec.clusterIP}')
want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" </dev/null)
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.5/chain.txt" 2>/dev/null)" "$want" """,
hint=r"""`kubectl -n kube-system get cm kube-proxy -o yaml | grep mode`. On the node: `sudo iptables-save -t nat | grep CLUSTER-IP` - the line for the Service jumps (`-j`) to its chain. In `nftables` mode the rules are in `sudo nft list ruleset` instead.""",
show=[cmd(r"""k -n kube-system get cm kube-proxy -o yaml | grep mode:"""),
hand(r"""echo MODE > {WS}/lesson-2.5/mode.txt""",
r"""`MODE`: its value.""",
run=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt"""),
cmd(r"""k -n shop get svc web""", r"""Its `CLUSTER-IP`:"""),
hand(r"""sudo iptables-save -t nat | grep IP""",
r"""`IP`: that address. The rule for it jumps (`-j`) to a `KUBE-SVC-...` chain.""",
run='ip=$(kubectl -n shop get svc web -o jsonpath=\'{.spec.clusterIP}\')\nsudo iptables-save -t nat | grep -- "-d $ip/32"'),
hand(r"""echo KUBE-SVC-NAME > {WS}/lesson-2.5/chain.txt""",
r"""The chain's full name.""",
run=r"""ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt""")]),
show=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt
ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32"
sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt
sudo iptables-save -t nat | grep "^-A $(cat {WS}/lesson-2.5/chain.txt)" """),
],
"outro": r"""You took a Service from "times out" to answering by fixing its selector, its target port and a NetworkPolicy, and followed it into kube-proxy's rules.

438
cka/lessons_work.py
View File

@ -1,5 +1,5 @@
"""CKA lessons of the Workloads and Scheduling domain (3.1-3.5). See cka/lessons.py for what a lesson is."""
from sim.bank import NS, step, cmd, hand
from sim.bank import NS, step
# ---- 3.1 rolling updates and rollbacks --------------------------------------------------------
L31 = {
@ -27,10 +27,8 @@ $K -n rollout rollout status deploy api --timeout=180s >/dev/null
r"""Change the update strategy of `api` so that an update first adds **one** Pod and never takes one away before its replacement is ready: `maxSurge: 1`, `maxUnavailable: 0`.""",
check=r"""eq "$(j -n rollout deploy api '{.spec.strategy.type} {.spec.strategy.rollingUpdate.maxSurge} {.spec.strategy.rollingUpdate.maxUnavailable}')" "RollingUpdate 1 0" """,
hint=r"""`kubectl -n rollout edit deploy api`, or `kubectl patch` with `{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}`.""",
show=[hand(r"""k -n rollout edit deploy api""",
r"""Under `strategy.rollingUpdate`, set `maxSurge: 1` and `maxUnavailable: 0`, then save and quit (`:wq`).""",
run=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' """),
cmd(r"""k -n rollout describe deploy api | grep -i strategy""")]),
show=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
kubectl -n rollout get deploy api -o jsonpath='{.spec.strategy}'; echo"""),
step("A release with a reason",
r"""Update `api` to `nginx:1.27`, and record why: the annotation `kubernetes.io/change-cause` on the Deployment, set to `upgrade to 1.27`. `kubectl rollout history` shows it beside the revision.""",
check=r"""
@ -38,10 +36,10 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin
eq "$(j -n rollout deploy api '{.metadata.annotations.kubernetes\.io/change-cause}')" "upgrade to 1.27" || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas}')" "4 4" """,
hint=r"""`kubectl set image deploy/api nginx=nginx:1.27` (the container is called `nginx`), then `kubectl annotate deploy api kubernetes.io/change-cause='...'`. The `--record` flag is gone.""",
show=[cmd(r"""k -n rollout set image deploy/api nginx=nginx:1.27
k -n rollout annotate deploy api kubernetes.io/change-cause='upgrade to 1.27' --overwrite
k -n rollout rollout status deploy api
k -n rollout rollout history deploy api""")]),
show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27
kubectl -n rollout annotate deploy api kubernetes.io/change-cause='upgrade to 1.27' --overwrite
kubectl -n rollout rollout status deploy api --timeout=180s
kubectl -n rollout rollout history deploy api"""),
step("A release that cannot start",
r"""Now a mistake: set the image of `api` to `nginx:1.27-broken`, a tag that does not exist.
@ -51,27 +49,29 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin
[ "$(j -n rollout deploy api '{.status.availableReplicas}')" = 4 ] || exit 1
$K -n rollout get pods -l app=api --no-headers 2>/dev/null | grep -Eq 'ErrImagePull|ImagePullBackOff' """,
hint=r"""`kubectl set image` again. `kubectl -n rollout get pods` shows the Pod that cannot pull; `kubectl -n rollout get rs` shows the two ReplicaSets side by side.""",
show=[cmd(r"""k -n rollout set image deploy/api nginx=nginx:1.27-broken
k -n rollout get po,rs""", r"""Run `get` again after a few seconds: one new Pod is stuck in `ErrImagePull`, and the four old ones still run.""")]),
show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27-broken
sleep 10
kubectl -n rollout get pods
kubectl -n rollout get rs"""),
step("Undo",
r"""Roll `api` back to the revision before - `nginx:1.27` - and let the rollout finish.""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
hint=r"""`kubectl rollout undo deploy/api`. It does not create a ReplicaSet: it scales the old one back up, and that revision gets a new number.""",
show=[cmd(r"""k -n rollout rollout undo deploy/api
k -n rollout rollout status deploy api
k -n rollout rollout history deploy api""")]),
show=r"""kubectl -n rollout rollout undo deploy/api
kubectl -n rollout rollout status deploy api --timeout=180s
kubectl -n rollout rollout history deploy api"""),
step("Back to a revision by number",
r"""Look at the history, then go back to the revision that ran **`nginx:1.26`**, by its number. `kubectl rollout history deploy api --revision=N` shows what a revision's template was.""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.26 || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
hint=r"""Revision 1 is where the Deployment started. `kubectl rollout undo deploy/api --to-revision=1`.""",
show=[cmd(r"""k -n rollout rollout history deploy api
k -n rollout rollout history deploy api --revision=1""", r"""Find the revision whose `Image:` is `nginx:1.26` - revision 1 here."""),
cmd(r"""k -n rollout rollout undo deploy/api --to-revision=1
k -n rollout rollout status deploy api""")]),
show=r"""kubectl -n rollout rollout history deploy api
kubectl -n rollout rollout history deploy api --revision=1 | grep Image
kubectl -n rollout rollout undo deploy/api --to-revision=1
kubectl -n rollout rollout status deploy api --timeout=180s"""),
step("Two changes, one rollout",
r"""Each `set` command starts a rollout of its own. To make two changes at once, **pause** the Deployment, make them - image `nginx:1.27`, and an environment variable `MODE=prod` in the container - then **resume**: one new revision, one rollout.""",
check=r"""
@ -80,12 +80,12 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].env[?(@.name=="MODE")].value}')" prod || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
hint=r"""`kubectl rollout pause deploy/api`, `kubectl set image ...`, `kubectl set env deploy/api MODE=prod`, `kubectl rollout resume deploy/api`.""",
show=[cmd(r"""k -n rollout rollout pause deploy/api
k -n rollout set image deploy/api nginx=nginx:1.27
k -n rollout set env deploy/api MODE=prod
k -n rollout rollout resume deploy/api
k -n rollout rollout status deploy api
k -n rollout rollout history deploy api""")]),
show=r"""kubectl -n rollout rollout pause deploy/api
kubectl -n rollout set image deploy/api nginx=nginx:1.27
kubectl -n rollout set env deploy/api MODE=prod
kubectl -n rollout rollout resume deploy/api
kubectl -n rollout rollout status deploy api --timeout=180s
kubectl -n rollout rollout history deploy api"""),
],
"outro": r"""You tuned the strategy, released with a recorded cause, watched a broken release stall safely, undid it, went back to a numbered revision, and batched two changes into one rollout.
@ -132,15 +132,15 @@ printf 'DB_USER=ledger\nDB_PASS=t0ps3cret\n' > "$WS/lesson-3.2/db.env"
check=r"""
eq "$(j -n conf cm app-config '{.data.MODE}')" prod && $K -n conf get cm app-config -o jsonpath='{.data.app\.properties}' | grep -q '^cache.size=128$'""",
hint=r"""`kubectl create configmap NAME --from-file=PATH --from-literal=KEY=VALUE`. With `--from-file`, the key is the file's name.""",
show=[cmd(r"""k -n conf create cm app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod
k -n conf describe cm app-config""")]),
show=r"""kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod
kubectl -n conf describe cm app-config"""),
step("A Secret from an env file",
r"""Create Secret `db` (type `Opaque`) in `conf` from `{WS}/lesson-3.2/db.env`, one key per line of the file: `DB_USER` and `DB_PASS`. Then read `DB_PASS` back, decoded.""",
check=r"""
eq "$(j -n conf secret db '{.type}')" Opaque && eq "$(j -n conf secret db '{.data.DB_USER}' | base64 -d)" ledger && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""",
hint=r"""`kubectl create secret generic NAME --from-env-file=PATH`. Reading one key: `-o jsonpath='{.data.DB_PASS}' | base64 -d`.""",
show=[cmd(r"""k -n conf create secret generic db --from-env-file={WS}/lesson-3.2/db.env
k -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d""")]),
show=r"""kubectl -n conf create secret generic db --from-env-file={WS}/lesson-3.2/db.env
kubectl -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d; echo"""),
step("Consume both",
r"""Run Pod `app` in `conf` (`busybox:1.36`, `sleep 86400`) that gets:
@ -151,10 +151,16 @@ k -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d""")]),
x() { $K -n conf exec app -- "$@" 2>/dev/null; }
eq "$(x printenv DB_PASS)" t0ps3cret && eq "$(x printenv APP_MODE)" prod && x cat /etc/app/app.properties | grep -q '^log.level=info$'""",
hint=r"""`envFrom: [{secretRef: {name: db}}]`, `env: [{name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: MODE}}}]`, and a `configMap` volume.""",
show=[cmd(r"""k -n conf run app --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > app.yaml"""),
hand(r"""vim app.yaml""",
r"""Add to the container, below `resources: {}` (the docs page *Configure a Pod to Use a ConfigMap* has each of these):
```
show=r"""kubectl -n conf apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: busybox:1.36
command: [sleep, '86400']
envFrom:
- secretRef: {name: db}
env:
@ -163,38 +169,33 @@ eq "$(x printenv DB_PASS)" t0ps3cret && eq "$(x printenv APP_MODE)" prod && x ca
configMapKeyRef: {name: app-config, key: MODE}
volumeMounts:
- {name: config, mountPath: /etc/app}
```
and to the Pod's `spec:`, beside `restartPolicy: Always` and indented like it:
```
volumes:
- {name: config, configMap: {name: app-config}}
```""",
run=r"""yq -i '.spec.containers[0].envFrom = [{"secretRef": {"name": "db"}}]
| .spec.containers[0].env = [{"name": "APP_MODE", "valueFrom": {"configMapKeyRef": {"name": "app-config", "key": "MODE"}}}]
| .spec.containers[0].volumeMounts = [{"name": "config", "mountPath": "/etc/app"}]
| .spec.volumes = [{"name": "config", "configMap": {"name": "app-config"}}]' app.yaml"""),
cmd(r"""k -n conf apply -f app.yaml
k -n conf exec app -- env | grep -e DB_ -e APP_""")]),
- name: config
configMap: {name: app-config}
EOF
kubectl -n conf wait --for=condition=Ready pod/app --timeout=120s
kubectl -n conf exec app -- sh -c 'printenv DB_USER APP_MODE; cat /etc/app/app.properties'"""),
step("Change the configuration, live",
r"""Change `cache.size` to `256` in ConfigMap `app-config` - edit the file and re-create the ConfigMap's key from it, or `kubectl edit`.
Then watch `/etc/app/app.properties` inside Pod `app`: within a minute the kubelet replaces the file. `APP_MODE` would not change, even if you edited `MODE`: the environment is fixed at start.""",
check=r"""$K -n conf exec app -- cat /etc/app/app.properties 2>/dev/null | grep -q '^cache.size=256$'""",
hint=r"""Regenerate and apply: `kubectl create configmap app-config --from-file=... --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f -` - the usual way to update a ConfigMap from files.""",
show=[hand(r"""k -n conf edit cm app-config""",
r"""In `app.properties`, change the line to `cache.size=256`, then `:wq`.""",
run=r"""kubectl -n conf get cm app-config -o yaml | sed 's/cache.size=.*/cache.size=256/' | kubectl replace -f -"""),
cmd(r"""k -n conf exec app -- cat /etc/app/app.properties""", r"""Again until it shows `cache.size=256`: the kubelet rewrites the file within a minute.""")]),
show=r"""sed -i 's/^cache.size=.*/cache.size=256/' {WS}/lesson-3.2/app.properties
kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f -
sleep 30
kubectl -n conf exec app -- cat /etc/app/app.properties"""),
step("A TLS Secret",
r"""Make a self-signed certificate for `shop.example.com` (`openssl` is on this host), and store it as Secret `shop-tls` in `conf`, of type `kubernetes.io/tls` - the type an Ingress or Gateway expects. Its keys are `tls.crt` and `tls.key`.""",
check=r"""
eq "$(j -n conf secret shop-tls '{.type}')" kubernetes.io/tls || exit 1
j -n conf secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'shop.example.com'""",
hint=r"""`openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com`, then `kubectl create secret tls NAME --cert=tls.crt --key=tls.key`.""",
show=[cmd(r"""cd {WS}/lesson-3.2
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com
k -n conf create secret tls shop-tls --cert=tls.crt --key=tls.key
cd""")]),
show=r"""cd {WS}/lesson-3.2
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
kubectl -n conf create secret tls shop-tls --cert=tls.crt --key=tls.key
kubectl -n conf get secret shop-tls
cd"""),
step("Encrypt Secrets at rest",
r"""Read Secret `db` straight out of etcd on `{CP}`, and you see `t0ps3cret` in plain text. Make the API server encrypt Secrets before it stores them:
@ -205,23 +206,7 @@ cd""")]),
Edit a copy of the manifest outside `/etc/kubernetes/manifests` and copy it back: the kubelet runs every file in that directory, backups too.""",
check=_ENC_CHECK + r""" && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""",
hint=r"""The docs page "Encrypting Confidential Data at Rest" has the file. Stored values of encrypted Secrets start with `k8s:enc:aescbc:v1:key1:`: `sudo ETCDCTL_API=3 etcdctl ... get /registry/secrets/conf/db | od -c | head` (or `hexdump -C`). Rewriting: `kubectl get secret db -o json | kubectl replace -f -`. If the API server does not come back, `sudo crictl ps -a` and the logs under `/var/log/pods/kube-system_kube-apiserver-*` say why.""",
show=[cmd(r"""head -c 32 /dev/urandom | base64""", r"""Copy the output: that is the key."""),
hand(r"""sudo mkdir -p /etc/kubernetes/enc
sudo vim /etc/kubernetes/enc/enc.yaml""",
r"""Paste the example from the docs page *Encrypting Confidential Data at Rest* (search for `aescbc`) and cut it down to:
```
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
providers:
- aescbc:
keys:
- name: key1
secret: <your key>
- identity: {}
```""",
run=r"""sudo mkdir -p /etc/kubernetes/enc
show=r"""sudo mkdir -p /etc/kubernetes/enc
sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
@ -233,30 +218,19 @@ resources:
- name: key1
secret: $(head -c 32 /dev/urandom | base64)
- identity: {}
EOF"""),
cmd(r"""sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/"""),
hand(r"""sudo vim ~/kube-apiserver.yaml""",
r"""Add three lines, each beside its kind (`yy` and `p` on an existing `k8s-certs` mount or volume, then change it):
```
- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
```
in the `command:` list,
```
- {name: enc, mountPath: /etc/kubernetes/enc, readOnly: true}
```
under `volumeMounts:`, and
```
- {name: enc, hostPath: {path: /etc/kubernetes/enc, type: DirectoryOrCreate}}
```
under `volumes:`.""",
run=r"""sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
EOF
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
| .spec.containers[0].volumeMounts += [{"name": "enc", "mountPath": "/etc/kubernetes/enc", "readOnly": true}]
| .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' ~/kube-apiserver.yaml"""),
cmd(r"""sudo cp ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
hand(r"""k get po -n kube-system""",
r"""Again until it answers: the API server is back in a minute or so.""",
run=r"""sleep 20; for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""k -n conf get secret db -o json | k replace -f -""", r"""Written again, the Secret is stored encrypted. Reading it from etcd would now show `k8s:enc:aescbc:v1:key1` and no plain text.""")]),
| .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' /tmp/kube-apiserver.yaml
sudo cp /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
sleep 20
for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
kubectl -n conf get secret db -o json | kubectl replace -f -
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
get /registry/secrets/conf/db | od -c | head -4"""),
],
"outro": r"""You built a ConfigMap and Secrets from files, consumed them three ways, updated configuration in a running Pod, and turned on encryption at rest for Secrets.
@ -329,8 +303,9 @@ $K -n scale rollout status deploy api --timeout=180s >/dev/null && $K -n scale r
eq "$(j -n scale hpa api '{.spec.minReplicas} {.spec.maxReplicas} {.spec.metrics[0].resource.name} {.spec.metrics[0].resource.target.averageUtilization}')" "2 6 cpu 50" || exit 1
[ -n "$(j -n scale hpa api '{.status.currentMetrics[0].resource.current.averageUtilization}')" ] && [ "$(j -n scale deploy api '{.status.readyReplicas}')" -ge 2 ] 2>/dev/null""",
hint=r"""`kubectl autoscale deploy api --min=2 --max=6 --cpu=50%` (kubectl before 1.34: `--cpu-percent=50`). `<unknown>` in the TARGETS column means no metrics yet - or a Pod without a CPU request.""",
show=[cmd(r"""k -n scale autoscale deploy api --min=2 --max=6 --cpu=50%
k -n scale get hpa api""", r"""Again in a minute: `TARGETS` shows a value once metrics arrive.""")]),
show=r"""kubectl -n scale autoscale deploy api --min=2 --max=6 --cpu=50%
sleep 45
kubectl -n scale get hpa api"""),
step("Scale down calmly",
r"""Give the HPA of `api` a `behavior`: when scaling **down**, wait **120** seconds of stable recommendations (`stabilizationWindowSeconds`), and remove at most **1** Pod per **60** seconds.
@ -339,27 +314,36 @@ k -n scale get hpa api""", r"""Again in a minute: `TARGETS` shows a value once m
eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.stabilizationWindowSeconds}')" 120 || exit 1
eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.policies[0].type} {.spec.behavior.scaleDown.policies[0].value} {.spec.behavior.scaleDown.policies[0].periodSeconds}')" "Pods 1 60" """,
hint=r"""Under `spec`: `behavior: {scaleDown: {stabilizationWindowSeconds: 120, policies: [{type: Pods, value: 1, periodSeconds: 60}]}}`. `kubectl explain hpa.spec.behavior.scaleDown`.""",
show=[hand(r"""k -n scale edit hpa api""",
r"""Add under `spec:` (the docs page *Horizontal Pod Autoscaling* has it under *Configurable scaling behavior*):
```
behavior:
scaleDown:
stabilizationWindowSeconds: 120
policies:
- {type: Pods, value: 1, periodSeconds: 60}
```""",
run=r"""kubectl -n scale patch hpa api --type merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":120,"policies":[{"type":"Pods","value":1,"periodSeconds":60}]}}}}' """),
cmd(r"""k -n scale describe hpa api""")]),
show=r"""kubectl -n scale patch hpa api --type merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":120,"policies":[{"type":"Pods","value":1,"periodSeconds":60}]}}}}'
kubectl -n scale describe hpa api | grep -A4 Behavior"""),
step("Watch it scale up",
r"""Write an HPA for `cruncher` as a manifest (`autoscaling/v2`): 1 to **4** replicas at **60%** average CPU. `cruncher` uses about three times its request, so the HPA soon adds Pods: wait until it wants more than one.""",
check=r"""
eq "$(j -n scale hpa cruncher '{.spec.maxReplicas} {.spec.metrics[0].resource.target.averageUtilization}')" "4 60" || exit 1
[ "$(j -n scale hpa cruncher '{.status.desiredReplicas}')" -gt 1 ] 2>/dev/null""",
hint=r"""`kubectl autoscale ... --dry-run=client -o yaml` writes the manifest for you. `kubectl -n scale get hpa cruncher -w` shows it react; `describe hpa` lists its decisions as events.""",
show=[cmd(r"""k -n scale autoscale deploy cruncher --min=1 --max=4 --cpu=60% --dry-run=client -o yaml > hpa.yaml
cat hpa.yaml""", r"""That is the manifest the step asks for: `autoscaling/v2`, `minReplicas`, `maxReplicas` and a `Resource` metric for CPU."""),
cmd(r"""k -n scale apply -f hpa.yaml
k -n scale get hpa cruncher""", r"""Again in a minute or two: `REPLICAS` goes above 1.""")]),
show=r"""kubectl -n scale apply -f - <<'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: cruncher
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: cruncher
minReplicas: 1
maxReplicas: 4
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
EOF
sleep 60
kubectl -n scale get hpa cruncher"""),
step("Resize a running Pod",
r"""Run Pod `sizer` in `scale` (`busybox:1.36`, `sleep 86400`) with CPU request `100m` and limit `200m`, memory request and limit `64Mi`.
@ -368,23 +352,11 @@ Then give it more CPU **without replacing it**: request `150m`, limit `300m`, th
eq "$(j -n scale pod sizer '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.limits.cpu}')" "150m 300m" || exit 1
eq "$(j -n scale pod sizer '{.status.containerStatuses[0].resources.limits.cpu} {.status.containerStatuses[0].restartCount}')" "300m 0" """,
hint=r"""`kubectl patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{...}}]}}'`. A plain `kubectl edit` of a Pod's resources is refused; the subresource is the way in. A container's `resizePolicy` decides whether a change needs a restart.""",
show=[cmd(r"""k -n scale run sizer --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > sizer.yaml"""),
hand(r"""vim sizer.yaml""",
r"""Make `resources: {}`:
```
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 200m, memory: 64Mi}
```""",
run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "100m", "memory": "64Mi"}, "limits": {"cpu": "200m", "memory": "64Mi"}}' sizer.yaml"""),
cmd(r"""k -n scale apply -f sizer.yaml"""),
hand(r"""k -n scale get po sizer""",
r"""Once it is `Running`:""",
run=r"""kubectl -n scale wait --for=condition=Ready pod/sizer --timeout=120s"""),
hand(r"""k -n scale edit po sizer --subresource resize""",
r"""Set the CPU request to `150m` and the CPU limit to `300m`, then `:wq`.""",
run=r"""kubectl -n scale patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{"requests":{"cpu":"150m"},"limits":{"cpu":"300m"}}}]}}' """),
cmd(r"""k -n scale describe po sizer | grep -A6 Limits""", r"""The new values, and `Restart Count: 0`.""")]),
show=r"""kubectl -n scale run sizer --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"sizer","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"200m","memory":"64Mi"}}}]}}'
kubectl -n scale wait --for=condition=Ready pod/sizer --timeout=120s
kubectl -n scale patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{"requests":{"cpu":"150m"},"limits":{"cpu":"300m"}}}]}}'
sleep 5
kubectl -n scale get pod sizer -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}{.status.containerStatuses[0].restartCount}{"\n"}'"""),
],
"outro": r"""You made HPAs both ways, slowed down their scale-down, watched one scale up, and resized a running Pod without replacing it.
@ -449,24 +421,17 @@ rs=$(tr -d ' \r\n' < "$WS/lesson-3.4/rs.txt" 2>/dev/null); [ -n "$rs" ] || exit
eq "$(j -n heal rs "$rs" '{.metadata.ownerReferences[0].name} {.status.readyReplicas}')" "web 3" || exit 1
now=$(j -n heal pods -l app=web '{.items[*].metadata.uid}'); for u in $now; do grep -q "$u" "$PRIV/lesson-3.4.uids" || exit 0; done; exit 1""",
hint=r"""`kubectl -n heal delete pod $(kubectl -n heal get pod -l app=web -o name | head -1)`. Pods of a Deployment are named DEPLOYMENT-RSHASH-RANDOM; the middle part is the ReplicaSet's suffix.""",
show=[cmd(r"""k -n heal get po -l app=web"""),
hand(r"""k -n heal delete po POD""",
r"""`POD`: any one of the three names.""",
run=r"""kubectl -n heal delete $(kubectl -n heal get pod -l app=web -o name | head -1)"""),
cmd(r"""k -n heal get po -l app=web
k -n heal get rs""", r"""A new Pod already, and one ReplicaSet behind them all: `web-` and a hash."""),
hand(r"""echo RS > {WS}/lesson-3.4/rs.txt""",
r"""`RS`: that ReplicaSet's name.""",
run=r"""kubectl -n heal get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].name}' > {WS}/lesson-3.4/rs.txt""")]),
show=r"""kubectl -n heal delete $(kubectl -n heal get pod -l app=web -o name | head -1)
kubectl -n heal get pods -l app=web
kubectl -n heal get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].name}' > {WS}/lesson-3.4/rs.txt
kubectl -n heal get rs"""),
step("A readiness probe that asks the wrong thing",
r"""The Pods of `api` run, but none is `Ready`: `kubectl -n heal describe pod -l app=api` shows the readiness probe failing with `404`. nginx has no `/healthz`. Make the probe ask for `/` instead, and both Pods become ready.""",
check=r"""eq "$(j -n heal deploy api '{.spec.template.spec.containers[0].readinessProbe.httpGet.path}')" / && eq "$(j -n heal deploy api '{.status.readyReplicas}')" 2""",
hint=r"""A JSON patch on `/spec/template/spec/containers/0/readinessProbe/httpGet/path`, or `kubectl edit deploy api`. The fix is a change of template: a rollout.""",
show=[cmd(r"""k -n heal describe po -l app=api | grep -i readiness"""),
hand(r"""k -n heal edit deploy api""",
r"""In the `readinessProbe`, change `path: /healthz` to `path: /`, then `:wq`.""",
run=r"""kubectl -n heal patch deploy api --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]' """),
cmd(r"""k -n heal rollout status deploy api""")]),
show=r"""kubectl -n heal describe pod -l app=api | grep -m2 Readiness
kubectl -n heal patch deploy api --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]'
kubectl -n heal rollout status deploy api --timeout=120s"""),
step("A liveness probe that restarts",
r"""Run Pod `fragile` in `heal` (`busybox:1.36`) with the command `sh -c 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400'` and a **liveness** probe that runs `cat /tmp/healthy` every 5 seconds, starting after 5.
@ -475,19 +440,24 @@ After 20 seconds the file is gone, the probe fails three times, and the kubelet
eq "$(j -n heal pod fragile '{.spec.containers[0].livenessProbe.exec.command[1]}')" /tmp/healthy || exit 1
[ "$(j -n heal pod fragile '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null""",
hint=r"""`livenessProbe: {exec: {command: [cat, /tmp/healthy]}, initialDelaySeconds: 5, periodSeconds: 5}`. `kubectl -n heal get pod fragile -w` shows RESTARTS go up; `describe` shows `Liveness probe failed`.""",
show=[cmd(r"""k -n heal run fragile --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400' > fragile.yaml"""),
hand(r"""vim fragile.yaml""",
r"""Add to the container, below `resources: {}` (the docs page *Configure Liveness, Readiness and Startup Probes* has it):
```
show=r"""kubectl -n heal apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: fragile
spec:
containers:
- name: fragile
image: busybox:1.36
command: [sh, -c, 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400']
livenessProbe:
exec:
command: [cat, /tmp/healthy]
initialDelaySeconds: 5
periodSeconds: 5
```""",
run=r"""yq -i '.spec.containers[0].livenessProbe = {"exec": {"command": ["cat", "/tmp/healthy"]}, "initialDelaySeconds": 5, "periodSeconds": 5}' fragile.yaml"""),
cmd(r"""k -n heal apply -f fragile.yaml
k -n heal get po fragile""", r"""Again after about 40 seconds: `RESTARTS` is 1.""")]),
EOF
sleep 50
kubectl -n heal get pod fragile"""),
step("A budget for disruptions",
r"""Create PodDisruptionBudget `web-pdb` in `heal`: at least **2** Pods of `app=web` must stay available.
@ -498,25 +468,20 @@ Then try to drain `{W1}`, where all three `web` Pods run: `kubectl drain {W1} --
eq "$(j -n heal pdb web-pdb '{.spec.minAvailable} {.spec.selector.matchLabels.app}')" "2 web" || exit 1
grep -qi 'disruption budget' "$WS/lesson-3.4/drain.txt" 2>/dev/null && [ "$(j node "$W1" '{.spec.unschedulable}')" != true ]""",
hint=r"""`kubectl create pdb web-pdb --selector=app=web --min-available=2`. Drain prints the refusals on stderr: `2>&1 | tee FILE`. `kubectl uncordon {W1}` afterwards - drain cordons first.""",
show=[cmd(r"""k -n heal create pdb web-pdb --selector=app=web --min-available=2
k -n heal delete po fragile --now
k drain {W1} --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 | tee {WS}/lesson-3.4/drain.txt
k uncordon {W1}""", r"""The drain evicts one `web` Pod, then refuses the others: "Cannot evict pod as it would violate the pod's disruption budget".""")]),
show=r"""kubectl -n heal create pdb web-pdb --selector=app=web --min-available=2
kubectl -n heal get pdb web-pdb
kubectl -n heal delete pod fragile --grace-period=1
kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 | tee {WS}/lesson-3.4/drain.txt | tail -3
kubectl uncordon {W1}"""),
step("A Job that runs to completion",
r"""Create Job `batch` in `heal`: `busybox:1.36` running `sh -c 'echo processed; sleep 2'`, **3** completions, **2** at a time, giving up after **2** retries (`backoffLimit`). Wait until it is complete.""",
check=r"""eq "$(j -n heal job batch '{.spec.completions} {.spec.parallelism} {.spec.backoffLimit} {.status.succeeded}')" "3 2 2 3" """,
hint=r"""`kubectl create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'`, then add `completions`, `parallelism` and `backoffLimit` under `spec`. `kubectl wait --for=condition=complete job/batch`.""",
show=[cmd(r"""k -n heal create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo processed; sleep 2' > batch.yaml"""),
hand(r"""vim batch.yaml""",
r"""Add to the Job's own `spec:`, above `template:`:
```
completions: 3
parallelism: 2
backoffLimit: 2
```""",
run=r"""yq -i '.spec.completions = 3 | .spec.parallelism = 2 | .spec.backoffLimit = 2' batch.yaml"""),
cmd(r"""k apply -f batch.yaml
k -n heal get job,po""", r"""Again until `COMPLETIONS` is `3/3`.""")]),
show=r"""kubectl -n heal create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo processed; sleep 2' > {WS}/lesson-3.4/batch.yaml
yq -i '.spec.completions = 3 | .spec.parallelism = 2 | .spec.backoffLimit = 2' {WS}/lesson-3.4/batch.yaml
kubectl -n heal apply -f {WS}/lesson-3.4/batch.yaml
kubectl -n heal wait --for=condition=complete job/batch --timeout=120s
kubectl -n heal get pods -l job-name=batch"""),
step("A CronJob, and a run on demand",
r"""Create CronJob `report` in `heal`: every 30 minutes (`*/30 * * * *`), `busybox:1.36` running `date`, never two runs at once (`concurrencyPolicy: Forbid`), keeping the last **2** successful Jobs.
@ -525,17 +490,11 @@ Do not wait half an hour: start a Job `report-now` from the CronJob yourself, an
eq "$(j -n heal cronjob report '{.spec.schedule} {.spec.concurrencyPolicy} {.spec.successfulJobsHistoryLimit}')" "*/30 * * * * Forbid 2" || exit 1
eq "$(j -n heal job report-now '{.metadata.annotations.cronjob\.kubernetes\.io/instantiate} {.status.succeeded}')" "manual 1" """,
hint=r"""`kubectl create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date`, then the two fields. `kubectl create job report-now --from=cronjob/report`.""",
show=[cmd(r"""k -n heal create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date > report.yaml"""),
hand(r"""vim report.yaml""",
r"""Add to the CronJob's own `spec:`, beside `schedule:`:
```
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 2
```""",
run=r"""yq -i '.spec.concurrencyPolicy = "Forbid" | .spec.successfulJobsHistoryLimit = 2' report.yaml"""),
cmd(r"""k apply -f report.yaml
k -n heal create job report-now --from=cronjob/report
k -n heal get job,po""", r"""Again until `report-now` shows `1/1`.""")]),
show=r"""kubectl -n heal create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date > {WS}/lesson-3.4/report.yaml
yq -i '.spec.concurrencyPolicy = "Forbid" | .spec.successfulJobsHistoryLimit = 2' {WS}/lesson-3.4/report.yaml
kubectl -n heal apply -f {WS}/lesson-3.4/report.yaml
kubectl -n heal create job report-now --from=cronjob/report
kubectl -n heal wait --for=condition=complete job/report-now --timeout=120s"""),
],
"outro": r"""You watched a ReplicaSet replace a Pod, fixed a readiness probe, saw a liveness probe restart a container, stopped a drain with a disruption budget, and ran a Job and a CronJob.
@ -578,48 +537,35 @@ mkns sched
check=r"""
for p in gold:Guaranteed silver:Burstable bronze:BestEffort; do eq "$(j -n sched pod "${p%%:*}" '{.status.qosClass}')" "${p##*:}" || exit 1; done""",
hint=r"""`kubectl run NAME --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400`, then `resources:` in the container. `.status.qosClass` shows the result.""",
show=[cmd(r"""k -n sched run bronze --image=busybox:1.36 -- sleep 86400
k -n sched run silver --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > silver.yaml"""),
hand(r"""vim silver.yaml""",
r"""Make `resources: {}`:
```
resources:
requests: {memory: 32Mi}
```""",
run=r"""yq -i '.spec.containers[0].resources = {"requests": {"memory": "32Mi"}}' silver.yaml"""),
cmd(r"""k -n sched apply -f silver.yaml
k -n sched run gold --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > gold.yaml"""),
hand(r"""vim gold.yaml""",
r"""Make `resources: {}`:
```
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 100m, memory: 64Mi}
```""",
run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "100m", "memory": "64Mi"}, "limits": {"cpu": "100m", "memory": "64Mi"}}' gold.yaml"""),
cmd(r"""k -n sched apply -f gold.yaml
k -n sched describe po | grep -e ^Name: -e QoS""")]),
show=r"""kubectl -n sched run bronze --image=busybox:1.36 -- sleep 86400
kubectl -n sched run silver --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"silver","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"memory":"32Mi"}}}]}}'
kubectl -n sched run gold --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"gold","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"64Mi"}}}]}}'
kubectl -n sched get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass"""),
step("Only on SSD nodes",
r"""Label `{W2}` with `disk=ssd`. Then run Pod `fast` in `sched` (`busybox:1.36`, `sleep 86400`) with a **required** node affinity for `disk In (ssd)`. It lands on `{W2}` - and if no node had the label, it would stay `Pending`.""",
check=r"""
eq "$(j node "$W2" '{.metadata.labels.disk}')" ssd && eq "$(j -n sched pod fast '{.spec.nodeName}')" "$W2" \
&& eq "$(j -n sched pod fast '{.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key}')" disk""",
hint=r"""`kubectl label node {W2} disk=ssd`. The affinity block: `affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[].matchExpressions[]` with `key`, `operator: In`, `values`. The docs page "Assign Pods to Nodes using Node Affinity" has it to copy.""",
show=[cmd(r"""k label node {W2} disk=ssd
k -n sched run fast --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > fast.yaml"""),
hand(r"""vim fast.yaml""",
r"""Add to the Pod's `spec:`, above `containers:` (the docs page *Assign Pods to Nodes using Node Affinity* has it):
```
show=r"""kubectl label node {W2} disk=ssd
kubectl -n sched apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: fast
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- {key: disk, operator: In, values: [ssd]}
```""",
run=r"""yq -i '.spec.affinity = {"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "disk", "operator": "In", "values": ["ssd"]}]}]}}}' fast.yaml"""),
cmd(r"""k -n sched apply -f fast.yaml
k -n sched get po fast -o wide""")]),
containers:
- name: fast
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n sched get pod fast -o wide"""),
step("One replica per node",
r"""Create Deployment `spread` in `sched` (2 replicas, `busybox:1.36`, `sleep 86400`, label `app=spread`) whose Pods **refuse** to share a node with each other: a required pod anti-affinity on `app=spread`, with `topologyKey: kubernetes.io/hostname`. One Pod lands on each worker; a third replica would stay `Pending`.""",
check=r"""
@ -627,20 +573,32 @@ eq "$(j -n sched deploy spread '{.status.readyReplicas}')" 2 || exit 1
eq "$(j -n sched deploy spread '{.spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey}')" kubernetes.io/hostname || exit 1
[ "$($K -n sched get pods -l app=spread -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 2 ]""",
hint=r"""`podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution: [{labelSelector: {matchLabels: {app: spread}}, topologyKey: kubernetes.io/hostname}]` in the Pod template. `topologySpreadConstraints` is the softer, counting way to spread.""",
show=[cmd(r"""k -n sched create deploy spread --image=busybox:1.36 --replicas=2 --dry-run=client -o yaml -- sleep 86400 > spread.yaml"""),
hand(r"""vim spread.yaml""",
r"""Add to the Pod template's `spec:`, above `containers:` (the docs page *Assigning Pods to Nodes* has an example):
```
show=r"""kubectl -n sched apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: spread
spec:
replicas: 2
selector:
matchLabels: {app: spread}
template:
metadata:
labels: {app: spread}
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels: {app: spread}
topologyKey: kubernetes.io/hostname
```""",
run=r"""yq -i '.spec.template.spec.affinity = {"podAntiAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": [{"labelSelector": {"matchLabels": {"app": "spread"}}, "topologyKey": "kubernetes.io/hostname"}]}}' spread.yaml"""),
cmd(r"""k apply -f spread.yaml
k -n sched get po -l app=spread -o wide""")]),
containers:
- name: spread
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n sched rollout status deploy spread --timeout=120s
kubectl -n sched get pods -l app=spread -o wide"""),
step("A node for batch work only",
r"""Taint `{W1}` with `dedicated=batch:NoSchedule`: from now on only Pods that tolerate it are scheduled there (Pods already running stay - `NoExecute` would evict them).
@ -649,19 +607,23 @@ Then run Pod `nightly` in `sched` (`busybox:1.36`, `sleep 86400`) that tolerates
[ -n "$($K get node "$W1" -o jsonpath='{range .spec.taints[*]}{.key}={.value}:{.effect}{"\n"}{end}' | grep -x 'dedicated=batch:NoSchedule')" ] || exit 1
eq "$(j -n sched pod nightly '{.spec.nodeName} {.status.phase}')" "$W1 Running" && eq "$(j -n sched pod nightly '{.spec.tolerations[?(@.key=="dedicated")].value}')" batch""",
hint=r"""`kubectl taint node {W1} dedicated=batch:NoSchedule`. The toleration: `{key: dedicated, operator: Equal, value: batch, effect: NoSchedule}`.""",
show=[cmd(r"""k taint node {W1} dedicated=batch:NoSchedule
k -n sched run nightly --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > nightly.yaml"""),
hand(r"""vim nightly.yaml""",
r"""Add to the Pod's `spec:`, above `containers:`:
```
show=r"""kubectl taint node {W1} dedicated=batch:NoSchedule
kubectl -n sched apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: nightly
spec:
nodeSelector:
kubernetes.io/hostname: {W1}
tolerations:
- {key: dedicated, operator: Equal, value: batch, effect: NoSchedule}
```""",
run=r"""yq -i '.spec.nodeSelector = {"kubernetes.io/hostname": "{W1}"} | .spec.tolerations = [{"key": "dedicated", "operator": "Equal", "value": "batch", "effect": "NoSchedule"}]' nightly.yaml"""),
cmd(r"""k -n sched apply -f nightly.yaml
k -n sched get po nightly -o wide""")]),
containers:
- name: nightly
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n sched get pod nightly -o wide"""),
step("Defaults for Pods that bring none",
r"""Create LimitRange `defaults` in `sched`: containers that say nothing get a **request** of `50m` CPU and `32Mi` memory and a **limit** of `200m` and `128Mi`.
@ -670,9 +632,7 @@ Then run Pod `plain` (`busybox:1.36`, `sleep 86400`, no resources) and look at w
eq "$(j -n sched limitrange defaults '{.spec.limits[0].type}')" Container || exit 1
eq "$(j -n sched pod plain '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.requests.memory} {.spec.containers[0].resources.limits.cpu} {.spec.containers[0].resources.limits.memory}')" "50m 32Mi 200m 128Mi" """,
hint=r"""`spec.limits: [{type: Container, defaultRequest: {cpu: 50m, memory: 32Mi}, default: {cpu: 200m, memory: 128Mi}}]` - `default` is the limit.""",
show=[hand(r"""vim lr.yaml""",
r"""No `kubectl create` for a LimitRange: paste the example from the docs page *Configure Default CPU Requests and Limits for a Namespace* and make it:
```
show=r"""kubectl -n sched apply -f - <<'EOF'
apiVersion: v1
kind: LimitRange
metadata:
@ -682,37 +642,19 @@ spec:
- type: Container
defaultRequest: {cpu: 50m, memory: 32Mi}
default: {cpu: 200m, memory: 128Mi}
```""",
run=r"""cat > lr.yaml <<'EOF'
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
spec:
limits:
- type: Container
defaultRequest: {cpu: 50m, memory: 32Mi}
default: {cpu: 200m, memory: 128Mi}
EOF"""),
cmd(r"""k -n sched apply -f lr.yaml
k -n sched run plain --image=busybox:1.36 -- sleep 86400
k -n sched describe po plain | grep -A6 Limits""")]),
EOF
kubectl -n sched run plain --image=busybox:1.36 -- sleep 86400
kubectl -n sched get pod plain -o jsonpath='{.spec.containers[0].resources}'; echo"""),
step("Past the scheduler",
r"""Cordon `{W2}`. Then create Pod `manual` in `sched` (`busybox:1.36`, `sleep 86400`) with **`nodeName: {W2}`**. It runs there anyway: `nodeName` hands the Pod straight to that node's kubelet, and cordoning only tells the *scheduler* to stay away.
Leave `{W2}` cordoned.""",
check=r"""eq "$(j node "$W2" '{.spec.unschedulable}')" true && eq "$(j -n sched pod manual '{.spec.nodeName} {.status.phase}')" "$W2 Running" """,
hint=r"""`kubectl cordon {W2}`, then a Pod manifest with `spec.nodeName`. Without a running scheduler, this is also how you place a Pod by hand.""",
show=[cmd(r"""k cordon {W2}
k -n sched run manual --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > manual.yaml"""),
hand(r"""vim manual.yaml""",
r"""Add to the Pod's `spec:`, above `containers:`:
```
nodeName: {W2}
```""",
run=r"""yq -i '.spec.nodeName = "{W2}"' manual.yaml"""),
cmd(r"""k -n sched apply -f manual.yaml
k -n sched get po manual -o wide""")]),
show=r"""kubectl cordon {W2}
kubectl -n sched run manual --image=busybox:1.36 --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 86400
kubectl -n sched get pod manual -o wide
kubectl get nodes"""),
],
"outro": r"""You produced all three QoS classes, steered Pods with node affinity, kept replicas apart, reserved a node with a taint, set defaults with a LimitRange, and bypassed the scheduler.

3
cka/questions.py
View File

@ -667,8 +667,7 @@ _ids = [q["id"] for q in QUESTIONS]
assert len(_ids) == len(set(_ids)), "duplicate question ids"
from cka.solutions import S as _SOL
for _q in QUESTIONS:
_s = _q.get("solution") or _SOL.get(_q["id"], "")
_q["solution"] = _s if isinstance(_s, list) else _s.strip() # a list: parts (sim/bank.py cmd, hand)
_q["solution"] = (_q.get("solution") or _SOL.get(_q["id"], "")).strip()
_q.setdefault("set", None)
_missing = [q["id"] for q in QUESTIONS if not q["solution"]]
assert not _missing, f"tasks without a solution: {_missing}"

3035
cka/solutions.py
View File

File diff suppressed because it is too large Load Diff

502
cka/solve.py
View File

@ -8,10 +8,6 @@ substituted as in the task. A task without a script is only audited for a clean
A command that reads stdin (ssh, kubectl run -i) would swallow the rest of the script: ssh -n,
</dev/null.
A task whose reference solution is in parts (sim/bank.py: cmd, hand) has no script here: the audit
runs the parts - what is typed as it stands, and for each edit by hand its script - so the solution
the candidate reads is the one that is proven.
SOLVE: task id -> bash
"""
@ -29,6 +25,142 @@ cpwait() {
SOLVE = {
# ---- 114-124: bank_storage.py
114: r"""
cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: {name: %{sc}}
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: %{sc}
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 300Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
terminationGracePeriodSeconds: 2
volumes: [{name: data, persistentVolumeClaim: {claimName: %{pvc}}}]
containers:
- name: writer
image: busybox:1.36
command: [sh, -c, "echo %{text} > /data/keep.txt; sleep 86400"]
volumeMounts: [{name: data, mountPath: /data}]
EOF
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q114/pv.txt
kubectl -n %{ns} delete pod %{pod}
kubectl -n %{ns} delete pvc %{pvc}
sleep 4
kubectl get pv $(cat {WS}/q114/pv.txt) -o jsonpath='{.status.phase}' > {WS}/q114/phase.txt
""",
115: r"""
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
volumes:
- name: scratch
ephemeral:
volumeClaimTemplate:
spec:
storageClassName: standard
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
- name: cache
emptyDir: {medium: Memory, sizeLimit: 32Mi}
containers:
- name: encoder
image: busybox:1.36
command: [sleep, "86400"]
volumeMounts:
- {name: scratch, mountPath: /scratch}
- {name: cache, mountPath: /cache}
EOF
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
kubectl -n %{ns} get pvc -o jsonpath='{.items[0].metadata.name}' > {WS}/q115/claim.txt
kubectl -n %{ns} get pvc %{pod}-scratch -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}' > {WS}/q115/owner.txt
""",
116: r"""
kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenScaled":"Delete","whenDeleted":"Retain"}}}'
kubectl -n %{ns} scale sts %{sts} --replicas=1
for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc --no-headers | wc -l)" = 1 ] && break; sleep 3; done
kubectl -n %{ns} get pvc -o name | cut -d/ -f2 > {WS}/q116/claims.txt
""",
117: r"""
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata: {name: %{pv}}
spec:
capacity: {storage: 1Gi}
accessModes: [ReadWriteOnce]
storageClassName: manual
persistentVolumeReclaimPolicy: Retain
hostPath: {path: /mnt/%{pv}}
claimRef: {namespace: %{ns}, name: %{pvc}}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: manual
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
EOF
sleep 5
""",
118: r"""
kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.volumeName}' > {WS}/q118/wrong-pv.txt
kubectl -n %{ns} scale deploy %{dep} --replicas=0
kubectl -n %{ns} wait --for=delete pod -l app=%{dep} --timeout=120s
kubectl -n %{ns} delete pvc %{pvc}
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: ""
volumeName: %{pv}
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 500Mi}}
EOF
kubectl -n %{ns} scale deploy %{dep} --replicas=1
kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
""",
119: r"""
printf 'reports capacity\narchive accessmode\nmedia class\n' > {WS}/q119/reasons.txt
kubectl -n %{ns} delete pvc reports archive media
for c in "reports 1Gi" "archive 2Gi" "media 1Gi"; do set -- $c; cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: $1, namespace: %{ns}}
spec:
storageClassName: manual
accessModes: [ReadWriteOnce]
selector: {matchLabels: {use: $1}}
resources: {requests: {storage: $2}}
EOF
done
sleep 5
""",
120: r"""
for i in $(seq 1 40); do
M=$(kubectl -n %{ns} get events --field-selector reason=FailedScheduling -o jsonpath='{.items[-1:].message}')
[ -n "$M" ] && break; sleep 3
done
echo "$M" > {WS}/q120/event.txt
kubectl -n %{ns} patch deploy %{dep} --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
""",
121: r"""
kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.metadata.finalizers[0]}' > {WS}/q121/finalizer.txt
echo deployment/%{dep} > {WS}/q121/used-by.txt
@ -44,6 +176,55 @@ for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.sta
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
echo rancher.io/local-path > {WS}/q122/provisioner.txt
""",
123: r"""
kubectl annotate sc %{sc} storageclass.kubernetes.io/is-default-class=true
for i in $(seq 1 40); do [ "$(kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.status.phase}')" = Bound ] && break; sleep 3; done
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
kubectl -n %{ns} get pvc %{pvc} -o jsonpath='{.spec.storageClassName}' > {WS}/q123/class.txt
""",
124: r"""
cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: {name: %{sc}}
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata: {name: %{pv}}
spec:
capacity: {storage: 1Gi}
accessModes: [ReadWriteOnce]
storageClassName: %{sc}
persistentVolumeReclaimPolicy: Retain
local: {path: /mnt/disks/%{pv}}
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions: [{key: kubernetes.io/hostname, operator: In, values: [{W2}]}]
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: %{pvc}, namespace: %{ns}}
spec:
storageClassName: %{sc}
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}}
spec:
volumes: [{name: disk, persistentVolumeClaim: {claimName: %{pvc}}}]
containers:
- name: prom
image: busybox:1.36
command: [sleep, "86400"]
volumeMounts: [{name: disk, mountPath: /disk}]
EOF
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=240s
""",
# ---- 125-128: bank_net.py
125: r"""
kubectl -n kube-system get cm coredns -o yaml > {WS}/q125/coredns-backup.yaml
@ -52,6 +233,36 @@ kubectl -n kube-system get cm coredns -o json | jq --arg add "$ADD" '.data.Coref
kubectl -n kube-system rollout restart deploy coredns
kubectl -n kube-system rollout status deploy coredns --timeout=240s
""",
126: r"""
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: {name: %{gw}, namespace: %{ns}}
spec:
gatewayClassName: %{gc}
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: %{host}.example.com
tls:
mode: Terminate
certificateRefs: [{kind: Secret, name: %{host}-tls}]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: %{host}, namespace: %{ns}}
spec:
parentRefs: [{name: %{gw}}]
hostnames: [%{host}.example.com]
rules:
- matches: [{path: {type: PathPrefix, value: /api}}]
backendRefs: [{name: api, port: 8080}]
- matches: [{path: {type: PathPrefix, value: /}}]
backendRefs: [{name: web, port: 80}]
EOF
kubectl -n %{ns} delete ingress %{host}
""",
127: r"""
kubectl annotate ingressclass %{old} ingressclass.kubernetes.io/is-default-class-
kubectl annotate ingressclass %{cls} ingressclass.kubernetes.io/is-default-class=true
@ -61,6 +272,16 @@ kubectl -n %{ns} patch ingress %{ing} --type=json -p '[
{"op": "replace", "path": "/spec/rules/0/http/paths/1/pathType", "value": "Prefix"}]'
kubectl apply -f {WS}/q127/blog-ingress.yaml
""",
128: r"""
kubectl -n %{ns} patch ds %{ds} --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/ports/0/hostPort", "value": %{hp}}]'
kubectl -n %{ns} rollout status ds %{ds} --timeout=240s
yq -i '.spec.dnsPolicy = "ClusterFirstWithHostNet"' {WS}/q128/collector.yaml
kubectl -n %{ns} delete pod %{pod} --grace-period=1
kubectl apply -f {WS}/q128/collector.yaml
kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=180s
kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP} {.status.hostIP}' > {WS}/q128/ips.txt
sleep 8
""",
# ---- 129-131: bank_tshoot.py
129: r"""
ssh -n {W1} 'sudo systemctl enable --now containerd; echo "runtime-endpoint: unix:///run/containerd/containerd.sock" | sudo tee /etc/crictl.yaml >/dev/null'
@ -100,10 +321,25 @@ sudo mkdir -p /opt/cka
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/cka/etcd-snapshot.db
""",
5: r"""
ssh -n {W3} 'sudo tee /etc/kubernetes/manifests/static-web.yaml >/dev/null <<EOF
apiVersion: v1
kind: Pod
metadata: {name: static-web, namespace: default}
spec:
containers: [{name: web, image: "nginx:1.27"}]
EOF'
for i in $(seq 1 60); do [ "$(kubectl get pod static-web-{W3} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
""",
17: r"""
ssh -n {W2} 'sudo systemctl enable --now kubelet'
for i in $(seq 1 40); do [ "$(kubectl get node {W2} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
18: r"""
kubectl -n troubleshoot set image deploy payments payments=nginx:1.27
kubectl -n troubleshoot patch svc payments -p '{"spec":{"selector":{"app":"payments"},"ports":[{"port":80,"targetPort":80}]}}'
kubectl -n troubleshoot rollout status deploy payments --timeout=240s
""",
20: CPWAIT + r"""
sudo sed -i 's/kube-schedulerr/kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
cpwait kube-scheduler
@ -123,6 +359,44 @@ kubectl -n kube-system rollout status deploy coredns --timeout=240s
sudo sed -i '/--leader-elect-lease-duration=1s/d' /etc/kubernetes/manifests/kube-controller-manager.yaml
cpwait kube-controller-manager
""",
39: r"""
ssh -n {W1} "sudo sed -i 's|ca-bundle.crt|ca.crt|' /var/lib/kubelet/config.yaml; sudo systemctl restart kubelet; sudo systemctl enable kubelet"
for i in $(seq 1 40); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
45: CPWAIT + r"""
sudo etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore
sudo sed -i 's|path: /var/lib/etcd$|path: /var/lib/etcd-restore|' /etc/kubernetes/manifests/etcd.yaml
sleep 30
for i in $(seq 1 80); do kubectl -n default get cm %{cm} >/dev/null 2>&1 && break; sleep 3; done
for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
""",
46: r"""
N=$(kubectl get nodes -l tier=edge -o jsonpath='{.items[0].metadata.name}')
D=$(ssh -n $N "sudo awk '/^staticPodPath:/ {print \$2}' /var/lib/kubelet/config.yaml")
kubectl create ns %{ns} 2>/dev/null
ssh -n $N "sudo tee $D/%{pod}.yaml >/dev/null <<EOF
apiVersion: v1
kind: Pod
metadata: {name: %{pod}, namespace: %{ns}, labels: {role: edge}}
spec:
containers:
- name: web
image: nginx:1.27
resources: {requests: {cpu: 50m, memory: 32Mi}}
EOF"
for i in $(seq 1 60); do [ "$(kubectl -n %{ns} get pod %{pod}-$N -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
""",
51: r"""
for r in $(kubectl -n %{ns} rollout history deploy %{dep} | awk '/^[0-9]/{print $1}'); do
kubectl -n %{ns} rollout history deploy %{dep} --revision=$r | grep -q 'nginx:1.25' && N=$r
done
kubectl -n %{ns} rollout undo deploy %{dep} --to-revision=$N
kubectl -n %{ns} scale deploy %{dep} --replicas=3
kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
kubectl -n %{ns} rollout pause deploy %{dep}
kubectl -n %{ns} set image deploy/%{dep} $(kubectl -n %{ns} get deploy %{dep} -o jsonpath='{.spec.template.spec.containers[0].name}')=nginx:1.27
kubectl -n %{ns} set env deploy/%{dep} RELEASE=next
""",
57: r"""
kubectl -n kube-system delete netpol maintenance-lockdown
kubectl -n kube-system scale deploy coredns --replicas=2
@ -130,6 +404,11 @@ kubectl -n kube-system get cm coredns -o json | jq '.data.Corefile |= sub("forwa
kubectl -n kube-system rollout restart deploy coredns
kubectl -n kube-system rollout status deploy coredns --timeout=240s
""",
60: r"""
ssh -n {W2} "sudo sed -i 's|ca-wrong.crt|ca.crt|' /var/lib/kubelet/config.yaml; sudo systemctl enable --now kubelet"
kubectl uncordon {W2}
for i in $(seq 1 40); do [ "$(kubectl get node {W2} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
61: CPWAIT + r"""
sudo sed -i 's/kube-schedulerr/kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
sudo sed -i 's|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)-broken|\1|' /etc/kubernetes/manifests/kube-controller-manager.yaml
@ -142,6 +421,17 @@ for i in $(seq 1 80); do kubectl get nodes >/dev/null 2>&1 && break; sleep 3; do
for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
sudo kubeadm certs check-expiration 2>/dev/null | grep '^apiserver ' > {WS}/q65/expiry.txt
""",
66: CPWAIT + r"""
sudo sed -i 's/kube-apiserverr/kube-apiserver/' /etc/kubernetes/manifests/kube-apiserver.yaml
for i in $(seq 1 80); do kubectl get nodes >/dev/null 2>&1 && break; sleep 3; done
for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do cpwait $c; done
""",
67: r"""
T=$(sudo kubeadm token create --ttl 2h)
H=$(sudo cat /etc/kubernetes/pki/ca.crt | openssl x509 -pubkey | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')
E=$(kubectl get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}')
printf 'JOIN_TOKEN=%s\nCA_HASH=sha256:%s\nAPI_ENDPOINT=%s\n' "$T" "$H" "$E" > {WS}/q67/node-join.env
""",
70: r"""
kubectl -n driver-onboard set image deploy/onboard-app onboard=nginx:1.25
kubectl -n driver-onboard patch deploy onboard-app --type=json -p '[
@ -149,6 +439,15 @@ kubectl -n driver-onboard patch deploy onboard-app --type=json -p '[
{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/"}]'
kubectl -n driver-onboard rollout status deploy onboard-app --timeout=240s
""",
71: r"""
kubectl -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
kubectl -n dispatch-hub rollout status deploy dispatch-api --timeout=240s
for i in 1 2 3 4 5; do
kubectl -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt </dev/null 2>/dev/null
grep -qi nginx {WS}/q71/test-result.txt && break; kubectl -n dispatch-hub delete pod probe --ignore-not-found >/dev/null 2>&1; sleep 3
done
kubectl -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt
""",
76: r"""
kubectl -n trip-tracker get deploy trip-logger -o json | jq '
.spec.template.spec.volumes += [{"name": "config-volume", "emptyDir": {}}]
@ -168,23 +467,79 @@ kubectl -n surge-pricing get deploy surge-engine -o json | jq '
| .spec.template.spec.volumes += [{"name": "config-vol", "configMap": {"name": "surge-config"}}]' | kubectl apply -f -
kubectl -n surge-pricing rollout status deploy surge-engine --timeout=240s
""",
81: r"""
kubectl -n fare-engine delete pvc fare-data-pvc
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: fare-data-pvc, namespace: fare-engine}
spec:
accessModes: [ReadWriteOnce]
resources: {requests: {storage: 100Mi}}
storageClassName: standard
EOF
kubectl label node {W1} disk=nvme
kubectl -n fare-engine rollout status deploy fare-calculator --timeout=240s
""",
84: r"""
kubectl -n booking-tls get cm booking-nginx-config -o json | jq '.data |= with_entries(.value |= sub("ssl_protocols TLSv1.2 TLSv1.3;"; "ssl_protocols TLSv1.3;"))' | kubectl apply -f -
kubectl -n booking-tls rollout restart deploy booking-nginx
kubectl -n booking-tls rollout status deploy booking-nginx --timeout=240s
echo 'ssl_protocols TLSv1.3;' > {WS}/q84/ssl-line.txt
""",
86: r"""
cat > {WS}/q86/controlplane.txt <<'EOF'
kubelet: process
kube-apiserver: static-pod
kube-scheduler: static-pod
kube-controller-manager: static-pod
etcd: static-pod
dns: deployment coredns
EOF
""",
87: r"""
C=$(kubectl get nodes --no-headers -l node-role.kubernetes.io/control-plane | wc -l)
W=$(kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' | wc -l)
CIDR=$(kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -o 'service-cluster-ip-range=[^ ]*' | head -1 | cut -d= -f2)
F=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
printf '1: %s\n2: %s\n3: %s\n4: kindnet, %s\n5: -{W3}\n' "$C" "$W" "$CIDR" "$F" > {WS}/q87/cluster-info.txt
""",
88: r"""
kubectl -n oslo-web run gatekeeper --image=busybox:1.36 --dry-run=client -o json -- sleep 86400 | jq '.spec.containers[0].readinessProbe = {"exec": {"command": ["sh", "-c", "wget -T2 -qO- http://ledger-api:8080"]}, "initialDelaySeconds": 5, "periodSeconds": 5}' | kubectl apply -f -
sleep 12
kubectl -n oslo-web expose deploy ledger --name=ledger-api --port=8080 --target-port=80
kubectl -n oslo-web wait --for=condition=Ready pod/gatekeeper --timeout=180s
""",
90: r"""
cat > {WS}/q90/cluster-events.sh <<'EOF'
kubectl get events -A --sort-by=.metadata.creationTimestamp
EOF
chmod +x {WS}/q90/cluster-events.sh
P=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system delete pod $P
for i in $(seq 1 40); do
[ "$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].status.phase}' 2>/dev/null)" = Running ] && break; sleep 3
done
sleep 5
kubectl -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
ssh -n {W2} 'sudo crictl stop $(sudo crictl ps --name kube-proxy -q)'
sleep 25
kubectl -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
echo container > {WS}/q90/restarts.txt
""",
92: r"""
ssh -n {W1} "sudo sed -i 's|^ExecStart=/usr/local/bin/kubelet |ExecStart=/usr/bin/kubelet |' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf; sudo systemctl daemon-reload; sudo systemctl restart kubelet"
echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q92/fixed-file.txt
for i in $(seq 1 40); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
94: r"""
kubectl run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon --dry-run=client -o yaml \
| yq '.spec.containers[0].resources = {"requests": {"cpu": "10m", "memory": "20Mi"}}' | sudo tee /etc/kubernetes/manifests/ops-beacon.yaml >/dev/null
for i in $(seq 1 60); do [ "$(kubectl -n default get pod ops-beacon-{CP} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done
kubectl -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
kubectl -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt
sleep 5
""",
95: r"""
CI=$(ssh -n {W2} 'sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -issuer' | sed 's/^issuer= *//')
SI=$(ssh -n {W2} 'sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -issuer' | sed 's/^issuer= *//')
@ -232,11 +587,47 @@ kubectl -n %{ns} patch deploy %{dep} --type=json -p '[
{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/%{ready}"}]'
kubectl -n %{ns} rollout status deploy %{dep} --timeout=240s
""",
107: r"""
ssh -n {W3} "printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/%{name}.conf >/dev/null
printf 'net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\n' | sudo tee /etc/sysctl.d/%{name}.conf >/dev/null"
IPF=$(ssh -n {W3} 'sudo sysctl -n net.ipv4.ip_forward')
SW=off; [ "$(ssh -n {W3} 'sed 1d /proc/swaps | grep -c .')" -gt 0 ] && SW=on
FS=$(ssh -n {W3} "sudo awk '/^failSwapOn:/ {print \$2}' /var/lib/kubelet/config.yaml"); [ -n "$FS" ] || FS=true
CG=$(ssh -n {W3} "sudo awk '/^cgroupDriver:/ {print \$2}' /var/lib/kubelet/config.yaml")
printf 'ip_forward=%s\nswap=%s\nfail-swap-on=%s\ncgroup-driver=%s\n' "$IPF" "$SW" "$FS" "$CG" > {WS}/q107/node.txt
""",
108: r"""
V=$(ssh -n {W2} 'sudo crictl version 2>/dev/null')
RN=$(echo "$V" | awk '/^RuntimeName:/ {print $2}'); RV=$(echo "$V" | awk '/^RuntimeVersion:/ {print $2}')
SOCK=$(ssh -n {W2} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
CID=$(kubectl -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's#^[a-z-]*://##')
PID=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.info.pid}}' $CID 2>/dev/null")
LOG=$(ssh -n {W2} "sudo crictl inspect -o go-template --template '{{.status.logPath}}' $CID 2>/dev/null")
printf 'runtime=%s %s\nsocket=%s\ncontainer-id=%s\npid=%s\nlog=%s\n' "$RN" "$RV" "$SOCK" "$CID" "$PID" "$LOG" > {WS}/q108/runtime.txt
""",
110: r"""
kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/^controlPlaneEndpoint:/{print $2}' | tr -d '"' > {WS}/q110/endpoint.txt
KEY=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
sudo kubeadm token create --print-join-command --certificate-key $KEY 2>/dev/null | grep 'kubeadm join' > {WS}/q110/join-control-plane.sh
""",
111: r"""
E='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
N=$(kubectl get nodes -l node-role.kubernetes.io/control-plane --no-headers | wc -l)
M=$($E member list | wc -l)
L=$($E member list | awk -F', ' 'NR==1 {print $3}')
S=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
C=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
printf 'control-plane-nodes=%s\netcd-topology=stacked\netcd-members=%s\netcd-leader=%s\nscheduler-leader=%s\ncontroller-manager-leader=%s\n' "$N" "$M" "$L" "$S" "$C" > {WS}/q111/ha.txt
""",
112: r"""
CONF=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
PL=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[].type] | join(",")')
IPAM=$(ssh -n {W1} "sudo cat $CONF" | jq -r '[.plugins[] | select(.ipam) | .ipam.type][0]')
SOCK=$(ssh -n {W1} "sudo awk '/^containerRuntimeEndpoint:/ {print \$2}' /var/lib/kubelet/config.yaml" | tr -d '"')
N=$(kubectl get csidrivers --no-headers 2>/dev/null | wc -l)
PROV=$(kubectl get sc -o json | jq -r '.items[] | select(.metadata.annotations["storageclass.kubernetes.io/is-default-class"] == "true") | .provisioner')
printf 'cni-config=%s\ncni-plugins=%s\ncni-ipam=%s\ncri-socket=%s\ncsi-drivers=%s\ndefault-provisioner=%s\n' "$CONF" "$PL" "$IPAM" "$SOCK" "$N" "$PROV" > {WS}/q112/interfaces.txt
""",
19: r"""
for i in $(seq 1 40); do kubectl -n troubleshoot logs crasher --previous 2>/dev/null | grep -q DB_HOST && break; sleep 3; done
kubectl -n troubleshoot logs crasher --previous > {WS}/q19/crasher.log
@ -253,6 +644,71 @@ KV=$(kubectl get node %{node} -o jsonpath='{.status.nodeInfo.kubeletVersion}')
R=$(kubectl get node %{node} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}')
printf 'apiserver=%s\netcd=%s\nkubelet=%s\nruntime=%s\n' "$A" "$E" "$KV" "$R" > {WS}/q41/versions.txt
""",
59: r"""
kubectl -n %{ns} apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: new-data, namespace: %{ns}}
spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 1Gi}}}
---
apiVersion: batch/v1
kind: Job
metadata: {name: migrate, namespace: %{ns}}
spec:
template:
spec:
restartPolicy: Never
volumes:
- {name: src, persistentVolumeClaim: {claimName: old-data}}
- {name: dst, persistentVolumeClaim: {claimName: new-data}}
containers:
- name: cp
image: busybox:1.36
command: [sh, -c, "cp -a /src/. /dst/ && ls /dst"]
volumeMounts: [{name: src, mountPath: /src}, {name: dst, mountPath: /dst}]
EOF
kubectl -n %{ns} wait --for=condition=complete job/migrate --timeout=240s
kubectl -n %{ns} get pod legacy -o json | jq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .status)
| (.spec.volumes[] | select(.persistentVolumeClaim.claimName == "old-data") | .persistentVolumeClaim.claimName) = "new-data"' > /tmp/legacy.json
kubectl -n %{ns} delete pod legacy --now
kubectl apply -f /tmp/legacy.json
kubectl -n %{ns} delete pod -l job-name=migrate
kubectl -n %{ns} delete pvc old-data
kubectl -n %{ns} wait --for=condition=Ready pod/legacy --timeout=180s
""",
77: r"""
kubectl -n kube-system get ds -o name | sed 's|.*/||' | grep -i kindnet > {WS}/q77/cni.txt
kubectl -n zero-trust run srv --image=nginx:1.27 --labels=app=srv
kubectl -n zero-trust expose pod srv --port 80
kubectl -n zero-trust run cli --image=busybox:1.36 -- sleep 3600
kubectl -n zero-trust wait --for=condition=Ready pod/srv pod/cli --timeout=180s
kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null && echo reachable
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-all, namespace: zero-trust}
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
EOF
sleep 6
if kubectl -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null 2>&1; then echo no; else echo yes; fi > {WS}/q77/enforced.txt
kubectl -n zero-trust delete pod srv cli --now
kubectl -n zero-trust delete svc srv
""",
91: r"""
kubectl api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q91/namespaced.txt
for ns in census-north census-south census-east census-west; do echo "$ns $(kubectl -n $ns get role --no-headers | wc -l)"; done | sort -k2 -n | tail -1 > {WS}/q91/most-roles.txt
""",
102: r"""
kubectl -n oslo-files get pvc shared-files -o json | jq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .metadata.annotations, .metadata.finalizers, .status)
| .spec.accessModes = ["ReadWriteOnce"]' > /tmp/pvc.json
kubectl -n oslo-files delete pvc shared-files
kubectl apply -f /tmp/pvc.json
kubectl -n oslo-files rollout restart deploy uploader
kubectl -n oslo-files rollout status deploy uploader --timeout=240s
echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
""",
93: r"""
kubectl -n oslo-secure create secret generic bundle --from-file={WS}/q93/tls-bundle.pem
kubectl -n oslo-secure create secret generic db-login --from-literal=user=svc-ledger --from-literal=pass='%{pw}'
@ -263,14 +719,48 @@ kubectl -n oslo-secure run vault-reader --image=busybox:1.36 --dry-run=client -o
{"name": "APP_PASS", "valueFrom": {"secretKeyRef": {"name": "db-login", "key": "pass"}}}]' | kubectl apply -f -
kubectl -n oslo-secure wait --for=condition=Ready pod/vault-reader --timeout=180s
""",
# ---- 132-145: bank_course.py (the others are solved by their solution as written, below)
132: r"""
apt_get() { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq -y -o Dpkg::Options::=--force-confold "$@" </dev/null >/dev/null; }
sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt_get update
sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null
sudo kubeadm upgrade apply v1.37.1 -y </dev/null | tail -2
kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl >/dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null
sudo systemctl daemon-reload && sudo systemctl restart kubelet
for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done
kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data
ssh {W1} sudo bash <<'EOF'
export DEBIAN_FRONTEND=noninteractive
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq </dev/null
apt-mark unhold kubeadm kubelet kubectl >/dev/null
apt-get install -y -qq kubeadm=1.37.1-1.1 </dev/null >/dev/null
kubeadm upgrade node </dev/null | tail -1
apt-get install -y -qq kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 </dev/null >/dev/null
apt-mark hold kubeadm kubelet kubectl >/dev/null
systemctl daemon-reload && systemctl restart kubelet
EOF
kubectl uncordon {W1}
""",
139: r"""
ssh {W1} sudo bash <<'EOF'
yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
systemctl restart kubelet
EOF
for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done
""",
}
# ---- 1-113: the banks carried over from cka-sim.
# For these tasks the reference solution is a script as it stands, so the audit runs exactly the
# text the candidate is shown - and a solution that stops working is found the next time it runs.
AS_WRITTEN = [1, 21, 40, 55, 69, 75, 85, 98, 104, 109,
AS_WRITTEN = [1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
40, 42, 43, 44, 47, 48, 49, 50, 52, 53, 54, 55, 56, 58, 62, 63, 64, 68, 69, 72, 73, 74, 75, 78, 80, 82, 83,
85, 89, 98, 99, 100, 101, 103, 104, 109, 113,
# bank_course.py: written to run as they stand
133]
133, 134, 135, 136, 137, 138, 140, 141, 142, 143, 144, 145, 146, 147]
def _as_written():

130
ksim
View File

@ -22,7 +22,7 @@ from collections.abc import Callable
from datetime import datetime
from typing import Any
VERSION = "2026.10.27.2"
VERSION = "2026.10.26.3"
ROOT = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, ROOT)
from sim import desktop, envs # noqa: E402
@ -96,11 +96,7 @@ def load_track(track_id):
q.setdefault("set", None)
if q["id"] in workers:
q["workers"] = workers[q["id"]]
if isinstance(q.get("solution"), list): # a solution in parts: the audit runs what it shows
if q["id"] in solve:
sys.exit(f"{track_id} task {q['id']}: its solution is in parts, which the audit runs - drop its script from {track_id}/solve.py")
q["solve"] = show_script({"show": q["solution"]})
elif q["id"] in solve:
if q["id"] in solve:
q["solve"] = solve[q["id"]]
# Every task has a cluster to itself, so nothing it breaks can reach another task: the flags
# that kept such tasks apart on a shared cluster ("breaks", "solo", "disruptive") are void.
@ -128,29 +124,7 @@ def lesson_task(l, objectives):
o = next((o for o in objectives if o["id"] == l["objective"]), None)
return dict(l, lesson=True, domain=o["domain"] if o else "", difficulty="normal", set=None,
checks=[{"name": s["title"], "pts": 1, "script": s["check"]} for s in l["steps"]], weight=len(l["steps"]),
task=l["intro"], solution=[p for s in l["steps"] for p in show_parts(s)])
def show_parts(s):
"""How to do a step, as parts (sim/bank.py: cmd, hand): commands typed as they stand, and what
is done by hand - an edit, a value read off the screen - with the script the audit runs for it."""
sh = s.get("show") or ""
return sh if isinstance(sh, list) else ([{"type": sh, "how": "", "run": sh}] if sh.strip() else [])
def show_md(s):
"""How to do a step, as you read it: each thing to type in a block of its own (a click copies
it), and beneath it what to do there by hand."""
return "\n\n".join(f"```\n{p['type'].strip(chr(10))}\n```" + (f"\n\n{p['how'].strip()}" if p["how"] else "")
for p in show_parts(s))
def show_script(s):
"""What the audit runs for a step: what you type, and for each part done by hand its script."""
return "\n".join(p["run"] for p in show_parts(s))
def solution_parts(q):
"""A task's reference solution as parts, like a lesson step's: a solution in parts (cmd, hand)
as it stands, one written as a single script as one part."""
return show_parts({"show": q.get("solution") or ""})
task=l["intro"], solution="\n\n".join(s["show"] for s in l["steps"] if s["show"]))
def set_questions(name):
@ -464,9 +438,7 @@ def task_text(q, params=None):
return prose(q["task"], params if params is not None else session_params(load_state(), q), q)
def solution_text(q, params=None):
"""A task's reference solution as you read it (Markdown): each thing to type in a block of its
own, and beneath it what to do there by hand."""
return prose(show_md({"show": solution_parts(q)}), params if params is not None else session_params(load_state(), q), q)
return prose(str(q.get("solution", "")), params if params is not None else session_params(load_state(), q), q)
def make_params(q, rng):
fn = q.get("params")
@ -776,15 +748,12 @@ def solution_allowed(st: dict[str, Any] | None, qid: int | str) -> bool:
def cmd_solution(args):
if not args:
die("usage: ./ksim solution N (drills: after grading the task; exams: after submit; --force to peek anyway)")
if args[0] == "lint": # no cluster needed, and no solution shown: only where one has you type a script
names = [a.upper() for a in args[1:]]
return print_solution_lint(solution_lint([q for q in QUESTIONS if not names or str(q["id"]) in names or q["domain"] in names]))
force = "--force" in args; args = [a for a in args if a != "--force"]
q = q_by_id(int(args[0])); st = load_state()
if not solution_allowed(st, int(q["id"])) and not force:
die("Not yet: in a drill, grade the task first (./ksim grade N); in an exam, submit first. --force overrides.")
print(bold(f"Solution - Q{q['id']} {q['title']}")); print("-" * 64)
print(cli_md(solution_text(q, session_params(st, q)))); print("-" * 64)
print(solution_text(q, session_params(st, q))); print("-" * 64)
print(dim(f"One correct way, not the only one. $do = --dry-run=client -o yaml, k = kubectl. Run it on the task's host: ssh {host_of(q)}"))
def cmd_progress(args):
@ -1597,9 +1566,7 @@ def begin_session(qs, mode, duration, force, log: Callable[..., Any] = _log_msg,
st = {"mode": mode, "track": TRACK_ID, "hard": hard, "ids": [q["id"] for q in qs],
"params": params_by_id, "duration": duration, "started_at": time.time(), "ended_at": None,
"k8s": k8s_version(), "results": {}, "flags": [], "notes": "",
"objective": objective, # the curriculum objective a drill or a lesson was started for, if any
# the tasks or lessons whose host failed its setup: the page and ./ksim lesson say so, and how to rebuild it
"broken": [q["id"] for q in qs if any(p.startswith(host_of(q) + ":") for p in problems)]}
"objective": objective} # the curriculum objective a drill or a lesson was started for, if any
lesson = mode == "lesson"
if lesson:
st["steps"] = {} # lesson id -> step index -> its latest check (a pass sticks)
@ -1775,9 +1742,6 @@ def cli_md(text):
def print_step(l, params, i, st=None):
s = l["steps"][i]
done = bool(st and steps_of(st, l).get(str(i), {}).get("pass"))
if st and l["id"] in st.get("broken", []):
print(red(f"This lesson's host failed its setup when the session started: it is not as the steps describe.\n"
f" Rebuild it (about a minute): ./ksim reset {l['id']}"))
print(bold(f"Step {i + 1} of {len(l['steps'])}: {render(s['title'], params, l)}") + (green(" ✓ checked") if done else ""))
print(cli_md(render(s["text"], params, l)))
print(dim(" ./ksim lesson check when it is done ./ksim lesson hint / show if you are stuck"))
@ -1806,9 +1770,6 @@ def cmd_lesson(args):
sub = args[0] if args else ""
if sub == "audit":
return lesson_audit(args[1:])
if sub == "lint": # no cluster needed: what each step has you type, against TYPING_TRAPS
names = [a for a in args[1:] if not a.startswith("--")]
return print_lint(lesson_lint(LESSONS if not names else [x for x in LESSONS if x["objective"] in names or x["domain"] in names]))
st = load_state()
live = lesson_of(st) if st and not st.get("ended_at") else None
if sub == "next":
@ -1833,9 +1794,9 @@ def cmd_lesson(args):
if sub == "step":
return print_step(live, params, i, st)
if sub in ("hint", "show"):
print(bold(f"{'Hint' if sub == 'hint' else 'How to do it'} - step {i + 1}: {render(s['title'], params, live)}"))
text = render(s["hint"] if sub == "hint" else show_md(s), params, live)
print(cli_md(text) or dim("Nothing for this step."))
print(bold(f"{'Hint' if sub == 'hint' else 'The commands'} - step {i + 1}: {render(s['title'], params, live)}"))
text = render(s[sub], params, live)
print((cli_md(text) if sub == "hint" else text) or dim("Nothing for this step."))
return
r, why = lesson_check(st, i)
if why:
@ -1879,51 +1840,6 @@ def cmd_lesson(args):
print(dim(f"\n ./ksim lesson {LESSONS[0]['objective']} starts that lesson ./ksim lesson audit proves them on real clusters"))
print(dim(f" ./ksim lesson {LESSONS[0]['domain']} a domain's lessons not done yet, back to back ({runs} done)"))
# What a step must not ask you to type: in the exam you have a keyboard, vim and the docs, and no
# time to compose scripts. Such things belong in the script of a part done by hand (sim/bank.py).
TYPING_TRAPS = [(re.compile(p, re.M), why) for p, why in (
(r"\b(sed|awk|yq|jq|perl|python3?)\b", "a text tool - edit in vim, or read the value and echo it"),
(r"<<", "a heredoc - generate the YAML, or paste the docs' example into vim"),
(r"^\s*(sleep|for|while|until)\b", "a pause or a loop - look again yourself"),
(r"\b(k|kubectl)\b[^|\n]* wait\b", "kubectl wait - look with get"),
(r"--overrides", "--overrides - generate the YAML and edit it"),
(r"\bpatch\b[^\n]*(-p|--patch)[ =]'[\[{]", "a JSON patch - kubectl edit"),
(r"jsonpath[^\n|]*>", "jsonpath into a file - read the value, then echo it"),
(r"\$\(", "command substitution - read the value, then type it"),
(r"(^|[|;&(]\s*)(sudo\s+)?kubectl\b", "kubectl in full - the exam has k"))] # the command, not the package
def lesson_lint(ls):
"""What each step has you type, against TYPING_TRAPS. Returns [(lesson, step number, why, line)]."""
out = []
for l in ls:
for i, s in enumerate(l["steps"]):
for p in show_parts(s):
for line in p["type"].splitlines():
for rx, why in TYPING_TRAPS:
if rx.search(line):
out.append((l, i + 1, why, line.strip()))
return out
def print_lint(problems):
for l, n, why, line in problems:
print(f" {yellow(l['objective'] + ' step ' + str(n))} {why}\n {dim(line[:150])}")
steps = len({(l["id"], n) for l, n, _, _ in problems})
print(bold(f"{steps} step(s) ask you to type what you would not type in the exam.") if steps
else green("Nothing a step asks you to type is a script: every step is done as in the exam."))
def solution_lint(qs):
"""What each task's reference solution has you type, its comments aside, against TYPING_TRAPS.
Returns [(task, why, line)]."""
return [(q, why, line.strip()) for q in qs for p in solution_parts(q) for line in p["type"].splitlines()
if not line.lstrip().startswith("#") for rx, why in TYPING_TRAPS if rx.search(line)]
def print_solution_lint(problems):
for q, why, line in problems:
print(f" {yellow('Q' + str(q['id']))} {why}\n {dim(line[:150])}")
tasks = len({q["id"] for q, _, _ in problems})
print(bold(f"{tasks} solution(s) ask you to type what you would not type in the exam.") if tasks
else green("Nothing a solution asks you to type is a script: every one is done as in the exam."))
def lesson_audit(args):
"""Prove lessons on real clusters: each is seeded on a host of its own; then every step must
fail its check, have its commands run as `candidate`, and pass its check - so a lesson can be
@ -1946,8 +1862,6 @@ def lesson_audit(args):
open(lock, "w").close()
import atexit
atexit.register(lambda: os.path.exists(lock) and os.remove(lock))
print(bold(f"What the {len(ls)} lesson(s) have you type:"))
print_lint(lesson_lint(ls))
print(bold(f"Lesson audit of {len(ls)} {TRACK_ID.upper()} lesson(s), {PARALLEL} at a time - each step must fail, be done by its own commands, then pass."))
def one(l):
params = make_params(l, random.Random(l["id"])); host = host_of(l); t0 = time.time()
@ -1969,7 +1883,7 @@ def lesson_audit(args):
if check_step(l, params, i)["pass"]:
res.update(verdict=f"step {i + 1} passes before its commands")
break
_, sout = solve_on_host(l, params, script=show_script(s))
_, sout = solve_on_host(l, params, script=s["show"])
# what the commands created may still be starting: a learner would watch it come up
r, waited = check_step(l, params, i), 0
while not r["pass"] and waited < 90:
@ -2241,24 +2155,13 @@ def cmd_history(args):
cells.append(f"{round(100 * de / dt):>3}%" if dt else " -")
print(f" {d:<7} " + " → ".join(cells))
def rebuild(st, q, log: Callable[..., Any] = _log_msg):
"""A task's or a lesson's host again from scratch, scenario included. A lesson's steps start over
on it; a host that comes up whole is no longer listed as broken. Returns what went wrong, or None."""
problem = provision(q, session_params(st, q), log=log)
st = load_state() or st
if q.get("lesson"):
st.get("steps", {}).pop(str(q["id"]), None)
st["broken"] = [i for i in st.get("broken", []) if i != q["id"]] + ([q["id"]] if problem else [])
save_state(st)
return problem
def cmd_reset(args):
"""Throw a task's cluster away and build it again, scenario included."""
if not args:
die("usage: ./ksim reset N (rebuild one task's host from scratch: new cluster, scenario seeded again)")
st = require_started(); q = q_by_id(int(args[0]))
print(dim(f" rebuilding {host_of(q)} - about a minute..."))
problem = rebuild(st, q)
problem = provision(q, session_params(st, q))
print(("rebuilt " if not problem else "problem rebuilding ") + f"Q{q['id']:02d}: {q['title']}")
# --------------------------------------------------------------- web ----
@ -2427,9 +2330,7 @@ class Api(http.server.BaseHTTPRequestHandler):
return self._json({"error": down_message(down)}, 503)
finish_session(st, log=lambda *_: None)
elif p.path == "/api/reset":
problem = rebuild(st, q_by_id(int(body["id"])), log=lambda *_: None)
if problem:
return self._json({"error": f"Rebuilding failed again ({problem}) - try ./ksim reset {body['id']} in a terminal to see why."}, 500)
q = q_by_id(int(body["id"])); provision(q, session_params(st, q), log=lambda *_: None)
else:
return self._json({"error": "not found"}, 404)
return self._json(self.session_payload())
@ -2467,7 +2368,6 @@ class Api(http.server.BaseHTTPRequestHandler):
"paused": bool(st.get("paused_at")), "paused_total": paused_total(st),
"notes": st.get("notes", ""),
"objective": st.get("objective"), "milestones": st.get("milestones", []), "aborted": bool(st.get("aborted")),
"broken": st.get("broken", []),
"results": st["results"], "score": {"earned": earned, "total": total, "pct": pct, "pass_pct": PASS_PCT},
"domains": self.domain_scores(st), "server_time": time.time()}
l = lesson_of(st)
@ -2489,7 +2389,7 @@ class Api(http.server.BaseHTTPRequestHandler):
r = lambda t: render(t, params, l).strip("\n")
return {"id": l["id"], "objective": l["objective"], "title": l["title"], "host": host_of(l), "nodes": 1 + workers_of(l),
"intro": r(l["intro"]), "outro": r(l["outro"]),
"steps": [{"title": r(s["title"]), "text": r(s["text"]), "hint": r(s["hint"]), "show": r(show_md(s))} for s in l["steps"]]}
"steps": [{"title": r(s["title"]), "text": r(s["text"]), "hint": r(s["hint"]), "show": r(s["show"])} for s in l["steps"]]}
def path_payload(self) -> dict[str, Any]:
"""The curriculum as the UI shows it: every objective with its notes, tasks and state, how
@ -2799,8 +2699,7 @@ def cmd_help(args):
" lesson DOMAIN [--all] teach a domain: its lessons not done yet (--all: every one) back to back, hosts made up front\n"
" lesson next | OBJ in a domain's run: on to its next lesson, or to any lesson of it\n"
" lesson step|check|hint|show [N] the step you are on, check it, a hint, the commands; ./ksim end finishes\n"
" lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after them\n"
" lesson lint [OBJ|DOMAIN..] what the steps have you type: no scripts, only what you would type in the exam\n")
" lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after them\n")
print(f"""usage: ./ksim [--track cka|ckad|cks] <command> (current track: {TRACK_ID}, {len(QUESTIONS)} tasks)
Every task runs on its own cluster at the exam's Kubernetes version ({TRACK_ID}: {k8s_version()}),
reached with `ssh <host>` from the remote desktop.
@ -2815,7 +2714,6 @@ def cmd_help(args):
whatever else you named: ./ksim next OBS:3
progress attempted/passed per domain, normal and hard
solution N reference solution (drills: after grading N; exams: after submit)
solution lint [N|DOMAIN..] what the solutions have you type: no scripts, only what you would type in the exam
sim {'|'.join(SETS) or '<set>'}{' ' * max(1, 20 - len('|'.join(SETS)))}fixed simulator set, exam rules, {TRACK['minutes']} min
start [--hard] full exam: {sum(QUOTA.values()) or '?'} tasks sampled by domain ({sum(QUOTA_HARD.values()) or '?'} in hard mode), {TRACK['minutes']} min.
Requires every domain drilled once first (--force skips). Hard mode adds traps,

16
sim/bank.py
View File

@ -21,22 +21,10 @@ def C(name, pts, script):
def step(title, text, check, hint="", show=""):
"""One step of a lesson (<track>/lessons.py): what to do (Markdown), the check that proves it
(bash, exit 0 = done, run like a task's checks), a hint, and how to do it as you would in the
exam - shown on request, and done by ./ksim lesson audit to prove the step works as shown.
`show` is commands you type as they stand, or a list of cmd() and hand() parts."""
(bash, exit 0 = done, run like a task's checks), a hint, and the commands that do it - shown on
request, and run by ./ksim lesson audit to prove the step can be done as shown."""
return {"title": title, "text": text, "check": check, "hint": hint, "show": show}
def cmd(text, how=""):
"""Commands you type as they stand, and what to look for (Markdown); the audit runs them as
they stand."""
return {"type": text, "how": how, "run": text}
def hand(type_, how, run):
"""What you do by hand: what you type to begin (`vim f`, `k edit ...`, `echo NAME > f`), what
to do there (Markdown), and a script that does the same, which the audit runs instead - an
editor needs a person, and so does a value read off the screen."""
return {"type": type_, "how": how, "run": run}
def ns_pick(r, base):
"""Namespace name with a random suffix so the same task never lands on the same namespace twice."""
return f"{base}-{r.choice(['alpha', 'bravo', 'delta', 'echo', 'kilo', 'nova', 'omega', 'zulu'])}"

44
web/index.html
View File

@ -607,10 +607,8 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
// minimal markdown: paragraphs, **bold**, `code` (click to copy), ```fenced blocks``` (kept as
// written, click to copy), and a paragraph made only of "- " lines as a list. Other lists stay as
// literal lines.
// `code` first, so that the stars of a cron schedule in it are not read as **bold** or *italics*
const inlineMd = s => s.split(/(`[^`]+`)/).map((p, i) => i % 2
? `<code class="ic" title="click to copy">${esc(p.slice(1, -1))}</code>`
: esc(p).replace(/\*\*(.+?)\*\*/g, '<b>$1</b>').replace(/(^|[^\w*])\*(?!\s)([^*]+?)\*(?![\w*])/g, '$1<i>$2</i>')).join('');
const inlineMd = s => esc(s).replace(/\*\*(.+?)\*\*/g, '<b>$1</b>')
.replace(/`([^`]+)`/g, (m, c) => `<code class="ic" title="click to copy">${c}</code>`);
const render = md => md.split(/^```[^\n]*\n([\s\S]*?)\n?^```[ \t]*$/m).map((part, i) => i % 2
? `<pre class="code" title="click to copy">${esc(part)}</pre>`
: part.split(/\n\s*\n/).filter(p => p.trim()).map(p => /^(- .*(\n|$))+$/.test(p.trim() + '\n')
@ -927,7 +925,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
$('#startBody').innerHTML = `<div class="alert alert-info"><b>No clock.</b> Take your time: the lessons teach, the drills test.</div>
<ul>
<li>every lesson has a host of its own, and all ${n} are up already: the top of each lesson says which to <code class="ic">ssh</code> to;</li>
<li>a lesson is an introduction, then steps - each with a <b>Check step</b> button, a hint, and <b>Show me how</b>: the exam way, if you are stuck; steps count in order;</li>
<li>a lesson is an introduction, then steps - each with a <b>Check step</b> button, a hint, and the commands if you are stuck; steps count in order;</li>
<li>a lesson's wrap-up leads to the next one, and the list at the top goes to any lesson of the run;</li>
<li>a lesson counts as done the moment its last step is checked; nothing goes into your history or scores;</li>
<li>finish the run from <b>Controls</b>, then drill ${esc(C.domain)} against the clock.</li>
@ -941,7 +939,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
$('#startBody').innerHTML = `<div class="alert alert-info"><b>No clock.</b> Take your time: a lesson teaches, the drills test.</div>
<ul>
<li>it runs on a host of its own: <code class="ic">${esc('ssh ' + L.host)}</code> in the desktop's terminal;</li>
<li>read the introduction, then do the ${L.steps.length} steps - each has a <b>Check step</b> button, a hint, and <b>Show me how</b>: the exam way, if you are stuck;</li>
<li>read the introduction, then do the ${L.steps.length} steps - each has a <b>Check step</b> button, a hint, and the commands if you are stuck;</li>
<li>steps count in order, and nothing goes into your history or scores;</li>
<li>the wrap-up sends you to the objective's timed drill.</li>
</ul>`;
@ -1017,7 +1015,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
const r = sess.results[q.id], show = r && (sess.mode === 'drill' || sess.ended_at);
const oids = q.objectives || [], oid = oids.includes(sess.objective) ? sess.objective : oids[0];
const obj = oid && (S.objectives || []).find(o => o.id === oid); // which objective a task belongs to is a hint: it follows the solution's rule
$('#qContent').innerHTML = brokenHost(q.id) + `
$('#qContent').innerHTML = `
<div class="qhead">
<div class="line"><b>Solve this question on instance:</b> <code class="ic" title="click to copy">ssh ${esc(q.host)}</code></div>
<div class="line sub">${q.nodes}-node cluster · task files in ${esc(S.workspace)} on that host</div>
@ -1030,10 +1028,9 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
<div class="qbody">${render(q.task)}</div>
${on && sess.mode === 'drill' ? '<div class="checkrow"><button class="btn btn-outline-primary btn-sm" id="btnCheck">Check this task</button><span class="muted">grades it per check; its reference solution opens once it is graded</span></div>' : ''}
${show ? `<div class="checks"><b>${r.earned}/${r.total} points</b>${r.checks.map(c => `<div class="${c.pass ? 'ok' : 'no'}">${esc(c.name)} <span class="pts">(${c.pts})</span></div>`).join('')}</div>` : ''}
${q.solution ? `<details class="fold"><summary>Reference solution</summary><div class="qbody" style="margin-top:10px">${render(q.solution)}</div></details>` : ''}
${q.solution ? `<details class="fold"><summary>Reference solution</summary><pre>${esc(q.solution)}</pre></details>` : ''}
${obj ? `<details class="fold"><summary>Objective ${obj.id} · ${esc(obj.text)}</summary><div class="notes">${notesHtml(obj)}</div></details>` : ''}`;
copyable($('#qContent'));
wireRebuild();
const bc = $('#btnCheck'); if (bc) bc.onclick = checkTask;
const fl = sess.flags.includes(q.id);
$('#btnFlag').className = 'btn ' + (fl ? 'btn-warning' : 'btn-outline-warning') + (on ? '' : ' hidden');
@ -1041,24 +1038,6 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
$('#btnSatisfied').disabled = i >= n - 1;
paintDesk();
}
// a host whose setup failed when the session started: say so where it is used, and offer to rebuild it
const brokenHost = id => live() && (S.session.broken || []).includes(id)
? `<div class="alert alert-danger"><b>This host's setup failed</b> when the session started, so it is not as the text describes.
<div style="margin-top:.5rem"><button class="btn btn-danger btn-sm" id="btnRebuild" data-id="${id}">Rebuild this host</button>
<span class="muted"> about a minute - or <code class="ic">./ksim reset ${id}</code> in a terminal</span></div></div>` : '';
function wireRebuild() {
const b = $('#btnRebuild'); if (!b) return;
b.onclick = async () => {
$('#busyTitle').textContent = 'Rebuilding the host';
$('#busyBody').innerHTML = '<p class="mb0">A new cluster, and its setup run again - about a minute.</p>';
openModal('#mBusy');
try { S = await api('/api/reset', {id: +b.dataset.id}); }
catch (e) { alert(e.message); }
finally { closeModal('#mBusy'); }
if (isLesson()) lpage = null;
paintExam();
};
}
// what the exam and a lesson share: the notepad, the remote desktop, the clock
function paintDesk() {
if (document.activeElement !== $('#notes')) $('#notes').value = S.session.notes || '';
@ -1122,7 +1101,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
+ C.lessons.map(x => { const c = x.steps.filter(s => s.pass).length;
return `<option value="${x.id}"${x.id === C.at ? ' selected' : ''}>${c === x.steps.length ? '✓' : '·'} ${esc(x.objective)} ${esc(x.title)} (${c}/${x.steps.length})</option>`; }).join('')
+ '</select></div>';
let body = brokenHost(L.id) + `<div class="qhead">${pick}<div class="line"><b>This lesson runs on:</b> <code class="ic" title="click to copy">${esc('ssh ' + L.host)}</code></div>
let body = `<div class="qhead">${pick}<div class="line"><b>This lesson runs on:</b> <code class="ic" title="click to copy">${esc('ssh ' + L.host)}</code></div>
<div class="line sub">${L.nodes}-node cluster · files in ${esc(S.workspace)} on that host · no clock</div></div><hr>`;
if (lpage === 0) {
body += `<div class="stepsum">Lesson ${esc(L.objective)} · ${n} steps</div><h2 class="qtitle">${esc(L.title)}</h2><div class="qbody">${render(L.intro)}</div>`;
@ -1141,11 +1120,10 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
+ (r && !r.pass && r.out ? `<details class="fold"><summary>What the check saw</summary><pre>${esc(r.out)}</pre></details>` : '');
} else if (r && r.pass) body += '<div class="lstate"><span class="ok">✓ Done</span></div>';
if (s.hint) body += `<details class="fold"><summary>Hint</summary><div class="qbody" style="margin-top:10px">${render(s.hint)}</div></details>`;
if (s.show) body += `<details class="fold"><summary>Show me how</summary><div class="qbody" style="margin-top:10px">${render(s.show)}</div></details>`;
if (s.show) body += `<details class="fold"><summary>Show me the commands</summary><pre class="code" title="click to copy" style="margin:10px 0 0">${esc(s.show)}</pre></details>`;
}
$('#qContent').innerHTML = body;
copyable($('#qContent'));
wireRebuild();
const bs = $('#btnStep'); if (bs) bs.onclick = () => checkStep(lpage - 1);
const bf = $('#btnFinish'); if (bf) bf.onclick = finishLesson;
const bn = $('#btnNextLesson'); if (bn) bn.onclick = () => goLesson(nx.id);
@ -1316,9 +1294,8 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
const cls = r.pass ? 't-pass' : (r.earned ? 't-partial' : 't-fail');
return `<div class="qcard ${cls}"><div class="qcard-h"><h3>Question ${i + 1} · Q${q.id} ${esc(q.title)}</h3><span class="qs">${r.earned} of ${r.total}</span></div>
<ul class="vitems">${r.checks.map(c => `<li class="vitem"><span>${esc(c.name)} <span class="muted">(${c.pts})</span></span><span class="vstat ${c.pass ? 'ok' : 'no'}">${c.pass ? '✓' : '✗'}</span></li>`).join('')}</ul>
${q.solution ? `<details class="fold"><summary>Reference solution</summary><div class="qbody" style="margin-top:10px">${render(q.solution)}</div></details>` : '<div class="pad"></div>'}</div>`;
${q.solution ? `<details class="fold"><summary>Reference solution</summary><pre>${esc(q.solution)}</pre></details>` : '<div class="pad"></div>'}</div>`;
}).join('');
copyable($('#resQuestions'));
const missed = S.questions.filter(q => s.results[q.id] && !s.results[q.id].pass).map(q => q.id);
$('#btnDrillMissed').disabled = !missed.length;
$('#btnDrillMissed').onclick = () => start('drill', missed);
@ -1326,9 +1303,8 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
function showAnswers() {
const s = S.session;
$('#answersBody').innerHTML = S.questions.filter(q => q.solution).map((q, i) =>
`<h4>Question ${S.questions.indexOf(q) + 1} · Q${q.id} ${esc(q.title)}${s && s.results[q.id] ? ` <span class="muted">(${s.results[q.id].earned}/${s.results[q.id].total})</span>` : ''}</h4><div class="qbody">${render(q.solution)}</div>`).join('')
`<h4>Question ${S.questions.indexOf(q) + 1} · Q${q.id} ${esc(q.title)}${s && s.results[q.id] ? ` <span class="muted">(${s.results[q.id].earned}/${s.results[q.id].total})</span>` : ''}</h4><pre>${esc(q.solution)}</pre>`).join('')
|| '<p class="muted">No reference solutions are open yet.</p>';
copyable($('#answersBody'));
openModal('#mAnswers');
}