vumast3r/kubestronaut-sim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: show every CKA lesson step the way the exam is done

Browse Source 
Show me how gave the commands the audit runs: sed, awk and yq edits,
heredoc manifests, JSON patches, --overrides, sleeps and wait loops -
nothing anyone types in the exam. A step's show is now a list of
parts (sim/bank.py): cmd() is what you type as it stands; hand() is
what you do by hand - vim on a generated skeleton or on a manifest
pasted from the docs page it names, kubectl edit, ssh NODE and sudo
-i, a value you read and echo - with a script that does the same for
the audit. All 135 CKA steps are rewritten so, with k for kubectl;
their checks, titles and texts are unchanged.

Add ./ksim lesson lint, which reads what every step has you type and
flags scripts. The lesson audit runs it first.

Record the hosts whose setup failed when a session started. The page
warns above that task or lesson and offers Rebuild this host; the CLI
names ./ksim reset N on each step.

Render *italics* in lessons and tasks, outside code spans.

Audited on v1.37: CKA lessons 27/27, 135/135 steps, done the exam
way; lint clean. kubectl edit --subresource resize, the one edit the
audit replaces with a patch, was proven on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
vumas3r 2026-10-26 20:34:36 -06:00
parent fe86b4be35
commit 226ef19ff9
11 changed files with 1620 additions and 715 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
CHANGELOG.md
View File

@ -4,6 +4,30 @@ Notable changes to kubestronaut-sim, newest first. A version is the `VERSION` st
(year.month.day.n); `./ksim help` and the top bar of the web page show the one you are running.
Every entry names the commit it came in.
## Unreleased
### Changed
- **Every CKA lesson step shows the exam way** (*Show me how*, `./ksim lesson show`): `k`, a
generator and the few lines to add in `vim`, a manifest pasted from the docs page it names,
`kubectl edit`, `ssh NODE` and `sudo -i`, or a value you read and `echo` - never a script. Each
command is a block of its own, which a click copies. The scripts the audit needs (`yq`, `sed`,
waits) are kept out of sight, one per part done by hand, so `./ksim lesson audit` still proves
every step: 135 of 135 on v1.37.
### Added
- `./ksim lesson lint`: what each step has you type, against what you would not type in the exam
(`sed`, `awk`, `yq`, heredocs, loops, `kubectl wait`, `--overrides`, JSON patches, `jsonpath`
into a file, `kubectl` in full). The lesson audit runs it first.
- A task's or a lesson's host whose setup failed when the session started is marked: the page
says so above it, with **Rebuild this host**, and `./ksim lesson` names `./ksim reset N`. Before,
only the log of the preparing window said so.
### Fixed
- The page shows `*italics*` in lessons and tasks instead of the stars.
## 2026.10.26.3 — 2026-10-26
Everything below came in `95f9d1b`.

23
README.md
View File

@ -155,7 +155,8 @@ host name such as `ckad0013` only resolves in the desktop's terminal: from your
**Lessons.** A lesson teaches one objective as a guided lab on a host of its own: an introduction,
then steps you do in the desktop's terminal, each with a **Check step** button, a hint and **Show me
the commands**, and a wrap-up that sends you to the objective's timed drill. Lessons are the one
how** (the exam way: what you type, what you edit in vim), and a wrap-up that sends you to
the objective's timed drill. Lessons are the one
untimed part of the simulator — they teach, the drills test — and they write nothing to the
history, the statistics or the weak list; a lesson whose every step was checked is marked done on
its objective (`lesson ✓` in the objective list). Steps count in order. Start one from an
@ -295,13 +296,19 @@ with a new Kubernetes release fails the next audit instead of misleading you aft
`intro`, `setup` (bash, as a task's), `steps` and `outro` - and, like a task, `k8s` or `image` if
it needs a host of its own kind (lesson 4.4 runs on v1.36 to upgrade it). Each step comes from
`step(title, text, check, hint, show)` in `sim/bank.py`: `check` runs like a task's check, and
`show` is what the learner would type. `./ksim lesson audit` seeds each lesson on its own host and,
step by step, requires the check to fail, runs `show` there as `candidate`, and requires the check
to pass — so a lesson can always be done exactly as it shows, and no step is done before you do it.
Work on another node is shown as `ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain
`ssh NODE` would read the rest of the commands as its own input. **As audited on 2026-10-19**
(v1.37, Ubuntu hosts): CKA's 27 lessons, 135 of 135 steps, and CKAD 2.1–2.4, 24 of 24 steps,
proven.
`show` is how you would do the step in the exam, as parts. `cmd(text)` is what you type as it
stands. `hand(text, how, run)` is what you do by hand: what you type to begin (`vim role.yaml`,
`k edit cm app-config`, `ssh NODE` and `sudo -i`, `echo NAME > file`), what to do there, and a script
that does the same unattended. `./ksim lesson audit` seeds each lesson on its own host and, step by
step, requires the check to fail, runs the commands (and each hand part's script) there as
`candidate`, and requires the check to pass, so a lesson can always be done as it shows, and no
step is done before you do it. `./ksim lesson lint` reads what every step has you type and refuses
scripts: no `sed`, `awk`, `yq`, heredocs, loops, `kubectl wait`, `--overrides`, JSON patches or
`jsonpath` into a file, and `k` rather than `kubectl`. In a script, a command on another node is
`ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain `ssh NODE` would read the rest of
the script as its own input. **As audited on 2026-10-26** (v1.37, Ubuntu hosts): CKA's 27 lessons,
done the exam way, 135 of 135 steps proven, and none has you type a script. CKAD 2.1–2.4 (24
steps, audited on 2026-10-19) still show their commands as scripts.
## What is different from the real exam

13
cka/lessons.py
View File

@ -10,11 +10,14 @@ Lessons write nothing to the history or the scores - only a "lesson done" mark o
./ksim lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after
A lesson's host is named after its id - lesson 9101 runs on `cka9101` - which never clashes with a
task's. Its files go under {WS}/lesson-<objective>/. A step's `show` commands are what the learner
types; the audit runs exactly them, as `candidate` on the host, so what a lesson shows always
works. Work on another node is shown as `ssh NODE sudo bash <<'EOF' ... EOF`: the same commands
you would type after `ssh NODE` and `sudo -i`, in a form that runs unattended. A single command
there is `ssh -n NODE 'COMMAND'`: without -n, ssh would read the rest of the commands as its input.
task's. Its files go under {WS}/lesson-<objective>/. A step's `show` is how you would do it in the
exam: cmd() for what you type as it stands, hand() for what you do by hand - `vim` a generated
skeleton or a manifest pasted from the docs, `kubectl edit`, `ssh NODE` and `sudo -i`, a value you
read and `echo` - each with a script that does the same unattended. The audit runs the commands
as `candidate` on the host and, for each hand() part, its script; `./ksim lesson lint` keeps
scripts (sed, awk, yq, heredocs, loops, JSON patches) out of what you type. In a script, a command
on another node is `ssh NODE sudo bash <<'EOF' ... EOF` or `ssh -n NODE 'COMMAND'`: without -n,
ssh would read the rest of the script as its input.
One file per domain, as with the task banks. Besides the curriculum's competencies, the lessons
cover what a full CKA course teaches under each of them (2026-10-19: the topics of KubeDojo's CKA

541
cka/lessons_arch.py
View File

@ -1,7 +1,7 @@
"""CKA lessons of the Cluster Architecture, Installation and Configuration domain (4.1-4.8). See
cka/lessons.py for what a lesson is."""
from cka.common import backup_operator
from sim.bank import CAN, NS, PY, step
from sim.bank import CAN, NS, PY, step, cmd, hand
_ETCDCTL = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
"--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key")
@ -38,23 +38,20 @@ S=system:serviceaccount:team:deployer
can --as=$S -- create deployments.apps -n team && can --as=$S -- list pods -n team && cannot --as=$S -- delete pods -n team \
&& cannot --as=$S -- create deployments.apps -n default""",
hint=r"""`kubectl create role deploy-edit --verb=... --resource=deployments.apps` - a Role with two resources and different verbs needs two rules, so write YAML or create it and `kubectl edit`. `kubectl create rolebinding NAME --role=ROLE --serviceaccount=team:deployer`.""",
show=r"""kubectl -n team create serviceaccount deployer
kubectl -n team apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deploy-edit
rules:
- apiGroups: [apps]
resources: [deployments]
verbs: [get, list, watch, create, update, patch, delete]
show=[cmd(r"""k -n team create sa deployer
k -n team create role deploy-edit --verb=get,list,watch,create,update,patch,delete --resource=deployments.apps --dry-run=client -o yaml > role.yaml"""),
hand(r"""vim role.yaml""",
r"""`kubectl create role` gives every resource the same verbs: add the second rule, for Pods, at the end of `rules:`:
```
- apiGroups: [""]
resources: [pods]
verbs: [get, list]
EOF
kubectl -n team create rolebinding deployer --role=deploy-edit --serviceaccount=team:deployer
kubectl auth can-i create deployments -n team --as=system:serviceaccount:team:deployer
kubectl auth can-i delete pods -n team --as=system:serviceaccount:team:deployer"""),
```""",
run=r"""yq -i '.rules += [{"apiGroups": [""], "resources": ["pods"], "verbs": ["get", "list"]}]' role.yaml"""),
cmd(r"""k apply -f role.yaml
k -n team create rolebinding deployer --role=deploy-edit --serviceaccount=team:deployer
k auth can-i create deploy -n team --as=system:serviceaccount:team:deployer
k auth can-i delete po -n team --as=system:serviceaccount:team:deployer""", r"""`yes`, then `no`.""")]),
step("One ClusterRole, one Namespace",
r"""Create ClusterRole `pod-reader` (`get`, `list`, `watch` on pods). Give it to the user **`jane`** in Namespace `team` only - with a **RoleBinding** `jane-reads-pods`, not a ClusterRoleBinding.
@ -64,10 +61,10 @@ eq "$(j clusterrole pod-reader '{.rules[0].verbs}')" '["get","list","watch"]' ||
eq "$(j -n team rolebinding jane-reads-pods '{.roleRef.kind}/{.roleRef.name}')" ClusterRole/pod-reader || exit 1
can --as=jane -- list pods -n team && cannot --as=jane -- list pods -n default && cannot --as=jane -- list pods -A""",
hint=r"""`kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods`, then `kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane`. Users exist only in certificates and tokens: nothing to create for `jane`.""",
show=r"""kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods
kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane
kubectl auth can-i list pods -n team --as=jane
kubectl auth can-i list pods -n default --as=jane"""),
show=[cmd(r"""k create clusterrole pod-reader --verb=get,list,watch --resource=pods
k -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane
k auth can-i list po -n team --as=jane
k auth can-i list po -n default --as=jane""", r"""`yes` in `team`, `no` elsewhere.""")]),
step("A token for the ServiceAccount",
r"""Issue a token for `deployer` that is valid for **one hour**, and save it in `{WS}/lesson-4.1/deployer.token`. Then use it - as the **only** credential: your kubeconfig carries the admin's client certificate, and the API server checks a certificate before a token, so `kubectl --token=...` alone would still be you. With no kubeconfig, just the server, the CA and the token, `kubectl auth whoami` says `system:serviceaccount:team:deployer`, `get deployments` works in `team` and `get secrets` is forbidden.""",
check=r"""
@ -79,44 +76,41 @@ import base64, json, sys
s = sys.stdin.read().strip(); c = json.loads(base64.b64decode(s + "=" * (-len(s) % 4)))
sys.exit(0 if c["exp"] - c["iat"] == 3600 else 1)'""",
hint=r"""`kubectl -n team create token deployer --duration=1h`. Then `kubectl --kubeconfig=/dev/null --server=URL --certificate-authority=/etc/kubernetes/pki/ca.crt --token=TOKEN ...`; the URL is in your kubeconfig. The token is a JWT: its middle part, base64-decoded, holds `sub`, `iat` and `exp`.""",
show=r"""kubectl -n team create token deployer --duration=1h > {WS}/lesson-4.1/deployer.token
server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
as_deployer() { kubectl --kubeconfig=/dev/null --server=$server --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$(cat {WS}/lesson-4.1/deployer.token) "$@"; }
as_deployer auth whoami
as_deployer -n team get deployments
as_deployer -n team get secrets || true"""),
show=[cmd(r"""k -n team create token deployer --duration=1h > {WS}/lesson-4.1/deployer.token
k config view --minify | grep server"""),
hand(r"""k --kubeconfig=/dev/null --server=SERVER --certificate-authority=/etc/kubernetes/pki/ca.crt --token=TOKEN auth whoami""",
r"""`SERVER`: the address above; `TOKEN`: the file's content (`cat` it, copy it). With `-n team get deploy` instead of `auth whoami` it works; `get secrets` is forbidden.""",
run=r"""server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
kubectl --kubeconfig=/dev/null --server=$server --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$(cat {WS}/lesson-4.1/deployer.token) auth whoami""")]),
step("A Pod without a token",
r"""Most Pods never talk to the API server, and do not need a token in their filesystem. Create ServiceAccount `quiet` in `team` with `automountServiceAccountToken: false`, and run Pod `notoken` (`busybox:1.36`, `sleep 86400`) as that ServiceAccount. Inside it, `/var/run/secrets/kubernetes.io/serviceaccount` does not exist.""",
check=r"""
eq "$(j -n team sa quiet '{.automountServiceAccountToken}')" false && eq "$(j -n team pod notoken '{.spec.serviceAccountName} {.status.phase}')" "quiet Running" || exit 1
! $K -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount >/dev/null 2>&1""",
hint=r"""The field sits at the top level of the ServiceAccount (and can be set on a Pod's `spec` too, which wins). The Pod names its ServiceAccount with `spec.serviceAccountName`.""",
show=r"""kubectl -n team apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: quiet
show=[cmd(r"""k -n team create sa quiet --dry-run=client -o yaml > quiet.yaml"""),
hand(r"""vim quiet.yaml""",
r"""Add, at the top level (no indent):
```
automountServiceAccountToken: false
---
apiVersion: v1
kind: Pod
metadata:
name: notoken
spec:
```""",
run=r"""yq -i '.automountServiceAccountToken = false' quiet.yaml"""),
cmd(r"""k apply -f quiet.yaml
k -n team run notoken --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > notoken.yaml"""),
hand(r"""vim notoken.yaml""",
r"""Add to the Pod's `spec:`, above `containers:`:
```
serviceAccountName: quiet
containers:
- name: app
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n team wait --for=condition=Ready pod/notoken --timeout=120s
kubectl -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount || echo 'no token mounted'"""),
```""",
run=r"""yq -i '.spec.serviceAccountName = "quiet"' notoken.yaml"""),
cmd(r"""k -n team apply -f notoken.yaml
k -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount""", r"""Once it runs: "No such file or directory" - no token.""")]),
step("Everything one subject may do",
r"""Write the complete list of what `deployer` may do in `team` to `{WS}/lesson-4.1/deployer-can.txt` - `kubectl auth can-i` prints it with `--list`. It includes what every authenticated subject gets from the built-in ClusterRoles, such as creating `selfsubjectreviews`.""",
check=r"""f="$WS/lesson-4.1/deployer-can.txt"; grep -q '^deployments.apps' "$f" 2>/dev/null && grep -q '^pods ' "$f" && grep -qi 'verbs' "$f" """,
hint=r"""`kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer`.""",
show=r"""kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer > {WS}/lesson-4.1/deployer-can.txt
cat {WS}/lesson-4.1/deployer-can.txt"""),
show=[cmd(r"""k auth can-i --list -n team --as=system:serviceaccount:team:deployer > {WS}/lesson-4.1/deployer-can.txt
cat {WS}/lesson-4.1/deployer-can.txt""")]),
],
"outro": r"""You gave a ServiceAccount a Role, reused a ClusterRole in one Namespace, issued and used a token, ran a Pod without one, and listed a subject's permissions.
@ -158,10 +152,18 @@ nx "$W1" 'rm -f /etc/modules-load.d/k8s.conf /etc/sysctl.d/k8s.conf' </dev/null
r"""On `{W1}`, make `overlay` and `br_netfilter` load at every boot: list them, one per line, in `/etc/modules-load.d/k8s.conf`.""",
check=r"""eq "$(nx "$W1" 'grep -v "^#" /etc/modules-load.d/k8s.conf 2>/dev/null | tr -d " " | grep . | sort | tr "\n" " "' </dev/null)" "br_netfilter overlay " """,
hint=r"""`printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/k8s.conf`. On a real machine you would also `modprobe` both now, and `lsmod | grep br_netfilter` would show it.""",
show=r"""ssh {W1} sudo bash <<'EOF'
show=[hand(r"""ssh {W1}
sudo -i
vim /etc/modules-load.d/k8s.conf""",
r"""Two lines (the docs page *Container Runtimes* has them, under *Install and configure prerequisites*):
```
overlay
br_netfilter
```
then `exit` twice, back to the control-plane node.""",
run=r"""ssh {W1} sudo bash <<'EOF'
printf 'overlay\nbr_netfilter\n' > /etc/modules-load.d/k8s.conf
cat /etc/modules-load.d/k8s.conf
EOF"""),
EOF""")]),
step("Kernel parameters, for good",
r"""On `{W1}`, put the three parameters Kubernetes needs in `/etc/sysctl.d/k8s.conf` - `net.bridge.bridge-nf-call-iptables = 1`, `net.bridge.bridge-nf-call-ip6tables = 1`, `net.ipv4.ip_forward = 1` - and apply every sysctl file now with `sysctl --system`.""",
check=r"""
@ -169,15 +171,29 @@ f=$(nx "$W1" 'cat /etc/sysctl.d/k8s.conf 2>/dev/null' </dev/null | tr -d ' ')
for p in net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1; do echo "$f" | grep -qx "$p" || exit 1; done
eq "$(nx "$W1" 'sysctl -n net.ipv4.ip_forward' </dev/null)" 1""",
hint=r"""A here-document into `sudo tee`, then `sudo sysctl --system` - it prints every file it reads, yours among them. (The bridge parameters only exist once `br_netfilter` is loaded.)""",
show=r"""ssh {W1} sudo bash <<'EOF'
show=[hand(r"""ssh {W1}
sudo -i
vim /etc/sysctl.d/k8s.conf""",
r"""Three lines (the same docs page has them):
```
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
```
then apply every sysctl file and look, still on `{W1}`:
```
sysctl --system
sysctl net.ipv4.ip_forward
```
and `exit` twice.""",
run=r"""ssh {W1} sudo bash <<'EOF'
cat > /etc/sysctl.d/k8s.conf <<'CONF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
CONF
sysctl --system 2>/dev/null | grep -A3 k8s.conf
sysctl net.ipv4.ip_forward
EOF"""),
sysctl --system >/dev/null 2>&1
EOF""")]),
step("Swap and the cgroup driver",
r"""Two more things kubeadm's preflight checks. Write to `{WS}/lesson-4.2/node.txt`, in this format:
@ -196,17 +212,30 @@ sc=$(nx "$W1" 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml |
want=$(printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s' "$swap" "${fso% *}" "${fso#* }" "$sc")
eq "$(tr -d ' \r' < "$WS/lesson-4.2/node.txt" 2>/dev/null | grep .)" "$want" """,
hint=r"""`cat /proc/swaps` (or `swapon --show`): any line below the header is swap. `grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml` - a field that is not there has its default (`failSwapOn` is `true`). `grep SystemdCgroup /etc/containerd/config.toml`; no such line means `false`.""",
show=r"""ssh {W1} sudo bash <<'EOF'
show=[hand(r"""ssh {W1}
sudo -i
cat /proc/swaps
grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml
grep -n SystemdCgroup /etc/containerd/config.toml || echo 'SystemdCgroup not set'
EOF
swap=$(ssh -n {W1} '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off')
grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml
grep SystemdCgroup /etc/containerd/config.toml""",
r"""- swap: `on` if `/proc/swaps` lists a device below its header line, `off` if it lists none;
- `failSwapOn`: `true` when the file does not set it - that is its default;
- `cgroupDriver`, and `SystemdCgroup` (`false` when it is not set).
Then `exit` twice.""",
run=r"""ssh -n {W1} 'sudo cat /proc/swaps; sudo grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml; sudo grep SystemdCgroup /etc/containerd/config.toml' """),
hand(r"""vim {WS}/lesson-4.2/node.txt""",
r"""The four lines, each `VALUE` what you found (`on` or `off`, `true` or `false`, `systemd` or `cgroupfs`):
```
swap=VALUE
failSwapOn=VALUE
kubelet-cgroup=VALUE
containerd-systemd-cgroup=VALUE
```""",
run=r"""swap=$(ssh -n {W1} '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off')
fso=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.failSwapOn)
cg=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver)
sc=$(ssh -n {W1} 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml | head -1 | tr -d " " | cut -d= -f2')
printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s\n' "$swap" "$fso" "$cg" "${sc:-false}" > {WS}/lesson-4.2/node.txt
cat {WS}/lesson-4.2/node.txt"""),
printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s\n' "$swap" "$fso" "$cg" "${sc:-false}" > {WS}/lesson-4.2/node.txt""")]),
step("The packages, as kubeadm wants them",
r"""On an exam node, kubeadm, kubelet and kubectl come from the repository of **one** minor at `pkgs.k8s.io`, and are **held** so that an `apt upgrade` never moves them by accident.
@ -215,9 +244,10 @@ Write to `{WS}/lesson-4.2/packages.txt`: the line of `{W1}`'s Kubernetes reposit
want=$(nx "$W1" 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' </dev/null | grep .)
[ -n "$want" ] && eq "$(grep . "$WS/lesson-4.2/packages.txt" 2>/dev/null | tr -d '\r')" "$want" """,
hint=r"""`cat /etc/apt/sources.list.d/kubernetes.list` and `apt-mark showhold`. `apt-cache madison kubeadm` lists the versions that repository offers; `dpkg -l | grep kube` what is installed.""",
show=r"""ssh -n {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt
cat {WS}/lesson-4.2/packages.txt
ssh -n {W1} 'dpkg -l | grep -E "kube(adm|let|ctl)"'"""),
show=[hand(r"""ssh {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt""",
r"""The repository line, then the three held packages.""",
run=r"""ssh -n {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt"""),
cmd(r"""cat {WS}/lesson-4.2/packages.txt""")]),
],
"outro": r"""You set up modules and sysctls that survive a reboot, checked swap and the cgroup drivers, and read how Kubernetes is installed on a node.
@ -259,9 +289,10 @@ nx "$W1" 'rm -f /etc/kubernetes/manifests/node-web.yaml' </dev/null
want=$($K -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}')
eq "$(tr -s ' \t' ' ' < "$WS/lesson-4.3/networks.txt" 2>/dev/null | tr -d '\r\n')" "$want" """,
hint=r"""`kubectl -n kube-system get cm kubeadm-config -o yaml` - look under `networking:`. `kubeadm config print init-defaults` shows every field such a file can have.""",
show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep -A4 networking
kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}' > {WS}/lesson-4.3/networks.txt
cat {WS}/lesson-4.3/networks.txt"""),
show=[cmd(r"""k -n kube-system get cm kubeadm-config -o yaml | grep -i subnet"""),
hand(r"""echo 'PODSUBNET SERVICESUBNET' > {WS}/lesson-4.3/networks.txt""",
r"""The two values, `podSubnet` first.""",
run=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}' > {WS}/lesson-4.3/networks.txt""")]),
step("A join command for a new worker",
r"""Create a new bootstrap token that expires in **2 hours**, and save the complete `kubeadm join` command for a worker in `{WS}/lesson-4.3/join.sh`, on one line. Run it on the control-plane node.""",
check=r"""
@ -270,9 +301,9 @@ id=$(grep -o -- '--token [a-z0-9]*\.[a-z0-9]*' "$f" | awk '{print $2}' | cut -d.
exp=$(j -n kube-system secret "bootstrap-token-$id" '{.data.expiration}' | base64 -d); [ -n "$exp" ] || exit 1
left=$(( $(date -d "$exp" +%s) - $(date +%s) )); [ "$left" -gt 6600 ] && [ "$left" -le 7300 ]""",
hint=r"""`sudo kubeadm token create --ttl 2h --print-join-command`. `kubeadm token list` shows the tokens and when they expire; each is a Secret `bootstrap-token-ID` in `kube-system`.""",
show=r"""sudo kubeadm token create --ttl 2h --print-join-command > {WS}/lesson-4.3/join.sh
show=[cmd(r"""sudo kubeadm token create --ttl 2h --print-join-command > {WS}/lesson-4.3/join.sh
cat {WS}/lesson-4.3/join.sh
sudo kubeadm token list"""),
sudo kubeadm token list""")]),
step("Renew the API server's certificate",
r"""Check the certificates' expiry dates, then **renew** the API server's serving certificate - only that one - and make the API server use it. A running process keeps the certificate it read at start, so the static Pod has to be restarted: move its manifest out of `/etc/kubernetes/manifests`, wait until the container is gone, and move it back.
@ -285,24 +316,32 @@ file=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -seri
served=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial' </dev/null)
[ -n "$file" ] && eq "$file" "$served" """,
hint=r"""`sudo kubeadm certs check-expiration`, `sudo kubeadm certs renew apiserver`. Restart: `sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/`, wait for `sudo crictl ps --name kube-apiserver` to be empty, move it back. Kubernetes renews the certificates itself on every `kubeadm upgrade`.""",
show=r"""sudo kubeadm certs check-expiration | head -8
show=[cmd(r"""sudo kubeadm certs check-expiration
sudo kubeadm certs renew apiserver
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name kube-apiserver -q)" ] && break; sleep 2; done
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
sudo kubeadm certs check-expiration | grep '^apiserver '"""),
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml ~/"""),
hand(r"""sudo crictl ps --name kube-apiserver""",
r"""Again until it lists nothing: the old API server is gone.""",
run=r"""for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name kube-apiserver -q)" ] && break; sleep 2; done"""),
cmd(r"""sudo mv ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
hand(r"""k get no""",
r"""Again until it answers: the API server is back with the new certificate.""",
run=r"""for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""sudo kubeadm certs check-expiration | grep apiserver""")]),
step("A static Pod on the worker",
r"""Every kubelet runs static Pods, not only the control plane's. On `{W1}`, find the kubelet's `staticPodPath` and put a manifest there for Pod `node-web` (`nginx:1.27`).
The API server shows it as a **mirror Pod** named after the node: `node-web-{W1}`. Deleting the mirror Pod with kubectl does nothing lasting - the kubelet puts it back; only removing the file does.""",
check=r"""eq "$(j -n default pod "node-web-$W1" '{.status.phase} {.metadata.annotations.kubernetes\.io/config\.source}')" "Running file" """,
hint=r"""`grep staticPodPath /var/lib/kubelet/config.yaml` on the node. `kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml` gives you the manifest; write it with `sudo tee`.""",
show=r"""kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml > {WS}/lesson-4.3/node-web.yaml
ssh -n {W1} 'grep staticPodPath /var/lib/kubelet/config.yaml'
ssh {W1} 'sudo tee /etc/kubernetes/manifests/node-web.yaml >/dev/null' < {WS}/lesson-4.3/node-web.yaml
sleep 15
kubectl get pods -o wide"""),
show=[cmd(r"""k run node-web --image=nginx:1.27 --dry-run=client -o yaml > node-web.yaml"""),
hand(r"""scp node-web.yaml {W1}:
ssh {W1}
sudo -i
grep staticPodPath /var/lib/kubelet/config.yaml
cp ~candidate/node-web.yaml /etc/kubernetes/manifests/""",
r"""The manifest goes into the `staticPodPath` directory; then `exit` twice.""",
run=r"""ssh {W1} 'sudo tee /etc/kubernetes/manifests/node-web.yaml >/dev/null' < node-web.yaml"""),
cmd(r"""k get po -o wide""", r"""Again in a few seconds: `node-web-{W1}`, the mirror Pod.""")]),
],
"outro": r"""You read the cluster's kubeadm configuration, made a join command, renewed a certificate and made the API server load it, and ran a static Pod on a worker.
@ -358,15 +397,14 @@ etcd only accepts clients with a certificate: give `etcdctl` the endpoint and th
nx "$CP" 'etcdutl snapshot status /opt/backup/etcd.db >/dev/null 2>&1' </dev/null && grep -qi 'hash' "$WS/lesson-4.4/snapshot.txt" 2>/dev/null \
&& grep -qi 'total key' "$WS/lesson-4.4/snapshot.txt" """,
hint=r"""`etcdctl --endpoints=https://127.0.0.1:2379 --cacert=.../ca.crt --cert=.../server.crt --key=.../server.key snapshot save FILE`, as root. Since etcd 3.6, `status` and `restore` are `etcdutl` only.""",
show=r"""sudo grep -E -- '--(listen-client-urls|cert-file|key-file|trusted-ca-file)' /etc/kubernetes/manifests/etcd.yaml
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup/etcd.db
sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/snapshot.txt"""),
show=[cmd(r"""sudo grep -e listen-client-urls -e cert-file -e key-file -e trusted-ca-file /etc/kubernetes/manifests/etcd.yaml""", r"""The endpoint, and the three files etcdctl needs: `--trusted-ca-file` is its `--cacert`, `--cert-file` its `--cert`, `--key-file` its `--key`."""),
cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup/etcd.db
sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/snapshot.txt""")]),
step("Lose something",
r"""Now the mistake a backup is for: delete ConfigMap `marker` in `life`.""",
check=r"""[ -z "$(j -n life cm marker '{.metadata.name}')" ] && nx "$CP" 'test -s /opt/backup/etcd.db' </dev/null""",
hint=r"""`kubectl -n life delete configmap marker`.""",
show=r"""kubectl -n life delete configmap marker"""),
show=[cmd(r"""k -n life delete cm marker""")]),
step("Restore the snapshot",
r"""Restore `/opt/backup/etcd.db` so that `marker` is back. Do it in place, so that etcd keeps its data directory `/var/lib/etcd`:
@ -377,13 +415,16 @@ sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/
Restoring into a **new** directory and pointing the manifest's `hostPath` at it works too, and is what many tasks ask for - but then the `dataDir` in kubeadm's configuration still names the old one, and the next `kubeadm upgrade` would write it back into the manifest.""",
check=r"""eq "$(j -n life cm marker '{.data.state}')" before-backup && nx "$CP" 'test -d /var/lib/etcd/member' </dev/null""",
hint=r"""`sudo etcdutl snapshot restore /opt/backup/etcd.db --data-dir /var/lib/etcd` - the target directory must not exist yet. `sudo crictl ps --name etcd` shows whether etcd is running.""",
show=r"""sudo mv /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml
for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name '^etcd$' -q)" ] && break; sleep 2; done
sudo mv /var/lib/etcd /var/lib/etcd.before-restore
show=[cmd(r"""sudo mv /etc/kubernetes/manifests/etcd.yaml ~/"""),
hand(r"""sudo crictl ps --name etcd""",
r"""Again until it lists nothing: etcd has stopped. (kubectl fails from now on until etcd is back.)""",
run=r"""for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name '^etcd$' -q)" ] && break; sleep 2; done"""),
cmd(r"""sudo mv /var/lib/etcd /var/lib/etcd.bak
sudo etcdutl snapshot restore /opt/backup/etcd.db --data-dir /var/lib/etcd
sudo mv /root/etcd.yaml /etc/kubernetes/manifests/etcd.yaml
for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
kubectl -n life get configmap marker -o yaml | grep state"""),
sudo mv ~/etcd.yaml /etc/kubernetes/manifests/"""),
hand(r"""k -n life get cm marker""",
r"""Again until it answers - a minute or two: etcd, then the API server come back, and `marker` with them.""",
run=r"""for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done""")]),
step("Plan the upgrade",
r"""On the control-plane node, point the Kubernetes repository at **v1.37**, install the **newest 1.37 kubeadm** the repository offers (unhold it first, hold it again after), and save the output of `kubeadm upgrade plan` in `{WS}/lesson-4.4/plan.txt`.
@ -392,13 +433,19 @@ The plan shows two choices: the newest 1.36 patch, and the 1.37 release your kub
t=$(newest); [ -n "$t" ] || exit 1
eq "$(ver "$CP" kubeadm)" "$t" && held "$CP" && grep -q "kubeadm upgrade apply v${t%%-*}" "$WS/lesson-4.4/plan.txt" 2>/dev/null""",
hint=r"""`sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list`, `sudo apt-get update`, `apt-cache madison kubeadm` (newest first), `sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.X-1.1 && sudo apt-mark hold kubeadm`.""",
show=r"""sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -qq
apt-cache madison kubeadm | head -3
v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo apt-mark unhold kubeadm && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$v && sudo apt-mark hold kubeadm
kubeadm version -o short
sudo kubeadm upgrade plan | tee {WS}/lesson-4.4/plan.txt | tail -12"""),
show=[hand(r"""sudo vim /etc/apt/sources.list.d/kubernetes.list""",
r"""In the URL, change `v1.36` to `v1.37`.""",
run=r"""sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list"""),
cmd(r"""sudo apt-get update
apt-cache madison kubeadm""", r"""The newest `1.37.x-1.1` (at the top) is what to install. The docs page *Upgrading kubeadm clusters* has the commands that follow."""),
hand(r"""sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=VERSION
sudo apt-mark hold kubeadm""",
r"""`VERSION`: that one, for example `1.37.1-1.1`.""",
run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo apt-mark unhold kubeadm && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$v && sudo apt-mark hold kubeadm"""),
cmd(r"""kubeadm version
sudo kubeadm upgrade plan | tee {WS}/lesson-4.4/plan.txt""")]),
step("Upgrade the control plane",
r"""Apply the upgrade to that 1.37 release (`kubeadm upgrade apply`, a few minutes - it upgrades etcd, the static Pods, CoreDNS and kube-proxy). Then drain the node, upgrade `kubelet` and `kubectl` to the same version, reload systemd, restart the kubelet, and uncordon.
@ -409,13 +456,23 @@ eq "$($K version -o json 2>/dev/null | python3 -c 'import json,sys; print(json.l
eq "$(j node "$CP" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$CP")" "$t $t $t " && held "$CP" \
&& [ "$(j node "$CP" '{.spec.unschedulable}')" != true ]""",
hint=r"""`sudo kubeadm upgrade apply v1.37.X -y`; `kubectl drain {CP} --ignore-daemonsets`; `sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=... kubectl=... && sudo apt-mark hold kubelet kubectl`; `sudo systemctl daemon-reload && sudo systemctl restart kubelet`; `kubectl uncordon {CP}`.""",
show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo kubeadm upgrade apply v${v%%-*} -y | tail -3
kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$v kubectl=$v && sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done
kubectl get nodes"""),
show=[hand(r"""sudo kubeadm upgrade apply vVERSION""",
r"""`vVERSION`: the one the plan names, for example `v1.37.1`. Answer `y`; it takes a few minutes.""",
run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo kubeadm upgrade apply v${v%%-*} -y | tail -3"""),
cmd(r"""k drain {CP} --ignore-daemonsets --delete-emptydir-data"""),
hand(r"""sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=VERSION kubectl=VERSION
sudo apt-mark hold kubelet kubectl""",
r"""`VERSION` as for kubeadm, for example `1.37.1-1.1`.""",
run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
sudo apt-mark unhold kubelet kubectl && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$v kubectl=$v && sudo apt-mark hold kubelet kubectl"""),
cmd(r"""sudo systemctl daemon-reload
sudo systemctl restart kubelet"""),
hand(r"""k uncordon {CP}""",
r"""Again if it fails at first: the kubelet takes a moment to come back.""",
run=r"""for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""k get no""")]),
step("Upgrade the worker",
r"""Now `{W1}`: drain it from the control-plane node; on `{W1}`, the repository line, kubeadm, **`kubeadm upgrade node`** (a worker has no control-plane Pods to replace - it only refreshes the kubelet's configuration), kubelet and kubectl, daemon-reload, restart; then uncordon it.
@ -425,8 +482,20 @@ t=$(newest); v="v${t%%-*}"
eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$W1")" "$t $t $t " && held "$W1" \
&& [ "$(j node "$W1" '{.spec.unschedulable}')" != true ] && eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""The same packages, from the same repository line, as on the control plane - only `kubeadm upgrade node` instead of `apply`. Drain and uncordon are kubectl commands: run them on the control-plane node.""",
show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data
show=[cmd(r"""k drain {W1} --ignore-daemonsets --delete-emptydir-data"""),
hand(r"""ssh {W1}
sudo -i
vim /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-mark unhold kubeadm kubelet kubectl
apt-get install -y kubeadm=VERSION
kubeadm upgrade node
apt-get install -y kubelet=VERSION kubectl=VERSION
apt-mark hold kubeadm kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet""",
r"""In the repository line `v1.36` becomes `v1.37`; `VERSION` as on the control-plane node. Then `exit` twice.""",
run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}')
ssh {W1} sudo bash -s $v <<'EOF'
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq
@ -436,9 +505,9 @@ kubeadm upgrade node | tail -2
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$1 kubectl=$1
apt-mark hold kubeadm kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet
EOF
kubectl uncordon {W1}
kubectl get nodes"""),
EOF"""),
cmd(r"""k uncordon {W1}
k get no""")]),
],
"outro": r"""You backed etcd up, restored it after a loss, and took a whole cluster from v1.36 to v1.37: control plane first, then the worker, with the packages held at every step.
@ -480,9 +549,11 @@ $K -n kube-system delete secret kubeadm-certs --ignore-not-found >/dev/null 2>&1
want=$($K -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}')
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-4.5/endpoint.txt" 2>/dev/null)" "$want" """,
hint=r"""`controlPlaneEndpoint:` in the `ClusterConfiguration` of ConfigMap `kubeadm-config`.""",
show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}' > {WS}/lesson-4.5/endpoint.txt
cat {WS}/lesson-4.5/endpoint.txt
sudo grep server: /etc/kubernetes/kubelet.conf"""),
show=[cmd(r"""k -n kube-system get cm kubeadm-config -o yaml | grep controlPlaneEndpoint"""),
hand(r"""echo ENDPOINT > {WS}/lesson-4.5/endpoint.txt""",
r"""`ENDPOINT`: that value, `host:port`.""",
run=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}' > {WS}/lesson-4.5/endpoint.txt"""),
cmd(r"""sudo grep server: /etc/kubernetes/kubelet.conf""")]),
step("etcd's members and its leader",
r"""Ask etcd itself. Write to `{WS}/lesson-4.5/etcd.txt`:
@ -498,13 +569,14 @@ import json, sys; m = json.load(sys.stdin)
st = json.loads(sys.argv[1])[0]["Status"]; print(next(x["name"] for x in m["members"] if x["ID"] == st["leader"]))' "$out")
eq "$(tr -d ' \r' < "$WS/lesson-4.5/etcd.txt" 2>/dev/null | grep .)" "$(printf 'members=%s\nleader=%s' "$n" "$lead")" """,
hint=r"""`etcdctl member list -w table` and `etcdctl endpoint status -w table` (the IS LEADER column), with the endpoint and certificates as for a snapshot. `export ETCDCTL_ENDPOINTS=... ETCDCTL_CACERT=...` saves typing.""",
show=r"""sudo bash <<'EOF'
export ETCDCTL_API=3 ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl member list -w table
etcdctl endpoint status -w table
EOF
printf 'members=1\nleader={CP}\n' > {WS}/lesson-4.5/etcd.txt"""),
show=[cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list -w table""", r"""For the next one, press ↑ and change the last words."""),
cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table""", r"""`IS LEADER` says which member leads."""),
hand(r"""vim {WS}/lesson-4.5/etcd.txt""",
r"""```
members=1
leader={CP}
```""",
run=r"""printf 'members=1\nleader={CP}\n' > {WS}/lesson-4.5/etcd.txt""")]),
step("Who leads the scheduler and the controller manager",
r"""Only one kube-scheduler and one kube-controller-manager act at a time; the others wait for the **Lease** in `kube-system` to expire. Write the node of each holder to `{WS}/lesson-4.5/leaders.txt`:
@ -516,20 +588,28 @@ controller-manager=<node>
s=$(j -n kube-system lease kube-scheduler '{.spec.holderIdentity}'); c=$(j -n kube-system lease kube-controller-manager '{.spec.holderIdentity}')
eq "$(tr -d ' \r' < "$WS/lesson-4.5/leaders.txt" 2>/dev/null | grep .)" "$(printf 'scheduler=%s\ncontroller-manager=%s' "${s%%_*}" "${c%%_*}")" """,
hint=r"""`kubectl -n kube-system get lease kube-scheduler kube-controller-manager` - HOLDER is `NODE_UUID`.""",
show=r"""kubectl -n kube-system get lease kube-scheduler kube-controller-manager
s=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}')
show=[cmd(r"""k -n kube-system get lease kube-scheduler kube-controller-manager""", r"""`HOLDER` is the node's name, then `_` and an ID: keep only the node's name."""),
hand(r"""vim {WS}/lesson-4.5/leaders.txt""",
r"""```
scheduler=NODE
controller-manager=NODE
```""",
run=r"""s=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}')
c=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}')
printf 'scheduler=%s\ncontroller-manager=%s\n' "${s%%_*}" "${c%%_*}" > {WS}/lesson-4.5/leaders.txt"""),
printf 'scheduler=%s\ncontroller-manager=%s\n' "${s%%_*}" "${c%%_*}" > {WS}/lesson-4.5/leaders.txt""")]),
step("What a second control-plane node needs",
r"""Upload the control-plane certificates for a joining node, and save the complete `kubeadm join` command a new **control-plane** node would run - new token, CA hash, `--control-plane` and the certificate key - to `{WS}/lesson-4.5/join-cp.sh`, on one line. Do not run it anywhere.""",
check=r"""
f="$WS/lesson-4.5/join-cp.sh"; grep -q '^kubeadm join .*--token .*--discovery-token-ca-cert-hash sha256:.*--control-plane' "$f" 2>/dev/null || exit 1
key=$(grep -o -- '--certificate-key [0-9a-f]\{64\}' "$f" | awk '{print $2}'); [ -n "$key" ] && [ -n "$(j -n kube-system secret kubeadm-certs '{.metadata.name}')" ]""",
hint=r"""`sudo kubeadm init phase upload-certs --upload-certs` prints the key as its last line; `sudo kubeadm token create --print-join-command --certificate-key KEY` adds `--control-plane --certificate-key` itself.""",
show=r"""key=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
sudo kubeadm token create --print-join-command --certificate-key $key > {WS}/lesson-4.5/join-cp.sh
cat {WS}/lesson-4.5/join-cp.sh
kubectl -n kube-system get secret kubeadm-certs"""),
show=[cmd(r"""sudo kubeadm init phase upload-certs --upload-certs""", r"""The last line is the certificate key."""),
hand(r"""sudo kubeadm token create --print-join-command --certificate-key KEY > {WS}/lesson-4.5/join-cp.sh""",
r"""`KEY`: that line.""",
run=r"""key=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1)
sudo kubeadm token create --print-join-command --certificate-key $key > {WS}/lesson-4.5/join-cp.sh"""),
cmd(r"""cat {WS}/lesson-4.5/join-cp.sh
k -n kube-system get secret kubeadm-certs""")]),
],
"outro": r"""You read the endpoint that makes HA possible, asked etcd for its members and leader, found the leaders of the scheduler and controller manager, and prepared a control-plane join.
@ -637,23 +717,23 @@ A chart is at `{WS}/lesson-4.6/chart` (values `replicaCount`, `image.tag`, `serv
r"""Render the chart as release `web` with `replicaCount=2`, without installing anything, and save the manifests in `{WS}/lesson-4.6/rendered.yaml`. That is how you check what a set of values does.""",
check=r"""f="$WS/lesson-4.6/rendered.yaml"; grep -q 'name: web-shop' "$f" 2>/dev/null && grep -q 'replicas: 2' "$f" && [ -z "$(j -n apps deploy web-shop '{.metadata.name}')" ]""",
hint=r"""`helm template RELEASE CHART --set KEY=VALUE`.""",
show=r"""helm template web {WS}/lesson-4.6/chart --set replicaCount=2 > {WS}/lesson-4.6/rendered.yaml
grep -E 'kind:|replicas:|image:' {WS}/lesson-4.6/rendered.yaml"""),
show=[cmd(r"""helm template web {WS}/lesson-4.6/chart --set replicaCount=2 > {WS}/lesson-4.6/rendered.yaml
grep -e kind: -e replicas: -e image: {WS}/lesson-4.6/rendered.yaml""")]),
step("Install a release",
r"""Install the chart as release `web` in Namespace `apps` (create it), with `replicaCount=2`.""",
check=r"""eq "$(helm -n apps status web -o json 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["info"]["status"], d["version"])')" "deployed 1" && eq "$(j -n apps deploy web-shop '{.status.readyReplicas}')" 2""",
hint=r"""`helm install web CHART -n apps --create-namespace --set replicaCount=2`; `helm list -n apps`.""",
show=r"""helm install web {WS}/lesson-4.6/chart -n apps --create-namespace --set replicaCount=2
show=[cmd(r"""helm install web {WS}/lesson-4.6/chart -n apps --create-namespace --set replicaCount=2
helm list -n apps
kubectl -n apps rollout status deploy web-shop --timeout=120s"""),
k -n apps rollout status deploy web-shop""")]),
step("Upgrade it, keeping what you set",
r"""Upgrade release `web` so that its Service becomes a `NodePort` - and keep `replicaCount=2`. A plain `helm upgrade --set service.type=NodePort` would quietly put the replica count back to the chart's default.""",
check=r"""eq "$(j -n apps svc web-shop '{.spec.type}')" NodePort && eq "$(j -n apps deploy web-shop '{.spec.replicas}')" 2 \
&& [ "$(helm -n apps history web -o json 2>/dev/null | python3 -c 'import json,sys; print(len(json.load(sys.stdin)))')" -ge 2 ]""",
hint=r"""`--reuse-values` keeps the values of the last release and adds yours; `helm get values web -n apps` shows what is set.""",
show=r"""helm upgrade web {WS}/lesson-4.6/chart -n apps --reuse-values --set service.type=NodePort
show=[cmd(r"""helm upgrade web {WS}/lesson-4.6/chart -n apps --reuse-values --set service.type=NodePort
helm get values web -n apps
kubectl -n apps get svc web-shop"""),
k -n apps get svc web-shop""")]),
step("Roll the release back",
r"""The NodePort was a mistake: roll release `web` back to **revision 1**. A rollback is itself a new revision.""",
check=r"""
@ -662,16 +742,18 @@ helm -n apps history web -o json 2>/dev/null | python3 -c '
import json, sys; h = json.load(sys.stdin)
sys.exit(0 if len(h) >= 3 and h[-1]["status"] == "deployed" and "Rollback to 1" in h[-1]["description"] else 1)'""",
hint=r"""`helm rollback web 1 -n apps`, then `helm history web -n apps`.""",
show=r"""helm rollback web 1 -n apps
show=[cmd(r"""helm rollback web 1 -n apps
helm history web -n apps
kubectl -n apps get svc web-shop"""),
k -n apps get svc web-shop""")]),
step("An overlay for production",
r"""Create a Kustomize overlay at `{WS}/lesson-4.6/kustomize/prod` on top of the base, which: puts everything in Namespace `prod`, prefixes names with `prod-`, runs **3** replicas, and uses image `nginx:1.27`. Apply it. Leave the base as it is.""",
check=r"""eq "$(j -n prod deploy prod-api '{.spec.replicas} {.spec.template.spec.containers[0].image} {.status.readyReplicas}')" "3 nginx:1.27 3" \
&& grep -q 'image: nginx:1.26' "$WS/lesson-4.6/kustomize/base/deployment.yaml" """,
hint=r"""A `kustomization.yaml` with `resources: [../base]`, `namespace: prod`, `namePrefix: prod-`, `replicas: [{name: api, count: 3}]`, `images: [{name: nginx, newTag: "1.27"}]`. The Namespace itself must exist (or be one of the resources). `kubectl kustomize DIR` shows the result first.""",
show=r"""mkdir -p {WS}/lesson-4.6/kustomize/prod
cat > {WS}/lesson-4.6/kustomize/prod/kustomization.yaml <<'EOF'
show=[cmd(r"""mkdir -p {WS}/lesson-4.6/kustomize/prod"""),
hand(r"""vim {WS}/lesson-4.6/kustomize/prod/kustomization.yaml""",
r"""(the docs page *Declarative Management of Kubernetes Objects Using Kustomize* has each field)
```
resources:
- ../base
namespace: prod
@ -682,11 +764,23 @@ replicas:
images:
- name: nginx
newTag: "1.27"
EOF
kubectl kustomize {WS}/lesson-4.6/kustomize/prod
kubectl create namespace prod
kubectl apply -k {WS}/lesson-4.6/kustomize/prod
kubectl -n prod rollout status deploy prod-api --timeout=120s"""),
```""",
run=r"""cat > {WS}/lesson-4.6/kustomize/prod/kustomization.yaml <<'EOF'
resources:
- ../base
namespace: prod
namePrefix: prod-
replicas:
- name: api
count: 3
images:
- name: nginx
newTag: "1.27"
EOF"""),
cmd(r"""k kustomize {WS}/lesson-4.6/kustomize/prod
k create ns prod
k apply -k {WS}/lesson-4.6/kustomize/prod
k -n prod rollout status deploy prod-api""")]),
],
"outro": r"""You rendered a chart, installed, upgraded and rolled back a release, and built a production overlay with Kustomize without touching its base.
@ -734,11 +828,18 @@ f=$(nx "$W1" 'ls /etc/cni/net.d/*.conf* 2>/dev/null | sort | head -1' </dev/null
p=$(nx "$W1" "cat $f" </dev/null | python3 -c 'import json,sys; c=json.load(sys.stdin); print(",".join(x["type"] for x in c.get("plugins", [c])))')
eq "$(tr -d ' \r' < "$WS/lesson-4.7/cni.txt" 2>/dev/null | grep .)" "$(printf 'config=%s\nplugins=%s' "$f" "$p")" """,
hint=r"""`ls /etc/cni/net.d/` and `cat` the file: a `.conflist` has a `"plugins"` list, each with a `"type"` - the binary under `/opt/cni/bin` that is run.""",
show=r"""ssh -n {W1} 'ls /etc/cni/net.d/; cat /etc/cni/net.d/10-kindnet.conflist; ls /opt/cni/bin'
f=$(ssh -n {W1} 'ls /etc/cni/net.d/*.conf* | sort | head -1')
show=[hand(r"""ssh {W1} ls /etc/cni/net.d/
ssh {W1} cat /etc/cni/net.d/10-kindnet.conflist""",
r"""The runtime uses the first file by name; the `type` of each entry of `plugins`, in order, is the chain.""",
run=r"""ssh -n {W1} 'ls /etc/cni/net.d/; cat /etc/cni/net.d/10-kindnet.conflist' """),
hand(r"""vim {WS}/lesson-4.7/cni.txt""",
r"""```
config=/etc/cni/net.d/10-kindnet.conflist
plugins=TYPE,TYPE,...
```""",
run=r"""f=$(ssh -n {W1} 'ls /etc/cni/net.d/*.conf* | sort | head -1')
p=$(ssh -n {W1} "cat $f" | jq -r '[.plugins[].type] | join(",")')
printf 'config=%s\nplugins=%s\n' "$f" "$p" > {WS}/lesson-4.7/cni.txt
cat {WS}/lesson-4.7/cni.txt"""),
printf 'config=%s\nplugins=%s\n' "$f" "$p" > {WS}/lesson-4.7/cni.txt""")]),
step("The runtime, through CRI",
r"""Write to `{WS}/lesson-4.7/cri.txt`:
@ -751,33 +852,63 @@ e=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" | python3 -c 'import json,sys
r=$(j node "$W1" '{.status.nodeInfo.containerRuntimeVersion}')
eq "$(tr -d ' \r' < "$WS/lesson-4.7/cri.txt" 2>/dev/null | grep .)" "$(printf 'endpoint=%s\nruntime=%s' "$e" "$r")" """,
hint=r"""`containerRuntimeEndpoint` in the kubelet's configuration (`/var/lib/kubelet/config.yaml`, or `configz`); `kubectl get node -o wide` shows CONTAINER-RUNTIME.""",
show=r"""e=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.containerRuntimeEndpoint)
show=[hand(r"""ssh {W1} grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml""",
r"""The endpoint the kubelet talks CRI to.""",
run=r"""ssh -n {W1} grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml"""),
cmd(r"""k get no -o wide""", r"""`CONTAINER-RUNTIME` is the runtime and its version."""),
hand(r"""vim {WS}/lesson-4.7/cri.txt""",
r"""```
endpoint=ENDPOINT
runtime=RUNTIME
```""",
run=r"""e=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.containerRuntimeEndpoint)
r=$(kubectl get node {W1} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}')
printf 'endpoint=%s\nruntime=%s\n' "$e" "$r" > {WS}/lesson-4.7/cri.txt
cat {WS}/lesson-4.7/cri.txt"""),
printf 'endpoint=%s\nruntime=%s\n' "$e" "$r" > {WS}/lesson-4.7/cri.txt""")]),
step("A Pod's container, from the runtime's side",
r"""Pod `probe` in `plug` runs on `{W1}`. Ask the runtime about it there, with `crictl`: find its container, and write the container's **process ID on the node** to `{WS}/lesson-4.7/pid.txt`.""",
check=r"""
want=$(nx "$W1" 'crictl inspect $(crictl ps --name probe -q | head -1) 2>/dev/null' </dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["info"]["pid"])')
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-4.7/pid.txt" 2>/dev/null)" "$want" """,
hint=r"""`sudo crictl pods --name probe`, `sudo crictl ps --name probe`, `sudo crictl inspect ID` - the PID is under `info.pid`. `crictl` talks to the same socket as the kubelet.""",
show=r"""ssh -n {W1} 'sudo crictl pods --name probe; sudo crictl ps --name probe'
ssh -n {W1} 'sudo crictl inspect $(sudo crictl ps --name probe -q | head -1)' | jq .info.pid > {WS}/lesson-4.7/pid.txt
cat {WS}/lesson-4.7/pid.txt"""),
show=[hand(r"""ssh {W1}
sudo -i
crictl ps --name probe
crictl inspect CONTAINER | grep -m1 '"pid"' """,
r"""`CONTAINER`: the ID `crictl ps` printed. Then `exit` twice.""",
run=r"""ssh -n {W1} 'sudo crictl pods --name probe; sudo crictl ps --name probe' """),
hand(r"""echo PID > {WS}/lesson-4.7/pid.txt""",
r"""`PID`: that number.""",
run=r"""ssh -n {W1} 'sudo crictl inspect $(sudo crictl ps --name probe -q | head -1)' | jq .info.pid > {WS}/lesson-4.7/pid.txt""")]),
step("A RuntimeClass",
r"""Create RuntimeClass `plain` for containerd's handler `runc`, and run Pod `classy` in `plug` (`busybox:1.36`, `sleep 86400`) with that runtime class. A Pod that names a handler the node does not have stays in `ContainerCreating`.""",
check=r"""eq "$(j runtimeclass plain '{.handler}')" runc && eq "$(j -n plug pod classy '{.spec.runtimeClassName} {.status.phase}')" "plain Running" """,
hint=r"""A RuntimeClass is `node.k8s.io/v1` with a top-level `handler:` and no `spec`. The Pod: `spec.runtimeClassName: plain`.""",
show=r"""kubectl apply -f - <<'EOF'
show=[hand(r"""vim rc.yaml""",
r"""No `kubectl create` for a RuntimeClass: paste the example from the docs page *Runtime Class*, and make it:
```
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: plain
handler: runc
EOF
kubectl -n plug run classy --image=busybox:1.36 --overrides='{"spec":{"runtimeClassName":"plain"}}' -- sleep 86400
kubectl -n plug wait --for=condition=Ready pod/classy --timeout=120s
kubectl get runtimeclass"""),
```""",
run=r"""cat > rc.yaml <<'EOF'
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: plain
handler: runc
EOF"""),
cmd(r"""k apply -f rc.yaml
k -n plug run classy --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > classy.yaml"""),
hand(r"""vim classy.yaml""",
r"""Add to the Pod's `spec:`, above `containers:`:
```
runtimeClassName: plain
```""",
run=r"""yq -i '.spec.runtimeClassName = "plain"' classy.yaml"""),
cmd(r"""k -n plug apply -f classy.yaml
k -n plug get po classy""")]),
step("Storage drivers",
r"""Write to `{WS}/lesson-4.7/csi.txt`:
@ -792,11 +923,16 @@ n=$($K get csidrivers --no-headers 2>/dev/null | grep -c .)
p=$($K get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.provisioner}{end}')
eq "$(tr -d ' \r' < "$WS/lesson-4.7/csi.txt" 2>/dev/null | grep .)" "$(printf 'csidrivers=%s\nprovisioner=%s' "$n" "$p")" """,
hint=r"""`kubectl get csidrivers`, `kubectl get csinodes`, `kubectl get sc`.""",
show=r"""kubectl get csidrivers
kubectl get csinodes
kubectl get sc
printf 'csidrivers=%s\nprovisioner=%s\n' "$(kubectl get csidrivers --no-headers 2>/dev/null | grep -c .)" "$(kubectl get sc standard -o jsonpath='{.provisioner}')" > {WS}/lesson-4.7/csi.txt
cat {WS}/lesson-4.7/csi.txt"""),
show=[cmd(r"""k get csidrivers
k get csinodes
k get sc"""),
hand(r"""vim {WS}/lesson-4.7/csi.txt""",
r"""```
csidrivers=N
provisioner=PROVISIONER
```
`N`: how many CSIDrivers `get csidrivers` listed (`0` for "No resources found"); `PROVISIONER`: the `(default)` class's.""",
run=r"""printf 'csidrivers=%s\nprovisioner=%s\n' "$(kubectl get csidrivers --no-headers 2>/dev/null | grep -c .)" "$(kubectl get sc standard -o jsonpath='{.provisioner}')" > {WS}/lesson-4.7/csi.txt""")]),
],
"outro": r"""You read the CNI chain on a node, the runtime the kubelet talks to, a container as the runtime sees it, ran a Pod with a RuntimeClass, and counted the storage drivers.
@ -850,7 +986,9 @@ py crd $c 's = o["spec"]["versions"][0]; p = s["schema"]["openAPIV3Schema"]["pro
s["served"] and s["storage"] and "status" in s.get("subresources", {}) and "schedule" in p.get("required", []) and p["properties"]["retention"].get("default") == 7 and p["properties"]["retention"].get("minimum") == 1' \
&& eq "$(j crd $c '{.status.conditions[?(@.type=="Established")].status}')" True""",
hint=r"""The docs page "Extend the Kubernetes API with CustomResourceDefinitions" has the skeleton. `required:` sits beside `properties:`, at the level of `spec`'s schema; `default:` and `minimum:` inside `retention`. `subresources: {status: {}}` goes under the version.""",
show=r"""kubectl apply -f - <<'EOF'
show=[hand(r"""vim crd.yaml""",
r"""Paste the example from the docs page *Extend the Kubernetes API with CustomResourceDefinitions*, and make it:
```
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
@ -879,54 +1017,101 @@ spec:
type: object
properties:
phase: {type: string}
EOF
kubectl wait --for condition=established crd/backups.data.example.com
kubectl api-resources --api-group=data.example.com"""),
```""",
run=r"""cat > crd.yaml <<'EOF'
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: backups.data.example.com
spec:
group: data.example.com
scope: Namespaced
names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]}
versions:
- name: v1
served: true
storage: true
subresources:
status: {}
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [schedule]
properties:
schedule: {type: string}
retention: {type: integer, minimum: 1, default: 7}
status:
type: object
properties:
phase: {type: string}
EOF"""),
cmd(r"""k apply -f crd.yaml
k api-resources --api-group=data.example.com""")]),
step("Defaults and validation",
r"""In Namespace `data`, create Backup `nightly` with `schedule: "0 2 * * *"` and no retention: the API server fills in `7`. Then try a Backup `broken` with `retention: 0` - it is refused. Save the refusal (kubectl writes it to **stderr**) in `{WS}/lesson-4.8/refused.txt`.""",
check=r"""
eq "$(j -n data bk nightly '{.spec.schedule} {.spec.retention}')" "0 2 * * * 7" && [ -z "$(j -n data bk broken '{.metadata.name}')" ] \
&& grep -q 'retention' "$WS/lesson-4.8/refused.txt" 2>/dev/null && grep -qi 'greater than or equal to 1' "$WS/lesson-4.8/refused.txt" """,
hint=r"""`apiVersion: data.example.com/v1`, `kind: Backup`. Redirect with `2>` - with `>` the file stays empty.""",
show=r"""kubectl -n data apply -f - <<'EOF'
show=[hand(r"""vim nightly.yaml""",
r"""```
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: nightly
spec:
schedule: "0 2 * * *"
EOF
kubectl -n data get bk nightly -o jsonpath='{.spec}'; echo
printf 'apiVersion: data.example.com/v1\nkind: Backup\nmetadata: {name: broken}\nspec: {schedule: "@hourly", retention: 0}\n' | kubectl -n data apply -f - 2> {WS}/lesson-4.8/refused.txt
cat {WS}/lesson-4.8/refused.txt"""),
```""",
run=r"""cat > nightly.yaml <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: nightly
spec:
schedule: "0 2 * * *"
EOF"""),
cmd(r"""k -n data apply -f nightly.yaml
k -n data get bk nightly -o yaml | grep -A2 '^spec:' """, r"""`retention: 7`, filled in by the API server."""),
hand(r"""cp nightly.yaml broken.yaml
vim broken.yaml""",
r"""Name it `broken`, and add `retention: 0` under `spec:`.""",
run=r"""printf 'apiVersion: data.example.com/v1\nkind: Backup\nmetadata: {name: broken}\nspec: {schedule: "@hourly", retention: 0}\n' > broken.yaml"""),
cmd(r"""k -n data apply -f broken.yaml 2> {WS}/lesson-4.8/refused.txt
cat {WS}/lesson-4.8/refused.txt""")]),
step("What the schema says",
r"""`kubectl explain` reads the CRD's schema like any built-in kind's. Save the output of `kubectl explain backup.spec` in `{WS}/lesson-4.8/explain.txt`.""",
check=r"""f="$WS/lesson-4.8/explain.txt"; grep -q 'retention' "$f" 2>/dev/null && grep -q 'schedule' "$f" && grep -qi 'required' "$f" """,
hint=r"""`kubectl explain backup.spec` - or `--recursive` for the whole tree.""",
show=r"""kubectl explain backup.spec | tee {WS}/lesson-4.8/explain.txt"""),
show=[cmd(r"""k explain backup.spec | tee {WS}/lesson-4.8/explain.txt""")]),
step("Install the operator",
r"""Install the operator from `{WS}/lesson-4.8/operator/` and wait until its controller runs. Read the manifests first: a Namespace, a ServiceAccount, a ClusterRole for `backups` and ConfigMaps, the binding, and the controller's Deployment.""",
check=r"""eq "$(j -n backup-operator deploy controller '{.status.readyReplicas}')" 1 && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller""",
hint=r"""`kubectl apply -f DIR` applies every file in it; `kubectl -n backup-operator rollout status deploy controller`.""",
show=r"""grep -E '^kind:|^ name:' {WS}/lesson-4.8/operator/operator.yaml
kubectl apply -f {WS}/lesson-4.8/operator/
kubectl -n backup-operator rollout status deploy controller --timeout=120s"""),
show=[cmd(r"""grep -e ^kind: -e '^ name:' {WS}/lesson-4.8/operator/operator.yaml
k apply -f {WS}/lesson-4.8/operator/
k -n backup-operator rollout status deploy controller""")]),
step("Let it work",
r"""Create Backup `weekly` in `data` with `schedule: "0 3 * * 0"`. Within a few seconds the controller has acted on it - and on `nightly`: each has `status.phase: Scheduled`, and a ConfigMap `backup-NAME` holds its schedule. `kubectl -n backup-operator logs deploy/controller` is where you would look if it did not.""",
check=r"""
eq "$(j -n data bk weekly '{.status.phase}')" Scheduled && eq "$(j -n data cm backup-weekly '{.data.schedule}')" "0 3 * * 0" && eq "$(j -n data bk nightly '{.status.phase}')" Scheduled""",
hint=r"""A Backup like `nightly`. `kubectl -n data get bk` - add `additionalPrinterColumns` to the CRD and the phase would show there too.""",
show=r"""kubectl -n data apply -f - <<'EOF'
show=[hand(r"""cp nightly.yaml weekly.yaml
vim weekly.yaml""",
r"""Name it `weekly`, with `schedule: "0 3 * * 0"`.""",
run=r"""cat > weekly.yaml <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: weekly
spec:
schedule: "0 3 * * 0"
EOF
sleep 10
kubectl -n data get bk -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PHASE:.status.phase
kubectl -n data get cm"""),
EOF"""),
cmd(r"""k -n data apply -f weekly.yaml
k -n data get bk -o yaml | grep phase
k -n data get cm""", r"""Again in a few seconds: both Backups `phase: Scheduled`, and a ConfigMap `backup-NAME` for each.""")]),
],
"outro": r"""You defined a kind with validation and defaults, saw the schema refuse bad objects, installed an operator with its RBAC, and watched it act on your custom resources.

479
cka/lessons_net.py
View File

@ -1,6 +1,6 @@
"""CKA lessons of the Services and Networking domain (5.1-5.6). See cka/lessons.py for what a lesson is."""
from cka.common import DNS
from sim.bank import NET, NS, PODS, step
from sim.bank import NET, NS, PODS, step, cmd, hand
# ---- 5.1 connectivity between Pods ------------------------------------------------------------
L51 = {
@ -35,35 +35,55 @@ ready net || exit 1
want=$($K get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' | sort)
eq "$(tr -s ' \t' ' ' < "$WS/lesson-5.1/cidrs.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """,
hint=r"""`.spec.podCIDR` of each node: `-o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}'`, or `custom-columns`. The cluster-wide range is the `podSubnet` in kubeadm's configuration.""",
show=r"""kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' > {WS}/lesson-5.1/cidrs.txt
show=[cmd(r"""k get no -o custom-columns=NODE:.metadata.name,CIDR:.spec.podCIDR --no-headers > {WS}/lesson-5.1/cidrs.txt
cat {WS}/lesson-5.1/cidrs.txt
kubectl -n net get pods -o wide"""),
k -n net get po -o wide""", r"""Every Pod's IP is from the range of the node it runs on.""")]),
step("Across nodes, Pod to Pod",
r"""From Pod `a`, send a request straight to Pod `b`'s **IP address** on port 80 - no Service - and save the answer in `{WS}/lesson-5.1/answer.txt`. It crossed from `{W1}` to `{W2}`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/answer.txt" 2>/dev/null)" b""",
hint=r"""`kubectl -n net get pod b -o jsonpath='{.status.podIP}'`, then `kubectl -n net exec a -- wget -qO- -T3 http://IP`.""",
show=r"""ip=$(kubectl -n net get pod b -o jsonpath='{.status.podIP}')
kubectl -n net exec a -- wget -qO- -T3 http://$ip | tee {WS}/lesson-5.1/answer.txt"""),
show=[cmd(r"""k -n net get po -o wide"""),
hand(r"""k -n net exec a -- wget -qO- -T3 IP > {WS}/lesson-5.1/answer.txt""",
r"""`IP`: Pod `b`'s address, from the list.""",
run=r"""ip=$(kubectl -n net get pod b -o jsonpath='{.status.podIP}')
kubectl -n net exec a -- wget -qO- -T3 http://$ip > {WS}/lesson-5.1/answer.txt"""),
cmd(r"""cat {WS}/lesson-5.1/answer.txt""")]),
step("The route that carries it",
r"""On `{W1}`, look at the routing table: there is a route for `{W2}`'s Pod CIDR, via `{W2}`'s node address. Write that route's destination and gateway to `{WS}/lesson-5.1/route.txt`, as `CIDR via ADDRESS`.""",
check=r"""
c=$(j node "$W2" '{.spec.podCIDR}'); gw=$(j node "$W2" '{.status.addresses[?(@.type=="InternalIP")].address}')
nx "$W1" "ip route | grep -q '^$c via $gw '" </dev/null && eq "$(tr -s ' ' < "$WS/lesson-5.1/route.txt" 2>/dev/null | tr -d '\r\n' | sed 's/ *$//')" "$c via $gw" """,
hint=r"""`ip route` on the node. The node's address is its `InternalIP` (`kubectl get nodes -o wide`).""",
show=r"""ssh -n {W1} ip route
c=$(kubectl get node {W2} -o jsonpath='{.spec.podCIDR}')
ssh -n {W1} ip route | grep "^$c " | awk '{print $1, $2, $3}' > {WS}/lesson-5.1/route.txt
cat {WS}/lesson-5.1/route.txt"""),
show=[cmd(r"""k get no -o wide
k get no -o custom-columns=NODE:.metadata.name,CIDR:.spec.podCIDR""", r"""`{W2}`'s address (`INTERNAL-IP`) and its Pod CIDR."""),
hand(r"""ssh {W1} ip route""",
r"""The line for `{W2}`'s Pod CIDR reads `CIDR via ADDRESS dev ...` - the route to the other node's Pods.""",
run=r"""ssh -n {W1} ip route"""),
hand(r"""echo 'CIDR via ADDRESS' > {WS}/lesson-5.1/route.txt""",
r"""The first three words of that line.""",
run=r"""c=$(kubectl get node {W2} -o jsonpath='{.spec.podCIDR}')
ssh -n {W1} ip route | grep "^$c " | awk '{print $1, $2, $3}' > {WS}/lesson-5.1/route.txt""")]),
step("A Pod on the host's network",
r"""Run Pod `hostnet` in `net` (`busybox:1.36`, `sleep 86400`) with `hostNetwork: true`, on `{W1}`. Its IP is not from the Pod CIDR: it is `{W1}`'s own address, and it sees the node's interfaces. Write its IP to `{WS}/lesson-5.1/hostnet.txt`.""",
check=r"""
want=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}')
eq "$(j -n net pod hostnet '{.spec.hostNetwork} {.status.podIP}')" "true $want" && eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/hostnet.txt" 2>/dev/null)" "$want" """,
hint=r"""`spec.hostNetwork: true` and `spec.nodeName: {W1}` (or a nodeSelector). `kubectl run ... --overrides='{"spec":{...}}'` adds both.""",
show=r"""kubectl -n net run hostnet --image=busybox:1.36 --overrides='{"spec":{"hostNetwork":true,"nodeName":"{W1}"}}' -- sleep 86400
kubectl -n net wait --for=condition=Ready pod/hostnet --timeout=120s
kubectl -n net get pod hostnet -o jsonpath='{.status.podIP}' > {WS}/lesson-5.1/hostnet.txt
kubectl -n net exec hostnet -- ip -4 addr show eth0"""),
show=[cmd(r"""k -n net run hostnet --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > hostnet.yaml"""),
hand(r"""vim hostnet.yaml""",
r"""Add to the Pod's `spec:`, above `containers:`:
```
hostNetwork: true
nodeName: {W1}
```""",
run=r"""yq -i '.spec.hostNetwork = true | .spec.nodeName = "{W1}"' hostnet.yaml"""),
cmd(r"""k -n net apply -f hostnet.yaml
k -n net get po hostnet -o wide""", r"""Its IP is `{W1}`'s own address, not one from the Pod CIDR."""),
hand(r"""echo IP > {WS}/lesson-5.1/hostnet.txt""",
r"""`IP`: that address.""",
run=r"""kubectl -n net wait --for=condition=Ready pod/hostnet --timeout=120s
kubectl -n net get pod hostnet -o jsonpath='{.status.podIP}' > {WS}/lesson-5.1/hostnet.txt"""),
cmd(r"""k -n net exec hostnet -- ip -4 addr show eth0""")]),
],
"outro": r"""You read how the Pod network is divided among nodes, sent a request from one node's Pod to another's, found the route that carried it, and ran a Pod on the host network.
@ -108,7 +128,9 @@ ready prod && ready dev || exit 1
eq "$(j -n prod netpol default-deny '{.spec.policyTypes[0]}')" Ingress && [ -z "$(j -n prod netpol default-deny '{.spec.ingress}')" ] || exit 1
blocked prod api http://db && blocked dev tester http://web.prod""",
hint=r"""`podSelector: {}` and `policyTypes: [Ingress]`, with no `ingress:` list. Test with `kubectl -n prod exec api -- wget -qO- -T3 db`.""",
show=r"""kubectl -n prod apply -f - <<'EOF'
show=[hand(r"""vim deny.yaml""",
r"""No `kubectl create` for a NetworkPolicy: paste *Default deny all ingress traffic* from the docs page *Network Policies*, and name it:
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@ -116,13 +138,25 @@ metadata:
spec:
podSelector: {}
policyTypes: [Ingress]
EOF
kubectl -n prod exec api -- wget -qO- -T3 db || echo blocked"""),
```""",
run=r"""cat > deny.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes: [Ingress]
EOF"""),
cmd(r"""k -n prod apply -f deny.yaml
k -n prod exec api -- wget -qO- -T3 db""", r"""It times out: blocked.""")]),
step("Let the API reach the database",
r"""Create NetworkPolicy `db-from-api` in `prod`: Pods `app=db` accept TCP 80 from Pods `app=api` **of `prod`**. `web` still cannot reach `db`, and neither can `dev`'s `api` - a `podSelector` alone means "in this Namespace".""",
check=NET + r"""okr prod api http://db && blocked prod web http://db && blocked dev api http://db.prod""",
hint=r"""`podSelector: {matchLabels: {app: db}}`, `ingress: [{from: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}]`.""",
show=r"""kubectl -n prod apply -f - <<'EOF'
show=[hand(r"""vim db-from-api.yaml""",
r"""Paste the example from the docs page *Network Policies* and cut it down to:
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@ -137,16 +171,36 @@ spec:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
EOF
kubectl -n prod exec api -- wget -qO- -T3 db
kubectl -n prod exec web -- wget -qO- -T3 db || echo blocked"""),
```""",
run=r"""cat > db-from-api.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: db-from-api
spec:
podSelector:
matchLabels: {app: db}
policyTypes: [Ingress]
ingress:
- from:
- podSelector:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
EOF"""),
cmd(r"""k -n prod apply -f db-from-api.yaml
k -n prod exec api -- wget -qO- -T3 db
k -n prod exec web -- wget -qO- -T3 db""", r"""`api` gets the page; `web` times out.""")]),
step("Those Pods in that Namespace",
r"""Create NetworkPolicy `web-from-dev-api` in `prod`: Pods `app=web` accept TCP 80 from Pods `app=api` **in** Namespaces labelled `team=dev` - both conditions in **one** element of `from`.
`dev`'s `api` gets in; `dev`'s `tester` does not, and neither does `prod`'s own `api`. Written as two elements, the rule would let every Pod of `dev` in, and every `api` Pod of `prod` too.""",
check=NET + r"""okr dev api http://web.prod && blocked dev tester http://web.prod && blocked prod api http://web""",
hint=r"""`from: [{namespaceSelector: {matchLabels: {team: dev}}, podSelector: {matchLabels: {app: api}}}]` - mind the single `-`.""",
show=r"""kubectl -n prod apply -f - <<'EOF'
show=[hand(r"""cp db-from-api.yaml web-from-dev-api.yaml
vim web-from-dev-api.yaml""",
r"""Make it - `namespaceSelector` and `podSelector` in **one** element of `from` (no `-` before `podSelector`):
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@ -163,9 +217,28 @@ spec:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
EOF
kubectl -n dev exec api -- wget -qO- -T3 web.prod
kubectl -n dev exec tester -- wget -qO- -T3 web.prod || echo blocked"""),
```""",
run=r"""cat > web-from-dev-api.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-from-dev-api
spec:
podSelector:
matchLabels: {app: web}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels: {team: dev}
podSelector:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
EOF"""),
cmd(r"""k -n prod apply -f web-from-dev-api.yaml
k -n dev exec api -- wget -qO- -T3 web.prod
k -n dev exec tester -- wget -qO- -T3 web.prod""", r"""`dev`'s `api` gets in; `tester` times out.""")]),
step("Egress, and DNS",
r"""Now limit what `tester` in `dev` may **send**: NetworkPolicy `tester-egress` in `dev` selects `app=tester` and allows egress only to `dev`'s `app=api` Pods on TCP 80 - and DNS, to `kube-dns` in `kube-system` on UDP and TCP 53.
@ -174,7 +247,9 @@ kubectl -n dev exec tester -- wget -qO- -T3 web.prod || echo blocked"""),
eq "$(j -n dev netpol tester-egress '{.spec.policyTypes[0]}')" Egress || exit 1
okr dev tester http://api && blocked dev tester http://echo""",
hint=r"""Two egress rules: `{to: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}` and `{to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}], ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]}`. Every Namespace carries the label `kubernetes.io/metadata.name`.""",
show=r"""kubectl -n dev apply -f - <<'EOF'
show=[hand(r"""vim tester-egress.yaml""",
r"""An egress policy - two elements of `egress`, one for `api`, one for DNS:
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@ -197,9 +272,34 @@ spec:
ports:
- {protocol: UDP, port: 53}
- {protocol: TCP, port: 53}
EOF
kubectl -n dev exec tester -- wget -qO- -T3 api
kubectl -n dev exec tester -- wget -qO- -T3 echo || echo blocked"""),
```""",
run=r"""cat > tester-egress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: tester-egress
spec:
podSelector:
matchLabels: {app: tester}
policyTypes: [Egress]
egress:
- to:
- podSelector:
matchLabels: {app: api}
ports:
- {protocol: TCP, port: 80}
- to:
- namespaceSelector:
matchLabels: {kubernetes.io/metadata.name: kube-system}
podSelector:
matchLabels: {k8s-app: kube-dns}
ports:
- {protocol: UDP, port: 53}
- {protocol: TCP, port: 53}
EOF"""),
cmd(r"""k -n dev apply -f tester-egress.yaml
k -n dev exec tester -- wget -qO- -T3 api
k -n dev exec tester -- wget -qO- -T3 echo""", r"""`api` answers; `echo` times out.""")]),
],
"outro": r"""You closed a Namespace, opened single paths into it, used the AND form of a rule, and limited a Pod's egress without breaking its DNS.
@ -243,20 +343,28 @@ $K -n svc rollout status deploy web --timeout=180s >/dev/null && ready svc
r"""Expose Deployment `web` as Service `web`, port 80. From `client`, `wget -qO- web` returns nginx's page.""",
check=NET + r"""eq "$(j -n svc svc web '{.spec.type} {.spec.ports[0].port}')" "ClusterIP 80" && okr svc client http://web""",
hint=r"""`kubectl expose deploy web --port=80` - it copies the Deployment's selector.""",
show=r"""kubectl -n svc expose deploy web --port=80
kubectl -n svc get svc,endpointslices
kubectl -n svc exec client -- wget -qO- -T3 web | head -4"""),
show=[cmd(r"""k -n svc expose deploy web --port=80
k -n svc get svc,endpointslices
k -n svc exec client -- wget -qO- -T3 web | head -4""")]),
step("NodePort",
r"""Create Service `web-np` of type **NodePort** for the same Pods, with node port **30080**. Then reach it from the control-plane node: `curl http://{W1}-ADDRESS:30080` - every node answers on that port, whether or not it runs one of the Pods.""",
check=r"""
eq "$(j -n svc svc web-np '{.spec.type} {.spec.ports[0].nodePort}')" "NodePort 30080" || exit 1
ip=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}'); nx "$CP" "curl -s -m 3 http://$ip:30080 | grep -q nginx" </dev/null""",
hint=r"""`kubectl create service nodeport web-np --tcp=80:80 --node-port=30080` makes a selector `app=web-np` - change it to `app=web`; or write the YAML, or `kubectl expose deploy web --name=web-np --type=NodePort` and then set the `nodePort`.""",
show=r"""kubectl -n svc expose deploy web --name=web-np --type=NodePort --port=80 --dry-run=client -o yaml > {WS}/lesson-5.3/web-np.yaml
yq -i '.spec.ports[0].nodePort = 30080' {WS}/lesson-5.3/web-np.yaml
kubectl -n svc apply -f {WS}/lesson-5.3/web-np.yaml
ip=$(kubectl get node {W1} -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
curl -s -m 3 http://$ip:30080 | head -4"""),
show=[cmd(r"""k -n svc expose deploy web --name=web-np --type=NodePort --port=80 --dry-run=client -o yaml > web-np.yaml"""),
hand(r"""vim web-np.yaml""",
r"""Add to the port:
```
nodePort: 30080
```""",
run=r"""yq -i '.spec.ports[0].nodePort = 30080' web-np.yaml"""),
cmd(r"""k -n svc apply -f web-np.yaml
k get no -o wide""", r"""`{W1}`'s `INTERNAL-IP`:"""),
hand(r"""curl IP:30080""",
r"""nginx's page, from the node port.""",
run=r"""ip=$(kubectl get node {W1} -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
curl -s -m 3 http://$ip:30080 | head -4""")]),
step("Headless",
r"""Create Service `web-headless` with `clusterIP: None` for the `web` Pods. Look it up from `client` with `nslookup web-headless`: you get the two Pod IPs, not a Service IP. Write them, one per line, to `{WS}/lesson-5.3/headless.txt`.""",
check=DNS + r"""
@ -264,18 +372,19 @@ eq "$(j -n svc svc web-headless '{.spec.clusterIP}')" None || exit 1
want=$($K -n svc get pods -l app=web -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' | sort)
eq "$(tr -d ' \r' < "$WS/lesson-5.3/headless.txt" 2>/dev/null | grep . | sort)" "$want" """,
hint=r"""`kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80`. In busybox's `nslookup` output the addresses follow the `Name:` line.""",
show=r"""kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80
sleep 5
kubectl -n svc exec client -- nslookup web-headless
kubectl -n svc exec client -- nslookup web-headless | awk '/^Name:/ {n=1} n && /^Address/ {print $NF}' > {WS}/lesson-5.3/headless.txt
cat {WS}/lesson-5.3/headless.txt"""),
show=[cmd(r"""k -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80
k -n svc exec client -- nslookup web-headless""", r"""Two addresses: the Pods' own. Again in a few seconds if it does not resolve yet."""),
hand(r"""vim {WS}/lesson-5.3/headless.txt""",
r"""The two addresses, one per line.""",
run=r"""sleep 5
kubectl -n svc exec client -- nslookup web-headless | awk '/^Name:/ {n=1} n && /^Address/ {print $NF}' > {WS}/lesson-5.3/headless.txt""")]),
step("ExternalName",
r"""Create Service `docs` of type **ExternalName** pointing at `kubernetes.io`. Inside the cluster, `docs.svc.svc.cluster.local` is now a CNAME for that name - DNS only: nothing is proxied, and the port does not matter.""",
check=r"""eq "$(j -n svc svc docs '{.spec.type} {.spec.externalName}')" "ExternalName kubernetes.io" """,
hint=r"""`kubectl -n svc create service externalname docs --external-name=kubernetes.io`.""",
show=r"""kubectl -n svc create service externalname docs --external-name=kubernetes.io
kubectl -n svc get svc docs
kubectl -n svc exec client -- nslookup docs | head -6"""),
show=[cmd(r"""k -n svc create svc externalname docs --external-name=kubernetes.io
k -n svc get svc docs
k -n svc exec client -- nslookup docs | head -6""")]),
step("A Service without a selector",
r"""Create Service `legacy` (port 80) with **no selector**, and an EndpointSlice for it by hand that holds `legacy-box`'s Pod IP on port 80. From `client`, `wget -qO- legacy` then answers `legacy`.
@ -284,8 +393,32 @@ An EndpointSlice belongs to a Service by the label `kubernetes.io/service-name`;
[ -z "$(j -n svc svc legacy '{.spec.selector}')" ] || exit 1
for i in 1 2 3 4 5; do [ "$($K -n svc exec client -- wget -qO- -T3 legacy 2>/dev/null)" = legacy ] && exit 0; sleep 3; done; exit 1""",
hint=r"""Service: `spec.ports: [{port: 80, targetPort: 80}]` and no `selector`. EndpointSlice (`discovery.k8s.io/v1`): `addressType: IPv4`, `endpoints: [{addresses: [IP]}]`, `ports: [{port: 80, protocol: TCP}]`, and the label.""",
show=r"""ip=$(kubectl -n svc get pod legacy-box -o jsonpath='{.status.podIP}')
kubectl -n svc apply -f - <<EOF
show=[cmd(r"""k -n svc get po legacy-box -o wide""", r"""Its IP goes into the EndpointSlice."""),
hand(r"""vim legacy.yaml""",
r"""No `kubectl create` makes a Service without a selector, nor an EndpointSlice: paste them from the docs page *Service* (*Services without selectors*), and make them, with that IP:
```
apiVersion: v1
kind: Service
metadata:
name: legacy
spec:
ports:
- {port: 80, targetPort: 80, protocol: TCP}
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: legacy-1
labels:
kubernetes.io/service-name: legacy
addressType: IPv4
endpoints:
- addresses: ["IP"]
ports:
- {port: 80, protocol: TCP}
```""",
run=r"""ip=$(kubectl -n svc get pod legacy-box -o jsonpath='{.status.podIP}')
cat > legacy.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
@ -305,8 +438,9 @@ endpoints:
- addresses: ["$ip"]
ports:
- {port: 80, protocol: TCP}
EOF
kubectl -n svc exec client -- wget -qO- -T3 legacy"""),
EOF"""),
cmd(r"""k -n svc apply -f legacy.yaml
k -n svc exec client -- wget -qO- -T3 legacy""")]),
],
"outro": r"""You exposed the same Pods four ways, saw what headless DNS returns, aliased an outside name, and backed a Service with endpoints you wrote yourself.
@ -348,7 +482,9 @@ mkdir -p "$WS/lesson-5.4"
eq "$(j gatewayclass example '{.spec.controllerName}')" example.com/gateway-controller || exit 1
eq "$(j -n gw gateway shop '{.spec.gatewayClassName} {.spec.listeners[0].name} {.spec.listeners[0].protocol} {.spec.listeners[0].port} {.spec.listeners[0].hostname} {.spec.listeners[0].allowedRoutes.namespaces.from}')" "example http HTTP 80 shop.example.com Same" """,
hint=r"""Both are `gateway.networking.k8s.io/v1`. `kubectl explain gateway.spec.listeners` once the CRDs are in; the docs at gateway-api.sigs.k8s.io have every example.""",
show=r"""kubectl apply -f - <<'EOF'
show=[hand(r"""vim gw.yaml""",
r"""No `kubectl create` for these: paste a GatewayClass and a Gateway from gateway-api.sigs.k8s.io (*API overview*), and make them:
```
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
@ -371,8 +507,33 @@ spec:
allowedRoutes:
namespaces:
from: Same
EOF
kubectl -n gw get gateway shop"""),
```""",
run=r"""cat > gw.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: example
spec:
controllerName: example.com/gateway-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: shop
namespace: gw
spec:
gatewayClassName: example
listeners:
- name: http
protocol: HTTP
port: 80
hostname: shop.example.com
allowedRoutes:
namespaces:
from: Same
EOF"""),
cmd(r"""k apply -f gw.yaml
k -n gw get gateway shop""")]),
step("Route by path",
r"""Create HTTPRoute `shop` in `gw`, attached to Gateway `shop`: requests for paths under `/api` go to Service `api` port 8080, everything else to `web` port 80.""",
check=r"""
@ -386,7 +547,9 @@ def to(path):
return [(b["name"], b.get("port")) for b in x["backendRefs"]]
sys.exit(0 if p == "shop" and to("/api") == [("api", 8080)] and to("/") == [("web", 80)] else 1)'""",
hint=r"""`parentRefs: [{name: shop}]`, then two `rules`, each with `matches: [{path: {type: PathPrefix, value: ...}}]` and `backendRefs: [{name: ..., port: ...}]`.""",
show=r"""kubectl -n gw apply -f - <<'EOF'
show=[hand(r"""vim route.yaml""",
r"""Paste an HTTPRoute from gateway-api.sigs.k8s.io (*HTTP routing*), and make it:
```
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
@ -404,8 +567,28 @@ spec:
- path: {type: PathPrefix, value: /}
backendRefs:
- {name: web, port: 80}
EOF
kubectl -n gw describe httproute shop | grep -A3 'Backend Refs' | head -8"""),
```""",
run=r"""cat > route.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop
spec:
parentRefs:
- name: shop
hostnames: [shop.example.com]
rules:
- matches:
- path: {type: PathPrefix, value: /api}
backendRefs:
- {name: api, port: 8080}
- matches:
- path: {type: PathPrefix, value: /}
backendRefs:
- {name: web, port: 80}
EOF"""),
cmd(r"""k -n gw apply -f route.yaml
k -n gw describe httproute shop""")]),
step("A canary by weight",
r"""Change the `/` rule of route `shop` so that **90%** of its requests go to `web` and **10%** to `web-v2` (both port 80). The `/api` rule stays as it is.""",
check=r"""
@ -416,10 +599,20 @@ root, api = by("/"), by("/api")
w = sorted((b["name"], b.get("weight", 1)) for b in root["backendRefs"]) if root else []
sys.exit(0 if w == [("web", 90), ("web-v2", 10)] and api and [b["name"] for b in api["backendRefs"]] == ["api"] else 1)'""",
hint=r"""Two `backendRefs` in the same rule, each with a `weight`. The weights are relative: 9 and 1 would do the same.""",
show=r"""kubectl -n gw get httproute shop -o yaml > {WS}/lesson-5.4/shop.yaml
show=[hand(r"""k -n gw edit httproute shop""",
r"""In the rule whose path is `/`, give `web` a `weight: 90` and add `web-v2` beside it:
```
- name: web
port: 80
weight: 90
- name: web-v2
port: 80
weight: 10
```""",
run=r"""kubectl -n gw get httproute shop -o yaml > {WS}/lesson-5.4/shop.yaml
yq -i '(.spec.rules[] | select(.matches[0].path.value == "/")).backendRefs = [{"name": "web", "port": 80, "weight": 90}, {"name": "web-v2", "port": 80, "weight": 10}]' {WS}/lesson-5.4/shop.yaml
kubectl apply -f {WS}/lesson-5.4/shop.yaml
kubectl -n gw get httproute shop -o jsonpath='{.spec.rules[1].backendRefs}'; echo"""),
kubectl apply -f {WS}/lesson-5.4/shop.yaml"""),
cmd(r"""k -n gw describe httproute shop""")]),
step("Beta testers by header",
r"""Add HTTPRoute `shop-beta` in `gw`, attached to `shop`: requests carrying the header `x-env: beta` go **all** to `web-v2`. A more specific match wins, so these requests no longer take part in the 90/10 split.""",
check=r"""
@ -428,7 +621,9 @@ import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0]
h = r["matches"][0].get("headers", [{}])[0]
sys.exit(0 if s["parentRefs"][0]["name"] == "shop" and h.get("name", "").lower() == "x-env" and h.get("value") == "beta" and [b["name"] for b in r["backendRefs"]] == ["web-v2"] else 1)'""",
hint=r"""`matches: [{headers: [{name: x-env, value: beta}]}]` (type `Exact` by default).""",
show=r"""kubectl -n gw apply -f - <<'EOF'
show=[hand(r"""vim beta.yaml""",
r"""Paste an HTTPRoute with a header match from gateway-api.sigs.k8s.io (*HTTP header matching*), and make it:
```
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
@ -443,8 +638,25 @@ spec:
- {name: x-env, value: beta}
backendRefs:
- {name: web-v2, port: 80}
EOF
kubectl -n gw get httproute"""),
```""",
run=r"""cat > beta.yaml <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop-beta
spec:
parentRefs:
- name: shop
hostnames: [shop.example.com]
rules:
- matches:
- headers:
- {name: x-env, value: beta}
backendRefs:
- {name: web-v2, port: 80}
EOF"""),
cmd(r"""k -n gw apply -f beta.yaml
k -n gw get httproute""")]),
step("HTTPS on the Gateway",
r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `gw`, and add a second listener to Gateway `shop`: `https`, protocol HTTPS, port 443, the same hostname, TLS mode `Terminate` with that Secret as its certificate.""",
check=r"""
@ -452,12 +664,27 @@ eq "$(j -n gw secret shop-tls '{.type}')" kubernetes.io/tls || exit 1
eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="https")].protocol} {.spec.listeners[?(@.name=="https")].port} {.spec.listeners[?(@.name=="https")].tls.mode} {.spec.listeners[?(@.name=="https")].tls.certificateRefs[0].name}')" "HTTPS 443 Terminate shop-tls" \
&& eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="http")].port}')" 80""",
hint=r"""`openssl req -x509 ...`, `kubectl create secret tls`, then a listener with `tls: {mode: Terminate, certificateRefs: [{name: shop-tls}]}`. Keep the `http` listener.""",
show=r"""cd {WS}/lesson-5.4
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
kubectl -n gw create secret tls shop-tls --cert=tls.crt --key=tls.key
kubectl -n gw patch gateway shop --type json -p '[{"op":"add","path":"/spec/listeners/-","value":{"name":"https","protocol":"HTTPS","port":443,"hostname":"shop.example.com","tls":{"mode":"Terminate","certificateRefs":[{"name":"shop-tls"}]},"allowedRoutes":{"namespaces":{"from":"Same"}}}}]'
kubectl -n gw get gateway shop -o jsonpath='{range .spec.listeners[*]}{.name} {.port}{"\n"}{end}'
show=[cmd(r"""cd {WS}/lesson-5.4
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com
k -n gw create secret tls shop-tls --cert=tls.crt --key=tls.key
cd"""),
hand(r"""k -n gw edit gateway shop""",
r"""Add a second listener below `http`, in `listeners:`:
```
- name: https
protocol: HTTPS
port: 443
hostname: shop.example.com
tls:
mode: Terminate
certificateRefs:
- name: shop-tls
allowedRoutes:
namespaces:
from: Same
```""",
run=r"""kubectl -n gw patch gateway shop --type json -p '[{"op":"add","path":"/spec/listeners/-","value":{"name":"https","protocol":"HTTPS","port":443,"hostname":"shop.example.com","tls":{"mode":"Terminate","certificateRefs":[{"name":"shop-tls"}]},"allowedRoutes":{"namespaces":{"from":"Same"}}}}]' """),
cmd(r"""k -n gw describe gateway shop | grep -A3 -i listener""")]),
],
"outro": r"""You built a Gateway with HTTP and HTTPS listeners, routed by path, split traffic by weight, and sent one header's requests elsewhere.
@ -498,7 +725,9 @@ mkdir -p "$WS/lesson-5.5"
r"""Create IngressClass `nginx` for controller `k8s.io/ingress-nginx`, and make it the cluster's **default** class: an Ingress without `ingressClassName` then belongs to it.""",
check=r"""eq "$(j ingressclass nginx '{.spec.controller} {.metadata.annotations.ingressclass\.kubernetes\.io/is-default-class}')" "k8s.io/ingress-nginx true" """,
hint=r"""`networking.k8s.io/v1`, kind `IngressClass`, `spec.controller`, and the annotation `ingressclass.kubernetes.io/is-default-class: "true"`. Installing ingress-nginx makes this object for you.""",
show=r"""kubectl apply -f - <<'EOF'
show=[hand(r"""vim ic.yaml""",
r"""No `kubectl create` for an IngressClass: paste the example from the docs page *Ingress* (*Default IngressClass*), and make it:
```
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
@ -507,8 +736,19 @@ metadata:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: k8s.io/ingress-nginx
EOF
kubectl get ingressclass"""),
```""",
run=r"""cat > ic.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: nginx
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: k8s.io/ingress-nginx
EOF"""),
cmd(r"""k apply -f ic.yaml
k get ingressclass""")]),
step("Host and paths, from the command line",
r"""Create Ingress `shop` in `ing` with class `nginx` and host `shop.example.com`: paths under `/api` go to `api:8080`, everything else to `web:80`. `kubectl create ingress` can do it in one line.""",
check=r"""
@ -517,8 +757,8 @@ import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0]
p = {x["path"]: (x["pathType"], x["backend"]["service"]["name"], x["backend"]["service"]["port"]["number"]) for x in r["http"]["paths"]}
sys.exit(0 if s.get("ingressClassName") == "nginx" and r["host"] == "shop.example.com" and p.get("/api") == ("Prefix", "api", 8080) and p.get("/") == ("Prefix", "web", 80) else 1)'""",
hint=r"""`kubectl create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'` - a trailing `*` means `Prefix`, without it `Exact`.""",
show=r"""kubectl -n ing create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'
kubectl -n ing describe ingress shop | sed -n '/Rules/,/Annotations/p'"""),
show=[cmd(r"""k -n ing create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'
k -n ing describe ingress shop""")]),
step("An exact path",
r"""Add a rule to Ingress `shop`: exactly `/healthz` (pathType `Exact`) goes to `health:8081`. `/healthz/deep` would not match it - it would fall through to the `/` prefix.""",
check=r"""
@ -527,26 +767,53 @@ import json, sys; ps = json.load(sys.stdin)["spec"]["rules"][0]["http"]["paths"]
h = [x for x in ps if x["path"] == "/healthz"]
sys.exit(0 if h and h[0]["pathType"] == "Exact" and h[0]["backend"]["service"]["name"] == "health" and h[0]["backend"]["service"]["port"]["number"] == 8081 and len(ps) == 3 else 1)'""",
hint=r"""`kubectl edit ingress shop` and add a third entry under `paths:`, or a JSON patch that appends to `/spec/rules/0/http/paths`.""",
show=r"""kubectl -n ing patch ingress shop --type json -p '[{"op":"add","path":"/spec/rules/0/http/paths/-","value":{"path":"/healthz","pathType":"Exact","backend":{"service":{"name":"health","port":{"number":8081}}}}}]'
kubectl -n ing get ingress shop -o jsonpath='{range .spec.rules[0].http.paths[*]}{.pathType} {.path} -> {.backend.service.name}{"\n"}{end}'"""),
show=[hand(r"""k -n ing edit ingress shop""",
r"""Add a third path to `paths:`, indented like the other two:
```
- path: /healthz
pathType: Exact
backend:
service:
name: health
port:
number: 8081
```""",
run=r"""kubectl -n ing patch ingress shop --type json -p '[{"op":"add","path":"/spec/rules/0/http/paths/-","value":{"path":"/healthz","pathType":"Exact","backend":{"service":{"name":"health","port":{"number":8081}}}}}]' """),
cmd(r"""k -n ing describe ingress shop""")]),
step("TLS",
r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `ing`, and add a `tls` section to Ingress `shop` for that host with that Secret. The controller would then serve HTTPS for it and, by default, redirect HTTP there.""",
check=r"""
eq "$(j -n ing secret shop-tls '{.type}')" kubernetes.io/tls && eq "$(j -n ing ingress shop '{.spec.tls[0].hosts[0]} {.spec.tls[0].secretName}')" "shop.example.com shop-tls" \
&& j -n ing secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q shop.example.com""",
hint=r"""`spec.tls: [{hosts: [shop.example.com], secretName: shop-tls}]`. The Secret must be in the Ingress's Namespace.""",
show=r"""cd {WS}/lesson-5.5
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
kubectl -n ing create secret tls shop-tls --cert=tls.crt --key=tls.key
kubectl -n ing patch ingress shop --type merge -p '{"spec":{"tls":[{"hosts":["shop.example.com"],"secretName":"shop-tls"}]}}'
kubectl -n ing get ingress shop
show=[cmd(r"""cd {WS}/lesson-5.5
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com
k -n ing create secret tls shop-tls --cert=tls.crt --key=tls.key
cd"""),
hand(r"""k -n ing edit ingress shop""",
r"""Add to `spec:`, beside `rules:`:
```
tls:
- hosts: [shop.example.com]
secretName: shop-tls
```""",
run=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"tls":[{"hosts":["shop.example.com"],"secretName":"shop-tls"}]}}' """),
cmd(r"""k -n ing get ingress shop""")]),
step("What matches nothing",
r"""Give Ingress `shop` a **default backend**: `web:80`, for requests whose host or path no rule matches.""",
check=r"""eq "$(j -n ing ingress shop '{.spec.defaultBackend.service.name} {.spec.defaultBackend.service.port.number}')" "web 80" """,
hint=r"""`spec.defaultBackend: {service: {name: web, port: {number: 80}}}`; `kubectl create ingress --default-backend=web:80` does it at creation.""",
show=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"defaultBackend":{"service":{"name":"web","port":{"number":80}}}}}'
kubectl -n ing describe ingress shop | head -12"""),
show=[hand(r"""k -n ing edit ingress shop""",
r"""Add to `spec:`, beside `rules:`:
```
defaultBackend:
service:
name: web
port:
number: 80
```""",
run=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"defaultBackend":{"service":{"name":"web","port":{"number":80}}}}}' """),
cmd(r"""k -n ing describe ingress shop""")]),
],
"outro": r"""You made a default IngressClass and an Ingress with host, prefix and exact paths, TLS and a default backend.
@ -608,16 +875,18 @@ ready dns-lab && $K -n dns-lab rollout status sts db --timeout=180s >/dev/null
r"""From `client`, look up `web` by its **fully qualified** name, and write the address you get to `{WS}/lesson-5.6/web.txt`. It is the Service's ClusterIP.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/web.txt" 2>/dev/null)" "$(j -n dns-lab svc web '{.spec.clusterIP}')" """,
hint=r"""`kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local`.""",
show=r"""kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local
kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local | awk '/^Name:/ {n=1} n && /^Address/ {print $NF; exit}' > {WS}/lesson-5.6/web.txt"""),
show=[cmd(r"""k -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local""", r"""The `Address` below `Name:` is the Service's ClusterIP."""),
hand(r"""echo IP > {WS}/lesson-5.6/web.txt""",
r"""`IP`: that address.""",
run=r"""kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local | awk '/^Name:/ {n=1} n && /^Address/ {print $NF; exit}' > {WS}/lesson-5.6/web.txt""")]),
step("One Pod of a StatefulSet",
r"""A StatefulSet's Pods get stable names through their headless Service. Write the fully qualified DNS name of Pod `db-0` to `{WS}/lesson-5.6/db0.txt`, and check from `client` that it resolves to `db-0`'s IP.""",
check=DNS + r"""
eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/db0.txt" 2>/dev/null | sed 's/\.$//')" db-0.db.dns-lab.svc.cluster.local && resolves dns-lab client db-0.db.dns-lab.svc.cluster.local "$(j -n dns-lab pod db-0 '{.status.podIP}')" """,
hint=r"""`POD.SERVICE.NAMESPACE.svc.cluster.local` - the Service is the StatefulSet's `serviceName`.""",
show=r"""echo db-0.db.dns-lab.svc.cluster.local > {WS}/lesson-5.6/db0.txt
kubectl -n dns-lab exec client -- nslookup db-0.db.dns-lab.svc.cluster.local
kubectl -n dns-lab get pod db-0 -o wide"""),
show=[cmd(r"""echo db-0.db.dns-lab.svc.cluster.local > {WS}/lesson-5.6/db0.txt
k -n dns-lab exec client -- nslookup db-0.db.dns-lab.svc.cluster.local
k -n dns-lab get po db-0 -o wide""")]),
step("Names of your own",
r"""Add a `hosts` block to CoreDNS's Corefile so that every Pod resolves **`legacy.internal`** to `10.96.0.250`, and every other name still resolves as before (`fallthrough`). Put it inside the `.:53 { ... }` server block, before `forward`.
@ -626,12 +895,19 @@ CoreDNS reloads the changed ConfigMap by itself within a minute; `client` then r
$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep -q 'kubernetes cluster.local' || exit 1
resolves dns-lab client legacy.internal 10.96.0.250 && resolves dns-lab client web.dns-lab.svc.cluster.local "$(j -n dns-lab svc web '{.spec.clusterIP}')" """,
hint=r"""`kubectl -n kube-system edit cm coredns`. The block: `hosts {` / `10.96.0.250 legacy.internal` / `fallthrough` / `}`. A syntax error leaves CoreDNS on its old configuration: `kubectl -n kube-system logs -l k8s-app=kube-dns` says so.""",
show=r"""kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > {WS}/lesson-5.6/Corefile
show=[hand(r"""k -n kube-system edit cm coredns""",
r"""In `Corefile`, add above the `forward` line, indented like it:
```
hosts {
10.96.0.250 legacy.internal
fallthrough
}
```""",
run=r"""kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > {WS}/lesson-5.6/Corefile
sed -i '0,/^ forward/s// hosts {\n 10.96.0.250 legacy.internal\n fallthrough\n }\n forward/' {WS}/lesson-5.6/Corefile
cat {WS}/lesson-5.6/Corefile
kubectl -n kube-system create configmap coredns --from-file=Corefile={WS}/lesson-5.6/Corefile --dry-run=client -o yaml | kubectl apply -f -
sleep 40
kubectl -n dns-lab exec client -- nslookup legacy.internal"""),
sleep 40"""),
cmd(r"""k -n dns-lab exec client -- nslookup legacy.internal""", r"""Again until it answers `10.96.0.250`: CoreDNS reloads the ConfigMap by itself within a minute or two.""")]),
step("A Pod's own resolver settings",
r"""Run Pod `tuned` in `dns-lab` (`busybox:1.36`, `sleep 86400`) whose resolver you set yourself: `dnsPolicy: None`, with the `kube-dns` Service's IP as its only nameserver, the search domains `dns-lab.svc.cluster.local` and `svc.cluster.local`, and the option `ndots: 2`. Its `/etc/resolv.conf` shows exactly that.""",
check=r"""
@ -640,26 +916,21 @@ r=$($K -n dns-lab exec tuned -- cat /etc/resolv.conf 2>/dev/null)
echo "$r" | grep -qx "nameserver $ip" && echo "$r" | grep -qx 'search dns-lab.svc.cluster.local svc.cluster.local' && echo "$r" | grep -q 'ndots:2' \
&& eq "$(j -n dns-lab pod tuned '{.spec.dnsPolicy}')" None""",
hint=r"""`spec.dnsPolicy: None` and `spec.dnsConfig: {nameservers: [IP], searches: [...], options: [{name: ndots, value: "2"}]}`. The other policies: `ClusterFirst` (the default), `Default` (the node's resolver), `ClusterFirstWithHostNet`.""",
show=r"""ip=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}')
kubectl -n dns-lab apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: tuned
spec:
dnsPolicy: None
show=[cmd(r"""k -n kube-system get svc kube-dns""", r"""Its `CLUSTER-IP` is the nameserver."""),
cmd(r"""k -n dns-lab run tuned --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > tuned.yaml"""),
hand(r"""vim tuned.yaml""",
r"""Change `dnsPolicy: ClusterFirst` to `dnsPolicy: None`, and add beside it (the docs page *DNS for Services and Pods* has it, under *Pod's DNS Config*):
```
dnsConfig:
nameservers: [$ip]
nameservers: [IP]
searches: [dns-lab.svc.cluster.local, svc.cluster.local]
options:
- {name: ndots, value: "2"}
containers:
- name: tuned
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n dns-lab wait --for=condition=Ready pod/tuned --timeout=120s
kubectl -n dns-lab exec tuned -- cat /etc/resolv.conf"""),
```""",
run=r"""export IP=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}')
yq -i '.spec.dnsPolicy = "None" | .spec.dnsConfig = {"nameservers": [strenv(IP)], "searches": ["dns-lab.svc.cluster.local", "svc.cluster.local"], "options": [{"name": "ndots", "value": "2"}]}' tuned.yaml"""),
cmd(r"""k -n dns-lab apply -f tuned.yaml
k -n dns-lab exec tuned -- cat /etc/resolv.conf""")]),
],
"outro": r"""You resolved a Service and a StatefulSet Pod by their full names, taught CoreDNS a name of your own, and set a Pod's resolver by hand.

431
cka/lessons_storage.py
View File

@ -1,5 +1,5 @@
"""CKA lessons of the Storage domain (1.1-1.3). See cka/lessons.py for what a lesson is."""
from sim.bank import NS, step
from sim.bank import NS, step, cmd, hand
# ---- 1.1 StorageClasses and dynamic provisioning ---------------------------------------------
L11 = {
@ -29,9 +29,10 @@ mkdir -p "$WS/lesson-1.1"
want="$(j sc standard '{.metadata.name} {.provisioner} {.volumeBindingMode}')"
[ -n "$want" ] && eq "$(tr -s ' \t' ' ' < "$WS/lesson-1.1/default.txt" 2>/dev/null | tr -d '\r\n' | sed 's/ *$//')" "$want" """,
hint=r"""`kubectl get sc` shows all three columns. For exactly the values: `-o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}'`.""",
show=r"""kubectl get sc
kubectl get sc standard -o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}' > {WS}/lesson-1.1/default.txt
cat {WS}/lesson-1.1/default.txt"""),
show=[cmd(r"""k get sc"""),
hand(r"""echo 'NAME PROVISIONER MODE' > {WS}/lesson-1.1/default.txt""",
r"""From the `(default)` line: its name, `PROVISIONER` and `VOLUMEBINDINGMODE`.""",
run=r"""kubectl get sc standard -o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}' > {WS}/lesson-1.1/default.txt""")]),
step("A class that keeps its volumes",
r"""Create StorageClass `keep`: the same provisioner as `standard` (`rancher.io/local-path`), binding mode `WaitForFirstConsumer`, and reclaim policy **`Retain`**, so a volume outlives its claim.
@ -40,7 +41,9 @@ There is no `kubectl create storageclass`: write the manifest. `kubectl get sc s
eq "$(j sc keep '{.provisioner} {.reclaimPolicy} {.volumeBindingMode}')" "rancher.io/local-path Retain WaitForFirstConsumer" \
&& [ "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""",
hint=r"""A StorageClass has no `spec`: `provisioner`, `reclaimPolicy` and `volumeBindingMode` sit at the top level, beside `metadata`. Its `apiVersion` is `storage.k8s.io/v1`.""",
show=r"""kubectl apply -f - <<'EOF'
show=[hand(r"""vim keep.yaml""",
r"""No `kubectl create` for a StorageClass: paste the example from the docs page *Storage Classes*, and make it:
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
@ -48,8 +51,18 @@ metadata:
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
EOF
kubectl get sc"""),
```""",
run=r"""cat > keep.yaml <<'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: keep
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
EOF"""),
cmd(r"""k apply -f keep.yaml
k get sc""")]),
step("A claim that waits",
r"""In Namespace `store`, create PersistentVolumeClaim `ledger`: class `keep`, `ReadWriteOnce`, `100Mi`.
@ -58,7 +71,9 @@ It stays `Pending`, and that is correct. `kubectl -n store describe pvc ledger`
eq "$(j -n store pvc ledger '{.spec.storageClassName}')" keep && eq "$(j -n store pvc ledger '{.spec.resources.requests.storage}')" 100Mi \
&& grep -qx 'WaitForFirstConsumer' <(tr -d ' \r' < "$WS/lesson-1.1/why.txt" 2>/dev/null)""",
hint=r"""The events at the bottom of `describe` have a Reason column. A claim is namespaced; the class is not.""",
show=r"""kubectl -n store apply -f - <<'EOF'
show=[hand(r"""vim ledger.yaml""",
r"""No `kubectl create` for a claim: paste the example from the docs page *Persistent Volumes*, and make it:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@ -69,9 +84,22 @@ spec:
resources:
requests:
storage: 100Mi
EOF
kubectl -n store describe pvc ledger | tail -4
echo WaitForFirstConsumer > {WS}/lesson-1.1/why.txt"""),
```""",
run=r"""cat > ledger.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger
spec:
storageClassName: keep
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 100Mi
EOF"""),
cmd(r"""k -n store apply -f ledger.yaml
k -n store describe pvc ledger | tail -4""", r"""The event's reason: `WaitForFirstConsumer`."""),
cmd(r"""echo WaitForFirstConsumer > {WS}/lesson-1.1/why.txt""")]),
step("Give it a consumer",
r"""Run Pod `bookkeeper` in `store` (image `busybox:1.36`) that mounts claim `ledger` at `/data` and runs `sh -c 'echo first > /data/entry.txt && sleep 86400'`.
@ -81,27 +109,22 @@ eq "$(j -n store pvc ledger '{.status.phase}')" Bound || exit 1
pv=$(j -n store pvc ledger '{.spec.volumeName}'); eq "$(j pv "$pv" '{.spec.persistentVolumeReclaimPolicy}')" Retain || exit 1
eq "$($K -n store exec bookkeeper -- cat /data/entry.txt 2>/dev/null)" first""",
hint=r"""`kubectl run bookkeeper --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'` gives you the Pod; add `volumes:` (with `persistentVolumeClaim.claimName`) at Pod level and `volumeMounts:` in the container.""",
show=r"""kubectl -n store apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: bookkeeper
spec:
containers:
- name: bookkeeper
image: busybox:1.36
command: [sh, -c, 'echo first > /data/entry.txt && sleep 86400']
show=[cmd(r"""k -n store run bookkeeper --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo first > /data/entry.txt && sleep 86400' > bk.yaml"""),
hand(r"""vim bk.yaml""",
r"""Add the mount to the container, below `resources: {}`:
```
volumeMounts:
- name: data
mountPath: /data
- {name: data, mountPath: /data}
```
and the volume to the Pod's `spec:`, beside `restartPolicy:`:
```
volumes:
- name: data
persistentVolumeClaim:
claimName: ledger
EOF
kubectl -n store wait --for=condition=Ready pod/bookkeeper --timeout=120s
kubectl -n store get pvc ledger
kubectl get pv"""),
- {name: data, persistentVolumeClaim: {claimName: ledger}}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}] | .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "ledger"}}]' bk.yaml"""),
cmd(r"""k -n store apply -f bk.yaml
k -n store get pvc ledger
k get pv""", r"""Again once the Pod runs: the claim is `Bound`, and its volume has `RECLAIM POLICY` `Retain`.""")]),
step("Delete the claim, keep the volume",
r"""Delete Pod `bookkeeper` and claim `ledger`. Because of `Retain`, the volume stays, with status `Released`: its data is still on disk, and no new claim can bind it until an admin says so.
@ -111,19 +134,22 @@ Write the PersistentVolume's name to `{WS}/lesson-1.1/released.txt`.""",
pv=$(tr -d ' \r\n' < "$WS/lesson-1.1/released.txt" 2>/dev/null); [ -n "$pv" ] || exit 1
eq "$(j pv "$pv" '{.status.phase} {.spec.claimRef.namespace}/{.spec.claimRef.name}')" "Released store/ledger" """,
hint=r"""Read the volume name off the claim before you delete it (`kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}'`), or find it afterwards in `kubectl get pv`: the CLAIM column still names `store/ledger`.""",
show=r"""kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}' > {WS}/lesson-1.1/released.txt
kubectl -n store delete pod bookkeeper --grace-period=1
kubectl -n store delete pvc ledger
kubectl get pv"""),
show=[cmd(r"""k -n store get pvc ledger""", r"""`VOLUME` is the PersistentVolume's name."""),
hand(r"""echo PV > {WS}/lesson-1.1/released.txt""",
r"""`PV`: that name.""",
run=r"""kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}' > {WS}/lesson-1.1/released.txt"""),
cmd(r"""k -n store delete po bookkeeper --now
k -n store delete pvc ledger
k get pv""", r"""The volume stays, `Released`.""")]),
step("Change the default class",
r"""Make `keep` the cluster's default class, and `standard` no longer the default. Two defaults would be ambiguous: set the annotation to `"true"` on one and `"false"` on the other.""",
check=r"""
eq "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" true \
&& [ "$(j sc standard '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""",
hint=r"""`kubectl annotate sc NAME storageclass.kubernetes.io/is-default-class=true --overwrite`, or `kubectl patch` with a `metadata.annotations` map.""",
show=r"""kubectl annotate sc standard storageclass.kubernetes.io/is-default-class=false --overwrite
kubectl annotate sc keep storageclass.kubernetes.io/is-default-class=true --overwrite
kubectl get sc"""),
show=[cmd(r"""k annotate sc standard storageclass.kubernetes.io/is-default-class=false --overwrite
k annotate sc keep storageclass.kubernetes.io/is-default-class=true --overwrite
k get sc""")]),
],
"outro": r"""You read a class, made one that keeps its volumes, watched a claim wait for its Pod, kept a volume past its claim, and moved the default.
@ -172,29 +198,29 @@ The reader sees what the writer wrote: an emptyDir is shared by every container
check=r"""eq "$($K -n vols exec pair -c reader -- cat /shared/msg 2>/dev/null)" hello \
&& [ -n "$(j -n vols pod pair '{.spec.volumes[?(@.name=="shared")].emptyDir}')" ]""",
hint=r"""Both containers list the same `volumeMounts` entry (`name: shared`); the Pod lists the volume once, as `emptyDir: {}`.""",
show=r"""kubectl -n vols apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: pair
spec:
show=[cmd(r"""k -n vols run pair --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo hello > /shared/msg && sleep 86400' > pair.yaml"""),
hand(r"""vim pair.yaml""",
r"""Name the container `writer` and give it the mount; add `reader` below it; add the volume:
```
containers:
- name: writer
image: busybox:1.36
command: [sh, -c, 'echo hello > /shared/msg && sleep 86400']
args: [sh, -c, 'echo hello > /shared/msg && sleep 86400']
volumeMounts:
- {name: shared, mountPath: /shared}
- name: reader
image: busybox:1.36
command: [sleep, '86400']
args: [sleep, '86400']
volumeMounts:
- {name: shared, mountPath: /shared}
volumes:
- name: shared
emptyDir: {}
EOF
kubectl -n vols wait --for=condition=Ready pod/pair --timeout=120s
kubectl -n vols exec pair -c reader -- cat /shared/msg"""),
- {name: shared, emptyDir: {}}
```""",
run=r"""yq -i '.spec.containers[0].name = "writer" | .spec.containers[0].volumeMounts = [{"name": "shared", "mountPath": "/shared"}]
| .spec.containers += [{"name": "reader", "image": "busybox:1.36", "args": ["sleep", "86400"], "volumeMounts": [{"name": "shared", "mountPath": "/shared"}]}]
| .spec.volumes = [{"name": "shared", "emptyDir": {}}]' pair.yaml"""),
cmd(r"""k -n vols apply -f pair.yaml
k -n vols exec pair -c reader -- cat /shared/msg""", r"""Once it runs: `hello`.""")]),
step("Scratch space in memory",
r"""Create Pod `scratch` in `vols` (`busybox:1.36`, `sleep 86400`) with an emptyDir at `/scratch` that is held in **memory**, at most `64Mi`.
@ -203,24 +229,21 @@ Inside the Pod, `/proc/mounts` shows the difference: `/scratch` is a `tmpfs`. Wh
eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.medium}')" Memory && eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.sizeLimit}')" 64Mi \
&& $K -n vols exec scratch -- grep -q ' /scratch tmpfs ' /proc/mounts 2>/dev/null""",
hint=r"""`emptyDir: {medium: Memory, sizeLimit: 64Mi}`.""",
show=r"""kubectl -n vols apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: scratch
spec:
containers:
- name: scratch
image: busybox:1.36
command: [sleep, '86400']
show=[cmd(r"""k -n vols run scratch --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > scratch.yaml"""),
hand(r"""vim scratch.yaml""",
r"""Add the mount to the container, below `resources: {}`:
```
volumeMounts:
- {name: tmp, mountPath: /scratch}
```
and the volume to the Pod's `spec:`, beside `restartPolicy:`:
```
volumes:
- name: tmp
emptyDir: {medium: Memory, sizeLimit: 64Mi}
EOF
kubectl -n vols wait --for=condition=Ready pod/scratch --timeout=120s
kubectl -n vols exec scratch -- grep /scratch /proc/mounts"""),
- {name: tmp, emptyDir: {medium: Memory, sizeLimit: 64Mi}}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "tmp", "mountPath": "/scratch"}] | .spec.volumes = [{"name": "tmp", "emptyDir": {"medium": "Memory", "sizeLimit": "64Mi"}}]' scratch.yaml"""),
cmd(r"""k -n vols apply -f scratch.yaml
k -n vols exec scratch -- grep /scratch /proc/mounts""", r"""Once it runs: a `tmpfs`.""")]),
step("One directory from three sources",
r"""Create Pod `bundle` in `vols` (`busybox:1.36`, `sleep 86400`) with **one** `projected` volume mounted at `/etc/bundle`, made of:
@ -233,18 +256,15 @@ Separate volumes would need three mount points; a projected volume puts them sid
x() { $K -n vols exec bundle -- cat "/etc/bundle/$1" 2>/dev/null; }
[ -n "$(j -n vols pod bundle '{.spec.volumes[0].projected}')" ] && eq "$(x mode)" prod && eq "$(x password)" s3cr3t && eq "$(x podname)" bundle""",
hint=r"""`projected: {sources: [...]}` - each source is `configMap: {name: ...}`, `secret: {name: ...}` or `downwardAPI: {items: [{path: podname, fieldRef: {fieldPath: metadata.name}}]}`. `kubectl explain pod.spec.volumes.projected.sources` lists them.""",
show=r"""kubectl -n vols apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: bundle
spec:
containers:
- name: app
image: busybox:1.36
command: [sleep, '86400']
show=[cmd(r"""k -n vols run bundle --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > bundle.yaml"""),
hand(r"""vim bundle.yaml""",
r"""Add the mount to the container, below `resources: {}`:
```
volumeMounts:
- {name: bundle, mountPath: /etc/bundle, readOnly: true}
```
and the volume to the Pod's `spec:`, beside `restartPolicy:` (the docs page *Projected Volumes* has an example):
```
volumes:
- name: bundle
projected:
@ -255,9 +275,11 @@ spec:
items:
- path: podname
fieldRef: {fieldPath: metadata.name}
EOF
kubectl -n vols wait --for=condition=Ready pod/bundle --timeout=120s
kubectl -n vols exec bundle -- ls /etc/bundle"""),
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "bundle", "mountPath": "/etc/bundle", "readOnly": true}]
| .spec.volumes = [{"name": "bundle", "projected": {"sources": [{"configMap": {"name": "app-config"}}, {"secret": {"name": "app-secret"}}, {"downwardAPI": {"items": [{"path": "podname", "fieldRef": {"fieldPath": "metadata.name"}}]}}]}}]' bundle.yaml"""),
cmd(r"""k -n vols apply -f bundle.yaml
k -n vols exec bundle -- ls /etc/bundle""", r"""Once it runs: `mode`, `password` and `podname`.""")]),
step("A volume for exactly one Pod",
r"""Create PersistentVolume `archive`: `hostPath` `/srv/archive` (type `DirectoryOrCreate`), `200Mi`, access mode **`ReadWriteOncePod`**, reclaim policy `Retain`, and `storageClassName: ""`.
@ -268,7 +290,9 @@ The empty class matters on both sides. A claim with no class at all gets the def
eq "$(j pv archive '{.spec.accessModes[0]} {.spec.persistentVolumeReclaimPolicy} {.spec.hostPath.path}')" "ReadWriteOncePod Retain /srv/archive" \
&& eq "$(j -n vols pvc archive-claim '{.status.phase} {.spec.volumeName}')" "Bound archive" """,
hint=r"""`storageClassName: ""` is spelled out, with the quotes, under `spec` of both objects. A PersistentVolume is cluster-wide: no namespace.""",
show=r"""kubectl apply -f - <<'EOF'
show=[hand(r"""vim archive.yaml""",
r"""No `kubectl create` for these: paste the PersistentVolume and the claim from the docs page *Configure a Pod to Use a PersistentVolume for Storage*, and make them:
```
apiVersion: v1
kind: PersistentVolume
metadata:
@ -290,9 +314,33 @@ spec:
accessModes: [ReadWriteOncePod]
resources:
requests: {storage: 200Mi}
EOF
kubectl get pv archive
kubectl -n vols get pvc archive-claim"""),
```""",
run=r"""cat > archive.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: archive
spec:
capacity: {storage: 200Mi}
accessModes: [ReadWriteOncePod]
persistentVolumeReclaimPolicy: Retain
storageClassName: ""
hostPath: {path: /srv/archive, type: DirectoryOrCreate}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archive-claim
namespace: vols
spec:
storageClassName: ""
accessModes: [ReadWriteOncePod]
resources:
requests: {storage: 200Mi}
EOF"""),
cmd(r"""k apply -f archive.yaml
k get pv archive
k -n vols get pvc archive-claim""", r"""Both `Bound`, to each other.""")]),
step("Hand a Released volume to a new claim",
r"""Delete claim `archive-claim`. The volume becomes `Released`: it still remembers its old claim in `spec.claimRef`, and nothing binds it while it does.
@ -300,10 +348,14 @@ Make it `Available` again by removing `spec.claimRef` from the PV, then create c
check=r"""
[ -z "$(j -n vols pvc archive-claim '{.metadata.name}')" ] && eq "$(j -n vols pvc archive-claim-2 '{.status.phase} {.spec.volumeName}')" "Bound archive" """,
hint=r"""`kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'` - or `kubectl edit pv archive` and delete the `claimRef:` block.""",
show=r"""kubectl -n vols delete pvc archive-claim
kubectl get pv archive
kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'
kubectl apply -f - <<'EOF'
show=[cmd(r"""k -n vols delete pvc archive-claim
k get pv archive""", r"""`Released`."""),
hand(r"""k edit pv archive""",
r"""Delete the whole `claimRef:` block, then `:wq`. The volume is `Available` again.""",
run=r"""kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' """),
hand(r"""vim claim2.yaml""",
r"""The claim of `archive.yaml`, named `archive-claim-2`:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@ -314,8 +366,21 @@ spec:
accessModes: [ReadWriteOncePod]
resources:
requests: {storage: 200Mi}
EOF
kubectl get pv archive"""),
```""",
run=r"""cat > claim2.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: archive-claim-2
namespace: vols
spec:
storageClassName: ""
accessModes: [ReadWriteOncePod]
resources:
requests: {storage: 200Mi}
EOF"""),
cmd(r"""k apply -f claim2.yaml
k get pv archive""")]),
],
"outro": r"""You shared an emptyDir between containers, put one in memory, combined three sources in one projected volume, and took a `Retain` volume through `Bound`, `Released` and back.
@ -401,7 +466,9 @@ $K -n claims wait --for=condition=Ready pod/grower --timeout=180s >/dev/null
Without the selector it would bind `pv-small`, the smallest volume that fits. With it, it binds `pv-large`.""",
check=r"""eq "$(j -n claims pvc gold-claim '{.status.phase} {.spec.volumeName}')" "Bound pv-large" """,
hint=r"""`spec.selector.matchLabels: {tier: gold}` on the claim.""",
show=r"""kubectl -n claims apply -f - <<'EOF'
show=[hand(r"""vim gold.yaml""",
r"""A claim with a `selector` (the docs page *Persistent Volumes* has one):
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@ -413,14 +480,30 @@ spec:
requests: {storage: 100Mi}
selector:
matchLabels: {tier: gold}
EOF
kubectl get pv"""),
```""",
run=r"""cat > gold.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gold-claim
spec:
storageClassName: manual
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 100Mi}
selector:
matchLabels: {tier: gold}
EOF"""),
cmd(r"""k -n claims apply -f gold.yaml
k get pv""", r"""`gold-claim` bound to `pv-large`.""")]),
step("Bind by name, and use it",
r"""Create claim `pinned` in `claims` that binds `pv-small` **by name** (class `manual`, `ReadWriteOnce`, `100Mi`), and Pod `user` (`busybox:1.36`, `sleep 86400`) that mounts it at `/data`.""",
check=r"""eq "$(j -n claims pvc pinned '{.status.phase} {.spec.volumeName}')" "Bound pv-small" && eq "$(j -n claims pod user '{.status.phase}')" Running \
&& eq "$(j -n claims pod user '{.spec.volumes[0].persistentVolumeClaim.claimName}')" pinned""",
hint=r"""`spec.volumeName: pv-small` on the claim. The volume's class and modes must still match the claim's.""",
show=r"""kubectl -n claims apply -f - <<'EOF'
show=[hand(r"""vim pinned.yaml""",
r"""A claim that names its volume:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@ -431,24 +514,35 @@ spec:
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 100Mi}
---
```""",
run=r"""cat > pinned.yaml <<'EOF'
apiVersion: v1
kind: Pod
kind: PersistentVolumeClaim
metadata:
name: user
name: pinned
spec:
containers:
- name: user
image: busybox:1.36
command: [sleep, '86400']
storageClassName: manual
volumeName: pv-small
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 100Mi}
EOF"""),
cmd(r"""k -n claims apply -f pinned.yaml
k -n claims run user --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > user.yaml"""),
hand(r"""vim user.yaml""",
r"""Add the mount to the container, below `resources: {}`:
```
volumeMounts:
- {name: data, mountPath: /data}
```
and the volume to the Pod's `spec:`, beside `restartPolicy:`:
```
volumes:
- name: data
persistentVolumeClaim: {claimName: pinned}
EOF
kubectl -n claims wait --for=condition=Ready pod/user --timeout=120s
kubectl -n claims get pvc"""),
- {name: data, persistentVolumeClaim: {claimName: pinned}}
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}] | .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "pinned"}}]' user.yaml"""),
cmd(r"""k -n claims apply -f user.yaml
k -n claims get pvc,po""")]),
step("A claim in use does not go",
r"""Delete claim `pinned` without waiting (`--wait=false`). It does not disappear: `kubectl -n claims get pvc` shows it `Terminating`, held by a **finalizer**.
@ -457,17 +551,19 @@ Write that finalizer's name to `{WS}/lesson-1.3/finalizer.txt`.""",
[ -n "$(j -n claims pvc pinned '{.metadata.deletionTimestamp}')" ] \
&& eq "$(tr -d ' \r\n' < "$WS/lesson-1.3/finalizer.txt" 2>/dev/null)" kubernetes.io/pvc-protection""",
hint=r"""`kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers}'`. Without `--wait=false`, `kubectl delete` would sit there until the claim is really gone.""",
show=r"""kubectl -n claims delete pvc pinned --wait=false
kubectl -n claims get pvc pinned
kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers[0]}' > {WS}/lesson-1.3/finalizer.txt
cat {WS}/lesson-1.3/finalizer.txt"""),
show=[cmd(r"""k -n claims delete pvc pinned --wait=false
k -n claims get pvc pinned
k -n claims get pvc pinned -o yaml | grep -A1 finalizers"""),
hand(r"""echo FINALIZER > {WS}/lesson-1.3/finalizer.txt""",
r"""`FINALIZER`: the name under `finalizers:`.""",
run=r"""kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers[0]}' > {WS}/lesson-1.3/finalizer.txt""")]),
step("Let it go",
r"""Delete Pod `user`. With no Pod using it any more, the protection lifts and claim `pinned` is removed. `pv-small` (reclaim policy `Retain`) is now `Released`.""",
check=r"""[ -z "$(j -n claims pod user '{.metadata.name}')" ] && [ -z "$(j -n claims pvc pinned '{.metadata.name}')" ] && eq "$(j pv pv-small '{.status.phase}')" Released""",
hint=r"""`kubectl -n claims delete pod user`, then `kubectl -n claims get pvc` and `kubectl get pv`.""",
show=r"""kubectl -n claims delete pod user --grace-period=1
kubectl -n claims get pvc
kubectl get pv pv-small"""),
show=[cmd(r"""k -n claims delete po user --now
k -n claims get pvc
k get pv pv-small""", r"""The claim is gone; `pv-small` is `Released`.""")]),
step("Claims made by a StatefulSet",
r"""Create a StatefulSet `db` in `claims` with 2 replicas (`busybox:1.36`, `sleep 86400`) and its headless Service `db`. Give it a `volumeClaimTemplates` entry `data` (`ReadWriteOnce`, `50Mi`, class `standard`) mounted at `/var/lib/db`, and a `persistentVolumeClaimRetentionPolicy` that **deletes** a replica's claim when the StatefulSet is scaled down and **retains** the claims when the StatefulSet is deleted.
@ -476,7 +572,9 @@ Each replica gets a claim of its own: `data-db-0`, `data-db-1`.""",
eq "$(j -n claims sts db '{.spec.persistentVolumeClaimRetentionPolicy.whenScaled} {.spec.persistentVolumeClaimRetentionPolicy.whenDeleted}')" "Delete Retain" || exit 1
eq "$(j -n claims sts db '{.status.readyReplicas}')" 2 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound && eq "$(j -n claims pvc data-db-1 '{.status.phase}')" Bound""",
hint=r"""`kubectl create deploy db --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400` is a start: change the kind, add `serviceName: db`, drop `strategy`, add `volumeClaimTemplates` beside `template`. The Service: `clusterIP: None`. `kubectl explain sts.spec.persistentVolumeClaimRetentionPolicy`.""",
show=r"""kubectl -n claims apply -f - <<'EOF'
show=[hand(r"""vim db.yaml""",
r"""No `kubectl create` for a StatefulSet: paste the example from the docs page *StatefulSets* (it comes with its headless Service), and make it:
```
apiVersion: v1
kind: Service
metadata:
@ -517,9 +615,52 @@ spec:
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
EOF
kubectl -n claims rollout status sts db --timeout=180s
kubectl -n claims get pvc"""),
```""",
run=r"""cat > db.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
name: db
spec:
clusterIP: None
selector: {app: db}
ports:
- port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
serviceName: db
replicas: 2
selector:
matchLabels: {app: db}
persistentVolumeClaimRetentionPolicy:
whenScaled: Delete
whenDeleted: Retain
template:
metadata:
labels: {app: db}
spec:
containers:
- name: db
image: busybox:1.36
command: [sleep, '86400']
volumeMounts:
- {name: data, mountPath: /var/lib/db}
volumeClaimTemplates:
- metadata:
name: data
spec:
storageClassName: standard
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
EOF"""),
cmd(r"""k -n claims apply -f db.yaml
k -n claims rollout status sts db
k -n claims get pvc""", r"""`data-db-0` and `data-db-1`.""")]),
step("Scale down, and the claim goes with the replica",
r"""Scale `db` to 1 replica. Pod `db-1` goes, and because `whenScaled` is `Delete`, so does `data-db-1`. `data-db-0` stays.
@ -528,23 +669,26 @@ The default for both fields is `Retain`: without the policy, `data-db-1` would w
eq "$(j -n claims sts db '{.spec.replicas}')" 1 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound || exit 1
for i in $(seq 1 10); do [ -z "$(j -n claims pvc data-db-1 '{.metadata.name}')" ] && exit 0; sleep 3; done; exit 1""",
hint=r"""`kubectl -n claims scale sts db --replicas=1`, then `kubectl -n claims get pvc -w`.""",
show=r"""kubectl -n claims scale sts db --replicas=1
kubectl -n claims rollout status sts db --timeout=120s
kubectl -n claims get pvc"""),
show=[cmd(r"""k -n claims scale sts db --replicas=1
k -n claims rollout status sts db
k -n claims get pvc""", r"""Again in a moment: `data-db-1` is gone, `data-db-0` stays.""")]),
step("Grow a claim in use",
r"""Claim `growing` is full. Its class allows expansion (`kubectl get sc csi-hostpath-sc`: `ALLOWVOLUMEEXPANSION true`), so make it **50Mi** - while Pod `grower` keeps running. Raise the request; the driver does the rest, and `status.capacity` follows.""",
check=r"""eq "$(j -n claims pvc growing '{.spec.resources.requests.storage} {.status.capacity.storage}')" "50Mi 50Mi" && eq "$(j -n claims pod grower '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """,
hint=r"""`kubectl -n claims edit pvc growing`, or `kubectl patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}'`. A claim never shrinks; with `standard` (local-path) the same patch is refused.""",
show=r"""kubectl get sc csi-hostpath-sc
kubectl -n claims patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}'
sleep 10
kubectl -n claims get pvc growing"""),
show=[cmd(r"""k get sc csi-hostpath-sc"""),
hand(r"""k -n claims edit pvc growing""",
r"""Make `storage:` under `spec.resources.requests` read `50Mi`, then `:wq`.""",
run=r"""kubectl -n claims patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}' """),
cmd(r"""k -n claims get pvc growing""", r"""Again until `CAPACITY` shows `50Mi`.""")]),
step("Snapshot it, and restore the snapshot",
r"""Take VolumeSnapshot `growing-snap` of claim `growing` (snapshot class `csi-hostpath-snapclass`), wait until it is ready to use, then create claim `restored` (`csi-hostpath-sc`, 50Mi) **from** the snapshot: its `dataSource` names the VolumeSnapshot. The new volume holds what `growing` held when the snapshot was taken.""",
check=r"""
eq "$(j -n claims volumesnapshot growing-snap '{.spec.source.persistentVolumeClaimName} {.status.readyToUse}')" "growing true" && eq "$(j -n claims pvc restored '{.spec.dataSource.kind}/{.spec.dataSource.name} {.status.phase}')" "VolumeSnapshot/growing-snap Bound" """,
hint=r"""`apiVersion: snapshot.storage.k8s.io/v1`, `kind: VolumeSnapshot`, `spec: {volumeSnapshotClassName: ..., source: {persistentVolumeClaimName: growing}}`. The claim: `dataSource: {apiGroup: snapshot.storage.k8s.io, kind: VolumeSnapshot, name: growing-snap}`. A clone is the same with `dataSource: {kind: PersistentVolumeClaim, name: growing}`.""",
show=r"""kubectl -n claims apply -f - <<'EOF'
show=[hand(r"""vim snap.yaml""",
r"""Paste a VolumeSnapshot from the docs page *Volume Snapshots*, and make it:
```
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
@ -553,9 +697,24 @@ spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: growing
EOF
kubectl -n claims wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/growing-snap --timeout=120s
kubectl -n claims apply -f - <<'EOF'
```""",
run=r"""cat > snap.yaml <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: growing-snap
spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: growing
EOF"""),
cmd(r"""k -n claims apply -f snap.yaml"""),
hand(r"""k -n claims get volumesnapshot""",
r"""Again until `READYTOUSE` is `true`.""",
run=r"""kubectl -n claims wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/growing-snap --timeout=120s"""),
hand(r"""vim restored.yaml""",
r"""A claim whose `dataSource` is the snapshot (the same docs page, *Provisioning Volumes from Snapshots*):
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@ -569,8 +728,24 @@ spec:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: growing-snap
EOF
kubectl -n claims get volumesnapshot,pvc"""),
```""",
run=r"""cat > restored.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: restored
spec:
storageClassName: csi-hostpath-sc
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 50Mi}
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: growing-snap
EOF"""),
cmd(r"""k -n claims apply -f restored.yaml
k -n claims get volumesnapshot,pvc""")]),
],
"outro": r"""You steered binding with a selector and with a volume's name, watched claim protection hold a claim until its Pod went, let a StatefulSet's claims follow its replicas, and grew, snapshotted and restored a claim on a CSI class.

240
cka/lessons_tshoot.py
View File

@ -1,5 +1,5 @@
"""CKA lessons of the Troubleshooting domain (2.1-2.5). See cka/lessons.py for what a lesson is."""
from sim.bank import NS, step
from sim.bank import NS, step, cmd, hand
# ---- 2.1 clusters and nodes -------------------------------------------------------------------
L21 = {
@ -35,9 +35,11 @@ echo "$W1 is still Ready a minute after its kubelet was stopped" >&2; exit 1
Then look at `kubectl describe node` for it: under **Conditions** every row says `Kubelet stopped posting node status` - the control plane has not heard from the kubelet, so it knows nothing more. The answer is on the node.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/node.txt" 2>/dev/null)" "$W1" """,
hint=r"""`kubectl get nodes`. `--no-headers` and `awk '$2 != "Ready" {print $1}'` give just the name.""",
show=r"""kubectl get nodes
kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt
kubectl describe node {W1} | grep -A8 Conditions"""),
show=[cmd(r"""k get no
k describe no {W1} | grep -A8 Conditions"""),
hand(r"""echo NODE > {WS}/lesson-2.1/node.txt""",
r"""`NODE`: the one that is not `Ready`.""",
run=r"""kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt""")]),
step("Bring the kubelet back - for good",
r"""On `{W1}` (`ssh {W1}`, then `sudo -i`), look at the kubelet: `systemctl status kubelet` says `inactive (dead)` and `disabled`. Start it, and enable it so that it survives a reboot - a fix that is gone after the next reboot is not a fix.
@ -46,12 +48,15 @@ Within a few seconds the node is `Ready` again.""",
nx "$W1" 'systemctl is-active -q kubelet && systemctl is-enabled -q kubelet' </dev/null || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""`systemctl enable --now kubelet` does both at once. `journalctl -u kubelet -n 20` tells you when it is running happily.""",
show=r"""ssh {W1} sudo bash <<'EOF'
systemctl status kubelet --no-pager | head -4
show=[hand(r"""ssh {W1}
sudo -i
systemctl status kubelet
systemctl enable --now kubelet""",
r"""Then `exit` twice.""",
run=r"""ssh {W1} sudo bash <<'EOF'
systemctl enable --now kubelet
systemctl is-active kubelet; systemctl is-enabled kubelet
EOF
kubectl get nodes"""),
EOF"""),
cmd(r"""k get no""", r"""Again in a few seconds: `Ready`.""")]),
step("Where the kubelet's configuration is",
r"""A kubelet reads its settings from the file named by its `--config` flag, and the kubeadm flags from `/var/lib/kubelet/kubeadm-flags.env`. On `{W1}`, find the file (`ps -o args= -C kubelet`, or the drop-in under `/etc/systemd/system/kubelet.service.d/`).
@ -60,9 +65,16 @@ The kubelet also tells the API server what it actually runs with: `kubectl get -
want=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["cgroupDriver"])')
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/cgroup.txt" 2>/dev/null)" "$want" """,
hint=r"""`configz` returns JSON: `| jq -r .kubeletconfig.cgroupDriver`. It must match containerd's `SystemdCgroup` setting, or Pods fail to start.""",
show=r"""ssh -n {W1} 'ps -o args= -C kubelet | tr " " "\n" | grep -- --config; sudo grep -n cgroupDriver /var/lib/kubelet/config.yaml'
kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt
cat {WS}/lesson-2.1/cgroup.txt"""),
show=[hand(r"""ssh {W1}
sudo -i
ps -o args= -C kubelet
grep cgroupDriver /var/lib/kubelet/config.yaml""",
r"""`--config=` names the kubelet's file. Then `exit` twice.""",
run=r"""ssh -n {W1} 'ps -o args= -C kubelet; sudo grep cgroupDriver /var/lib/kubelet/config.yaml' """),
cmd(r"""k get --raw /api/v1/nodes/{W1}/proxy/configz | tr , '\n' | grep cgroupDriver""", r"""The live value."""),
hand(r"""echo VALUE > {WS}/lesson-2.1/cgroup.txt""",
r"""`VALUE`: `systemd` or `cgroupfs`.""",
run=r"""kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt""")]),
step("Set the eviction thresholds",
r"""Look at `evictionHard` in `/var/lib/kubelet/config.yaml` on `{W1}`: kind sets every threshold to `0%`, which means "never evict" - fine for a test cluster, not for a real one.
@ -74,15 +86,26 @@ e = json.load(sys.stdin)["kubeletconfig"].get("evictionHard") or {}
sys.exit(0 if e.get("memory.available") == "100Mi" and e.get("nodefs.available") == "10%" else 1)' || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""Edit the file with `vim`, or `sed`: change the `nodefs.available` line and add a `memory.available` line inside the `evictionHard:` block (two spaces of indent). A kubelet that does not come back after the restart: `journalctl -u kubelet -n 20` shows the line it could not parse.""",
show=r"""ssh {W1} sudo bash <<'EOF'
show=[hand(r"""ssh {W1}
sudo -i
vim /var/lib/kubelet/config.yaml""",
r"""Under `evictionHard:`, set `nodefs.available` to `10%` and add `memory.available`, indented like the others:
```
evictionHard:
memory.available: 100Mi
nodefs.available: 10%
```
then restart the kubelet, and `exit` twice:
```
systemctl restart kubelet
```""",
run=r"""ssh {W1} sudo bash <<'EOF'
f=/var/lib/kubelet/config.yaml
grep -n -A4 '^evictionHard:' $f
sed -i 's/^ nodefs.available: .*/ nodefs.available: 10%/' $f
sed -i '/^evictionHard:/a\ memory.available: 100Mi' $f
grep -n -A5 '^evictionHard:' $f
systemctl restart kubelet
EOF
kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq .kubeletconfig.evictionHard"""),
EOF"""),
cmd(r"""k get --raw /api/v1/nodes/{W1}/proxy/configz | tr , '\n' | grep -A4 evictionHard""", r"""Again if the kubelet is not back yet.""")]),
step("Read the pressure conditions",
r"""The kubelet reports the result of those thresholds as node conditions. Write the status of `{W1}`'s three pressure conditions to `{WS}/lesson-2.1/pressure.txt`, one per line, as `TYPE STATUS` - for example `MemoryPressure False`.
@ -91,8 +114,15 @@ kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq .kubeletconfig.evictionH
want=$(for c in MemoryPressure DiskPressure PIDPressure; do echo "$c $(j node "$W1" "{.status.conditions[?(@.type==\"$c\")].status}")"; done | sort)
eq "$(tr -s ' \t' ' ' < "$WS/lesson-2.1/pressure.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """,
hint=r"""`kubectl describe node {W1}` lists them under Conditions. As a list: `-o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}'`, then keep the three `*Pressure` lines.""",
show=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt
cat {WS}/lesson-2.1/pressure.txt"""),
show=[cmd(r"""k describe no {W1} | grep Pressure"""),
hand(r"""vim {WS}/lesson-2.1/pressure.txt""",
r"""Three lines, `TYPE STATUS`, with the statuses you read:
```
MemoryPressure False
DiskPressure False
PIDPressure False
```""",
run=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt""")]),
],
"outro": r"""You found a `NotReady` node, brought its kubelet back for good, read its configuration from the file and from the running kubelet, set eviction thresholds and read what the kubelet reports about them.
@ -144,10 +174,12 @@ Two things in Namespace `ops` do not work: Deployment `queue` has no Pods, and P
`kubectl -n kube-system get pods` shows them; on the control-plane node, `sudo crictl ps -a` shows their containers, and `ls /etc/kubernetes/manifests` what the kubelet is told to run.""",
check=r"""eq "$(tr -d ' \r' < "$WS/lesson-2.2/down.txt" 2>/dev/null | grep . | sort | tr '\n' ' ')" "kube-controller-manager kube-scheduler " """,
hint=r"""A Pod that is not `Running` with all its containers ready. The Pod names end in the node's name; the component is the part before it.""",
show=r"""kubectl -n kube-system get pods
show=[cmd(r"""k -n kube-system get po
ls /etc/kubernetes/manifests
sudo crictl ps -a | grep kube-
printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt"""),
sudo crictl ps -a | grep kube-"""),
hand(r"""vim {WS}/lesson-2.2/down.txt""",
r"""The components that do not run, one per line.""",
run=r"""printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt""")]),
step("Repair the scheduler",
r"""Find out why `kube-scheduler` does not start, and fix its manifest. `kubectl -n kube-system describe pod kube-scheduler-{CP}` shows the error in its events, and so do `sudo crictl ps -a --name kube-scheduler` and `sudo journalctl -u kubelet`.
@ -156,30 +188,33 @@ The kubelet notices the saved file on its own: no restart, no `kubectl apply`. O
nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml && [ -n "$(crictl ps --name kube-scheduler -q)" ]' </dev/null || exit 1
[ -n "$(j -n ops pod waiting '{.spec.nodeName}')" ]""",
hint=r"""Look at the first entry of the container's `command:` - the program the container runs. A static Pod's manifest is edited in place (`sudo vim`), never with `kubectl edit`.""",
show=r"""kubectl -n kube-system describe pod kube-scheduler-{CP} | tail -5
sudo grep -n -A2 'command:' /etc/kubernetes/manifests/kube-scheduler.yaml
sudo sed -i 's/- kube-schedulerr$/- kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
sudo crictl ps --name kube-scheduler
kubectl -n ops get pod waiting -o wide"""),
show=[cmd(r"""k -n kube-system describe po kube-scheduler-{CP} | tail -5""", r"""The command it cannot find: a typo."""),
hand(r"""sudo vim /etc/kubernetes/manifests/kube-scheduler.yaml""",
r"""Under `command:`, make `- kube-schedulerr` read `- kube-scheduler`, then `:wq`.""",
run=r"""sudo sed -i 's/- kube-schedulerr$/- kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml"""),
cmd(r"""sudo crictl ps --name kube-scheduler
k -n ops get po waiting -o wide""", r"""Again in a few seconds: the scheduler runs, and `waiting` has a node.""")]),
step("Repair the controller manager",
r"""Now `kube-controller-manager`. Its Pod is not crashing - its container never starts. `describe` on it tells you which step fails. Fix the manifest; once it runs, Deployment `queue` gets its ReplicaSet and both Pods.""",
check=r"""
nx "$CP" '! grep -q -- "-broken" /etc/kubernetes/manifests/kube-controller-manager.yaml && [ -n "$(crictl ps --name kube-controller-manager -q)" ]' </dev/null || exit 1
eq "$(j -n ops deploy queue '{.status.availableReplicas}')" 2""",
hint=r"""`ErrImagePull` / `ImagePullBackOff`: the image tag does not exist. Every control-plane image runs the cluster's version - compare with `kube-apiserver.yaml`.""",
show=r"""kubectl -n kube-system describe pod kube-controller-manager-{CP} | tail -5
sudo grep -n 'image:' /etc/kubernetes/manifests/*.yaml
sudo sed -i 's/-broken$//' /etc/kubernetes/manifests/kube-controller-manager.yaml
sudo crictl ps --name kube-controller-manager
kubectl -n ops get deploy queue"""),
show=[cmd(r"""k -n kube-system describe po kube-controller-manager-{CP} | tail -5
sudo grep image: /etc/kubernetes/manifests/*.yaml""", r"""Its image tag ends in `-broken`; the other components show the right tag."""),
hand(r"""sudo vim /etc/kubernetes/manifests/kube-controller-manager.yaml""",
r"""Remove `-broken` from the end of the `image:` line, then `:wq`.""",
run=r"""sudo sed -i 's/-broken$//' /etc/kubernetes/manifests/kube-controller-manager.yaml"""),
cmd(r"""sudo crictl ps --name kube-controller-manager
k -n ops get deploy queue""", r"""Again until `queue` shows `2/2`.""")]),
step("Ask the API server how it feels",
r"""The API server has health endpoints of its own: `/livez` (restart me if this fails) and `/readyz` (send me traffic). With `?verbose` they list every check.
Save the output of `kubectl get --raw '/readyz?verbose'` in `{WS}/lesson-2.2/readyz.txt`.""",
check=r"""grep -q '^\[+\]etcd ok' "$WS/lesson-2.2/readyz.txt" 2>/dev/null && grep -q 'readyz check passed' "$WS/lesson-2.2/readyz.txt" """,
hint=r"""`kubectl get --raw` sends a GET to any API path, with your credentials. Quote the path: `?` means something to the shell.""",
show=r"""kubectl get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt | tail -3
kubectl get --raw '/livez'; echo"""),
show=[cmd(r"""k get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt
k get --raw /livez""")]),
],
"outro": r"""You read the control plane from both sides - `kubectl` and the node's container runtime - and repaired two static Pods by editing their manifests in place.
@ -261,38 +296,46 @@ echo "kubectl top shows no metrics for the lesson's Pods" >&2; exit 1
r"""Write the name of the Pod in `metrics` that uses the most memory to `{WS}/lesson-2.3/memory.txt`. `kubectl top` can sort for you.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/memory.txt" 2>/dev/null)" hoarder""",
hint=r"""`kubectl top pod -n metrics --sort-by=memory`. `--no-headers | head -1 | awk '{print $1}'` keeps just the name.""",
show=r"""kubectl top pod -n metrics --sort-by=memory
kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt"""),
show=[cmd(r"""k top po -n metrics --sort-by=memory"""),
hand(r"""echo POD > {WS}/lesson-2.3/memory.txt""",
r"""`POD`: the first one.""",
run=r"""kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt""")]),
step("The busiest Pod in the cluster",
r"""Across **all** Namespaces, which Pod uses the most CPU? Write it as `NAMESPACE/NAME` to `{WS}/lesson-2.3/cpu.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/cpu.txt" 2>/dev/null)" metrics/burner""",
hint=r"""`kubectl top pod -A --sort-by=cpu`; with `-A` the first column is the Namespace.""",
show=r"""kubectl top pod -A --sort-by=cpu | head -5
kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt"""),
show=[cmd(r"""k top po -A --sort-by=cpu | head -5"""),
hand(r"""echo NAMESPACE/POD > {WS}/lesson-2.3/cpu.txt""",
r"""From the first line.""",
run=r"""kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt""")]),
step("Inside one Pod",
r"""`kubectl top pod` adds up a Pod's containers. Which container of Pod `duo` uses the memory? Write its name to `{WS}/lesson-2.3/container.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/container.txt" 2>/dev/null)" cache""",
hint=r"""`--containers` splits the Pod into its containers.""",
show=r"""kubectl top pod duo -n metrics --containers
kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt"""),
show=[cmd(r"""k top po duo -n metrics --containers"""),
hand(r"""echo CONTAINER > {WS}/lesson-2.3/container.txt""",
r"""The container with the most `MEMORY(bytes)`.""",
run=r"""kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt""")]),
step("Usage against what is reserved",
r"""`kubectl top node` shows use; `kubectl describe node {W1}`, under **Allocated resources**, shows what Pods have **requested** - which is what decides whether another Pod fits.
Write the memory that `{W1}` can give to Pods - its **allocatable** memory, as the API reports it (for example `8012345Ki`) - to `{WS}/lesson-2.3/allocatable.txt`.""",
check=r"""want=$(j node "$W1" '{.status.allocatable.memory}'); [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/allocatable.txt" 2>/dev/null)" "$want" """,
hint=r"""Allocatable is capacity minus what the node keeps for itself and for the eviction threshold: `.status.allocatable.memory`.""",
show=r"""kubectl top node
kubectl describe node {W1} | grep -A9 'Allocated resources'
kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt"""),
show=[cmd(r"""k top no
k describe no {W1} | grep -A6 -e Allocatable: -e 'Allocated resources' """, r"""`memory:` under `Allocatable:` is what `{W1}` can give to Pods, for example `8012345Ki`."""),
hand(r"""echo MEMORY > {WS}/lesson-2.3/allocatable.txt""",
r"""`MEMORY`: that value, as it stands.""",
run=r"""kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt""")]),
step("The metrics API itself",
r"""`kubectl top` only reads an API. Read it yourself: save what `/apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder` returns in `{WS}/lesson-2.3/raw.json`.
If this path answers `NotFound` or `ServiceUnavailable` on some cluster, `kubectl top` fails there too: look at the `metrics-server` Deployment in `kube-system`, and at `kubectl get apiservice v1beta1.metrics.k8s.io`.""",
check=r"""python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); sys.exit(0 if d.get("kind")=="PodMetrics" and d["metadata"]["name"]=="hoarder" and d.get("containers") else 1)' "$WS/lesson-2.3/raw.json" 2>/dev/null""",
hint=r"""`kubectl get --raw PATH`.""",
show=r"""kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json
jq '.containers' {WS}/lesson-2.3/raw.json
kubectl get apiservice v1beta1.metrics.k8s.io"""),
show=[cmd(r"""k get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json
cat {WS}/lesson-2.3/raw.json
k get apiservice v1beta1.metrics.k8s.io""")]),
],
"outro": r"""You ranked Pods by memory and CPU, looked inside a Pod, compared use with what a node can give, and read the metrics API that `kubectl top` uses.
@ -361,16 +404,17 @@ echo "flaky did not restart" >&2; exit 1
Save those in `{WS}/lesson-2.4/previous.log`.""",
check=r"""grep -q 'FATAL config /etc/flaky.conf not found' "$WS/lesson-2.4/previous.log" 2>/dev/null && ! grep -q 'second attempt' "$WS/lesson-2.4/previous.log" """,
hint=r"""`kubectl logs --previous` (or `-p`).""",
show=r"""kubectl -n logs logs flaky
kubectl -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log
cat {WS}/lesson-2.4/previous.log"""),
show=[cmd(r"""k -n logs logs flaky
k -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log
cat {WS}/lesson-2.4/previous.log""")]),
step("The one replica that complains",
r"""Pods `api-1`, `api-2` and `api-3` all carry `app=api`. One of them logs errors. Read all three at once, find it, and write its name to `{WS}/lesson-2.4/erroring.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/erroring.txt" 2>/dev/null)" api-2""",
hint=r"""`kubectl logs -l app=api --prefix` puts `[pod/NAME/CONTAINER]` before each line. With a selector, kubectl shows only the last 10 lines of each Pod unless you say `--tail`.""",
show=r"""kubectl -n logs logs -l app=api --prefix --tail=2
kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt
cat {WS}/lesson-2.4/erroring.txt"""),
show=[cmd(r"""k -n logs logs -l app=api --prefix --tail=20 | grep ERROR""", r"""`[pod/NAME/...]` at the start of the line names the Pod."""),
hand(r"""echo POD > {WS}/lesson-2.4/erroring.txt""",
r"""`POD`: that name.""",
run=r"""kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt""")]),
step("The log file on the node",
r"""The node keeps one file per container instance. Write the full path of the file that holds the output of `flaky`'s **current** instance to `{WS}/lesson-2.4/logfile.txt`.
@ -380,11 +424,15 @@ p=$(tr -d ' \r\n' < "$WS/lesson-2.4/logfile.txt" 2>/dev/null)
uid=$(j -n logs pod flaky '{.metadata.uid}'); n=$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}'); node=$(j -n logs pod flaky '{.spec.nodeName}')
eq "$p" "/var/log/pods/logs_flaky_$uid/flaky/$n.log" && nx "$node" "grep -q 'second attempt' '$p'" </dev/null""",
hint=r"""`/var/log/pods/NAMESPACE_POD_UID/CONTAINER/RESTARTS.log`. `sudo ls -R /var/log/pods/ | grep flaky` on the node gets you close.""",
show=r"""kubectl -n logs get pod flaky -o wide
uid=$(kubectl -n logs get pod flaky -o jsonpath='{.metadata.uid}')
ssh -n {W1} "sudo ls /var/log/pods/logs_flaky_$uid/flaky/"
echo "/var/log/pods/logs_flaky_$uid/flaky/1.log" > {WS}/lesson-2.4/logfile.txt
ssh -n {W1} "sudo cat $(cat {WS}/lesson-2.4/logfile.txt)"""),
show=[cmd(r"""k -n logs get po flaky -o wide
k -n logs get po flaky -o yaml | grep uid:""", r"""Its node, and its UID."""),
hand(r"""ssh {W1} sudo ls /var/log/pods/""",
r"""`logs_flaky_UID` holds `flaky/`, one file per instance: `0.log` the first, `1.log` the current one.""",
run=r"""ssh -n {W1} sudo ls /var/log/pods/"""),
hand(r"""echo /var/log/pods/logs_flaky_UID/flaky/1.log > {WS}/lesson-2.4/logfile.txt""",
r"""With the Pod's UID.""",
run=r"""uid=$(kubectl -n logs get pod flaky -o jsonpath='{.metadata.uid}')
echo "/var/log/pods/logs_flaky_$uid/flaky/1.log" > {WS}/lesson-2.4/logfile.txt""")]),
step("Debug a container that has no shell",
r"""`kubectl -n logs exec quiet -- sh` fails: the `pause` image has nothing but its one program. Add an ephemeral container to the running Pod instead: name `dbg`, image `busybox:1.36`, **targeting** container `quiet` so that it sees its processes, running `sleep 3600`.
@ -393,10 +441,13 @@ Then run `ps` in `dbg` and write the command of the target's process - the first
eq "$(j -n logs pod quiet '{.spec.ephemeralContainers[0].name} {.spec.ephemeralContainers[0].targetContainerName}')" "dbg quiet" || exit 1
grep -qx '/pause' <(tr -d ' \r' < "$WS/lesson-2.4/target.txt" 2>/dev/null)""",
hint=r"""`kubectl debug POD --image=IMAGE --target=CONTAINER -c NAME -- COMMAND`. Interactively you would add `-it` and use the shell straight away; without it, `kubectl exec POD -c dbg -- ps` reaches it once it runs. Ephemeral containers cannot be removed - the Pod keeps them until it goes.""",
show=r"""kubectl -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600
for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done
kubectl -n logs exec quiet -c dbg -- ps
kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt"""),
show=[cmd(r"""k -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600"""),
hand(r"""k -n logs exec quiet -c dbg -- ps""",
r"""Again if `dbg` is not up yet. PID 1 is the target's process.""",
run=r"""for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done"""),
hand(r"""echo COMMAND > {WS}/lesson-2.4/target.txt""",
r"""`COMMAND`: PID 1's command, as `ps` shows it.""",
run=r"""kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt""")]),
step("Debug the node itself",
r"""`kubectl debug node/{W1} --image=busybox:1.36` starts a Pod on that node with the node's whole filesystem mounted at `/host` - useful when you cannot `ssh` to a node, but can reach the API.
@ -406,11 +457,16 @@ want=$(nx "$W1" '. /etc/os-release; echo "$PRETTY_NAME"' </dev/null); [ -n "$wan
eq "$(tr -d '\r"' < "$WS/lesson-2.4/os.txt" 2>/dev/null | sed 's/^PRETTY_NAME=//; s/ *$//')" "$want" \
&& [ -n "$($K get pods -A -o name 2>/dev/null | grep node-debugger)" ]""",
hint=r"""Without `-it`, give it a command and read its output with `kubectl logs`: the Pod is called `node-debugger-NODE-xxxxx`, in your current Namespace. Delete it afterwards.""",
show=r"""kubectl debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release
pod=$(kubectl get pods -o name | grep node-debugger | head -1)
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s
kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt
cat {WS}/lesson-2.4/os.txt"""),
show=[cmd(r"""k debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release
k get po""", r"""The `node-debugger-...` Pod: once it is `Completed`, its log is the file."""),
hand(r"""k logs node-debugger-NAME""",
r"""`node-debugger-NAME`: its full name, from `get po`.""",
run=r"""pod=$(kubectl get pods -o name | grep node-debugger | head -1)
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s"""),
hand(r"""echo 'VALUE' > {WS}/lesson-2.4/os.txt""",
r"""`VALUE`: what `PRETTY_NAME=` says, without the double quotes.""",
run=r"""pod=$(kubectl get pods -o name | grep node-debugger | head -1)
kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt""")]),
],
"outro": r"""You read a crashed instance's logs, searched several Pods at once, found the log file on the node, and debugged both a Pod without a shell and a node.
@ -477,19 +533,20 @@ eq "$(j -n shop svc web '{.spec.selector.app}')" web || exit 1
n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' | grep -c .)
[ "$n" -eq 2 ]""",
hint=r"""Compare `kubectl -n shop get svc web -o wide` (SELECTOR) with `kubectl -n shop get pods --show-labels`. `kubectl set selector` or `kubectl edit svc`.""",
show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
kubectl -n shop get svc web -o wide
kubectl -n shop get pods --show-labels
kubectl -n shop set selector svc web app=web
kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""),
show=[cmd(r"""k -n shop get endpointslices -l kubernetes.io/service-name=web
k -n shop get svc web -o wide
k -n shop get po --show-labels
k -n shop set selector svc web app=web
k -n shop get endpointslices -l kubernetes.io/service-name=web""")]),
step("The wrong port",
r"""Now the Service has endpoints, but they point at port `8080` - and nginx listens on `80`. Make the Service send traffic to the port the container serves.""",
check=r"""
eq "$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{.items[0].ports[0].port}')" 80 && eq "$(j -n shop svc web '{.spec.ports[0].port}')" 80""",
hint=r"""`targetPort` is the container's port; `port` is the Service's own. `kubectl -n shop edit svc web`, or a JSON patch on `/spec/ports/0/targetPort`.""",
show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]'
kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""),
show=[hand(r"""k -n shop edit svc web""",
r"""Make `targetPort: 8080` read `targetPort: 80`, then `:wq`.""",
run=r"""kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]' """),
cmd(r"""k -n shop get endpointslices -l kubernetes.io/service-name=web""")]),
step("Let the right client in",
r"""Still nothing: NetworkPolicy `lockdown` selects the `web` Pods and allows no ingress at all.
@ -500,7 +557,17 @@ ok() { $K -n shop exec "$1" -- wget -qO- -T 3 web >/dev/null 2>&1; }
for i in 1 2 3 4 5; do ok client && break; sleep 3; done; ok client || exit 1
! ok stranger""",
hint=r"""Add an `ingress` rule to `lockdown`: `from: [{podSelector: {matchLabels: {app: client}}}]` with `ports: [{protocol: TCP, port: 80}]`. A NetworkPolicy's ports are the Pod's, not the Service's.""",
show=r"""kubectl -n shop apply -f - <<'EOF'
show=[hand(r"""k -n shop edit netpol lockdown""",
r"""Add the rule to its `spec:`, beside `policyTypes:`:
```
ingress:
- from:
- podSelector:
matchLabels: {app: client}
ports:
- {protocol: TCP, port: 80}
```""",
run=r"""kubectl -n shop apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@ -515,9 +582,9 @@ spec:
matchLabels: {app: client}
ports:
- {protocol: TCP, port: 80}
EOF
kubectl -n shop exec client -- wget -qO- -T3 web | head -4
kubectl -n shop exec stranger -- wget -qO- -T3 web || echo refused"""),
EOF"""),
cmd(r"""k -n shop exec client -- wget -qO- -T3 web | head -4
k -n shop exec stranger -- wget -qO- -T3 web""", r"""`client` gets the page; `stranger` times out.""")]),
step("How kube-proxy routes it",
r"""kube-proxy turned Service `web` into rules on every node. Find out how: its mode is in ConfigMap `kube-proxy` in `kube-system`. Write the mode to `{WS}/lesson-2.5/mode.txt`.
@ -529,11 +596,18 @@ ip=$(j -n shop svc web '{.spec.clusterIP}')
want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" </dev/null)
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.5/chain.txt" 2>/dev/null)" "$want" """,
hint=r"""`kubectl -n kube-system get cm kube-proxy -o yaml | grep mode`. On the node: `sudo iptables-save -t nat | grep CLUSTER-IP` - the line for the Service jumps (`-j`) to its chain. In `nftables` mode the rules are in `sudo nft list ruleset` instead.""",
show=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt
ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32"
sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt
sudo iptables-save -t nat | grep "^-A $(cat {WS}/lesson-2.5/chain.txt)" """),
show=[cmd(r"""k -n kube-system get cm kube-proxy -o yaml | grep mode:"""),
hand(r"""echo MODE > {WS}/lesson-2.5/mode.txt""",
r"""`MODE`: its value.""",
run=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt"""),
cmd(r"""k -n shop get svc web""", r"""Its `CLUSTER-IP`:"""),
hand(r"""sudo iptables-save -t nat | grep IP""",
r"""`IP`: that address. The rule for it jumps (`-j`) to a `KUBE-SVC-...` chain.""",
run='ip=$(kubectl -n shop get svc web -o jsonpath=\'{.spec.clusterIP}\')\nsudo iptables-save -t nat | grep -- "-d $ip/32"'),
hand(r"""echo KUBE-SVC-NAME > {WS}/lesson-2.5/chain.txt""",
r"""The chain's full name.""",
run=r"""ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt""")]),
],
"outro": r"""You took a Service from "times out" to answering by fixing its selector, its target port and a NetworkPolicy, and followed it into kube-proxy's rules.

438
cka/lessons_work.py
View File

@ -1,5 +1,5 @@
"""CKA lessons of the Workloads and Scheduling domain (3.1-3.5). See cka/lessons.py for what a lesson is."""
from sim.bank import NS, step
from sim.bank import NS, step, cmd, hand
# ---- 3.1 rolling updates and rollbacks --------------------------------------------------------
L31 = {
@ -27,8 +27,10 @@ $K -n rollout rollout status deploy api --timeout=180s >/dev/null
r"""Change the update strategy of `api` so that an update first adds **one** Pod and never takes one away before its replacement is ready: `maxSurge: 1`, `maxUnavailable: 0`.""",
check=r"""eq "$(j -n rollout deploy api '{.spec.strategy.type} {.spec.strategy.rollingUpdate.maxSurge} {.spec.strategy.rollingUpdate.maxUnavailable}')" "RollingUpdate 1 0" """,
hint=r"""`kubectl -n rollout edit deploy api`, or `kubectl patch` with `{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}`.""",
show=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}'
kubectl -n rollout get deploy api -o jsonpath='{.spec.strategy}'; echo"""),
show=[hand(r"""k -n rollout edit deploy api""",
r"""Under `strategy.rollingUpdate`, set `maxSurge: 1` and `maxUnavailable: 0`, then save and quit (`:wq`).""",
run=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' """),
cmd(r"""k -n rollout describe deploy api | grep -i strategy""")]),
step("A release with a reason",
r"""Update `api` to `nginx:1.27`, and record why: the annotation `kubernetes.io/change-cause` on the Deployment, set to `upgrade to 1.27`. `kubectl rollout history` shows it beside the revision.""",
check=r"""
@ -36,10 +38,10 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin
eq "$(j -n rollout deploy api '{.metadata.annotations.kubernetes\.io/change-cause}')" "upgrade to 1.27" || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas}')" "4 4" """,
hint=r"""`kubectl set image deploy/api nginx=nginx:1.27` (the container is called `nginx`), then `kubectl annotate deploy api kubernetes.io/change-cause='...'`. The `--record` flag is gone.""",
show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27
kubectl -n rollout annotate deploy api kubernetes.io/change-cause='upgrade to 1.27' --overwrite
kubectl -n rollout rollout status deploy api --timeout=180s
kubectl -n rollout rollout history deploy api"""),
show=[cmd(r"""k -n rollout set image deploy/api nginx=nginx:1.27
k -n rollout annotate deploy api kubernetes.io/change-cause='upgrade to 1.27' --overwrite
k -n rollout rollout status deploy api
k -n rollout rollout history deploy api""")]),
step("A release that cannot start",
r"""Now a mistake: set the image of `api` to `nginx:1.27-broken`, a tag that does not exist.
@ -49,29 +51,27 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin
[ "$(j -n rollout deploy api '{.status.availableReplicas}')" = 4 ] || exit 1
$K -n rollout get pods -l app=api --no-headers 2>/dev/null | grep -Eq 'ErrImagePull|ImagePullBackOff' """,
hint=r"""`kubectl set image` again. `kubectl -n rollout get pods` shows the Pod that cannot pull; `kubectl -n rollout get rs` shows the two ReplicaSets side by side.""",
show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27-broken
sleep 10
kubectl -n rollout get pods
kubectl -n rollout get rs"""),
show=[cmd(r"""k -n rollout set image deploy/api nginx=nginx:1.27-broken
k -n rollout get po,rs""", r"""Run `get` again after a few seconds: one new Pod is stuck in `ErrImagePull`, and the four old ones still run.""")]),
step("Undo",
r"""Roll `api` back to the revision before - `nginx:1.27` - and let the rollout finish.""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
hint=r"""`kubectl rollout undo deploy/api`. It does not create a ReplicaSet: it scales the old one back up, and that revision gets a new number.""",
show=r"""kubectl -n rollout rollout undo deploy/api
kubectl -n rollout rollout status deploy api --timeout=180s
kubectl -n rollout rollout history deploy api"""),
show=[cmd(r"""k -n rollout rollout undo deploy/api
k -n rollout rollout status deploy api
k -n rollout rollout history deploy api""")]),
step("Back to a revision by number",
r"""Look at the history, then go back to the revision that ran **`nginx:1.26`**, by its number. `kubectl rollout history deploy api --revision=N` shows what a revision's template was.""",
check=r"""
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.26 || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
hint=r"""Revision 1 is where the Deployment started. `kubectl rollout undo deploy/api --to-revision=1`.""",
show=r"""kubectl -n rollout rollout history deploy api
kubectl -n rollout rollout history deploy api --revision=1 | grep Image
kubectl -n rollout rollout undo deploy/api --to-revision=1
kubectl -n rollout rollout status deploy api --timeout=180s"""),
show=[cmd(r"""k -n rollout rollout history deploy api
k -n rollout rollout history deploy api --revision=1""", r"""Find the revision whose `Image:` is `nginx:1.26` - revision 1 here."""),
cmd(r"""k -n rollout rollout undo deploy/api --to-revision=1
k -n rollout rollout status deploy api""")]),
step("Two changes, one rollout",
r"""Each `set` command starts a rollout of its own. To make two changes at once, **pause** the Deployment, make them - image `nginx:1.27`, and an environment variable `MODE=prod` in the container - then **resume**: one new revision, one rollout.""",
check=r"""
@ -80,12 +80,12 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin
eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].env[?(@.name=="MODE")].value}')" prod || exit 1
eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """,
hint=r"""`kubectl rollout pause deploy/api`, `kubectl set image ...`, `kubectl set env deploy/api MODE=prod`, `kubectl rollout resume deploy/api`.""",
show=r"""kubectl -n rollout rollout pause deploy/api
kubectl -n rollout set image deploy/api nginx=nginx:1.27
kubectl -n rollout set env deploy/api MODE=prod
kubectl -n rollout rollout resume deploy/api
kubectl -n rollout rollout status deploy api --timeout=180s
kubectl -n rollout rollout history deploy api"""),
show=[cmd(r"""k -n rollout rollout pause deploy/api
k -n rollout set image deploy/api nginx=nginx:1.27
k -n rollout set env deploy/api MODE=prod
k -n rollout rollout resume deploy/api
k -n rollout rollout status deploy api
k -n rollout rollout history deploy api""")]),
],
"outro": r"""You tuned the strategy, released with a recorded cause, watched a broken release stall safely, undid it, went back to a numbered revision, and batched two changes into one rollout.
@ -132,15 +132,15 @@ printf 'DB_USER=ledger\nDB_PASS=t0ps3cret\n' > "$WS/lesson-3.2/db.env"
check=r"""
eq "$(j -n conf cm app-config '{.data.MODE}')" prod && $K -n conf get cm app-config -o jsonpath='{.data.app\.properties}' | grep -q '^cache.size=128$'""",
hint=r"""`kubectl create configmap NAME --from-file=PATH --from-literal=KEY=VALUE`. With `--from-file`, the key is the file's name.""",
show=r"""kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod
kubectl -n conf describe cm app-config"""),
show=[cmd(r"""k -n conf create cm app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod
k -n conf describe cm app-config""")]),
step("A Secret from an env file",
r"""Create Secret `db` (type `Opaque`) in `conf` from `{WS}/lesson-3.2/db.env`, one key per line of the file: `DB_USER` and `DB_PASS`. Then read `DB_PASS` back, decoded.""",
check=r"""
eq "$(j -n conf secret db '{.type}')" Opaque && eq "$(j -n conf secret db '{.data.DB_USER}' | base64 -d)" ledger && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""",
hint=r"""`kubectl create secret generic NAME --from-env-file=PATH`. Reading one key: `-o jsonpath='{.data.DB_PASS}' | base64 -d`.""",
show=r"""kubectl -n conf create secret generic db --from-env-file={WS}/lesson-3.2/db.env
kubectl -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d; echo"""),
show=[cmd(r"""k -n conf create secret generic db --from-env-file={WS}/lesson-3.2/db.env
k -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d""")]),
step("Consume both",
r"""Run Pod `app` in `conf` (`busybox:1.36`, `sleep 86400`) that gets:
@ -151,16 +151,10 @@ kubectl -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d; echo"""
x() { $K -n conf exec app -- "$@" 2>/dev/null; }
eq "$(x printenv DB_PASS)" t0ps3cret && eq "$(x printenv APP_MODE)" prod && x cat /etc/app/app.properties | grep -q '^log.level=info$'""",
hint=r"""`envFrom: [{secretRef: {name: db}}]`, `env: [{name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: MODE}}}]`, and a `configMap` volume.""",
show=r"""kubectl -n conf apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
image: busybox:1.36
command: [sleep, '86400']
show=[cmd(r"""k -n conf run app --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > app.yaml"""),
hand(r"""vim app.yaml""",
r"""Add to the container, below `resources: {}` (the docs page *Configure a Pod to Use a ConfigMap* has each of these):
```
envFrom:
- secretRef: {name: db}
env:
@ -169,33 +163,38 @@ spec:
configMapKeyRef: {name: app-config, key: MODE}
volumeMounts:
- {name: config, mountPath: /etc/app}
```
and to the Pod's `spec:`, beside `restartPolicy: Always` and indented like it:
```
volumes:
- name: config
configMap: {name: app-config}
EOF
kubectl -n conf wait --for=condition=Ready pod/app --timeout=120s
kubectl -n conf exec app -- sh -c 'printenv DB_USER APP_MODE; cat /etc/app/app.properties'"""),
- {name: config, configMap: {name: app-config}}
```""",
run=r"""yq -i '.spec.containers[0].envFrom = [{"secretRef": {"name": "db"}}]
| .spec.containers[0].env = [{"name": "APP_MODE", "valueFrom": {"configMapKeyRef": {"name": "app-config", "key": "MODE"}}}]
| .spec.containers[0].volumeMounts = [{"name": "config", "mountPath": "/etc/app"}]
| .spec.volumes = [{"name": "config", "configMap": {"name": "app-config"}}]' app.yaml"""),
cmd(r"""k -n conf apply -f app.yaml
k -n conf exec app -- env | grep -e DB_ -e APP_""")]),
step("Change the configuration, live",
r"""Change `cache.size` to `256` in ConfigMap `app-config` - edit the file and re-create the ConfigMap's key from it, or `kubectl edit`.
Then watch `/etc/app/app.properties` inside Pod `app`: within a minute the kubelet replaces the file. `APP_MODE` would not change, even if you edited `MODE`: the environment is fixed at start.""",
check=r"""$K -n conf exec app -- cat /etc/app/app.properties 2>/dev/null | grep -q '^cache.size=256$'""",
hint=r"""Regenerate and apply: `kubectl create configmap app-config --from-file=... --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f -` - the usual way to update a ConfigMap from files.""",
show=r"""sed -i 's/^cache.size=.*/cache.size=256/' {WS}/lesson-3.2/app.properties
kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f -
sleep 30
kubectl -n conf exec app -- cat /etc/app/app.properties"""),
show=[hand(r"""k -n conf edit cm app-config""",
r"""In `app.properties`, change the line to `cache.size=256`, then `:wq`.""",
run=r"""kubectl -n conf get cm app-config -o yaml | sed 's/cache.size=.*/cache.size=256/' | kubectl replace -f -"""),
cmd(r"""k -n conf exec app -- cat /etc/app/app.properties""", r"""Again until it shows `cache.size=256`: the kubelet rewrites the file within a minute.""")]),
step("A TLS Secret",
r"""Make a self-signed certificate for `shop.example.com` (`openssl` is on this host), and store it as Secret `shop-tls` in `conf`, of type `kubernetes.io/tls` - the type an Ingress or Gateway expects. Its keys are `tls.crt` and `tls.key`.""",
check=r"""
eq "$(j -n conf secret shop-tls '{.type}')" kubernetes.io/tls || exit 1
j -n conf secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'shop.example.com'""",
hint=r"""`openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com`, then `kubectl create secret tls NAME --cert=tls.crt --key=tls.key`.""",
show=r"""cd {WS}/lesson-3.2
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null
kubectl -n conf create secret tls shop-tls --cert=tls.crt --key=tls.key
kubectl -n conf get secret shop-tls
cd"""),
show=[cmd(r"""cd {WS}/lesson-3.2
openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com
k -n conf create secret tls shop-tls --cert=tls.crt --key=tls.key
cd""")]),
step("Encrypt Secrets at rest",
r"""Read Secret `db` straight out of etcd on `{CP}`, and you see `t0ps3cret` in plain text. Make the API server encrypt Secrets before it stores them:
@ -206,7 +205,23 @@ cd"""),
Edit a copy of the manifest outside `/etc/kubernetes/manifests` and copy it back: the kubelet runs every file in that directory, backups too.""",
check=_ENC_CHECK + r""" && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""",
hint=r"""The docs page "Encrypting Confidential Data at Rest" has the file. Stored values of encrypted Secrets start with `k8s:enc:aescbc:v1:key1:`: `sudo ETCDCTL_API=3 etcdctl ... get /registry/secrets/conf/db | od -c | head` (or `hexdump -C`). Rewriting: `kubectl get secret db -o json | kubectl replace -f -`. If the API server does not come back, `sudo crictl ps -a` and the logs under `/var/log/pods/kube-system_kube-apiserver-*` say why.""",
show=r"""sudo mkdir -p /etc/kubernetes/enc
show=[cmd(r"""head -c 32 /dev/urandom | base64""", r"""Copy the output: that is the key."""),
hand(r"""sudo mkdir -p /etc/kubernetes/enc
sudo vim /etc/kubernetes/enc/enc.yaml""",
r"""Paste the example from the docs page *Encrypting Confidential Data at Rest* (search for `aescbc`) and cut it down to:
```
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
providers:
- aescbc:
keys:
- name: key1
secret: <your key>
- identity: {}
```""",
run=r"""sudo mkdir -p /etc/kubernetes/enc
sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
@ -218,19 +233,30 @@ resources:
- name: key1
secret: $(head -c 32 /dev/urandom | base64)
- identity: {}
EOF
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
EOF"""),
cmd(r"""sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/"""),
hand(r"""sudo vim ~/kube-apiserver.yaml""",
r"""Add three lines, each beside its kind (`yy` and `p` on an existing `k8s-certs` mount or volume, then change it):
```
- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
```
in the `command:` list,
```
- {name: enc, mountPath: /etc/kubernetes/enc, readOnly: true}
```
under `volumeMounts:`, and
```
- {name: enc, hostPath: {path: /etc/kubernetes/enc, type: DirectoryOrCreate}}
```
under `volumes:`.""",
run=r"""sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
| .spec.containers[0].volumeMounts += [{"name": "enc", "mountPath": "/etc/kubernetes/enc", "readOnly": true}]
| .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' /tmp/kube-apiserver.yaml
sudo cp /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
sleep 20
for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done
kubectl -n conf get secret db -o json | kubectl replace -f -
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
get /registry/secrets/conf/db | od -c | head -4"""),
| .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' ~/kube-apiserver.yaml"""),
cmd(r"""sudo cp ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
hand(r"""k get po -n kube-system""",
r"""Again until it answers: the API server is back in a minute or so.""",
run=r"""sleep 20; for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""k -n conf get secret db -o json | k replace -f -""", r"""Written again, the Secret is stored encrypted. Reading it from etcd would now show `k8s:enc:aescbc:v1:key1` and no plain text.""")]),
],
"outro": r"""You built a ConfigMap and Secrets from files, consumed them three ways, updated configuration in a running Pod, and turned on encryption at rest for Secrets.
@ -303,9 +329,8 @@ $K -n scale rollout status deploy api --timeout=180s >/dev/null && $K -n scale r
eq "$(j -n scale hpa api '{.spec.minReplicas} {.spec.maxReplicas} {.spec.metrics[0].resource.name} {.spec.metrics[0].resource.target.averageUtilization}')" "2 6 cpu 50" || exit 1
[ -n "$(j -n scale hpa api '{.status.currentMetrics[0].resource.current.averageUtilization}')" ] && [ "$(j -n scale deploy api '{.status.readyReplicas}')" -ge 2 ] 2>/dev/null""",
hint=r"""`kubectl autoscale deploy api --min=2 --max=6 --cpu=50%` (kubectl before 1.34: `--cpu-percent=50`). `<unknown>` in the TARGETS column means no metrics yet - or a Pod without a CPU request.""",
show=r"""kubectl -n scale autoscale deploy api --min=2 --max=6 --cpu=50%
sleep 45
kubectl -n scale get hpa api"""),
show=[cmd(r"""k -n scale autoscale deploy api --min=2 --max=6 --cpu=50%
k -n scale get hpa api""", r"""Again in a minute: `TARGETS` shows a value once metrics arrive.""")]),
step("Scale down calmly",
r"""Give the HPA of `api` a `behavior`: when scaling **down**, wait **120** seconds of stable recommendations (`stabilizationWindowSeconds`), and remove at most **1** Pod per **60** seconds.
@ -314,36 +339,27 @@ kubectl -n scale get hpa api"""),
eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.stabilizationWindowSeconds}')" 120 || exit 1
eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.policies[0].type} {.spec.behavior.scaleDown.policies[0].value} {.spec.behavior.scaleDown.policies[0].periodSeconds}')" "Pods 1 60" """,
hint=r"""Under `spec`: `behavior: {scaleDown: {stabilizationWindowSeconds: 120, policies: [{type: Pods, value: 1, periodSeconds: 60}]}}`. `kubectl explain hpa.spec.behavior.scaleDown`.""",
show=r"""kubectl -n scale patch hpa api --type merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":120,"policies":[{"type":"Pods","value":1,"periodSeconds":60}]}}}}'
kubectl -n scale describe hpa api | grep -A4 Behavior"""),
show=[hand(r"""k -n scale edit hpa api""",
r"""Add under `spec:` (the docs page *Horizontal Pod Autoscaling* has it under *Configurable scaling behavior*):
```
behavior:
scaleDown:
stabilizationWindowSeconds: 120
policies:
- {type: Pods, value: 1, periodSeconds: 60}
```""",
run=r"""kubectl -n scale patch hpa api --type merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":120,"policies":[{"type":"Pods","value":1,"periodSeconds":60}]}}}}' """),
cmd(r"""k -n scale describe hpa api""")]),
step("Watch it scale up",
r"""Write an HPA for `cruncher` as a manifest (`autoscaling/v2`): 1 to **4** replicas at **60%** average CPU. `cruncher` uses about three times its request, so the HPA soon adds Pods: wait until it wants more than one.""",
check=r"""
eq "$(j -n scale hpa cruncher '{.spec.maxReplicas} {.spec.metrics[0].resource.target.averageUtilization}')" "4 60" || exit 1
[ "$(j -n scale hpa cruncher '{.status.desiredReplicas}')" -gt 1 ] 2>/dev/null""",
hint=r"""`kubectl autoscale ... --dry-run=client -o yaml` writes the manifest for you. `kubectl -n scale get hpa cruncher -w` shows it react; `describe hpa` lists its decisions as events.""",
show=r"""kubectl -n scale apply -f - <<'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: cruncher
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: cruncher
minReplicas: 1
maxReplicas: 4
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
EOF
sleep 60
kubectl -n scale get hpa cruncher"""),
show=[cmd(r"""k -n scale autoscale deploy cruncher --min=1 --max=4 --cpu=60% --dry-run=client -o yaml > hpa.yaml
cat hpa.yaml""", r"""That is the manifest the step asks for: `autoscaling/v2`, `minReplicas`, `maxReplicas` and a `Resource` metric for CPU."""),
cmd(r"""k -n scale apply -f hpa.yaml
k -n scale get hpa cruncher""", r"""Again in a minute or two: `REPLICAS` goes above 1.""")]),
step("Resize a running Pod",
r"""Run Pod `sizer` in `scale` (`busybox:1.36`, `sleep 86400`) with CPU request `100m` and limit `200m`, memory request and limit `64Mi`.
@ -352,11 +368,23 @@ Then give it more CPU **without replacing it**: request `150m`, limit `300m`, th
eq "$(j -n scale pod sizer '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.limits.cpu}')" "150m 300m" || exit 1
eq "$(j -n scale pod sizer '{.status.containerStatuses[0].resources.limits.cpu} {.status.containerStatuses[0].restartCount}')" "300m 0" """,
hint=r"""`kubectl patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{...}}]}}'`. A plain `kubectl edit` of a Pod's resources is refused; the subresource is the way in. A container's `resizePolicy` decides whether a change needs a restart.""",
show=r"""kubectl -n scale run sizer --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"sizer","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"200m","memory":"64Mi"}}}]}}'
kubectl -n scale wait --for=condition=Ready pod/sizer --timeout=120s
kubectl -n scale patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{"requests":{"cpu":"150m"},"limits":{"cpu":"300m"}}}]}}'
sleep 5
kubectl -n scale get pod sizer -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}{.status.containerStatuses[0].restartCount}{"\n"}'"""),
show=[cmd(r"""k -n scale run sizer --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > sizer.yaml"""),
hand(r"""vim sizer.yaml""",
r"""Make `resources: {}`:
```
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 200m, memory: 64Mi}
```""",
run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "100m", "memory": "64Mi"}, "limits": {"cpu": "200m", "memory": "64Mi"}}' sizer.yaml"""),
cmd(r"""k -n scale apply -f sizer.yaml"""),
hand(r"""k -n scale get po sizer""",
r"""Once it is `Running`:""",
run=r"""kubectl -n scale wait --for=condition=Ready pod/sizer --timeout=120s"""),
hand(r"""k -n scale edit po sizer --subresource resize""",
r"""Set the CPU request to `150m` and the CPU limit to `300m`, then `:wq`.""",
run=r"""kubectl -n scale patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{"requests":{"cpu":"150m"},"limits":{"cpu":"300m"}}}]}}' """),
cmd(r"""k -n scale describe po sizer | grep -A6 Limits""", r"""The new values, and `Restart Count: 0`.""")]),
],
"outro": r"""You made HPAs both ways, slowed down their scale-down, watched one scale up, and resized a running Pod without replacing it.
@ -421,17 +449,24 @@ rs=$(tr -d ' \r\n' < "$WS/lesson-3.4/rs.txt" 2>/dev/null); [ -n "$rs" ] || exit
eq "$(j -n heal rs "$rs" '{.metadata.ownerReferences[0].name} {.status.readyReplicas}')" "web 3" || exit 1
now=$(j -n heal pods -l app=web '{.items[*].metadata.uid}'); for u in $now; do grep -q "$u" "$PRIV/lesson-3.4.uids" || exit 0; done; exit 1""",
hint=r"""`kubectl -n heal delete pod $(kubectl -n heal get pod -l app=web -o name | head -1)`. Pods of a Deployment are named DEPLOYMENT-RSHASH-RANDOM; the middle part is the ReplicaSet's suffix.""",
show=r"""kubectl -n heal delete $(kubectl -n heal get pod -l app=web -o name | head -1)
kubectl -n heal get pods -l app=web
kubectl -n heal get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].name}' > {WS}/lesson-3.4/rs.txt
kubectl -n heal get rs"""),
show=[cmd(r"""k -n heal get po -l app=web"""),
hand(r"""k -n heal delete po POD""",
r"""`POD`: any one of the three names.""",
run=r"""kubectl -n heal delete $(kubectl -n heal get pod -l app=web -o name | head -1)"""),
cmd(r"""k -n heal get po -l app=web
k -n heal get rs""", r"""A new Pod already, and one ReplicaSet behind them all: `web-` and a hash."""),
hand(r"""echo RS > {WS}/lesson-3.4/rs.txt""",
r"""`RS`: that ReplicaSet's name.""",
run=r"""kubectl -n heal get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].name}' > {WS}/lesson-3.4/rs.txt""")]),
step("A readiness probe that asks the wrong thing",
r"""The Pods of `api` run, but none is `Ready`: `kubectl -n heal describe pod -l app=api` shows the readiness probe failing with `404`. nginx has no `/healthz`. Make the probe ask for `/` instead, and both Pods become ready.""",
check=r"""eq "$(j -n heal deploy api '{.spec.template.spec.containers[0].readinessProbe.httpGet.path}')" / && eq "$(j -n heal deploy api '{.status.readyReplicas}')" 2""",
hint=r"""A JSON patch on `/spec/template/spec/containers/0/readinessProbe/httpGet/path`, or `kubectl edit deploy api`. The fix is a change of template: a rollout.""",
show=r"""kubectl -n heal describe pod -l app=api | grep -m2 Readiness
kubectl -n heal patch deploy api --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]'
kubectl -n heal rollout status deploy api --timeout=120s"""),
show=[cmd(r"""k -n heal describe po -l app=api | grep -i readiness"""),
hand(r"""k -n heal edit deploy api""",
r"""In the `readinessProbe`, change `path: /healthz` to `path: /`, then `:wq`.""",
run=r"""kubectl -n heal patch deploy api --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]' """),
cmd(r"""k -n heal rollout status deploy api""")]),
step("A liveness probe that restarts",
r"""Run Pod `fragile` in `heal` (`busybox:1.36`) with the command `sh -c 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400'` and a **liveness** probe that runs `cat /tmp/healthy` every 5 seconds, starting after 5.
@ -440,24 +475,19 @@ After 20 seconds the file is gone, the probe fails three times, and the kubelet
eq "$(j -n heal pod fragile '{.spec.containers[0].livenessProbe.exec.command[1]}')" /tmp/healthy || exit 1
[ "$(j -n heal pod fragile '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null""",
hint=r"""`livenessProbe: {exec: {command: [cat, /tmp/healthy]}, initialDelaySeconds: 5, periodSeconds: 5}`. `kubectl -n heal get pod fragile -w` shows RESTARTS go up; `describe` shows `Liveness probe failed`.""",
show=r"""kubectl -n heal apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: fragile
spec:
containers:
- name: fragile
image: busybox:1.36
command: [sh, -c, 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400']
show=[cmd(r"""k -n heal run fragile --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400' > fragile.yaml"""),
hand(r"""vim fragile.yaml""",
r"""Add to the container, below `resources: {}` (the docs page *Configure Liveness, Readiness and Startup Probes* has it):
```
livenessProbe:
exec:
command: [cat, /tmp/healthy]
initialDelaySeconds: 5
periodSeconds: 5
EOF
sleep 50
kubectl -n heal get pod fragile"""),
```""",
run=r"""yq -i '.spec.containers[0].livenessProbe = {"exec": {"command": ["cat", "/tmp/healthy"]}, "initialDelaySeconds": 5, "periodSeconds": 5}' fragile.yaml"""),
cmd(r"""k -n heal apply -f fragile.yaml
k -n heal get po fragile""", r"""Again after about 40 seconds: `RESTARTS` is 1.""")]),
step("A budget for disruptions",
r"""Create PodDisruptionBudget `web-pdb` in `heal`: at least **2** Pods of `app=web` must stay available.
@ -468,20 +498,25 @@ Then try to drain `{W1}`, where all three `web` Pods run: `kubectl drain {W1} --
eq "$(j -n heal pdb web-pdb '{.spec.minAvailable} {.spec.selector.matchLabels.app}')" "2 web" || exit 1
grep -qi 'disruption budget' "$WS/lesson-3.4/drain.txt" 2>/dev/null && [ "$(j node "$W1" '{.spec.unschedulable}')" != true ]""",
hint=r"""`kubectl create pdb web-pdb --selector=app=web --min-available=2`. Drain prints the refusals on stderr: `2>&1 | tee FILE`. `kubectl uncordon {W1}` afterwards - drain cordons first.""",
show=r"""kubectl -n heal create pdb web-pdb --selector=app=web --min-available=2
kubectl -n heal get pdb web-pdb
kubectl -n heal delete pod fragile --grace-period=1
kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 | tee {WS}/lesson-3.4/drain.txt | tail -3
kubectl uncordon {W1}"""),
show=[cmd(r"""k -n heal create pdb web-pdb --selector=app=web --min-available=2
k -n heal delete po fragile --now
k drain {W1} --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 | tee {WS}/lesson-3.4/drain.txt
k uncordon {W1}""", r"""The drain evicts one `web` Pod, then refuses the others: "Cannot evict pod as it would violate the pod's disruption budget".""")]),
step("A Job that runs to completion",
r"""Create Job `batch` in `heal`: `busybox:1.36` running `sh -c 'echo processed; sleep 2'`, **3** completions, **2** at a time, giving up after **2** retries (`backoffLimit`). Wait until it is complete.""",
check=r"""eq "$(j -n heal job batch '{.spec.completions} {.spec.parallelism} {.spec.backoffLimit} {.status.succeeded}')" "3 2 2 3" """,
hint=r"""`kubectl create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'`, then add `completions`, `parallelism` and `backoffLimit` under `spec`. `kubectl wait --for=condition=complete job/batch`.""",
show=r"""kubectl -n heal create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo processed; sleep 2' > {WS}/lesson-3.4/batch.yaml
yq -i '.spec.completions = 3 | .spec.parallelism = 2 | .spec.backoffLimit = 2' {WS}/lesson-3.4/batch.yaml
kubectl -n heal apply -f {WS}/lesson-3.4/batch.yaml
kubectl -n heal wait --for=condition=complete job/batch --timeout=120s
kubectl -n heal get pods -l job-name=batch"""),
show=[cmd(r"""k -n heal create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo processed; sleep 2' > batch.yaml"""),
hand(r"""vim batch.yaml""",
r"""Add to the Job's own `spec:`, above `template:`:
```
completions: 3
parallelism: 2
backoffLimit: 2
```""",
run=r"""yq -i '.spec.completions = 3 | .spec.parallelism = 2 | .spec.backoffLimit = 2' batch.yaml"""),
cmd(r"""k apply -f batch.yaml
k -n heal get job,po""", r"""Again until `COMPLETIONS` is `3/3`.""")]),
step("A CronJob, and a run on demand",
r"""Create CronJob `report` in `heal`: every 30 minutes (`*/30 * * * *`), `busybox:1.36` running `date`, never two runs at once (`concurrencyPolicy: Forbid`), keeping the last **2** successful Jobs.
@ -490,11 +525,17 @@ Do not wait half an hour: start a Job `report-now` from the CronJob yourself, an
eq "$(j -n heal cronjob report '{.spec.schedule} {.spec.concurrencyPolicy} {.spec.successfulJobsHistoryLimit}')" "*/30 * * * * Forbid 2" || exit 1
eq "$(j -n heal job report-now '{.metadata.annotations.cronjob\.kubernetes\.io/instantiate} {.status.succeeded}')" "manual 1" """,
hint=r"""`kubectl create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date`, then the two fields. `kubectl create job report-now --from=cronjob/report`.""",
show=r"""kubectl -n heal create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date > {WS}/lesson-3.4/report.yaml
yq -i '.spec.concurrencyPolicy = "Forbid" | .spec.successfulJobsHistoryLimit = 2' {WS}/lesson-3.4/report.yaml
kubectl -n heal apply -f {WS}/lesson-3.4/report.yaml
kubectl -n heal create job report-now --from=cronjob/report
kubectl -n heal wait --for=condition=complete job/report-now --timeout=120s"""),
show=[cmd(r"""k -n heal create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date > report.yaml"""),
hand(r"""vim report.yaml""",
r"""Add to the CronJob's own `spec:`, beside `schedule:`:
```
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 2
```""",
run=r"""yq -i '.spec.concurrencyPolicy = "Forbid" | .spec.successfulJobsHistoryLimit = 2' report.yaml"""),
cmd(r"""k apply -f report.yaml
k -n heal create job report-now --from=cronjob/report
k -n heal get job,po""", r"""Again until `report-now` shows `1/1`.""")]),
],
"outro": r"""You watched a ReplicaSet replace a Pod, fixed a readiness probe, saw a liveness probe restart a container, stopped a drain with a disruption budget, and ran a Job and a CronJob.
@ -537,35 +578,48 @@ mkns sched
check=r"""
for p in gold:Guaranteed silver:Burstable bronze:BestEffort; do eq "$(j -n sched pod "${p%%:*}" '{.status.qosClass}')" "${p##*:}" || exit 1; done""",
hint=r"""`kubectl run NAME --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400`, then `resources:` in the container. `.status.qosClass` shows the result.""",
show=r"""kubectl -n sched run bronze --image=busybox:1.36 -- sleep 86400
kubectl -n sched run silver --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"silver","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"memory":"32Mi"}}}]}}'
kubectl -n sched run gold --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"gold","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"64Mi"}}}]}}'
kubectl -n sched get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass"""),
show=[cmd(r"""k -n sched run bronze --image=busybox:1.36 -- sleep 86400
k -n sched run silver --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > silver.yaml"""),
hand(r"""vim silver.yaml""",
r"""Make `resources: {}`:
```
resources:
requests: {memory: 32Mi}
```""",
run=r"""yq -i '.spec.containers[0].resources = {"requests": {"memory": "32Mi"}}' silver.yaml"""),
cmd(r"""k -n sched apply -f silver.yaml
k -n sched run gold --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > gold.yaml"""),
hand(r"""vim gold.yaml""",
r"""Make `resources: {}`:
```
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 100m, memory: 64Mi}
```""",
run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "100m", "memory": "64Mi"}, "limits": {"cpu": "100m", "memory": "64Mi"}}' gold.yaml"""),
cmd(r"""k -n sched apply -f gold.yaml
k -n sched describe po | grep -e ^Name: -e QoS""")]),
step("Only on SSD nodes",
r"""Label `{W2}` with `disk=ssd`. Then run Pod `fast` in `sched` (`busybox:1.36`, `sleep 86400`) with a **required** node affinity for `disk In (ssd)`. It lands on `{W2}` - and if no node had the label, it would stay `Pending`.""",
check=r"""
eq "$(j node "$W2" '{.metadata.labels.disk}')" ssd && eq "$(j -n sched pod fast '{.spec.nodeName}')" "$W2" \
&& eq "$(j -n sched pod fast '{.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key}')" disk""",
hint=r"""`kubectl label node {W2} disk=ssd`. The affinity block: `affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[].matchExpressions[]` with `key`, `operator: In`, `values`. The docs page "Assign Pods to Nodes using Node Affinity" has it to copy.""",
show=r"""kubectl label node {W2} disk=ssd
kubectl -n sched apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: fast
spec:
show=[cmd(r"""k label node {W2} disk=ssd
k -n sched run fast --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > fast.yaml"""),
hand(r"""vim fast.yaml""",
r"""Add to the Pod's `spec:`, above `containers:` (the docs page *Assign Pods to Nodes using Node Affinity* has it):
```
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- {key: disk, operator: In, values: [ssd]}
containers:
- name: fast
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n sched get pod fast -o wide"""),
```""",
run=r"""yq -i '.spec.affinity = {"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "disk", "operator": "In", "values": ["ssd"]}]}]}}}' fast.yaml"""),
cmd(r"""k -n sched apply -f fast.yaml
k -n sched get po fast -o wide""")]),
step("One replica per node",
r"""Create Deployment `spread` in `sched` (2 replicas, `busybox:1.36`, `sleep 86400`, label `app=spread`) whose Pods **refuse** to share a node with each other: a required pod anti-affinity on `app=spread`, with `topologyKey: kubernetes.io/hostname`. One Pod lands on each worker; a third replica would stay `Pending`.""",
check=r"""
@ -573,32 +627,20 @@ eq "$(j -n sched deploy spread '{.status.readyReplicas}')" 2 || exit 1
eq "$(j -n sched deploy spread '{.spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey}')" kubernetes.io/hostname || exit 1
[ "$($K -n sched get pods -l app=spread -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 2 ]""",
hint=r"""`podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution: [{labelSelector: {matchLabels: {app: spread}}, topologyKey: kubernetes.io/hostname}]` in the Pod template. `topologySpreadConstraints` is the softer, counting way to spread.""",
show=r"""kubectl -n sched apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: spread
spec:
replicas: 2
selector:
matchLabels: {app: spread}
template:
metadata:
labels: {app: spread}
spec:
show=[cmd(r"""k -n sched create deploy spread --image=busybox:1.36 --replicas=2 --dry-run=client -o yaml -- sleep 86400 > spread.yaml"""),
hand(r"""vim spread.yaml""",
r"""Add to the Pod template's `spec:`, above `containers:` (the docs page *Assigning Pods to Nodes* has an example):
```
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels: {app: spread}
topologyKey: kubernetes.io/hostname
containers:
- name: spread
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n sched rollout status deploy spread --timeout=120s
kubectl -n sched get pods -l app=spread -o wide"""),
```""",
run=r"""yq -i '.spec.template.spec.affinity = {"podAntiAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": [{"labelSelector": {"matchLabels": {"app": "spread"}}, "topologyKey": "kubernetes.io/hostname"}]}}' spread.yaml"""),
cmd(r"""k apply -f spread.yaml
k -n sched get po -l app=spread -o wide""")]),
step("A node for batch work only",
r"""Taint `{W1}` with `dedicated=batch:NoSchedule`: from now on only Pods that tolerate it are scheduled there (Pods already running stay - `NoExecute` would evict them).
@ -607,23 +649,19 @@ Then run Pod `nightly` in `sched` (`busybox:1.36`, `sleep 86400`) that tolerates
[ -n "$($K get node "$W1" -o jsonpath='{range .spec.taints[*]}{.key}={.value}:{.effect}{"\n"}{end}' | grep -x 'dedicated=batch:NoSchedule')" ] || exit 1
eq "$(j -n sched pod nightly '{.spec.nodeName} {.status.phase}')" "$W1 Running" && eq "$(j -n sched pod nightly '{.spec.tolerations[?(@.key=="dedicated")].value}')" batch""",
hint=r"""`kubectl taint node {W1} dedicated=batch:NoSchedule`. The toleration: `{key: dedicated, operator: Equal, value: batch, effect: NoSchedule}`.""",
show=r"""kubectl taint node {W1} dedicated=batch:NoSchedule
kubectl -n sched apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: nightly
spec:
show=[cmd(r"""k taint node {W1} dedicated=batch:NoSchedule
k -n sched run nightly --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > nightly.yaml"""),
hand(r"""vim nightly.yaml""",
r"""Add to the Pod's `spec:`, above `containers:`:
```
nodeSelector:
kubernetes.io/hostname: {W1}
tolerations:
- {key: dedicated, operator: Equal, value: batch, effect: NoSchedule}
containers:
- name: nightly
image: busybox:1.36
command: [sleep, '86400']
EOF
kubectl -n sched get pod nightly -o wide"""),
```""",
run=r"""yq -i '.spec.nodeSelector = {"kubernetes.io/hostname": "{W1}"} | .spec.tolerations = [{"key": "dedicated", "operator": "Equal", "value": "batch", "effect": "NoSchedule"}]' nightly.yaml"""),
cmd(r"""k -n sched apply -f nightly.yaml
k -n sched get po nightly -o wide""")]),
step("Defaults for Pods that bring none",
r"""Create LimitRange `defaults` in `sched`: containers that say nothing get a **request** of `50m` CPU and `32Mi` memory and a **limit** of `200m` and `128Mi`.
@ -632,7 +670,9 @@ Then run Pod `plain` (`busybox:1.36`, `sleep 86400`, no resources) and look at w
eq "$(j -n sched limitrange defaults '{.spec.limits[0].type}')" Container || exit 1
eq "$(j -n sched pod plain '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.requests.memory} {.spec.containers[0].resources.limits.cpu} {.spec.containers[0].resources.limits.memory}')" "50m 32Mi 200m 128Mi" """,
hint=r"""`spec.limits: [{type: Container, defaultRequest: {cpu: 50m, memory: 32Mi}, default: {cpu: 200m, memory: 128Mi}}]` - `default` is the limit.""",
show=r"""kubectl -n sched apply -f - <<'EOF'
show=[hand(r"""vim lr.yaml""",
r"""No `kubectl create` for a LimitRange: paste the example from the docs page *Configure Default CPU Requests and Limits for a Namespace* and make it:
```
apiVersion: v1
kind: LimitRange
metadata:
@ -642,19 +682,37 @@ spec:
- type: Container
defaultRequest: {cpu: 50m, memory: 32Mi}
default: {cpu: 200m, memory: 128Mi}
EOF
kubectl -n sched run plain --image=busybox:1.36 -- sleep 86400
kubectl -n sched get pod plain -o jsonpath='{.spec.containers[0].resources}'; echo"""),
```""",
run=r"""cat > lr.yaml <<'EOF'
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
spec:
limits:
- type: Container
defaultRequest: {cpu: 50m, memory: 32Mi}
default: {cpu: 200m, memory: 128Mi}
EOF"""),
cmd(r"""k -n sched apply -f lr.yaml
k -n sched run plain --image=busybox:1.36 -- sleep 86400
k -n sched describe po plain | grep -A6 Limits""")]),
step("Past the scheduler",
r"""Cordon `{W2}`. Then create Pod `manual` in `sched` (`busybox:1.36`, `sleep 86400`) with **`nodeName: {W2}`**. It runs there anyway: `nodeName` hands the Pod straight to that node's kubelet, and cordoning only tells the *scheduler* to stay away.
Leave `{W2}` cordoned.""",
check=r"""eq "$(j node "$W2" '{.spec.unschedulable}')" true && eq "$(j -n sched pod manual '{.spec.nodeName} {.status.phase}')" "$W2 Running" """,
hint=r"""`kubectl cordon {W2}`, then a Pod manifest with `spec.nodeName`. Without a running scheduler, this is also how you place a Pod by hand.""",
show=r"""kubectl cordon {W2}
kubectl -n sched run manual --image=busybox:1.36 --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 86400
kubectl -n sched get pod manual -o wide
kubectl get nodes"""),
show=[cmd(r"""k cordon {W2}
k -n sched run manual --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > manual.yaml"""),
hand(r"""vim manual.yaml""",
r"""Add to the Pod's `spec:`, above `containers:`:
```
nodeName: {W2}
```""",
run=r"""yq -i '.spec.nodeName = "{W2}"' manual.yaml"""),
cmd(r"""k -n sched apply -f manual.yaml
k -n sched get po manual -o wide""")]),
],
"outro": r"""You produced all three QoS classes, steered Pods with node affinity, kept replicas apart, reserved a node with a taint, set defaults with a LimitRange, and bypassed the scheduler.

94
ksim
View File

@ -124,7 +124,24 @@ def lesson_task(l, objectives):
o = next((o for o in objectives if o["id"] == l["objective"]), None)
return dict(l, lesson=True, domain=o["domain"] if o else "", difficulty="normal", set=None,
checks=[{"name": s["title"], "pts": 1, "script": s["check"]} for s in l["steps"]], weight=len(l["steps"]),
task=l["intro"], solution="\n\n".join(s["show"] for s in l["steps"] if s["show"]))
task=l["intro"], solution="\n\n".join(show_md(s) for s in l["steps"] if show_parts(s)))
def show_parts(s):
"""How to do a step, as parts (sim/bank.py: cmd, hand): commands typed as they stand, and what
is done by hand - an edit, a value read off the screen - with the script the audit runs for it."""
sh = s.get("show") or ""
return sh if isinstance(sh, list) else ([{"type": sh, "how": "", "run": sh}] if sh.strip() else [])
def show_md(s):
"""How to do a step, as you read it: each thing to type in a block of its own (a click copies
it), and beneath it what to do there by hand."""
return "\n\n".join(f"```\n{p['type'].strip(chr(10))}\n```" + (f"\n\n{p['how'].strip()}" if p["how"] else "")
for p in show_parts(s))
def show_script(s):
"""What the audit runs for a step: what you type, and for each part done by hand its script."""
return "\n".join(p["run"] for p in show_parts(s))
def set_questions(name):
@ -1566,7 +1583,9 @@ def begin_session(qs, mode, duration, force, log: Callable[..., Any] = _log_msg,
st = {"mode": mode, "track": TRACK_ID, "hard": hard, "ids": [q["id"] for q in qs],
"params": params_by_id, "duration": duration, "started_at": time.time(), "ended_at": None,
"k8s": k8s_version(), "results": {}, "flags": [], "notes": "",
"objective": objective} # the curriculum objective a drill or a lesson was started for, if any
"objective": objective, # the curriculum objective a drill or a lesson was started for, if any
# the tasks or lessons whose host failed its setup: the page and ./ksim lesson say so, and how to rebuild it
"broken": [q["id"] for q in qs if any(p.startswith(host_of(q) + ":") for p in problems)]}
lesson = mode == "lesson"
if lesson:
st["steps"] = {} # lesson id -> step index -> its latest check (a pass sticks)
@ -1742,6 +1761,9 @@ def cli_md(text):
def print_step(l, params, i, st=None):
s = l["steps"][i]
done = bool(st and steps_of(st, l).get(str(i), {}).get("pass"))
if st and l["id"] in st.get("broken", []):
print(red(f"This lesson's host failed its setup when the session started: it is not as the steps describe.\n"
f" Rebuild it (about a minute): ./ksim reset {l['id']}"))
print(bold(f"Step {i + 1} of {len(l['steps'])}: {render(s['title'], params, l)}") + (green(" ✓ checked") if done else ""))
print(cli_md(render(s["text"], params, l)))
print(dim(" ./ksim lesson check when it is done ./ksim lesson hint / show if you are stuck"))
@ -1770,6 +1792,9 @@ def cmd_lesson(args):
sub = args[0] if args else ""
if sub == "audit":
return lesson_audit(args[1:])
if sub == "lint": # no cluster needed: what each step has you type, against TYPING_TRAPS
names = [a for a in args[1:] if not a.startswith("--")]
return print_lint(lesson_lint(LESSONS if not names else [x for x in LESSONS if x["objective"] in names or x["domain"] in names]))
st = load_state()
live = lesson_of(st) if st and not st.get("ended_at") else None
if sub == "next":
@ -1794,9 +1819,9 @@ def cmd_lesson(args):
if sub == "step":
return print_step(live, params, i, st)
if sub in ("hint", "show"):
print(bold(f"{'Hint' if sub == 'hint' else 'The commands'} - step {i + 1}: {render(s['title'], params, live)}"))
text = render(s[sub], params, live)
print((cli_md(text) if sub == "hint" else text) or dim("Nothing for this step."))
print(bold(f"{'Hint' if sub == 'hint' else 'How to do it'} - step {i + 1}: {render(s['title'], params, live)}"))
text = render(s["hint"] if sub == "hint" else show_md(s), params, live)
print(cli_md(text) or dim("Nothing for this step."))
return
r, why = lesson_check(st, i)
if why:
@ -1840,6 +1865,38 @@ def cmd_lesson(args):
print(dim(f"\n ./ksim lesson {LESSONS[0]['objective']} starts that lesson ./ksim lesson audit proves them on real clusters"))
print(dim(f" ./ksim lesson {LESSONS[0]['domain']} a domain's lessons not done yet, back to back ({runs} done)"))
# What a step must not ask you to type: in the exam you have a keyboard, vim and the docs, and no
# time to compose scripts. Such things belong in the script of a part done by hand (sim/bank.py).
TYPING_TRAPS = [(re.compile(p, re.M), why) for p, why in (
(r"\b(sed|awk|yq|jq|perl|python3?)\b", "a text tool - edit in vim, or read the value and echo it"),
(r"<<", "a heredoc - generate the YAML, or paste the docs' example into vim"),
(r"^\s*(sleep|for|while|until)\b", "a pause or a loop - look again yourself"),
(r"\b(k|kubectl)\b[^|\n]* wait\b", "kubectl wait - look with get"),
(r"--overrides", "--overrides - generate the YAML and edit it"),
(r"\bpatch\b[^\n]*(-p|--patch)[ =]'[\[{]", "a JSON patch - kubectl edit"),
(r"jsonpath[^\n|]*>", "jsonpath into a file - read the value, then echo it"),
(r"\$\(", "command substitution - read the value, then type it"),
(r"(^|[|;&(]\s*)(sudo\s+)?kubectl\b", "kubectl in full - the exam has k"))] # the command, not the package
def lesson_lint(ls):
"""What each step has you type, against TYPING_TRAPS. Returns [(lesson, step number, why, line)]."""
out = []
for l in ls:
for i, s in enumerate(l["steps"]):
for p in show_parts(s):
for line in p["type"].splitlines():
for rx, why in TYPING_TRAPS:
if rx.search(line):
out.append((l, i + 1, why, line.strip()))
return out
def print_lint(problems):
for l, n, why, line in problems:
print(f" {yellow(l['objective'] + ' step ' + str(n))} {why}\n {dim(line[:150])}")
steps = len({(l["id"], n) for l, n, _, _ in problems})
print(bold(f"{steps} step(s) ask you to type what you would not type in the exam.") if steps
else green("Nothing a step asks you to type is a script: every step is done as in the exam."))
def lesson_audit(args):
"""Prove lessons on real clusters: each is seeded on a host of its own; then every step must
fail its check, have its commands run as `candidate`, and pass its check - so a lesson can be
@ -1862,6 +1919,8 @@ def lesson_audit(args):
open(lock, "w").close()
import atexit
atexit.register(lambda: os.path.exists(lock) and os.remove(lock))
print(bold(f"What the {len(ls)} lesson(s) have you type:"))
print_lint(lesson_lint(ls))
print(bold(f"Lesson audit of {len(ls)} {TRACK_ID.upper()} lesson(s), {PARALLEL} at a time - each step must fail, be done by its own commands, then pass."))
def one(l):
params = make_params(l, random.Random(l["id"])); host = host_of(l); t0 = time.time()
@ -1883,7 +1942,7 @@ def lesson_audit(args):
if check_step(l, params, i)["pass"]:
res.update(verdict=f"step {i + 1} passes before its commands")
break
_, sout = solve_on_host(l, params, script=s["show"])
_, sout = solve_on_host(l, params, script=show_script(s))
# what the commands created may still be starting: a learner would watch it come up
r, waited = check_step(l, params, i), 0
while not r["pass"] and waited < 90:
@ -2155,13 +2214,24 @@ def cmd_history(args):
cells.append(f"{round(100 * de / dt):>3}%" if dt else " -")
print(f" {d:<7} " + " → ".join(cells))
def rebuild(st, q, log: Callable[..., Any] = _log_msg):
"""A task's or a lesson's host again from scratch, scenario included. A lesson's steps start over
on it; a host that comes up whole is no longer listed as broken. Returns what went wrong, or None."""
problem = provision(q, session_params(st, q), log=log)
st = load_state() or st
if q.get("lesson"):
st.get("steps", {}).pop(str(q["id"]), None)
st["broken"] = [i for i in st.get("broken", []) if i != q["id"]] + ([q["id"]] if problem else [])
save_state(st)
return problem
def cmd_reset(args):
"""Throw a task's cluster away and build it again, scenario included."""
if not args:
die("usage: ./ksim reset N (rebuild one task's host from scratch: new cluster, scenario seeded again)")
st = require_started(); q = q_by_id(int(args[0]))
print(dim(f" rebuilding {host_of(q)} - about a minute..."))
problem = provision(q, session_params(st, q))
problem = rebuild(st, q)
print(("rebuilt " if not problem else "problem rebuilding ") + f"Q{q['id']:02d}: {q['title']}")
# --------------------------------------------------------------- web ----
@ -2330,7 +2400,9 @@ class Api(http.server.BaseHTTPRequestHandler):
return self._json({"error": down_message(down)}, 503)
finish_session(st, log=lambda *_: None)
elif p.path == "/api/reset":
q = q_by_id(int(body["id"])); provision(q, session_params(st, q), log=lambda *_: None)
problem = rebuild(st, q_by_id(int(body["id"])), log=lambda *_: None)
if problem:
return self._json({"error": f"Rebuilding failed again ({problem}) - try ./ksim reset {body['id']} in a terminal to see why."}, 500)
else:
return self._json({"error": "not found"}, 404)
return self._json(self.session_payload())
@ -2368,6 +2440,7 @@ class Api(http.server.BaseHTTPRequestHandler):
"paused": bool(st.get("paused_at")), "paused_total": paused_total(st),
"notes": st.get("notes", ""),
"objective": st.get("objective"), "milestones": st.get("milestones", []), "aborted": bool(st.get("aborted")),
"broken": st.get("broken", []),
"results": st["results"], "score": {"earned": earned, "total": total, "pct": pct, "pass_pct": PASS_PCT},
"domains": self.domain_scores(st), "server_time": time.time()}
l = lesson_of(st)
@ -2389,7 +2462,7 @@ class Api(http.server.BaseHTTPRequestHandler):
r = lambda t: render(t, params, l).strip("\n")
return {"id": l["id"], "objective": l["objective"], "title": l["title"], "host": host_of(l), "nodes": 1 + workers_of(l),
"intro": r(l["intro"]), "outro": r(l["outro"]),
"steps": [{"title": r(s["title"]), "text": r(s["text"]), "hint": r(s["hint"]), "show": r(s["show"])} for s in l["steps"]]}
"steps": [{"title": r(s["title"]), "text": r(s["text"]), "hint": r(s["hint"]), "show": r(show_md(s))} for s in l["steps"]]}
def path_payload(self) -> dict[str, Any]:
"""The curriculum as the UI shows it: every objective with its notes, tasks and state, how
@ -2699,7 +2772,8 @@ def cmd_help(args):
" lesson DOMAIN [--all] teach a domain: its lessons not done yet (--all: every one) back to back, hosts made up front\n"
" lesson next | OBJ in a domain's run: on to its next lesson, or to any lesson of it\n"
" lesson step|check|hint|show [N] the step you are on, check it, a hint, the commands; ./ksim end finishes\n"
" lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after them\n")
" lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after them\n"
" lesson lint [OBJ|DOMAIN..] what the steps have you type: no scripts, only what you would type in the exam\n")
print(f"""usage: ./ksim [--track cka|ckad|cks] <command> (current track: {TRACK_ID}, {len(QUESTIONS)} tasks)
Every task runs on its own cluster at the exam's Kubernetes version ({TRACK_ID}: {k8s_version()}),
reached with `ssh <host>` from the remote desktop.

16
sim/bank.py
View File

@ -21,10 +21,22 @@ def C(name, pts, script):
def step(title, text, check, hint="", show=""):
"""One step of a lesson (<track>/lessons.py): what to do (Markdown), the check that proves it
(bash, exit 0 = done, run like a task's checks), a hint, and the commands that do it - shown on
request, and run by ./ksim lesson audit to prove the step can be done as shown."""
(bash, exit 0 = done, run like a task's checks), a hint, and how to do it as you would in the
exam - shown on request, and done by ./ksim lesson audit to prove the step works as shown.
`show` is commands you type as they stand, or a list of cmd() and hand() parts."""
return {"title": title, "text": text, "check": check, "hint": hint, "show": show}
def cmd(text, how=""):
"""Commands you type as they stand, and what to look for (Markdown); the audit runs them as
they stand."""
return {"type": text, "how": how, "run": text}
def hand(type_, how, run):
"""What you do by hand: what you type to begin (`vim f`, `k edit ...`, `echo NAME > f`), what
to do there (Markdown), and a script that does the same, which the audit runs instead - an
editor needs a person, and so does a value read off the screen."""
return {"type": type_, "how": how, "run": run}
def ns_pick(r, base):
"""Namespace name with a random suffix so the same task never lands on the same namespace twice."""
return f"{base}-{r.choice(['alpha', 'bravo', 'delta', 'echo', 'kilo', 'nova', 'omega', 'zulu'])}"

36
web/index.html
View File

@ -607,8 +607,10 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
// minimal markdown: paragraphs, **bold**, `code` (click to copy), ```fenced blocks``` (kept as
// written, click to copy), and a paragraph made only of "- " lines as a list. Other lists stay as
// literal lines.
const inlineMd = s => esc(s).replace(/\*\*(.+?)\*\*/g, '<b>$1</b>')
.replace(/`([^`]+)`/g, (m, c) => `<code class="ic" title="click to copy">${c}</code>`);
// `code` first, so that the stars of a cron schedule in it are not read as **bold** or *italics*
const inlineMd = s => s.split(/(`[^`]+`)/).map((p, i) => i % 2
? `<code class="ic" title="click to copy">${esc(p.slice(1, -1))}</code>`
: esc(p).replace(/\*\*(.+?)\*\*/g, '<b>$1</b>').replace(/(^|[^\w*])\*(?!\s)([^*]+?)\*(?![\w*])/g, '$1<i>$2</i>')).join('');
const render = md => md.split(/^```[^\n]*\n([\s\S]*?)\n?^```[ \t]*$/m).map((part, i) => i % 2
? `<pre class="code" title="click to copy">${esc(part)}</pre>`
: part.split(/\n\s*\n/).filter(p => p.trim()).map(p => /^(- .*(\n|$))+$/.test(p.trim() + '\n')
@ -925,7 +927,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
$('#startBody').innerHTML = `<div class="alert alert-info"><b>No clock.</b> Take your time: the lessons teach, the drills test.</div>
<ul>
<li>every lesson has a host of its own, and all ${n} are up already: the top of each lesson says which to <code class="ic">ssh</code> to;</li>
<li>a lesson is an introduction, then steps - each with a <b>Check step</b> button, a hint, and the commands if you are stuck; steps count in order;</li>
<li>a lesson is an introduction, then steps - each with a <b>Check step</b> button, a hint, and <b>Show me how</b>: the exam way, if you are stuck; steps count in order;</li>
<li>a lesson's wrap-up leads to the next one, and the list at the top goes to any lesson of the run;</li>
<li>a lesson counts as done the moment its last step is checked; nothing goes into your history or scores;</li>
<li>finish the run from <b>Controls</b>, then drill ${esc(C.domain)} against the clock.</li>
@ -939,7 +941,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
$('#startBody').innerHTML = `<div class="alert alert-info"><b>No clock.</b> Take your time: a lesson teaches, the drills test.</div>
<ul>
<li>it runs on a host of its own: <code class="ic">${esc('ssh ' + L.host)}</code> in the desktop's terminal;</li>
<li>read the introduction, then do the ${L.steps.length} steps - each has a <b>Check step</b> button, a hint, and the commands if you are stuck;</li>
<li>read the introduction, then do the ${L.steps.length} steps - each has a <b>Check step</b> button, a hint, and <b>Show me how</b>: the exam way, if you are stuck;</li>
<li>steps count in order, and nothing goes into your history or scores;</li>
<li>the wrap-up sends you to the objective's timed drill.</li>
</ul>`;
@ -1015,7 +1017,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
const r = sess.results[q.id], show = r && (sess.mode === 'drill' || sess.ended_at);
const oids = q.objectives || [], oid = oids.includes(sess.objective) ? sess.objective : oids[0];
const obj = oid && (S.objectives || []).find(o => o.id === oid); // which objective a task belongs to is a hint: it follows the solution's rule
$('#qContent').innerHTML = `
$('#qContent').innerHTML = brokenHost(q.id) + `
<div class="qhead">
<div class="line"><b>Solve this question on instance:</b> <code class="ic" title="click to copy">ssh ${esc(q.host)}</code></div>
<div class="line sub">${q.nodes}-node cluster · task files in ${esc(S.workspace)} on that host</div>
@ -1031,6 +1033,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
${q.solution ? `<details class="fold"><summary>Reference solution</summary><pre>${esc(q.solution)}</pre></details>` : ''}
${obj ? `<details class="fold"><summary>Objective ${obj.id} · ${esc(obj.text)}</summary><div class="notes">${notesHtml(obj)}</div></details>` : ''}`;
copyable($('#qContent'));
wireRebuild();
const bc = $('#btnCheck'); if (bc) bc.onclick = checkTask;
const fl = sess.flags.includes(q.id);
$('#btnFlag').className = 'btn ' + (fl ? 'btn-warning' : 'btn-outline-warning') + (on ? '' : ' hidden');
@ -1038,6 +1041,24 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
$('#btnSatisfied').disabled = i >= n - 1;
paintDesk();
}
// a host whose setup failed when the session started: say so where it is used, and offer to rebuild it
const brokenHost = id => live() && (S.session.broken || []).includes(id)
? `<div class="alert alert-danger"><b>This host's setup failed</b> when the session started, so it is not as the text describes.
<div style="margin-top:.5rem"><button class="btn btn-danger btn-sm" id="btnRebuild" data-id="${id}">Rebuild this host</button>
<span class="muted"> about a minute - or <code class="ic">./ksim reset ${id}</code> in a terminal</span></div></div>` : '';
function wireRebuild() {
const b = $('#btnRebuild'); if (!b) return;
b.onclick = async () => {
$('#busyTitle').textContent = 'Rebuilding the host';
$('#busyBody').innerHTML = '<p class="mb0">A new cluster, and its setup run again - about a minute.</p>';
openModal('#mBusy');
try { S = await api('/api/reset', {id: +b.dataset.id}); }
catch (e) { alert(e.message); }
finally { closeModal('#mBusy'); }
if (isLesson()) lpage = null;
paintExam();
};
}
// what the exam and a lesson share: the notepad, the remote desktop, the clock
function paintDesk() {
if (document.activeElement !== $('#notes')) $('#notes').value = S.session.notes || '';
@ -1101,7 +1122,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
+ C.lessons.map(x => { const c = x.steps.filter(s => s.pass).length;
return `<option value="${x.id}"${x.id === C.at ? ' selected' : ''}>${c === x.steps.length ? '✓' : '·'} ${esc(x.objective)} ${esc(x.title)} (${c}/${x.steps.length})</option>`; }).join('')
+ '</select></div>';
let body = `<div class="qhead">${pick}<div class="line"><b>This lesson runs on:</b> <code class="ic" title="click to copy">${esc('ssh ' + L.host)}</code></div>
let body = brokenHost(L.id) + `<div class="qhead">${pick}<div class="line"><b>This lesson runs on:</b> <code class="ic" title="click to copy">${esc('ssh ' + L.host)}</code></div>
<div class="line sub">${L.nodes}-node cluster · files in ${esc(S.workspace)} on that host · no clock</div></div><hr>`;
if (lpage === 0) {
body += `<div class="stepsum">Lesson ${esc(L.objective)} · ${n} steps</div><h2 class="qtitle">${esc(L.title)}</h2><div class="qbody">${render(L.intro)}</div>`;
@ -1120,10 +1141,11 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === '
+ (r && !r.pass && r.out ? `<details class="fold"><summary>What the check saw</summary><pre>${esc(r.out)}</pre></details>` : '');
} else if (r && r.pass) body += '<div class="lstate"><span class="ok">✓ Done</span></div>';
if (s.hint) body += `<details class="fold"><summary>Hint</summary><div class="qbody" style="margin-top:10px">${render(s.hint)}</div></details>`;
if (s.show) body += `<details class="fold"><summary>Show me the commands</summary><pre class="code" title="click to copy" style="margin:10px 0 0">${esc(s.show)}</pre></details>`;
if (s.show) body += `<details class="fold"><summary>Show me how</summary><div class="qbody" style="margin-top:10px">${render(s.show)}</div></details>`;
}
$('#qContent').innerHTML = body;
copyable($('#qContent'));
wireRebuild();
const bs = $('#btnStep'); if (bs) bs.onclick = () => checkStep(lpage - 1);
const bf = $('#btnFinish'); if (bf) bf.onclick = finishLesson;
const bn = $('#btnNextLesson'); if (bn) bn.onclick = () => goLesson(nx.id);