From 226ef19ff97f3c6cabc0c281de2fbd6aead77855 Mon Sep 17 00:00:00 2001 From: vumas3r Date: Mon, 26 Oct 2026 20:34:36 -0600 Subject: [PATCH] feat: show every CKA lesson step the way the exam is done Show me how gave the commands the audit runs: sed, awk and yq edits, heredoc manifests, JSON patches, --overrides, sleeps and wait loops - nothing anyone types in the exam. A step's show is now a list of parts (sim/bank.py): cmd() is what you type as it stands; hand() is what you do by hand - vim on a generated skeleton or on a manifest pasted from the docs page it names, kubectl edit, ssh NODE and sudo -i, a value you read and echo - with a script that does the same for the audit. All 135 CKA steps are rewritten so, with k for kubectl; their checks, titles and texts are unchanged. Add ./ksim lesson lint, which reads what every step has you type and flags scripts. The lesson audit runs it first. Record the hosts whose setup failed when a session started. The page warns above that task or lesson and offers Rebuild this host; the CLI names ./ksim reset N on each step. Render *italics* in lessons and tasks, outside code spans. Audited on v1.37: CKA lessons 27/27, 135/135 steps, done the exam way; lint clean. kubectl edit --subresource resize, the one edit the audit replaces with a patch, was proven on its own. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 24 ++ README.md | 23 +- cka/lessons.py | 13 +- cka/lessons_arch.py | 541 +++++++++++++++++++++++++++-------------- cka/lessons_net.py | 479 ++++++++++++++++++++++++++++-------- cka/lessons_storage.py | 431 ++++++++++++++++++++++---------- cka/lessons_tshoot.py | 240 +++++++++++------- cka/lessons_work.py | 438 ++++++++++++++++++--------------- ksim | 94 ++++++- sim/bank.py | 16 +- web/index.html | 36 ++- 11 files changed, 1620 insertions(+), 715 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ea56d6b..98d782f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,30 @@ Notable changes to kubestronaut-sim, newest first. A version is the `VERSION` st (year.month.day.n); `./ksim help` and the top bar of the web page show the one you are running. Every entry names the commit it came in. +## Unreleased + +### Changed + +- **Every CKA lesson step shows the exam way** (*Show me how*, `./ksim lesson show`): `k`, a + generator and the few lines to add in `vim`, a manifest pasted from the docs page it names, + `kubectl edit`, `ssh NODE` and `sudo -i`, or a value you read and `echo` - never a script. Each + command is a block of its own, which a click copies. The scripts the audit needs (`yq`, `sed`, + waits) are kept out of sight, one per part done by hand, so `./ksim lesson audit` still proves + every step: 135 of 135 on v1.37. + +### Added + +- `./ksim lesson lint`: what each step has you type, against what you would not type in the exam + (`sed`, `awk`, `yq`, heredocs, loops, `kubectl wait`, `--overrides`, JSON patches, `jsonpath` + into a file, `kubectl` in full). The lesson audit runs it first. +- A task's or a lesson's host whose setup failed when the session started is marked: the page + says so above it, with **Rebuild this host**, and `./ksim lesson` names `./ksim reset N`. Before, + only the log of the preparing window said so. + +### Fixed + +- The page shows `*italics*` in lessons and tasks instead of the stars. + ## 2026.10.26.3 — 2026-10-26 Everything below came in `95f9d1b`. diff --git a/README.md b/README.md index ebbe76a..a207827 100644 --- a/README.md +++ b/README.md @@ -155,7 +155,8 @@ host name such as `ckad0013` only resolves in the desktop's terminal: from your **Lessons.** A lesson teaches one objective as a guided lab on a host of its own: an introduction, then steps you do in the desktop's terminal, each with a **Check step** button, a hint and **Show me -the commands**, and a wrap-up that sends you to the objective's timed drill. Lessons are the one +how** (the exam way: what you type, what you edit in vim), and a wrap-up that sends you to +the objective's timed drill. Lessons are the one untimed part of the simulator — they teach, the drills test — and they write nothing to the history, the statistics or the weak list; a lesson whose every step was checked is marked done on its objective (`lesson ✓` in the objective list). Steps count in order. Start one from an @@ -295,13 +296,19 @@ with a new Kubernetes release fails the next audit instead of misleading you aft `intro`, `setup` (bash, as a task's), `steps` and `outro` - and, like a task, `k8s` or `image` if it needs a host of its own kind (lesson 4.4 runs on v1.36 to upgrade it). Each step comes from `step(title, text, check, hint, show)` in `sim/bank.py`: `check` runs like a task's check, and -`show` is what the learner would type. `./ksim lesson audit` seeds each lesson on its own host and, -step by step, requires the check to fail, runs `show` there as `candidate`, and requires the check -to pass — so a lesson can always be done exactly as it shows, and no step is done before you do it. -Work on another node is shown as `ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain -`ssh NODE` would read the rest of the commands as its own input. **As audited on 2026-10-19** -(v1.37, Ubuntu hosts): CKA's 27 lessons, 135 of 135 steps, and CKAD 2.1–2.4, 24 of 24 steps, -proven. +`show` is how you would do the step in the exam, as parts. `cmd(text)` is what you type as it +stands. `hand(text, how, run)` is what you do by hand: what you type to begin (`vim role.yaml`, +`k edit cm app-config`, `ssh NODE` and `sudo -i`, `echo NAME > file`), what to do there, and a script +that does the same unattended. `./ksim lesson audit` seeds each lesson on its own host and, step by +step, requires the check to fail, runs the commands (and each hand part's script) there as +`candidate`, and requires the check to pass, so a lesson can always be done as it shows, and no +step is done before you do it. `./ksim lesson lint` reads what every step has you type and refuses +scripts: no `sed`, `awk`, `yq`, heredocs, loops, `kubectl wait`, `--overrides`, JSON patches or +`jsonpath` into a file, and `k` rather than `kubectl`. In a script, a command on another node is +`ssh NODE sudo bash <<'EOF'` or `ssh -n NODE 'COMMAND'`: a plain `ssh NODE` would read the rest of +the script as its own input. **As audited on 2026-10-26** (v1.37, Ubuntu hosts): CKA's 27 lessons, +done the exam way, 135 of 135 steps proven, and none has you type a script. CKAD 2.1–2.4 (24 +steps, audited on 2026-10-19) still show their commands as scripts. ## What is different from the real exam diff --git a/cka/lessons.py b/cka/lessons.py index 2ab35ef..bccb14a 100644 --- a/cka/lessons.py +++ b/cka/lessons.py @@ -10,11 +10,14 @@ Lessons write nothing to the history or the scores - only a "lesson done" mark o ./ksim lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after A lesson's host is named after its id - lesson 9101 runs on `cka9101` - which never clashes with a -task's. Its files go under {WS}/lesson-/. A step's `show` commands are what the learner -types; the audit runs exactly them, as `candidate` on the host, so what a lesson shows always -works. Work on another node is shown as `ssh NODE sudo bash <<'EOF' ... EOF`: the same commands -you would type after `ssh NODE` and `sudo -i`, in a form that runs unattended. A single command -there is `ssh -n NODE 'COMMAND'`: without -n, ssh would read the rest of the commands as its input. +task's. Its files go under {WS}/lesson-/. A step's `show` is how you would do it in the +exam: cmd() for what you type as it stands, hand() for what you do by hand - `vim` a generated +skeleton or a manifest pasted from the docs, `kubectl edit`, `ssh NODE` and `sudo -i`, a value you +read and `echo` - each with a script that does the same unattended. The audit runs the commands +as `candidate` on the host and, for each hand() part, its script; `./ksim lesson lint` keeps +scripts (sed, awk, yq, heredocs, loops, JSON patches) out of what you type. In a script, a command +on another node is `ssh NODE sudo bash <<'EOF' ... EOF` or `ssh -n NODE 'COMMAND'`: without -n, +ssh would read the rest of the script as its input. One file per domain, as with the task banks. Besides the curriculum's competencies, the lessons cover what a full CKA course teaches under each of them (2026-10-19: the topics of KubeDojo's CKA diff --git a/cka/lessons_arch.py b/cka/lessons_arch.py index f6a0a60..2835eef 100644 --- a/cka/lessons_arch.py +++ b/cka/lessons_arch.py @@ -1,7 +1,7 @@ """CKA lessons of the Cluster Architecture, Installation and Configuration domain (4.1-4.8). See cka/lessons.py for what a lesson is.""" from cka.common import backup_operator -from sim.bank import CAN, NS, PY, step +from sim.bank import CAN, NS, PY, step, cmd, hand _ETCDCTL = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt " "--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key") @@ -38,23 +38,20 @@ S=system:serviceaccount:team:deployer can --as=$S -- create deployments.apps -n team && can --as=$S -- list pods -n team && cannot --as=$S -- delete pods -n team \ && cannot --as=$S -- create deployments.apps -n default""", hint=r"""`kubectl create role deploy-edit --verb=... --resource=deployments.apps` - a Role with two resources and different verbs needs two rules, so write YAML or create it and `kubectl edit`. `kubectl create rolebinding NAME --role=ROLE --serviceaccount=team:deployer`.""", - show=r"""kubectl -n team create serviceaccount deployer -kubectl -n team apply -f - <<'EOF' -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: deploy-edit -rules: -- apiGroups: [apps] - resources: [deployments] - verbs: [get, list, watch, create, update, patch, delete] + show=[cmd(r"""k -n team create sa deployer +k -n team create role deploy-edit --verb=get,list,watch,create,update,patch,delete --resource=deployments.apps --dry-run=client -o yaml > role.yaml"""), + hand(r"""vim role.yaml""", + r"""`kubectl create role` gives every resource the same verbs: add the second rule, for Pods, at the end of `rules:`: +``` - apiGroups: [""] resources: [pods] verbs: [get, list] -EOF -kubectl -n team create rolebinding deployer --role=deploy-edit --serviceaccount=team:deployer -kubectl auth can-i create deployments -n team --as=system:serviceaccount:team:deployer -kubectl auth can-i delete pods -n team --as=system:serviceaccount:team:deployer"""), +```""", + run=r"""yq -i '.rules += [{"apiGroups": [""], "resources": ["pods"], "verbs": ["get", "list"]}]' role.yaml"""), + cmd(r"""k apply -f role.yaml +k -n team create rolebinding deployer --role=deploy-edit --serviceaccount=team:deployer +k auth can-i create deploy -n team --as=system:serviceaccount:team:deployer +k auth can-i delete po -n team --as=system:serviceaccount:team:deployer""", r"""`yes`, then `no`.""")]), step("One ClusterRole, one Namespace", r"""Create ClusterRole `pod-reader` (`get`, `list`, `watch` on pods). Give it to the user **`jane`** in Namespace `team` only - with a **RoleBinding** `jane-reads-pods`, not a ClusterRoleBinding. @@ -64,10 +61,10 @@ eq "$(j clusterrole pod-reader '{.rules[0].verbs}')" '["get","list","watch"]' || eq "$(j -n team rolebinding jane-reads-pods '{.roleRef.kind}/{.roleRef.name}')" ClusterRole/pod-reader || exit 1 can --as=jane -- list pods -n team && cannot --as=jane -- list pods -n default && cannot --as=jane -- list pods -A""", hint=r"""`kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods`, then `kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane`. Users exist only in certificates and tokens: nothing to create for `jane`.""", - show=r"""kubectl create clusterrole pod-reader --verb=get,list,watch --resource=pods -kubectl -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane -kubectl auth can-i list pods -n team --as=jane -kubectl auth can-i list pods -n default --as=jane"""), + show=[cmd(r"""k create clusterrole pod-reader --verb=get,list,watch --resource=pods +k -n team create rolebinding jane-reads-pods --clusterrole=pod-reader --user=jane +k auth can-i list po -n team --as=jane +k auth can-i list po -n default --as=jane""", r"""`yes` in `team`, `no` elsewhere.""")]), step("A token for the ServiceAccount", r"""Issue a token for `deployer` that is valid for **one hour**, and save it in `{WS}/lesson-4.1/deployer.token`. Then use it - as the **only** credential: your kubeconfig carries the admin's client certificate, and the API server checks a certificate before a token, so `kubectl --token=...` alone would still be you. With no kubeconfig, just the server, the CA and the token, `kubectl auth whoami` says `system:serviceaccount:team:deployer`, `get deployments` works in `team` and `get secrets` is forbidden.""", check=r""" @@ -79,44 +76,41 @@ import base64, json, sys s = sys.stdin.read().strip(); c = json.loads(base64.b64decode(s + "=" * (-len(s) % 4))) sys.exit(0 if c["exp"] - c["iat"] == 3600 else 1)'""", hint=r"""`kubectl -n team create token deployer --duration=1h`. Then `kubectl --kubeconfig=/dev/null --server=URL --certificate-authority=/etc/kubernetes/pki/ca.crt --token=TOKEN ...`; the URL is in your kubeconfig. The token is a JWT: its middle part, base64-decoded, holds `sub`, `iat` and `exp`.""", - show=r"""kubectl -n team create token deployer --duration=1h > {WS}/lesson-4.1/deployer.token -server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}') -as_deployer() { kubectl --kubeconfig=/dev/null --server=$server --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$(cat {WS}/lesson-4.1/deployer.token) "$@"; } -as_deployer auth whoami -as_deployer -n team get deployments -as_deployer -n team get secrets || true"""), + show=[cmd(r"""k -n team create token deployer --duration=1h > {WS}/lesson-4.1/deployer.token +k config view --minify | grep server"""), + hand(r"""k --kubeconfig=/dev/null --server=SERVER --certificate-authority=/etc/kubernetes/pki/ca.crt --token=TOKEN auth whoami""", + r"""`SERVER`: the address above; `TOKEN`: the file's content (`cat` it, copy it). With `-n team get deploy` instead of `auth whoami` it works; `get secrets` is forbidden.""", + run=r"""server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}') +kubectl --kubeconfig=/dev/null --server=$server --certificate-authority=/etc/kubernetes/pki/ca.crt --token=$(cat {WS}/lesson-4.1/deployer.token) auth whoami""")]), step("A Pod without a token", r"""Most Pods never talk to the API server, and do not need a token in their filesystem. Create ServiceAccount `quiet` in `team` with `automountServiceAccountToken: false`, and run Pod `notoken` (`busybox:1.36`, `sleep 86400`) as that ServiceAccount. Inside it, `/var/run/secrets/kubernetes.io/serviceaccount` does not exist.""", check=r""" eq "$(j -n team sa quiet '{.automountServiceAccountToken}')" false && eq "$(j -n team pod notoken '{.spec.serviceAccountName} {.status.phase}')" "quiet Running" || exit 1 ! $K -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount >/dev/null 2>&1""", hint=r"""The field sits at the top level of the ServiceAccount (and can be set on a Pod's `spec` too, which wins). The Pod names its ServiceAccount with `spec.serviceAccountName`.""", - show=r"""kubectl -n team apply -f - <<'EOF' -apiVersion: v1 -kind: ServiceAccount -metadata: - name: quiet + show=[cmd(r"""k -n team create sa quiet --dry-run=client -o yaml > quiet.yaml"""), + hand(r"""vim quiet.yaml""", + r"""Add, at the top level (no indent): +``` automountServiceAccountToken: false ---- -apiVersion: v1 -kind: Pod -metadata: - name: notoken -spec: +```""", + run=r"""yq -i '.automountServiceAccountToken = false' quiet.yaml"""), + cmd(r"""k apply -f quiet.yaml +k -n team run notoken --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > notoken.yaml"""), + hand(r"""vim notoken.yaml""", + r"""Add to the Pod's `spec:`, above `containers:`: +``` serviceAccountName: quiet - containers: - - name: app - image: busybox:1.36 - command: [sleep, '86400'] -EOF -kubectl -n team wait --for=condition=Ready pod/notoken --timeout=120s -kubectl -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount || echo 'no token mounted'"""), +```""", + run=r"""yq -i '.spec.serviceAccountName = "quiet"' notoken.yaml"""), + cmd(r"""k -n team apply -f notoken.yaml +k -n team exec notoken -- ls /var/run/secrets/kubernetes.io/serviceaccount""", r"""Once it runs: "No such file or directory" - no token.""")]), step("Everything one subject may do", r"""Write the complete list of what `deployer` may do in `team` to `{WS}/lesson-4.1/deployer-can.txt` - `kubectl auth can-i` prints it with `--list`. It includes what every authenticated subject gets from the built-in ClusterRoles, such as creating `selfsubjectreviews`.""", check=r"""f="$WS/lesson-4.1/deployer-can.txt"; grep -q '^deployments.apps' "$f" 2>/dev/null && grep -q '^pods ' "$f" && grep -qi 'verbs' "$f" """, hint=r"""`kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer`.""", - show=r"""kubectl auth can-i --list -n team --as=system:serviceaccount:team:deployer > {WS}/lesson-4.1/deployer-can.txt -cat {WS}/lesson-4.1/deployer-can.txt"""), + show=[cmd(r"""k auth can-i --list -n team --as=system:serviceaccount:team:deployer > {WS}/lesson-4.1/deployer-can.txt +cat {WS}/lesson-4.1/deployer-can.txt""")]), ], "outro": r"""You gave a ServiceAccount a Role, reused a ClusterRole in one Namespace, issued and used a token, ran a Pod without one, and listed a subject's permissions. @@ -158,10 +152,18 @@ nx "$W1" 'rm -f /etc/modules-load.d/k8s.conf /etc/sysctl.d/k8s.conf' /dev/null | tr -d " " | grep . | sort | tr "\n" " "' /etc/modules-load.d/k8s.conf -cat /etc/modules-load.d/k8s.conf -EOF"""), +EOF""")]), step("Kernel parameters, for good", r"""On `{W1}`, put the three parameters Kubernetes needs in `/etc/sysctl.d/k8s.conf` - `net.bridge.bridge-nf-call-iptables = 1`, `net.bridge.bridge-nf-call-ip6tables = 1`, `net.ipv4.ip_forward = 1` - and apply every sysctl file now with `sysctl --system`.""", check=r""" @@ -169,15 +171,29 @@ f=$(nx "$W1" 'cat /etc/sysctl.d/k8s.conf 2>/dev/null' /etc/sysctl.d/k8s.conf <<'CONF' net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 CONF -sysctl --system 2>/dev/null | grep -A3 k8s.conf -sysctl net.ipv4.ip_forward -EOF"""), +sysctl --system >/dev/null 2>&1 +EOF""")]), step("Swap and the cgroup driver", r"""Two more things kubeadm's preflight checks. Write to `{WS}/lesson-4.2/node.txt`, in this format: @@ -196,17 +212,30 @@ sc=$(nx "$W1" 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml | want=$(printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s' "$swap" "${fso% *}" "${fso#* }" "$sc") eq "$(tr -d ' \r' < "$WS/lesson-4.2/node.txt" 2>/dev/null | grep .)" "$want" """, hint=r"""`cat /proc/swaps` (or `swapon --show`): any line below the header is swap. `grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml` - a field that is not there has its default (`failSwapOn` is `true`). `grep SystemdCgroup /etc/containerd/config.toml`; no such line means `false`.""", - show=r"""ssh {W1} sudo bash <<'EOF' + show=[hand(r"""ssh {W1} +sudo -i cat /proc/swaps -grep -E 'failSwapOn|cgroupDriver' /var/lib/kubelet/config.yaml -grep -n SystemdCgroup /etc/containerd/config.toml || echo 'SystemdCgroup not set' -EOF -swap=$(ssh -n {W1} '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off') +grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml +grep SystemdCgroup /etc/containerd/config.toml""", + r"""- swap: `on` if `/proc/swaps` lists a device below its header line, `off` if it lists none; +- `failSwapOn`: `true` when the file does not set it - that is its default; +- `cgroupDriver`, and `SystemdCgroup` (`false` when it is not set). + +Then `exit` twice.""", + run=r"""ssh -n {W1} 'sudo cat /proc/swaps; sudo grep -e failSwapOn -e cgroupDriver /var/lib/kubelet/config.yaml; sudo grep SystemdCgroup /etc/containerd/config.toml' """), + hand(r"""vim {WS}/lesson-4.2/node.txt""", + r"""The four lines, each `VALUE` what you found (`on` or `off`, `true` or `false`, `systemd` or `cgroupfs`): +``` +swap=VALUE +failSwapOn=VALUE +kubelet-cgroup=VALUE +containerd-systemd-cgroup=VALUE +```""", + run=r"""swap=$(ssh -n {W1} '[ "$(grep -c . /proc/swaps)" -gt 1 ] && echo on || echo off') fso=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.failSwapOn) cg=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver) sc=$(ssh -n {W1} 'grep -o "SystemdCgroup *= *[a-z]*" /etc/containerd/config.toml | head -1 | tr -d " " | cut -d= -f2') -printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s\n' "$swap" "$fso" "$cg" "${sc:-false}" > {WS}/lesson-4.2/node.txt -cat {WS}/lesson-4.2/node.txt"""), +printf 'swap=%s\nfailSwapOn=%s\nkubelet-cgroup=%s\ncontainerd-systemd-cgroup=%s\n' "$swap" "$fso" "$cg" "${sc:-false}" > {WS}/lesson-4.2/node.txt""")]), step("The packages, as kubeadm wants them", r"""On an exam node, kubeadm, kubelet and kubectl come from the repository of **one** minor at `pkgs.k8s.io`, and are **held** so that an `apt upgrade` never moves them by accident. @@ -215,9 +244,10 @@ Write to `{WS}/lesson-4.2/packages.txt`: the line of `{W1}`'s Kubernetes reposit want=$(nx "$W1" 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' /dev/null | tr -d '\r')" "$want" """, hint=r"""`cat /etc/apt/sources.list.d/kubernetes.list` and `apt-mark showhold`. `apt-cache madison kubeadm` lists the versions that repository offers; `dpkg -l | grep kube` what is installed.""", - show=r"""ssh -n {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt -cat {WS}/lesson-4.2/packages.txt -ssh -n {W1} 'dpkg -l | grep -E "kube(adm|let|ctl)"'"""), + show=[hand(r"""ssh {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt""", + r"""The repository line, then the three held packages.""", + run=r"""ssh -n {W1} 'cat /etc/apt/sources.list.d/kubernetes.list; apt-mark showhold' > {WS}/lesson-4.2/packages.txt"""), + cmd(r"""cat {WS}/lesson-4.2/packages.txt""")]), ], "outro": r"""You set up modules and sysctls that survive a reboot, checked swap and the cgroup drivers, and read how Kubernetes is installed on a node. @@ -259,9 +289,10 @@ nx "$W1" 'rm -f /etc/kubernetes/manifests/node-web.yaml' /dev/null | tr -d '\r\n')" "$want" """, hint=r"""`kubectl -n kube-system get cm kubeadm-config -o yaml` - look under `networking:`. `kubeadm config print init-defaults` shows every field such a file can have.""", - show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | grep -A4 networking -kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}' > {WS}/lesson-4.3/networks.txt -cat {WS}/lesson-4.3/networks.txt"""), + show=[cmd(r"""k -n kube-system get cm kubeadm-config -o yaml | grep -i subnet"""), + hand(r"""echo 'PODSUBNET SERVICESUBNET' > {WS}/lesson-4.3/networks.txt""", + r"""The two values, `podSubnet` first.""", + run=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/podSubnet:/ {p=$2} /serviceSubnet:/ {s=$2} END {print p, s}' > {WS}/lesson-4.3/networks.txt""")]), step("A join command for a new worker", r"""Create a new bootstrap token that expires in **2 hours**, and save the complete `kubeadm join` command for a worker in `{WS}/lesson-4.3/join.sh`, on one line. Run it on the control-plane node.""", check=r""" @@ -270,9 +301,9 @@ id=$(grep -o -- '--token [a-z0-9]*\.[a-z0-9]*' "$f" | awk '{print $2}' | cut -d. exp=$(j -n kube-system secret "bootstrap-token-$id" '{.data.expiration}' | base64 -d); [ -n "$exp" ] || exit 1 left=$(( $(date -d "$exp" +%s) - $(date +%s) )); [ "$left" -gt 6600 ] && [ "$left" -le 7300 ]""", hint=r"""`sudo kubeadm token create --ttl 2h --print-join-command`. `kubeadm token list` shows the tokens and when they expire; each is a Secret `bootstrap-token-ID` in `kube-system`.""", - show=r"""sudo kubeadm token create --ttl 2h --print-join-command > {WS}/lesson-4.3/join.sh + show=[cmd(r"""sudo kubeadm token create --ttl 2h --print-join-command > {WS}/lesson-4.3/join.sh cat {WS}/lesson-4.3/join.sh -sudo kubeadm token list"""), +sudo kubeadm token list""")]), step("Renew the API server's certificate", r"""Check the certificates' expiry dates, then **renew** the API server's serving certificate - only that one - and make the API server use it. A running process keeps the certificate it read at start, so the static Pod has to be restarted: move its manifest out of `/etc/kubernetes/manifests`, wait until the container is gone, and move it back. @@ -285,24 +316,32 @@ file=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -seri served=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial' /dev/null 2>&1 && break; sleep 3; done -sudo kubeadm certs check-expiration | grep '^apiserver '"""), +sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml ~/"""), + hand(r"""sudo crictl ps --name kube-apiserver""", + r"""Again until it lists nothing: the old API server is gone.""", + run=r"""for i in $(seq 1 30); do [ -z "$(sudo crictl ps --name kube-apiserver -q)" ] && break; sleep 2; done"""), + cmd(r"""sudo mv ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""), + hand(r"""k get no""", + r"""Again until it answers: the API server is back with the new certificate.""", + run=r"""for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""), + cmd(r"""sudo kubeadm certs check-expiration | grep apiserver""")]), step("A static Pod on the worker", r"""Every kubelet runs static Pods, not only the control plane's. On `{W1}`, find the kubelet's `staticPodPath` and put a manifest there for Pod `node-web` (`nginx:1.27`). The API server shows it as a **mirror Pod** named after the node: `node-web-{W1}`. Deleting the mirror Pod with kubectl does nothing lasting - the kubelet puts it back; only removing the file does.""", check=r"""eq "$(j -n default pod "node-web-$W1" '{.status.phase} {.metadata.annotations.kubernetes\.io/config\.source}')" "Running file" """, hint=r"""`grep staticPodPath /var/lib/kubelet/config.yaml` on the node. `kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml` gives you the manifest; write it with `sudo tee`.""", - show=r"""kubectl run node-web --image=nginx:1.27 --dry-run=client -o yaml > {WS}/lesson-4.3/node-web.yaml -ssh -n {W1} 'grep staticPodPath /var/lib/kubelet/config.yaml' -ssh {W1} 'sudo tee /etc/kubernetes/manifests/node-web.yaml >/dev/null' < {WS}/lesson-4.3/node-web.yaml -sleep 15 -kubectl get pods -o wide"""), + show=[cmd(r"""k run node-web --image=nginx:1.27 --dry-run=client -o yaml > node-web.yaml"""), + hand(r"""scp node-web.yaml {W1}: +ssh {W1} +sudo -i +grep staticPodPath /var/lib/kubelet/config.yaml +cp ~candidate/node-web.yaml /etc/kubernetes/manifests/""", + r"""The manifest goes into the `staticPodPath` directory; then `exit` twice.""", + run=r"""ssh {W1} 'sudo tee /etc/kubernetes/manifests/node-web.yaml >/dev/null' < node-web.yaml"""), + cmd(r"""k get po -o wide""", r"""Again in a few seconds: `node-web-{W1}`, the mirror Pod.""")]), ], "outro": r"""You read the cluster's kubeadm configuration, made a join command, renewed a certificate and made the API server load it, and ran a static Pod on a worker. @@ -358,15 +397,14 @@ etcd only accepts clients with a certificate: give `etcdctl` the endpoint and th nx "$CP" 'etcdutl snapshot status /opt/backup/etcd.db >/dev/null 2>&1' /dev/null \ && grep -qi 'total key' "$WS/lesson-4.4/snapshot.txt" """, hint=r"""`etcdctl --endpoints=https://127.0.0.1:2379 --cacert=.../ca.crt --cert=.../server.crt --key=.../server.key snapshot save FILE`, as root. Since etcd 3.6, `status` and `restore` are `etcdutl` only.""", - show=r"""sudo grep -E -- '--(listen-client-urls|cert-file|key-file|trusted-ca-file)' /etc/kubernetes/manifests/etcd.yaml -sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \ - --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup/etcd.db -sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/snapshot.txt"""), + show=[cmd(r"""sudo grep -e listen-client-urls -e cert-file -e key-file -e trusted-ca-file /etc/kubernetes/manifests/etcd.yaml""", r"""The endpoint, and the three files etcdctl needs: `--trusted-ca-file` is its `--cacert`, `--cert-file` its `--cert`, `--key-file` its `--key`."""), + cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/backup/etcd.db +sudo etcdutl snapshot status /opt/backup/etcd.db -w table | tee {WS}/lesson-4.4/snapshot.txt""")]), step("Lose something", r"""Now the mistake a backup is for: delete ConfigMap `marker` in `life`.""", check=r"""[ -z "$(j -n life cm marker '{.metadata.name}')" ] && nx "$CP" 'test -s /opt/backup/etcd.db' /dev/null 2>&1 && break; sleep 3; done -kubectl -n life get configmap marker -o yaml | grep state"""), +sudo mv ~/etcd.yaml /etc/kubernetes/manifests/"""), + hand(r"""k -n life get cm marker""", + r"""Again until it answers - a minute or two: etcd, then the API server come back, and `marker` with them.""", + run=r"""for i in $(seq 1 60); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done""")]), step("Plan the upgrade", r"""On the control-plane node, point the Kubernetes repository at **v1.37**, install the **newest 1.37 kubeadm** the repository offers (unhold it first, hold it again after), and save the output of `kubeadm upgrade plan` in `{WS}/lesson-4.4/plan.txt`. @@ -392,13 +433,19 @@ The plan shows two choices: the newest 1.36 patch, and the 1.37 release your kub t=$(newest); [ -n "$t" ] || exit 1 eq "$(ver "$CP" kubeadm)" "$t" && held "$CP" && grep -q "kubeadm upgrade apply v${t%%-*}" "$WS/lesson-4.4/plan.txt" 2>/dev/null""", hint=r"""`sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list`, `sudo apt-get update`, `apt-cache madison kubeadm` (newest first), `sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.37.X-1.1 && sudo apt-mark hold kubeadm`.""", - show=r"""sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list -sudo apt-get update -qq -apt-cache madison kubeadm | head -3 -v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}') -sudo apt-mark unhold kubeadm && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$v && sudo apt-mark hold kubeadm -kubeadm version -o short -sudo kubeadm upgrade plan | tee {WS}/lesson-4.4/plan.txt | tail -12"""), + show=[hand(r"""sudo vim /etc/apt/sources.list.d/kubernetes.list""", + r"""In the URL, change `v1.36` to `v1.37`.""", + run=r"""sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list"""), + cmd(r"""sudo apt-get update +apt-cache madison kubeadm""", r"""The newest `1.37.x-1.1` (at the top) is what to install. The docs page *Upgrading kubeadm clusters* has the commands that follow."""), + hand(r"""sudo apt-mark unhold kubeadm +sudo apt-get install -y kubeadm=VERSION +sudo apt-mark hold kubeadm""", + r"""`VERSION`: that one, for example `1.37.1-1.1`.""", + run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}') +sudo apt-mark unhold kubeadm && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubeadm=$v && sudo apt-mark hold kubeadm"""), + cmd(r"""kubeadm version +sudo kubeadm upgrade plan | tee {WS}/lesson-4.4/plan.txt""")]), step("Upgrade the control plane", r"""Apply the upgrade to that 1.37 release (`kubeadm upgrade apply`, a few minutes - it upgrades etcd, the static Pods, CoreDNS and kube-proxy). Then drain the node, upgrade `kubelet` and `kubectl` to the same version, reload systemd, restart the kubelet, and uncordon. @@ -409,13 +456,23 @@ eq "$($K version -o json 2>/dev/null | python3 -c 'import json,sys; print(json.l eq "$(j node "$CP" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$CP")" "$t $t $t " && held "$CP" \ && [ "$(j node "$CP" '{.spec.unschedulable}')" != true ]""", hint=r"""`sudo kubeadm upgrade apply v1.37.X -y`; `kubectl drain {CP} --ignore-daemonsets`; `sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=... kubectl=... && sudo apt-mark hold kubelet kubectl`; `sudo systemctl daemon-reload && sudo systemctl restart kubelet`; `kubectl uncordon {CP}`.""", - show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}') -sudo kubeadm upgrade apply v${v%%-*} -y | tail -3 -kubectl drain {CP} --ignore-daemonsets --delete-emptydir-data -sudo apt-mark unhold kubelet kubectl && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$v kubectl=$v && sudo apt-mark hold kubelet kubectl -sudo systemctl daemon-reload && sudo systemctl restart kubelet -for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done -kubectl get nodes"""), + show=[hand(r"""sudo kubeadm upgrade apply vVERSION""", + r"""`vVERSION`: the one the plan names, for example `v1.37.1`. Answer `y`; it takes a few minutes.""", + run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}') +sudo kubeadm upgrade apply v${v%%-*} -y | tail -3"""), + cmd(r"""k drain {CP} --ignore-daemonsets --delete-emptydir-data"""), + hand(r"""sudo apt-mark unhold kubelet kubectl +sudo apt-get install -y kubelet=VERSION kubectl=VERSION +sudo apt-mark hold kubelet kubectl""", + r"""`VERSION` as for kubeadm, for example `1.37.1-1.1`.""", + run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}') +sudo apt-mark unhold kubelet kubectl && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$v kubectl=$v && sudo apt-mark hold kubelet kubectl"""), + cmd(r"""sudo systemctl daemon-reload +sudo systemctl restart kubelet"""), + hand(r"""k uncordon {CP}""", + r"""Again if it fails at first: the kubelet takes a moment to come back.""", + run=r"""for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done"""), + cmd(r"""k get no""")]), step("Upgrade the worker", r"""Now `{W1}`: drain it from the control-plane node; on `{W1}`, the repository line, kubeadm, **`kubeadm upgrade node`** (a worker has no control-plane Pods to replace - it only refreshes the kubelet's configuration), kubelet and kubectl, daemon-reload, restart; then uncordon it. @@ -425,8 +482,20 @@ t=$(newest); v="v${t%%-*}" eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" "$v" && eq "$(pkgs "$W1")" "$t $t $t " && held "$W1" \ && [ "$(j node "$W1" '{.spec.unschedulable}')" != true ] && eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""", hint=r"""The same packages, from the same repository line, as on the control plane - only `kubeadm upgrade node` instead of `apply`. Drain and uncordon are kubectl commands: run them on the control-plane node.""", - show=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}') -kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data + show=[cmd(r"""k drain {W1} --ignore-daemonsets --delete-emptydir-data"""), + hand(r"""ssh {W1} +sudo -i +vim /etc/apt/sources.list.d/kubernetes.list +apt-get update +apt-mark unhold kubeadm kubelet kubectl +apt-get install -y kubeadm=VERSION +kubeadm upgrade node +apt-get install -y kubelet=VERSION kubectl=VERSION +apt-mark hold kubeadm kubelet kubectl +systemctl daemon-reload +systemctl restart kubelet""", + r"""In the repository line `v1.36` becomes `v1.37`; `VERSION` as on the control-plane node. Then `exit` twice.""", + run=r"""v=$(apt-cache madison kubeadm | awk '$3 ~ /^1\.37\./ {print $3; exit}') ssh {W1} sudo bash -s $v <<'EOF' sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list apt-get update -qq @@ -436,9 +505,9 @@ kubeadm upgrade node | tail -2 DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet=$1 kubectl=$1 apt-mark hold kubeadm kubelet kubectl systemctl daemon-reload && systemctl restart kubelet -EOF -kubectl uncordon {W1} -kubectl get nodes"""), +EOF"""), + cmd(r"""k uncordon {W1} +k get no""")]), ], "outro": r"""You backed etcd up, restored it after a loss, and took a whole cluster from v1.36 to v1.37: control plane first, then the worker, with the packages held at every step. @@ -480,9 +549,11 @@ $K -n kube-system delete secret kubeadm-certs --ignore-not-found >/dev/null 2>&1 want=$($K -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}') [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-4.5/endpoint.txt" 2>/dev/null)" "$want" """, hint=r"""`controlPlaneEndpoint:` in the `ClusterConfiguration` of ConfigMap `kubeadm-config`.""", - show=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}' > {WS}/lesson-4.5/endpoint.txt -cat {WS}/lesson-4.5/endpoint.txt -sudo grep server: /etc/kubernetes/kubelet.conf"""), + show=[cmd(r"""k -n kube-system get cm kubeadm-config -o yaml | grep controlPlaneEndpoint"""), + hand(r"""echo ENDPOINT > {WS}/lesson-4.5/endpoint.txt""", + r"""`ENDPOINT`: that value, `host:port`.""", + run=r"""kubectl -n kube-system get cm kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' | awk '/controlPlaneEndpoint:/ {print $2}' > {WS}/lesson-4.5/endpoint.txt"""), + cmd(r"""sudo grep server: /etc/kubernetes/kubelet.conf""")]), step("etcd's members and its leader", r"""Ask etcd itself. Write to `{WS}/lesson-4.5/etcd.txt`: @@ -498,13 +569,14 @@ import json, sys; m = json.load(sys.stdin) st = json.loads(sys.argv[1])[0]["Status"]; print(next(x["name"] for x in m["members"] if x["ID"] == st["leader"]))' "$out") eq "$(tr -d ' \r' < "$WS/lesson-4.5/etcd.txt" 2>/dev/null | grep .)" "$(printf 'members=%s\nleader=%s' "$n" "$lead")" """, hint=r"""`etcdctl member list -w table` and `etcdctl endpoint status -w table` (the IS LEADER column), with the endpoint and certificates as for a snapshot. `export ETCDCTL_ENDPOINTS=... ETCDCTL_CACERT=...` saves typing.""", - show=r"""sudo bash <<'EOF' -export ETCDCTL_API=3 ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \ - ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key -etcdctl member list -w table -etcdctl endpoint status -w table -EOF -printf 'members=1\nleader={CP}\n' > {WS}/lesson-4.5/etcd.txt"""), + show=[cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list -w table""", r"""For the next one, press ↑ and change the last words."""), + cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status -w table""", r"""`IS LEADER` says which member leads."""), + hand(r"""vim {WS}/lesson-4.5/etcd.txt""", + r"""``` +members=1 +leader={CP} +```""", + run=r"""printf 'members=1\nleader={CP}\n' > {WS}/lesson-4.5/etcd.txt""")]), step("Who leads the scheduler and the controller manager", r"""Only one kube-scheduler and one kube-controller-manager act at a time; the others wait for the **Lease** in `kube-system` to expire. Write the node of each holder to `{WS}/lesson-4.5/leaders.txt`: @@ -516,20 +588,28 @@ controller-manager= s=$(j -n kube-system lease kube-scheduler '{.spec.holderIdentity}'); c=$(j -n kube-system lease kube-controller-manager '{.spec.holderIdentity}') eq "$(tr -d ' \r' < "$WS/lesson-4.5/leaders.txt" 2>/dev/null | grep .)" "$(printf 'scheduler=%s\ncontroller-manager=%s' "${s%%_*}" "${c%%_*}")" """, hint=r"""`kubectl -n kube-system get lease kube-scheduler kube-controller-manager` - HOLDER is `NODE_UUID`.""", - show=r"""kubectl -n kube-system get lease kube-scheduler kube-controller-manager -s=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}') + show=[cmd(r"""k -n kube-system get lease kube-scheduler kube-controller-manager""", r"""`HOLDER` is the node's name, then `_` and an ID: keep only the node's name."""), + hand(r"""vim {WS}/lesson-4.5/leaders.txt""", + r"""``` +scheduler=NODE +controller-manager=NODE +```""", + run=r"""s=$(kubectl -n kube-system get lease kube-scheduler -o jsonpath='{.spec.holderIdentity}') c=$(kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.holderIdentity}') -printf 'scheduler=%s\ncontroller-manager=%s\n' "${s%%_*}" "${c%%_*}" > {WS}/lesson-4.5/leaders.txt"""), +printf 'scheduler=%s\ncontroller-manager=%s\n' "${s%%_*}" "${c%%_*}" > {WS}/lesson-4.5/leaders.txt""")]), step("What a second control-plane node needs", r"""Upload the control-plane certificates for a joining node, and save the complete `kubeadm join` command a new **control-plane** node would run - new token, CA hash, `--control-plane` and the certificate key - to `{WS}/lesson-4.5/join-cp.sh`, on one line. Do not run it anywhere.""", check=r""" f="$WS/lesson-4.5/join-cp.sh"; grep -q '^kubeadm join .*--token .*--discovery-token-ca-cert-hash sha256:.*--control-plane' "$f" 2>/dev/null || exit 1 key=$(grep -o -- '--certificate-key [0-9a-f]\{64\}' "$f" | awk '{print $2}'); [ -n "$key" ] && [ -n "$(j -n kube-system secret kubeadm-certs '{.metadata.name}')" ]""", hint=r"""`sudo kubeadm init phase upload-certs --upload-certs` prints the key as its last line; `sudo kubeadm token create --print-join-command --certificate-key KEY` adds `--control-plane --certificate-key` itself.""", - show=r"""key=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1) -sudo kubeadm token create --print-join-command --certificate-key $key > {WS}/lesson-4.5/join-cp.sh -cat {WS}/lesson-4.5/join-cp.sh -kubectl -n kube-system get secret kubeadm-certs"""), + show=[cmd(r"""sudo kubeadm init phase upload-certs --upload-certs""", r"""The last line is the certificate key."""), + hand(r"""sudo kubeadm token create --print-join-command --certificate-key KEY > {WS}/lesson-4.5/join-cp.sh""", + r"""`KEY`: that line.""", + run=r"""key=$(sudo kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -1) +sudo kubeadm token create --print-join-command --certificate-key $key > {WS}/lesson-4.5/join-cp.sh"""), + cmd(r"""cat {WS}/lesson-4.5/join-cp.sh +k -n kube-system get secret kubeadm-certs""")]), ], "outro": r"""You read the endpoint that makes HA possible, asked etcd for its members and leader, found the leaders of the scheduler and controller manager, and prepared a control-plane join. @@ -637,23 +717,23 @@ A chart is at `{WS}/lesson-4.6/chart` (values `replicaCount`, `image.tag`, `serv r"""Render the chart as release `web` with `replicaCount=2`, without installing anything, and save the manifests in `{WS}/lesson-4.6/rendered.yaml`. That is how you check what a set of values does.""", check=r"""f="$WS/lesson-4.6/rendered.yaml"; grep -q 'name: web-shop' "$f" 2>/dev/null && grep -q 'replicas: 2' "$f" && [ -z "$(j -n apps deploy web-shop '{.metadata.name}')" ]""", hint=r"""`helm template RELEASE CHART --set KEY=VALUE`.""", - show=r"""helm template web {WS}/lesson-4.6/chart --set replicaCount=2 > {WS}/lesson-4.6/rendered.yaml -grep -E 'kind:|replicas:|image:' {WS}/lesson-4.6/rendered.yaml"""), + show=[cmd(r"""helm template web {WS}/lesson-4.6/chart --set replicaCount=2 > {WS}/lesson-4.6/rendered.yaml +grep -e kind: -e replicas: -e image: {WS}/lesson-4.6/rendered.yaml""")]), step("Install a release", r"""Install the chart as release `web` in Namespace `apps` (create it), with `replicaCount=2`.""", check=r"""eq "$(helm -n apps status web -o json 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["info"]["status"], d["version"])')" "deployed 1" && eq "$(j -n apps deploy web-shop '{.status.readyReplicas}')" 2""", hint=r"""`helm install web CHART -n apps --create-namespace --set replicaCount=2`; `helm list -n apps`.""", - show=r"""helm install web {WS}/lesson-4.6/chart -n apps --create-namespace --set replicaCount=2 + show=[cmd(r"""helm install web {WS}/lesson-4.6/chart -n apps --create-namespace --set replicaCount=2 helm list -n apps -kubectl -n apps rollout status deploy web-shop --timeout=120s"""), +k -n apps rollout status deploy web-shop""")]), step("Upgrade it, keeping what you set", r"""Upgrade release `web` so that its Service becomes a `NodePort` - and keep `replicaCount=2`. A plain `helm upgrade --set service.type=NodePort` would quietly put the replica count back to the chart's default.""", check=r"""eq "$(j -n apps svc web-shop '{.spec.type}')" NodePort && eq "$(j -n apps deploy web-shop '{.spec.replicas}')" 2 \ && [ "$(helm -n apps history web -o json 2>/dev/null | python3 -c 'import json,sys; print(len(json.load(sys.stdin)))')" -ge 2 ]""", hint=r"""`--reuse-values` keeps the values of the last release and adds yours; `helm get values web -n apps` shows what is set.""", - show=r"""helm upgrade web {WS}/lesson-4.6/chart -n apps --reuse-values --set service.type=NodePort + show=[cmd(r"""helm upgrade web {WS}/lesson-4.6/chart -n apps --reuse-values --set service.type=NodePort helm get values web -n apps -kubectl -n apps get svc web-shop"""), +k -n apps get svc web-shop""")]), step("Roll the release back", r"""The NodePort was a mistake: roll release `web` back to **revision 1**. A rollback is itself a new revision.""", check=r""" @@ -662,16 +742,18 @@ helm -n apps history web -o json 2>/dev/null | python3 -c ' import json, sys; h = json.load(sys.stdin) sys.exit(0 if len(h) >= 3 and h[-1]["status"] == "deployed" and "Rollback to 1" in h[-1]["description"] else 1)'""", hint=r"""`helm rollback web 1 -n apps`, then `helm history web -n apps`.""", - show=r"""helm rollback web 1 -n apps + show=[cmd(r"""helm rollback web 1 -n apps helm history web -n apps -kubectl -n apps get svc web-shop"""), +k -n apps get svc web-shop""")]), step("An overlay for production", r"""Create a Kustomize overlay at `{WS}/lesson-4.6/kustomize/prod` on top of the base, which: puts everything in Namespace `prod`, prefixes names with `prod-`, runs **3** replicas, and uses image `nginx:1.27`. Apply it. Leave the base as it is.""", check=r"""eq "$(j -n prod deploy prod-api '{.spec.replicas} {.spec.template.spec.containers[0].image} {.status.readyReplicas}')" "3 nginx:1.27 3" \ && grep -q 'image: nginx:1.26' "$WS/lesson-4.6/kustomize/base/deployment.yaml" """, hint=r"""A `kustomization.yaml` with `resources: [../base]`, `namespace: prod`, `namePrefix: prod-`, `replicas: [{name: api, count: 3}]`, `images: [{name: nginx, newTag: "1.27"}]`. The Namespace itself must exist (or be one of the resources). `kubectl kustomize DIR` shows the result first.""", - show=r"""mkdir -p {WS}/lesson-4.6/kustomize/prod -cat > {WS}/lesson-4.6/kustomize/prod/kustomization.yaml <<'EOF' + show=[cmd(r"""mkdir -p {WS}/lesson-4.6/kustomize/prod"""), + hand(r"""vim {WS}/lesson-4.6/kustomize/prod/kustomization.yaml""", + r"""(the docs page *Declarative Management of Kubernetes Objects Using Kustomize* has each field) +``` resources: - ../base namespace: prod @@ -682,11 +764,23 @@ replicas: images: - name: nginx newTag: "1.27" -EOF -kubectl kustomize {WS}/lesson-4.6/kustomize/prod -kubectl create namespace prod -kubectl apply -k {WS}/lesson-4.6/kustomize/prod -kubectl -n prod rollout status deploy prod-api --timeout=120s"""), +```""", + run=r"""cat > {WS}/lesson-4.6/kustomize/prod/kustomization.yaml <<'EOF' +resources: +- ../base +namespace: prod +namePrefix: prod- +replicas: +- name: api + count: 3 +images: +- name: nginx + newTag: "1.27" +EOF"""), + cmd(r"""k kustomize {WS}/lesson-4.6/kustomize/prod +k create ns prod +k apply -k {WS}/lesson-4.6/kustomize/prod +k -n prod rollout status deploy prod-api""")]), ], "outro": r"""You rendered a chart, installed, upgraded and rolled back a release, and built a production overlay with Kustomize without touching its base. @@ -734,11 +828,18 @@ f=$(nx "$W1" 'ls /etc/cni/net.d/*.conf* 2>/dev/null | sort | head -1' /dev/null | grep .)" "$(printf 'config=%s\nplugins=%s' "$f" "$p")" """, hint=r"""`ls /etc/cni/net.d/` and `cat` the file: a `.conflist` has a `"plugins"` list, each with a `"type"` - the binary under `/opt/cni/bin` that is run.""", - show=r"""ssh -n {W1} 'ls /etc/cni/net.d/; cat /etc/cni/net.d/10-kindnet.conflist; ls /opt/cni/bin' -f=$(ssh -n {W1} 'ls /etc/cni/net.d/*.conf* | sort | head -1') + show=[hand(r"""ssh {W1} ls /etc/cni/net.d/ +ssh {W1} cat /etc/cni/net.d/10-kindnet.conflist""", + r"""The runtime uses the first file by name; the `type` of each entry of `plugins`, in order, is the chain.""", + run=r"""ssh -n {W1} 'ls /etc/cni/net.d/; cat /etc/cni/net.d/10-kindnet.conflist' """), + hand(r"""vim {WS}/lesson-4.7/cni.txt""", + r"""``` +config=/etc/cni/net.d/10-kindnet.conflist +plugins=TYPE,TYPE,... +```""", + run=r"""f=$(ssh -n {W1} 'ls /etc/cni/net.d/*.conf* | sort | head -1') p=$(ssh -n {W1} "cat $f" | jq -r '[.plugins[].type] | join(",")') -printf 'config=%s\nplugins=%s\n' "$f" "$p" > {WS}/lesson-4.7/cni.txt -cat {WS}/lesson-4.7/cni.txt"""), +printf 'config=%s\nplugins=%s\n' "$f" "$p" > {WS}/lesson-4.7/cni.txt""")]), step("The runtime, through CRI", r"""Write to `{WS}/lesson-4.7/cri.txt`: @@ -751,33 +852,63 @@ e=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" | python3 -c 'import json,sys r=$(j node "$W1" '{.status.nodeInfo.containerRuntimeVersion}') eq "$(tr -d ' \r' < "$WS/lesson-4.7/cri.txt" 2>/dev/null | grep .)" "$(printf 'endpoint=%s\nruntime=%s' "$e" "$r")" """, hint=r"""`containerRuntimeEndpoint` in the kubelet's configuration (`/var/lib/kubelet/config.yaml`, or `configz`); `kubectl get node -o wide` shows CONTAINER-RUNTIME.""", - show=r"""e=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.containerRuntimeEndpoint) + show=[hand(r"""ssh {W1} grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml""", + r"""The endpoint the kubelet talks CRI to.""", + run=r"""ssh -n {W1} grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml"""), + cmd(r"""k get no -o wide""", r"""`CONTAINER-RUNTIME` is the runtime and its version."""), + hand(r"""vim {WS}/lesson-4.7/cri.txt""", + r"""``` +endpoint=ENDPOINT +runtime=RUNTIME +```""", + run=r"""e=$(kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.containerRuntimeEndpoint) r=$(kubectl get node {W1} -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}') -printf 'endpoint=%s\nruntime=%s\n' "$e" "$r" > {WS}/lesson-4.7/cri.txt -cat {WS}/lesson-4.7/cri.txt"""), +printf 'endpoint=%s\nruntime=%s\n' "$e" "$r" > {WS}/lesson-4.7/cri.txt""")]), step("A Pod's container, from the runtime's side", r"""Pod `probe` in `plug` runs on `{W1}`. Ask the runtime about it there, with `crictl`: find its container, and write the container's **process ID on the node** to `{WS}/lesson-4.7/pid.txt`.""", check=r""" want=$(nx "$W1" 'crictl inspect $(crictl ps --name probe -q | head -1) 2>/dev/null' /dev/null)" "$want" """, hint=r"""`sudo crictl pods --name probe`, `sudo crictl ps --name probe`, `sudo crictl inspect ID` - the PID is under `info.pid`. `crictl` talks to the same socket as the kubelet.""", - show=r"""ssh -n {W1} 'sudo crictl pods --name probe; sudo crictl ps --name probe' -ssh -n {W1} 'sudo crictl inspect $(sudo crictl ps --name probe -q | head -1)' | jq .info.pid > {WS}/lesson-4.7/pid.txt -cat {WS}/lesson-4.7/pid.txt"""), + show=[hand(r"""ssh {W1} +sudo -i +crictl ps --name probe +crictl inspect CONTAINER | grep -m1 '"pid"' """, + r"""`CONTAINER`: the ID `crictl ps` printed. Then `exit` twice.""", + run=r"""ssh -n {W1} 'sudo crictl pods --name probe; sudo crictl ps --name probe' """), + hand(r"""echo PID > {WS}/lesson-4.7/pid.txt""", + r"""`PID`: that number.""", + run=r"""ssh -n {W1} 'sudo crictl inspect $(sudo crictl ps --name probe -q | head -1)' | jq .info.pid > {WS}/lesson-4.7/pid.txt""")]), step("A RuntimeClass", r"""Create RuntimeClass `plain` for containerd's handler `runc`, and run Pod `classy` in `plug` (`busybox:1.36`, `sleep 86400`) with that runtime class. A Pod that names a handler the node does not have stays in `ContainerCreating`.""", check=r"""eq "$(j runtimeclass plain '{.handler}')" runc && eq "$(j -n plug pod classy '{.spec.runtimeClassName} {.status.phase}')" "plain Running" """, hint=r"""A RuntimeClass is `node.k8s.io/v1` with a top-level `handler:` and no `spec`. The Pod: `spec.runtimeClassName: plain`.""", - show=r"""kubectl apply -f - <<'EOF' + show=[hand(r"""vim rc.yaml""", + r"""No `kubectl create` for a RuntimeClass: paste the example from the docs page *Runtime Class*, and make it: +``` apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: plain handler: runc -EOF -kubectl -n plug run classy --image=busybox:1.36 --overrides='{"spec":{"runtimeClassName":"plain"}}' -- sleep 86400 -kubectl -n plug wait --for=condition=Ready pod/classy --timeout=120s -kubectl get runtimeclass"""), +```""", + run=r"""cat > rc.yaml <<'EOF' +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: plain +handler: runc +EOF"""), + cmd(r"""k apply -f rc.yaml +k -n plug run classy --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > classy.yaml"""), + hand(r"""vim classy.yaml""", + r"""Add to the Pod's `spec:`, above `containers:`: +``` + runtimeClassName: plain +```""", + run=r"""yq -i '.spec.runtimeClassName = "plain"' classy.yaml"""), + cmd(r"""k -n plug apply -f classy.yaml +k -n plug get po classy""")]), step("Storage drivers", r"""Write to `{WS}/lesson-4.7/csi.txt`: @@ -792,11 +923,16 @@ n=$($K get csidrivers --no-headers 2>/dev/null | grep -c .) p=$($K get sc -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.provisioner}{end}') eq "$(tr -d ' \r' < "$WS/lesson-4.7/csi.txt" 2>/dev/null | grep .)" "$(printf 'csidrivers=%s\nprovisioner=%s' "$n" "$p")" """, hint=r"""`kubectl get csidrivers`, `kubectl get csinodes`, `kubectl get sc`.""", - show=r"""kubectl get csidrivers -kubectl get csinodes -kubectl get sc -printf 'csidrivers=%s\nprovisioner=%s\n' "$(kubectl get csidrivers --no-headers 2>/dev/null | grep -c .)" "$(kubectl get sc standard -o jsonpath='{.provisioner}')" > {WS}/lesson-4.7/csi.txt -cat {WS}/lesson-4.7/csi.txt"""), + show=[cmd(r"""k get csidrivers +k get csinodes +k get sc"""), + hand(r"""vim {WS}/lesson-4.7/csi.txt""", + r"""``` +csidrivers=N +provisioner=PROVISIONER +``` +`N`: how many CSIDrivers `get csidrivers` listed (`0` for "No resources found"); `PROVISIONER`: the `(default)` class's.""", + run=r"""printf 'csidrivers=%s\nprovisioner=%s\n' "$(kubectl get csidrivers --no-headers 2>/dev/null | grep -c .)" "$(kubectl get sc standard -o jsonpath='{.provisioner}')" > {WS}/lesson-4.7/csi.txt""")]), ], "outro": r"""You read the CNI chain on a node, the runtime the kubelet talks to, a container as the runtime sees it, ran a Pod with a RuntimeClass, and counted the storage drivers. @@ -850,7 +986,9 @@ py crd $c 's = o["spec"]["versions"][0]; p = s["schema"]["openAPIV3Schema"]["pro s["served"] and s["storage"] and "status" in s.get("subresources", {}) and "schedule" in p.get("required", []) and p["properties"]["retention"].get("default") == 7 and p["properties"]["retention"].get("minimum") == 1' \ && eq "$(j crd $c '{.status.conditions[?(@.type=="Established")].status}')" True""", hint=r"""The docs page "Extend the Kubernetes API with CustomResourceDefinitions" has the skeleton. `required:` sits beside `properties:`, at the level of `spec`'s schema; `default:` and `minimum:` inside `retention`. `subresources: {status: {}}` goes under the version.""", - show=r"""kubectl apply -f - <<'EOF' + show=[hand(r"""vim crd.yaml""", + r"""Paste the example from the docs page *Extend the Kubernetes API with CustomResourceDefinitions*, and make it: +``` apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: @@ -879,54 +1017,101 @@ spec: type: object properties: phase: {type: string} -EOF -kubectl wait --for condition=established crd/backups.data.example.com -kubectl api-resources --api-group=data.example.com"""), +```""", + run=r"""cat > crd.yaml <<'EOF' +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: backups.data.example.com +spec: + group: data.example.com + scope: Namespaced + names: {kind: Backup, plural: backups, singular: backup, shortNames: [bk]} + versions: + - name: v1 + served: true + storage: true + subresources: + status: {} + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + required: [schedule] + properties: + schedule: {type: string} + retention: {type: integer, minimum: 1, default: 7} + status: + type: object + properties: + phase: {type: string} +EOF"""), + cmd(r"""k apply -f crd.yaml +k api-resources --api-group=data.example.com""")]), step("Defaults and validation", r"""In Namespace `data`, create Backup `nightly` with `schedule: "0 2 * * *"` and no retention: the API server fills in `7`. Then try a Backup `broken` with `retention: 0` - it is refused. Save the refusal (kubectl writes it to **stderr**) in `{WS}/lesson-4.8/refused.txt`.""", check=r""" eq "$(j -n data bk nightly '{.spec.schedule} {.spec.retention}')" "0 2 * * * 7" && [ -z "$(j -n data bk broken '{.metadata.name}')" ] \ && grep -q 'retention' "$WS/lesson-4.8/refused.txt" 2>/dev/null && grep -qi 'greater than or equal to 1' "$WS/lesson-4.8/refused.txt" """, hint=r"""`apiVersion: data.example.com/v1`, `kind: Backup`. Redirect with `2>` - with `>` the file stays empty.""", - show=r"""kubectl -n data apply -f - <<'EOF' + show=[hand(r"""vim nightly.yaml""", + r"""``` apiVersion: data.example.com/v1 kind: Backup metadata: name: nightly spec: schedule: "0 2 * * *" -EOF -kubectl -n data get bk nightly -o jsonpath='{.spec}'; echo -printf 'apiVersion: data.example.com/v1\nkind: Backup\nmetadata: {name: broken}\nspec: {schedule: "@hourly", retention: 0}\n' | kubectl -n data apply -f - 2> {WS}/lesson-4.8/refused.txt -cat {WS}/lesson-4.8/refused.txt"""), +```""", + run=r"""cat > nightly.yaml <<'EOF' +apiVersion: data.example.com/v1 +kind: Backup +metadata: + name: nightly +spec: + schedule: "0 2 * * *" +EOF"""), + cmd(r"""k -n data apply -f nightly.yaml +k -n data get bk nightly -o yaml | grep -A2 '^spec:' """, r"""`retention: 7`, filled in by the API server."""), + hand(r"""cp nightly.yaml broken.yaml +vim broken.yaml""", + r"""Name it `broken`, and add `retention: 0` under `spec:`.""", + run=r"""printf 'apiVersion: data.example.com/v1\nkind: Backup\nmetadata: {name: broken}\nspec: {schedule: "@hourly", retention: 0}\n' > broken.yaml"""), + cmd(r"""k -n data apply -f broken.yaml 2> {WS}/lesson-4.8/refused.txt +cat {WS}/lesson-4.8/refused.txt""")]), step("What the schema says", r"""`kubectl explain` reads the CRD's schema like any built-in kind's. Save the output of `kubectl explain backup.spec` in `{WS}/lesson-4.8/explain.txt`.""", check=r"""f="$WS/lesson-4.8/explain.txt"; grep -q 'retention' "$f" 2>/dev/null && grep -q 'schedule' "$f" && grep -qi 'required' "$f" """, hint=r"""`kubectl explain backup.spec` - or `--recursive` for the whole tree.""", - show=r"""kubectl explain backup.spec | tee {WS}/lesson-4.8/explain.txt"""), + show=[cmd(r"""k explain backup.spec | tee {WS}/lesson-4.8/explain.txt""")]), step("Install the operator", r"""Install the operator from `{WS}/lesson-4.8/operator/` and wait until its controller runs. Read the manifests first: a Namespace, a ServiceAccount, a ClusterRole for `backups` and ConfigMaps, the binding, and the controller's Deployment.""", check=r"""eq "$(j -n backup-operator deploy controller '{.status.readyReplicas}')" 1 && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller""", hint=r"""`kubectl apply -f DIR` applies every file in it; `kubectl -n backup-operator rollout status deploy controller`.""", - show=r"""grep -E '^kind:|^ name:' {WS}/lesson-4.8/operator/operator.yaml -kubectl apply -f {WS}/lesson-4.8/operator/ -kubectl -n backup-operator rollout status deploy controller --timeout=120s"""), + show=[cmd(r"""grep -e ^kind: -e '^ name:' {WS}/lesson-4.8/operator/operator.yaml +k apply -f {WS}/lesson-4.8/operator/ +k -n backup-operator rollout status deploy controller""")]), step("Let it work", r"""Create Backup `weekly` in `data` with `schedule: "0 3 * * 0"`. Within a few seconds the controller has acted on it - and on `nightly`: each has `status.phase: Scheduled`, and a ConfigMap `backup-NAME` holds its schedule. `kubectl -n backup-operator logs deploy/controller` is where you would look if it did not.""", check=r""" eq "$(j -n data bk weekly '{.status.phase}')" Scheduled && eq "$(j -n data cm backup-weekly '{.data.schedule}')" "0 3 * * 0" && eq "$(j -n data bk nightly '{.status.phase}')" Scheduled""", hint=r"""A Backup like `nightly`. `kubectl -n data get bk` - add `additionalPrinterColumns` to the CRD and the phase would show there too.""", - show=r"""kubectl -n data apply -f - <<'EOF' + show=[hand(r"""cp nightly.yaml weekly.yaml +vim weekly.yaml""", + r"""Name it `weekly`, with `schedule: "0 3 * * 0"`.""", + run=r"""cat > weekly.yaml <<'EOF' apiVersion: data.example.com/v1 kind: Backup metadata: name: weekly spec: schedule: "0 3 * * 0" -EOF -sleep 10 -kubectl -n data get bk -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,PHASE:.status.phase -kubectl -n data get cm"""), +EOF"""), + cmd(r"""k -n data apply -f weekly.yaml +k -n data get bk -o yaml | grep phase +k -n data get cm""", r"""Again in a few seconds: both Backups `phase: Scheduled`, and a ConfigMap `backup-NAME` for each.""")]), ], "outro": r"""You defined a kind with validation and defaults, saw the schema refuse bad objects, installed an operator with its RBAC, and watched it act on your custom resources. diff --git a/cka/lessons_net.py b/cka/lessons_net.py index f529fd3..3e7324e 100644 --- a/cka/lessons_net.py +++ b/cka/lessons_net.py @@ -1,6 +1,6 @@ """CKA lessons of the Services and Networking domain (5.1-5.6). See cka/lessons.py for what a lesson is.""" from cka.common import DNS -from sim.bank import NET, NS, PODS, step +from sim.bank import NET, NS, PODS, step, cmd, hand # ---- 5.1 connectivity between Pods ------------------------------------------------------------ L51 = { @@ -35,35 +35,55 @@ ready net || exit 1 want=$($K get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' | sort) eq "$(tr -s ' \t' ' ' < "$WS/lesson-5.1/cidrs.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """, hint=r"""`.spec.podCIDR` of each node: `-o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}'`, or `custom-columns`. The cluster-wide range is the `podSubnet` in kubeadm's configuration.""", - show=r"""kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.spec.podCIDR}{"\n"}{end}' > {WS}/lesson-5.1/cidrs.txt + show=[cmd(r"""k get no -o custom-columns=NODE:.metadata.name,CIDR:.spec.podCIDR --no-headers > {WS}/lesson-5.1/cidrs.txt cat {WS}/lesson-5.1/cidrs.txt -kubectl -n net get pods -o wide"""), +k -n net get po -o wide""", r"""Every Pod's IP is from the range of the node it runs on.""")]), step("Across nodes, Pod to Pod", r"""From Pod `a`, send a request straight to Pod `b`'s **IP address** on port 80 - no Service - and save the answer in `{WS}/lesson-5.1/answer.txt`. It crossed from `{W1}` to `{W2}`.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/answer.txt" 2>/dev/null)" b""", hint=r"""`kubectl -n net get pod b -o jsonpath='{.status.podIP}'`, then `kubectl -n net exec a -- wget -qO- -T3 http://IP`.""", - show=r"""ip=$(kubectl -n net get pod b -o jsonpath='{.status.podIP}') -kubectl -n net exec a -- wget -qO- -T3 http://$ip | tee {WS}/lesson-5.1/answer.txt"""), + show=[cmd(r"""k -n net get po -o wide"""), + hand(r"""k -n net exec a -- wget -qO- -T3 IP > {WS}/lesson-5.1/answer.txt""", + r"""`IP`: Pod `b`'s address, from the list.""", + run=r"""ip=$(kubectl -n net get pod b -o jsonpath='{.status.podIP}') +kubectl -n net exec a -- wget -qO- -T3 http://$ip > {WS}/lesson-5.1/answer.txt"""), + cmd(r"""cat {WS}/lesson-5.1/answer.txt""")]), step("The route that carries it", r"""On `{W1}`, look at the routing table: there is a route for `{W2}`'s Pod CIDR, via `{W2}`'s node address. Write that route's destination and gateway to `{WS}/lesson-5.1/route.txt`, as `CIDR via ADDRESS`.""", check=r""" c=$(j node "$W2" '{.spec.podCIDR}'); gw=$(j node "$W2" '{.status.addresses[?(@.type=="InternalIP")].address}') nx "$W1" "ip route | grep -q '^$c via $gw '" /dev/null | tr -d '\r\n' | sed 's/ *$//')" "$c via $gw" """, hint=r"""`ip route` on the node. The node's address is its `InternalIP` (`kubectl get nodes -o wide`).""", - show=r"""ssh -n {W1} ip route -c=$(kubectl get node {W2} -o jsonpath='{.spec.podCIDR}') -ssh -n {W1} ip route | grep "^$c " | awk '{print $1, $2, $3}' > {WS}/lesson-5.1/route.txt -cat {WS}/lesson-5.1/route.txt"""), + show=[cmd(r"""k get no -o wide +k get no -o custom-columns=NODE:.metadata.name,CIDR:.spec.podCIDR""", r"""`{W2}`'s address (`INTERNAL-IP`) and its Pod CIDR."""), + hand(r"""ssh {W1} ip route""", + r"""The line for `{W2}`'s Pod CIDR reads `CIDR via ADDRESS dev ...` - the route to the other node's Pods.""", + run=r"""ssh -n {W1} ip route"""), + hand(r"""echo 'CIDR via ADDRESS' > {WS}/lesson-5.1/route.txt""", + r"""The first three words of that line.""", + run=r"""c=$(kubectl get node {W2} -o jsonpath='{.spec.podCIDR}') +ssh -n {W1} ip route | grep "^$c " | awk '{print $1, $2, $3}' > {WS}/lesson-5.1/route.txt""")]), step("A Pod on the host's network", r"""Run Pod `hostnet` in `net` (`busybox:1.36`, `sleep 86400`) with `hostNetwork: true`, on `{W1}`. Its IP is not from the Pod CIDR: it is `{W1}`'s own address, and it sees the node's interfaces. Write its IP to `{WS}/lesson-5.1/hostnet.txt`.""", check=r""" want=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}') eq "$(j -n net pod hostnet '{.spec.hostNetwork} {.status.podIP}')" "true $want" && eq "$(tr -d ' \r\n' < "$WS/lesson-5.1/hostnet.txt" 2>/dev/null)" "$want" """, hint=r"""`spec.hostNetwork: true` and `spec.nodeName: {W1}` (or a nodeSelector). `kubectl run ... --overrides='{"spec":{...}}'` adds both.""", - show=r"""kubectl -n net run hostnet --image=busybox:1.36 --overrides='{"spec":{"hostNetwork":true,"nodeName":"{W1}"}}' -- sleep 86400 -kubectl -n net wait --for=condition=Ready pod/hostnet --timeout=120s -kubectl -n net get pod hostnet -o jsonpath='{.status.podIP}' > {WS}/lesson-5.1/hostnet.txt -kubectl -n net exec hostnet -- ip -4 addr show eth0"""), + show=[cmd(r"""k -n net run hostnet --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > hostnet.yaml"""), + hand(r"""vim hostnet.yaml""", + r"""Add to the Pod's `spec:`, above `containers:`: +``` + hostNetwork: true + nodeName: {W1} +```""", + run=r"""yq -i '.spec.hostNetwork = true | .spec.nodeName = "{W1}"' hostnet.yaml"""), + cmd(r"""k -n net apply -f hostnet.yaml +k -n net get po hostnet -o wide""", r"""Its IP is `{W1}`'s own address, not one from the Pod CIDR."""), + hand(r"""echo IP > {WS}/lesson-5.1/hostnet.txt""", + r"""`IP`: that address.""", + run=r"""kubectl -n net wait --for=condition=Ready pod/hostnet --timeout=120s +kubectl -n net get pod hostnet -o jsonpath='{.status.podIP}' > {WS}/lesson-5.1/hostnet.txt"""), + cmd(r"""k -n net exec hostnet -- ip -4 addr show eth0""")]), ], "outro": r"""You read how the Pod network is divided among nodes, sent a request from one node's Pod to another's, found the route that carried it, and ran a Pod on the host network. @@ -108,7 +128,9 @@ ready prod && ready dev || exit 1 eq "$(j -n prod netpol default-deny '{.spec.policyTypes[0]}')" Ingress && [ -z "$(j -n prod netpol default-deny '{.spec.ingress}')" ] || exit 1 blocked prod api http://db && blocked dev tester http://web.prod""", hint=r"""`podSelector: {}` and `policyTypes: [Ingress]`, with no `ingress:` list. Test with `kubectl -n prod exec api -- wget -qO- -T3 db`.""", - show=r"""kubectl -n prod apply -f - <<'EOF' + show=[hand(r"""vim deny.yaml""", + r"""No `kubectl create` for a NetworkPolicy: paste *Default deny all ingress traffic* from the docs page *Network Policies*, and name it: +``` apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: @@ -116,13 +138,25 @@ metadata: spec: podSelector: {} policyTypes: [Ingress] -EOF -kubectl -n prod exec api -- wget -qO- -T3 db || echo blocked"""), +```""", + run=r"""cat > deny.yaml <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny +spec: + podSelector: {} + policyTypes: [Ingress] +EOF"""), + cmd(r"""k -n prod apply -f deny.yaml +k -n prod exec api -- wget -qO- -T3 db""", r"""It times out: blocked.""")]), step("Let the API reach the database", r"""Create NetworkPolicy `db-from-api` in `prod`: Pods `app=db` accept TCP 80 from Pods `app=api` **of `prod`**. `web` still cannot reach `db`, and neither can `dev`'s `api` - a `podSelector` alone means "in this Namespace".""", check=NET + r"""okr prod api http://db && blocked prod web http://db && blocked dev api http://db.prod""", hint=r"""`podSelector: {matchLabels: {app: db}}`, `ingress: [{from: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}]`.""", - show=r"""kubectl -n prod apply -f - <<'EOF' + show=[hand(r"""vim db-from-api.yaml""", + r"""Paste the example from the docs page *Network Policies* and cut it down to: +``` apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: @@ -137,16 +171,36 @@ spec: matchLabels: {app: api} ports: - {protocol: TCP, port: 80} -EOF -kubectl -n prod exec api -- wget -qO- -T3 db -kubectl -n prod exec web -- wget -qO- -T3 db || echo blocked"""), +```""", + run=r"""cat > db-from-api.yaml <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: db-from-api +spec: + podSelector: + matchLabels: {app: db} + policyTypes: [Ingress] + ingress: + - from: + - podSelector: + matchLabels: {app: api} + ports: + - {protocol: TCP, port: 80} +EOF"""), + cmd(r"""k -n prod apply -f db-from-api.yaml +k -n prod exec api -- wget -qO- -T3 db +k -n prod exec web -- wget -qO- -T3 db""", r"""`api` gets the page; `web` times out.""")]), step("Those Pods in that Namespace", r"""Create NetworkPolicy `web-from-dev-api` in `prod`: Pods `app=web` accept TCP 80 from Pods `app=api` **in** Namespaces labelled `team=dev` - both conditions in **one** element of `from`. `dev`'s `api` gets in; `dev`'s `tester` does not, and neither does `prod`'s own `api`. Written as two elements, the rule would let every Pod of `dev` in, and every `api` Pod of `prod` too.""", check=NET + r"""okr dev api http://web.prod && blocked dev tester http://web.prod && blocked prod api http://web""", hint=r"""`from: [{namespaceSelector: {matchLabels: {team: dev}}, podSelector: {matchLabels: {app: api}}}]` - mind the single `-`.""", - show=r"""kubectl -n prod apply -f - <<'EOF' + show=[hand(r"""cp db-from-api.yaml web-from-dev-api.yaml +vim web-from-dev-api.yaml""", + r"""Make it - `namespaceSelector` and `podSelector` in **one** element of `from` (no `-` before `podSelector`): +``` apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: @@ -163,9 +217,28 @@ spec: matchLabels: {app: api} ports: - {protocol: TCP, port: 80} -EOF -kubectl -n dev exec api -- wget -qO- -T3 web.prod -kubectl -n dev exec tester -- wget -qO- -T3 web.prod || echo blocked"""), +```""", + run=r"""cat > web-from-dev-api.yaml <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: web-from-dev-api +spec: + podSelector: + matchLabels: {app: web} + policyTypes: [Ingress] + ingress: + - from: + - namespaceSelector: + matchLabels: {team: dev} + podSelector: + matchLabels: {app: api} + ports: + - {protocol: TCP, port: 80} +EOF"""), + cmd(r"""k -n prod apply -f web-from-dev-api.yaml +k -n dev exec api -- wget -qO- -T3 web.prod +k -n dev exec tester -- wget -qO- -T3 web.prod""", r"""`dev`'s `api` gets in; `tester` times out.""")]), step("Egress, and DNS", r"""Now limit what `tester` in `dev` may **send**: NetworkPolicy `tester-egress` in `dev` selects `app=tester` and allows egress only to `dev`'s `app=api` Pods on TCP 80 - and DNS, to `kube-dns` in `kube-system` on UDP and TCP 53. @@ -174,7 +247,9 @@ kubectl -n dev exec tester -- wget -qO- -T3 web.prod || echo blocked"""), eq "$(j -n dev netpol tester-egress '{.spec.policyTypes[0]}')" Egress || exit 1 okr dev tester http://api && blocked dev tester http://echo""", hint=r"""Two egress rules: `{to: [{podSelector: {matchLabels: {app: api}}}], ports: [{protocol: TCP, port: 80}]}` and `{to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}], ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]}`. Every Namespace carries the label `kubernetes.io/metadata.name`.""", - show=r"""kubectl -n dev apply -f - <<'EOF' + show=[hand(r"""vim tester-egress.yaml""", + r"""An egress policy - two elements of `egress`, one for `api`, one for DNS: +``` apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: @@ -197,9 +272,34 @@ spec: ports: - {protocol: UDP, port: 53} - {protocol: TCP, port: 53} -EOF -kubectl -n dev exec tester -- wget -qO- -T3 api -kubectl -n dev exec tester -- wget -qO- -T3 echo || echo blocked"""), +```""", + run=r"""cat > tester-egress.yaml <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: tester-egress +spec: + podSelector: + matchLabels: {app: tester} + policyTypes: [Egress] + egress: + - to: + - podSelector: + matchLabels: {app: api} + ports: + - {protocol: TCP, port: 80} + - to: + - namespaceSelector: + matchLabels: {kubernetes.io/metadata.name: kube-system} + podSelector: + matchLabels: {k8s-app: kube-dns} + ports: + - {protocol: UDP, port: 53} + - {protocol: TCP, port: 53} +EOF"""), + cmd(r"""k -n dev apply -f tester-egress.yaml +k -n dev exec tester -- wget -qO- -T3 api +k -n dev exec tester -- wget -qO- -T3 echo""", r"""`api` answers; `echo` times out.""")]), ], "outro": r"""You closed a Namespace, opened single paths into it, used the AND form of a rule, and limited a Pod's egress without breaking its DNS. @@ -243,20 +343,28 @@ $K -n svc rollout status deploy web --timeout=180s >/dev/null && ready svc r"""Expose Deployment `web` as Service `web`, port 80. From `client`, `wget -qO- web` returns nginx's page.""", check=NET + r"""eq "$(j -n svc svc web '{.spec.type} {.spec.ports[0].port}')" "ClusterIP 80" && okr svc client http://web""", hint=r"""`kubectl expose deploy web --port=80` - it copies the Deployment's selector.""", - show=r"""kubectl -n svc expose deploy web --port=80 -kubectl -n svc get svc,endpointslices -kubectl -n svc exec client -- wget -qO- -T3 web | head -4"""), + show=[cmd(r"""k -n svc expose deploy web --port=80 +k -n svc get svc,endpointslices +k -n svc exec client -- wget -qO- -T3 web | head -4""")]), step("NodePort", r"""Create Service `web-np` of type **NodePort** for the same Pods, with node port **30080**. Then reach it from the control-plane node: `curl http://{W1}-ADDRESS:30080` - every node answers on that port, whether or not it runs one of the Pods.""", check=r""" eq "$(j -n svc svc web-np '{.spec.type} {.spec.ports[0].nodePort}')" "NodePort 30080" || exit 1 ip=$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}'); nx "$CP" "curl -s -m 3 http://$ip:30080 | grep -q nginx" {WS}/lesson-5.3/web-np.yaml -yq -i '.spec.ports[0].nodePort = 30080' {WS}/lesson-5.3/web-np.yaml -kubectl -n svc apply -f {WS}/lesson-5.3/web-np.yaml -ip=$(kubectl get node {W1} -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') -curl -s -m 3 http://$ip:30080 | head -4"""), + show=[cmd(r"""k -n svc expose deploy web --name=web-np --type=NodePort --port=80 --dry-run=client -o yaml > web-np.yaml"""), + hand(r"""vim web-np.yaml""", + r"""Add to the port: +``` + nodePort: 30080 +```""", + run=r"""yq -i '.spec.ports[0].nodePort = 30080' web-np.yaml"""), + cmd(r"""k -n svc apply -f web-np.yaml +k get no -o wide""", r"""`{W1}`'s `INTERNAL-IP`:"""), + hand(r"""curl IP:30080""", + r"""nginx's page, from the node port.""", + run=r"""ip=$(kubectl get node {W1} -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') +curl -s -m 3 http://$ip:30080 | head -4""")]), step("Headless", r"""Create Service `web-headless` with `clusterIP: None` for the `web` Pods. Look it up from `client` with `nslookup web-headless`: you get the two Pod IPs, not a Service IP. Write them, one per line, to `{WS}/lesson-5.3/headless.txt`.""", check=DNS + r""" @@ -264,18 +372,19 @@ eq "$(j -n svc svc web-headless '{.spec.clusterIP}')" None || exit 1 want=$($K -n svc get pods -l app=web -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' | sort) eq "$(tr -d ' \r' < "$WS/lesson-5.3/headless.txt" 2>/dev/null | grep . | sort)" "$want" """, hint=r"""`kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80`. In busybox's `nslookup` output the addresses follow the `Name:` line.""", - show=r"""kubectl -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80 -sleep 5 -kubectl -n svc exec client -- nslookup web-headless -kubectl -n svc exec client -- nslookup web-headless | awk '/^Name:/ {n=1} n && /^Address/ {print $NF}' > {WS}/lesson-5.3/headless.txt -cat {WS}/lesson-5.3/headless.txt"""), + show=[cmd(r"""k -n svc expose deploy web --name=web-headless --cluster-ip=None --port=80 +k -n svc exec client -- nslookup web-headless""", r"""Two addresses: the Pods' own. Again in a few seconds if it does not resolve yet."""), + hand(r"""vim {WS}/lesson-5.3/headless.txt""", + r"""The two addresses, one per line.""", + run=r"""sleep 5 +kubectl -n svc exec client -- nslookup web-headless | awk '/^Name:/ {n=1} n && /^Address/ {print $NF}' > {WS}/lesson-5.3/headless.txt""")]), step("ExternalName", r"""Create Service `docs` of type **ExternalName** pointing at `kubernetes.io`. Inside the cluster, `docs.svc.svc.cluster.local` is now a CNAME for that name - DNS only: nothing is proxied, and the port does not matter.""", check=r"""eq "$(j -n svc svc docs '{.spec.type} {.spec.externalName}')" "ExternalName kubernetes.io" """, hint=r"""`kubectl -n svc create service externalname docs --external-name=kubernetes.io`.""", - show=r"""kubectl -n svc create service externalname docs --external-name=kubernetes.io -kubectl -n svc get svc docs -kubectl -n svc exec client -- nslookup docs | head -6"""), + show=[cmd(r"""k -n svc create svc externalname docs --external-name=kubernetes.io +k -n svc get svc docs +k -n svc exec client -- nslookup docs | head -6""")]), step("A Service without a selector", r"""Create Service `legacy` (port 80) with **no selector**, and an EndpointSlice for it by hand that holds `legacy-box`'s Pod IP on port 80. From `client`, `wget -qO- legacy` then answers `legacy`. @@ -284,8 +393,32 @@ An EndpointSlice belongs to a Service by the label `kubernetes.io/service-name`; [ -z "$(j -n svc svc legacy '{.spec.selector}')" ] || exit 1 for i in 1 2 3 4 5; do [ "$($K -n svc exec client -- wget -qO- -T3 legacy 2>/dev/null)" = legacy ] && exit 0; sleep 3; done; exit 1""", hint=r"""Service: `spec.ports: [{port: 80, targetPort: 80}]` and no `selector`. EndpointSlice (`discovery.k8s.io/v1`): `addressType: IPv4`, `endpoints: [{addresses: [IP]}]`, `ports: [{port: 80, protocol: TCP}]`, and the label.""", - show=r"""ip=$(kubectl -n svc get pod legacy-box -o jsonpath='{.status.podIP}') -kubectl -n svc apply -f - < legacy.yaml < gw.yaml <<'EOF' +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass +metadata: + name: example +spec: + controllerName: example.com/gateway-controller +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: shop + namespace: gw +spec: + gatewayClassName: example + listeners: + - name: http + protocol: HTTP + port: 80 + hostname: shop.example.com + allowedRoutes: + namespaces: + from: Same +EOF"""), + cmd(r"""k apply -f gw.yaml +k -n gw get gateway shop""")]), step("Route by path", r"""Create HTTPRoute `shop` in `gw`, attached to Gateway `shop`: requests for paths under `/api` go to Service `api` port 8080, everything else to `web` port 80.""", check=r""" @@ -386,7 +547,9 @@ def to(path): return [(b["name"], b.get("port")) for b in x["backendRefs"]] sys.exit(0 if p == "shop" and to("/api") == [("api", 8080)] and to("/") == [("web", 80)] else 1)'""", hint=r"""`parentRefs: [{name: shop}]`, then two `rules`, each with `matches: [{path: {type: PathPrefix, value: ...}}]` and `backendRefs: [{name: ..., port: ...}]`.""", - show=r"""kubectl -n gw apply -f - <<'EOF' + show=[hand(r"""vim route.yaml""", + r"""Paste an HTTPRoute from gateway-api.sigs.k8s.io (*HTTP routing*), and make it: +``` apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: @@ -404,8 +567,28 @@ spec: - path: {type: PathPrefix, value: /} backendRefs: - {name: web, port: 80} -EOF -kubectl -n gw describe httproute shop | grep -A3 'Backend Refs' | head -8"""), +```""", + run=r"""cat > route.yaml <<'EOF' +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: shop +spec: + parentRefs: + - name: shop + hostnames: [shop.example.com] + rules: + - matches: + - path: {type: PathPrefix, value: /api} + backendRefs: + - {name: api, port: 8080} + - matches: + - path: {type: PathPrefix, value: /} + backendRefs: + - {name: web, port: 80} +EOF"""), + cmd(r"""k -n gw apply -f route.yaml +k -n gw describe httproute shop""")]), step("A canary by weight", r"""Change the `/` rule of route `shop` so that **90%** of its requests go to `web` and **10%** to `web-v2` (both port 80). The `/api` rule stays as it is.""", check=r""" @@ -416,10 +599,20 @@ root, api = by("/"), by("/api") w = sorted((b["name"], b.get("weight", 1)) for b in root["backendRefs"]) if root else [] sys.exit(0 if w == [("web", 90), ("web-v2", 10)] and api and [b["name"] for b in api["backendRefs"]] == ["api"] else 1)'""", hint=r"""Two `backendRefs` in the same rule, each with a `weight`. The weights are relative: 9 and 1 would do the same.""", - show=r"""kubectl -n gw get httproute shop -o yaml > {WS}/lesson-5.4/shop.yaml + show=[hand(r"""k -n gw edit httproute shop""", + r"""In the rule whose path is `/`, give `web` a `weight: 90` and add `web-v2` beside it: +``` + - name: web + port: 80 + weight: 90 + - name: web-v2 + port: 80 + weight: 10 +```""", + run=r"""kubectl -n gw get httproute shop -o yaml > {WS}/lesson-5.4/shop.yaml yq -i '(.spec.rules[] | select(.matches[0].path.value == "/")).backendRefs = [{"name": "web", "port": 80, "weight": 90}, {"name": "web-v2", "port": 80, "weight": 10}]' {WS}/lesson-5.4/shop.yaml -kubectl apply -f {WS}/lesson-5.4/shop.yaml -kubectl -n gw get httproute shop -o jsonpath='{.spec.rules[1].backendRefs}'; echo"""), +kubectl apply -f {WS}/lesson-5.4/shop.yaml"""), + cmd(r"""k -n gw describe httproute shop""")]), step("Beta testers by header", r"""Add HTTPRoute `shop-beta` in `gw`, attached to `shop`: requests carrying the header `x-env: beta` go **all** to `web-v2`. A more specific match wins, so these requests no longer take part in the 90/10 split.""", check=r""" @@ -428,7 +621,9 @@ import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0] h = r["matches"][0].get("headers", [{}])[0] sys.exit(0 if s["parentRefs"][0]["name"] == "shop" and h.get("name", "").lower() == "x-env" and h.get("value") == "beta" and [b["name"] for b in r["backendRefs"]] == ["web-v2"] else 1)'""", hint=r"""`matches: [{headers: [{name: x-env, value: beta}]}]` (type `Exact` by default).""", - show=r"""kubectl -n gw apply -f - <<'EOF' + show=[hand(r"""vim beta.yaml""", + r"""Paste an HTTPRoute with a header match from gateway-api.sigs.k8s.io (*HTTP header matching*), and make it: +``` apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: @@ -443,8 +638,25 @@ spec: - {name: x-env, value: beta} backendRefs: - {name: web-v2, port: 80} -EOF -kubectl -n gw get httproute"""), +```""", + run=r"""cat > beta.yaml <<'EOF' +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: shop-beta +spec: + parentRefs: + - name: shop + hostnames: [shop.example.com] + rules: + - matches: + - headers: + - {name: x-env, value: beta} + backendRefs: + - {name: web-v2, port: 80} +EOF"""), + cmd(r"""k -n gw apply -f beta.yaml +k -n gw get httproute""")]), step("HTTPS on the Gateway", r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `gw`, and add a second listener to Gateway `shop`: `https`, protocol HTTPS, port 443, the same hostname, TLS mode `Terminate` with that Secret as its certificate.""", check=r""" @@ -452,12 +664,27 @@ eq "$(j -n gw secret shop-tls '{.type}')" kubernetes.io/tls || exit 1 eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="https")].protocol} {.spec.listeners[?(@.name=="https")].port} {.spec.listeners[?(@.name=="https")].tls.mode} {.spec.listeners[?(@.name=="https")].tls.certificateRefs[0].name}')" "HTTPS 443 Terminate shop-tls" \ && eq "$(j -n gw gateway shop '{.spec.listeners[?(@.name=="http")].port}')" 80""", hint=r"""`openssl req -x509 ...`, `kubectl create secret tls`, then a listener with `tls: {mode: Terminate, certificateRefs: [{name: shop-tls}]}`. Keep the `http` listener.""", - show=r"""cd {WS}/lesson-5.4 -openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null -kubectl -n gw create secret tls shop-tls --cert=tls.crt --key=tls.key -kubectl -n gw patch gateway shop --type json -p '[{"op":"add","path":"/spec/listeners/-","value":{"name":"https","protocol":"HTTPS","port":443,"hostname":"shop.example.com","tls":{"mode":"Terminate","certificateRefs":[{"name":"shop-tls"}]},"allowedRoutes":{"namespaces":{"from":"Same"}}}}]' -kubectl -n gw get gateway shop -o jsonpath='{range .spec.listeners[*]}{.name} {.port}{"\n"}{end}' + show=[cmd(r"""cd {WS}/lesson-5.4 +openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com +k -n gw create secret tls shop-tls --cert=tls.crt --key=tls.key cd"""), + hand(r"""k -n gw edit gateway shop""", + r"""Add a second listener below `http`, in `listeners:`: +``` + - name: https + protocol: HTTPS + port: 443 + hostname: shop.example.com + tls: + mode: Terminate + certificateRefs: + - name: shop-tls + allowedRoutes: + namespaces: + from: Same +```""", + run=r"""kubectl -n gw patch gateway shop --type json -p '[{"op":"add","path":"/spec/listeners/-","value":{"name":"https","protocol":"HTTPS","port":443,"hostname":"shop.example.com","tls":{"mode":"Terminate","certificateRefs":[{"name":"shop-tls"}]},"allowedRoutes":{"namespaces":{"from":"Same"}}}}]' """), + cmd(r"""k -n gw describe gateway shop | grep -A3 -i listener""")]), ], "outro": r"""You built a Gateway with HTTP and HTTPS listeners, routed by path, split traffic by weight, and sent one header's requests elsewhere. @@ -498,7 +725,9 @@ mkdir -p "$WS/lesson-5.5" r"""Create IngressClass `nginx` for controller `k8s.io/ingress-nginx`, and make it the cluster's **default** class: an Ingress without `ingressClassName` then belongs to it.""", check=r"""eq "$(j ingressclass nginx '{.spec.controller} {.metadata.annotations.ingressclass\.kubernetes\.io/is-default-class}')" "k8s.io/ingress-nginx true" """, hint=r"""`networking.k8s.io/v1`, kind `IngressClass`, `spec.controller`, and the annotation `ingressclass.kubernetes.io/is-default-class: "true"`. Installing ingress-nginx makes this object for you.""", - show=r"""kubectl apply -f - <<'EOF' + show=[hand(r"""vim ic.yaml""", + r"""No `kubectl create` for an IngressClass: paste the example from the docs page *Ingress* (*Default IngressClass*), and make it: +``` apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: @@ -507,8 +736,19 @@ metadata: ingressclass.kubernetes.io/is-default-class: "true" spec: controller: k8s.io/ingress-nginx -EOF -kubectl get ingressclass"""), +```""", + run=r"""cat > ic.yaml <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: IngressClass +metadata: + name: nginx + annotations: + ingressclass.kubernetes.io/is-default-class: "true" +spec: + controller: k8s.io/ingress-nginx +EOF"""), + cmd(r"""k apply -f ic.yaml +k get ingressclass""")]), step("Host and paths, from the command line", r"""Create Ingress `shop` in `ing` with class `nginx` and host `shop.example.com`: paths under `/api` go to `api:8080`, everything else to `web:80`. `kubectl create ingress` can do it in one line.""", check=r""" @@ -517,8 +757,8 @@ import json, sys; s = json.load(sys.stdin)["spec"]; r = s["rules"][0] p = {x["path"]: (x["pathType"], x["backend"]["service"]["name"], x["backend"]["service"]["port"]["number"]) for x in r["http"]["paths"]} sys.exit(0 if s.get("ingressClassName") == "nginx" and r["host"] == "shop.example.com" and p.get("/api") == ("Prefix", "api", 8080) and p.get("/") == ("Prefix", "web", 80) else 1)'""", hint=r"""`kubectl create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80'` - a trailing `*` means `Prefix`, without it `Exact`.""", - show=r"""kubectl -n ing create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80' -kubectl -n ing describe ingress shop | sed -n '/Rules/,/Annotations/p'"""), + show=[cmd(r"""k -n ing create ingress shop --class=nginx --rule='shop.example.com/api*=api:8080' --rule='shop.example.com/*=web:80' +k -n ing describe ingress shop""")]), step("An exact path", r"""Add a rule to Ingress `shop`: exactly `/healthz` (pathType `Exact`) goes to `health:8081`. `/healthz/deep` would not match it - it would fall through to the `/` prefix.""", check=r""" @@ -527,26 +767,53 @@ import json, sys; ps = json.load(sys.stdin)["spec"]["rules"][0]["http"]["paths"] h = [x for x in ps if x["path"] == "/healthz"] sys.exit(0 if h and h[0]["pathType"] == "Exact" and h[0]["backend"]["service"]["name"] == "health" and h[0]["backend"]["service"]["port"]["number"] == 8081 and len(ps) == 3 else 1)'""", hint=r"""`kubectl edit ingress shop` and add a third entry under `paths:`, or a JSON patch that appends to `/spec/rules/0/http/paths`.""", - show=r"""kubectl -n ing patch ingress shop --type json -p '[{"op":"add","path":"/spec/rules/0/http/paths/-","value":{"path":"/healthz","pathType":"Exact","backend":{"service":{"name":"health","port":{"number":8081}}}}}]' -kubectl -n ing get ingress shop -o jsonpath='{range .spec.rules[0].http.paths[*]}{.pathType} {.path} -> {.backend.service.name}{"\n"}{end}'"""), + show=[hand(r"""k -n ing edit ingress shop""", + r"""Add a third path to `paths:`, indented like the other two: +``` + - path: /healthz + pathType: Exact + backend: + service: + name: health + port: + number: 8081 +```""", + run=r"""kubectl -n ing patch ingress shop --type json -p '[{"op":"add","path":"/spec/rules/0/http/paths/-","value":{"path":"/healthz","pathType":"Exact","backend":{"service":{"name":"health","port":{"number":8081}}}}}]' """), + cmd(r"""k -n ing describe ingress shop""")]), step("TLS", r"""Make a self-signed certificate for `shop.example.com`, store it as TLS Secret `shop-tls` in `ing`, and add a `tls` section to Ingress `shop` for that host with that Secret. The controller would then serve HTTPS for it and, by default, redirect HTTP there.""", check=r""" eq "$(j -n ing secret shop-tls '{.type}')" kubernetes.io/tls && eq "$(j -n ing ingress shop '{.spec.tls[0].hosts[0]} {.spec.tls[0].secretName}')" "shop.example.com shop-tls" \ && j -n ing secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q shop.example.com""", hint=r"""`spec.tls: [{hosts: [shop.example.com], secretName: shop-tls}]`. The Secret must be in the Ingress's Namespace.""", - show=r"""cd {WS}/lesson-5.5 -openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null -kubectl -n ing create secret tls shop-tls --cert=tls.crt --key=tls.key -kubectl -n ing patch ingress shop --type merge -p '{"spec":{"tls":[{"hosts":["shop.example.com"],"secretName":"shop-tls"}]}}' -kubectl -n ing get ingress shop + show=[cmd(r"""cd {WS}/lesson-5.5 +openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com +k -n ing create secret tls shop-tls --cert=tls.crt --key=tls.key cd"""), + hand(r"""k -n ing edit ingress shop""", + r"""Add to `spec:`, beside `rules:`: +``` + tls: + - hosts: [shop.example.com] + secretName: shop-tls +```""", + run=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"tls":[{"hosts":["shop.example.com"],"secretName":"shop-tls"}]}}' """), + cmd(r"""k -n ing get ingress shop""")]), step("What matches nothing", r"""Give Ingress `shop` a **default backend**: `web:80`, for requests whose host or path no rule matches.""", check=r"""eq "$(j -n ing ingress shop '{.spec.defaultBackend.service.name} {.spec.defaultBackend.service.port.number}')" "web 80" """, hint=r"""`spec.defaultBackend: {service: {name: web, port: {number: 80}}}`; `kubectl create ingress --default-backend=web:80` does it at creation.""", - show=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"defaultBackend":{"service":{"name":"web","port":{"number":80}}}}}' -kubectl -n ing describe ingress shop | head -12"""), + show=[hand(r"""k -n ing edit ingress shop""", + r"""Add to `spec:`, beside `rules:`: +``` + defaultBackend: + service: + name: web + port: + number: 80 +```""", + run=r"""kubectl -n ing patch ingress shop --type merge -p '{"spec":{"defaultBackend":{"service":{"name":"web","port":{"number":80}}}}}' """), + cmd(r"""k -n ing describe ingress shop""")]), ], "outro": r"""You made a default IngressClass and an Ingress with host, prefix and exact paths, TLS and a default backend. @@ -608,16 +875,18 @@ ready dns-lab && $K -n dns-lab rollout status sts db --timeout=180s >/dev/null r"""From `client`, look up `web` by its **fully qualified** name, and write the address you get to `{WS}/lesson-5.6/web.txt`. It is the Service's ClusterIP.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/web.txt" 2>/dev/null)" "$(j -n dns-lab svc web '{.spec.clusterIP}')" """, hint=r"""`kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local`.""", - show=r"""kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local -kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local | awk '/^Name:/ {n=1} n && /^Address/ {print $NF; exit}' > {WS}/lesson-5.6/web.txt"""), + show=[cmd(r"""k -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local""", r"""The `Address` below `Name:` is the Service's ClusterIP."""), + hand(r"""echo IP > {WS}/lesson-5.6/web.txt""", + r"""`IP`: that address.""", + run=r"""kubectl -n dns-lab exec client -- nslookup web.dns-lab.svc.cluster.local | awk '/^Name:/ {n=1} n && /^Address/ {print $NF; exit}' > {WS}/lesson-5.6/web.txt""")]), step("One Pod of a StatefulSet", r"""A StatefulSet's Pods get stable names through their headless Service. Write the fully qualified DNS name of Pod `db-0` to `{WS}/lesson-5.6/db0.txt`, and check from `client` that it resolves to `db-0`'s IP.""", check=DNS + r""" eq "$(tr -d ' \r\n' < "$WS/lesson-5.6/db0.txt" 2>/dev/null | sed 's/\.$//')" db-0.db.dns-lab.svc.cluster.local && resolves dns-lab client db-0.db.dns-lab.svc.cluster.local "$(j -n dns-lab pod db-0 '{.status.podIP}')" """, hint=r"""`POD.SERVICE.NAMESPACE.svc.cluster.local` - the Service is the StatefulSet's `serviceName`.""", - show=r"""echo db-0.db.dns-lab.svc.cluster.local > {WS}/lesson-5.6/db0.txt -kubectl -n dns-lab exec client -- nslookup db-0.db.dns-lab.svc.cluster.local -kubectl -n dns-lab get pod db-0 -o wide"""), + show=[cmd(r"""echo db-0.db.dns-lab.svc.cluster.local > {WS}/lesson-5.6/db0.txt +k -n dns-lab exec client -- nslookup db-0.db.dns-lab.svc.cluster.local +k -n dns-lab get po db-0 -o wide""")]), step("Names of your own", r"""Add a `hosts` block to CoreDNS's Corefile so that every Pod resolves **`legacy.internal`** to `10.96.0.250`, and every other name still resolves as before (`fallthrough`). Put it inside the `.:53 { ... }` server block, before `forward`. @@ -626,12 +895,19 @@ CoreDNS reloads the changed ConfigMap by itself within a minute; `client` then r $K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' | grep -q 'kubernetes cluster.local' || exit 1 resolves dns-lab client legacy.internal 10.96.0.250 && resolves dns-lab client web.dns-lab.svc.cluster.local "$(j -n dns-lab svc web '{.spec.clusterIP}')" """, hint=r"""`kubectl -n kube-system edit cm coredns`. The block: `hosts {` / `10.96.0.250 legacy.internal` / `fallthrough` / `}`. A syntax error leaves CoreDNS on its old configuration: `kubectl -n kube-system logs -l k8s-app=kube-dns` says so.""", - show=r"""kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > {WS}/lesson-5.6/Corefile + show=[hand(r"""k -n kube-system edit cm coredns""", + r"""In `Corefile`, add above the `forward` line, indented like it: +``` + hosts { + 10.96.0.250 legacy.internal + fallthrough + } +```""", + run=r"""kubectl -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > {WS}/lesson-5.6/Corefile sed -i '0,/^ forward/s// hosts {\n 10.96.0.250 legacy.internal\n fallthrough\n }\n forward/' {WS}/lesson-5.6/Corefile -cat {WS}/lesson-5.6/Corefile kubectl -n kube-system create configmap coredns --from-file=Corefile={WS}/lesson-5.6/Corefile --dry-run=client -o yaml | kubectl apply -f - -sleep 40 -kubectl -n dns-lab exec client -- nslookup legacy.internal"""), +sleep 40"""), + cmd(r"""k -n dns-lab exec client -- nslookup legacy.internal""", r"""Again until it answers `10.96.0.250`: CoreDNS reloads the ConfigMap by itself within a minute or two.""")]), step("A Pod's own resolver settings", r"""Run Pod `tuned` in `dns-lab` (`busybox:1.36`, `sleep 86400`) whose resolver you set yourself: `dnsPolicy: None`, with the `kube-dns` Service's IP as its only nameserver, the search domains `dns-lab.svc.cluster.local` and `svc.cluster.local`, and the option `ndots: 2`. Its `/etc/resolv.conf` shows exactly that.""", check=r""" @@ -640,26 +916,21 @@ r=$($K -n dns-lab exec tuned -- cat /etc/resolv.conf 2>/dev/null) echo "$r" | grep -qx "nameserver $ip" && echo "$r" | grep -qx 'search dns-lab.svc.cluster.local svc.cluster.local' && echo "$r" | grep -q 'ndots:2' \ && eq "$(j -n dns-lab pod tuned '{.spec.dnsPolicy}')" None""", hint=r"""`spec.dnsPolicy: None` and `spec.dnsConfig: {nameservers: [IP], searches: [...], options: [{name: ndots, value: "2"}]}`. The other policies: `ClusterFirst` (the default), `Default` (the node's resolver), `ClusterFirstWithHostNet`.""", - show=r"""ip=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}') -kubectl -n dns-lab apply -f - < tuned.yaml"""), + hand(r"""vim tuned.yaml""", + r"""Change `dnsPolicy: ClusterFirst` to `dnsPolicy: None`, and add beside it (the docs page *DNS for Services and Pods* has it, under *Pod's DNS Config*): +``` dnsConfig: - nameservers: [$ip] + nameservers: [IP] searches: [dns-lab.svc.cluster.local, svc.cluster.local] options: - {name: ndots, value: "2"} - containers: - - name: tuned - image: busybox:1.36 - command: [sleep, '86400'] -EOF -kubectl -n dns-lab wait --for=condition=Ready pod/tuned --timeout=120s -kubectl -n dns-lab exec tuned -- cat /etc/resolv.conf"""), +```""", + run=r"""export IP=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}') +yq -i '.spec.dnsPolicy = "None" | .spec.dnsConfig = {"nameservers": [strenv(IP)], "searches": ["dns-lab.svc.cluster.local", "svc.cluster.local"], "options": [{"name": "ndots", "value": "2"}]}' tuned.yaml"""), + cmd(r"""k -n dns-lab apply -f tuned.yaml +k -n dns-lab exec tuned -- cat /etc/resolv.conf""")]), ], "outro": r"""You resolved a Service and a StatefulSet Pod by their full names, taught CoreDNS a name of your own, and set a Pod's resolver by hand. diff --git a/cka/lessons_storage.py b/cka/lessons_storage.py index a75c4c1..93d8449 100644 --- a/cka/lessons_storage.py +++ b/cka/lessons_storage.py @@ -1,5 +1,5 @@ """CKA lessons of the Storage domain (1.1-1.3). See cka/lessons.py for what a lesson is.""" -from sim.bank import NS, step +from sim.bank import NS, step, cmd, hand # ---- 1.1 StorageClasses and dynamic provisioning --------------------------------------------- L11 = { @@ -29,9 +29,10 @@ mkdir -p "$WS/lesson-1.1" want="$(j sc standard '{.metadata.name} {.provisioner} {.volumeBindingMode}')" [ -n "$want" ] && eq "$(tr -s ' \t' ' ' < "$WS/lesson-1.1/default.txt" 2>/dev/null | tr -d '\r\n' | sed 's/ *$//')" "$want" """, hint=r"""`kubectl get sc` shows all three columns. For exactly the values: `-o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}'`.""", - show=r"""kubectl get sc -kubectl get sc standard -o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}' > {WS}/lesson-1.1/default.txt -cat {WS}/lesson-1.1/default.txt"""), + show=[cmd(r"""k get sc"""), + hand(r"""echo 'NAME PROVISIONER MODE' > {WS}/lesson-1.1/default.txt""", + r"""From the `(default)` line: its name, `PROVISIONER` and `VOLUMEBINDINGMODE`.""", + run=r"""kubectl get sc standard -o jsonpath='{.metadata.name} {.provisioner} {.volumeBindingMode}' > {WS}/lesson-1.1/default.txt""")]), step("A class that keeps its volumes", r"""Create StorageClass `keep`: the same provisioner as `standard` (`rancher.io/local-path`), binding mode `WaitForFirstConsumer`, and reclaim policy **`Retain`**, so a volume outlives its claim. @@ -40,7 +41,9 @@ There is no `kubectl create storageclass`: write the manifest. `kubectl get sc s eq "$(j sc keep '{.provisioner} {.reclaimPolicy} {.volumeBindingMode}')" "rancher.io/local-path Retain WaitForFirstConsumer" \ && [ "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""", hint=r"""A StorageClass has no `spec`: `provisioner`, `reclaimPolicy` and `volumeBindingMode` sit at the top level, beside `metadata`. Its `apiVersion` is `storage.k8s.io/v1`.""", - show=r"""kubectl apply -f - <<'EOF' + show=[hand(r"""vim keep.yaml""", + r"""No `kubectl create` for a StorageClass: paste the example from the docs page *Storage Classes*, and make it: +``` apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: @@ -48,8 +51,18 @@ metadata: provisioner: rancher.io/local-path reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer -EOF -kubectl get sc"""), +```""", + run=r"""cat > keep.yaml <<'EOF' +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: keep +provisioner: rancher.io/local-path +reclaimPolicy: Retain +volumeBindingMode: WaitForFirstConsumer +EOF"""), + cmd(r"""k apply -f keep.yaml +k get sc""")]), step("A claim that waits", r"""In Namespace `store`, create PersistentVolumeClaim `ledger`: class `keep`, `ReadWriteOnce`, `100Mi`. @@ -58,7 +71,9 @@ It stays `Pending`, and that is correct. `kubectl -n store describe pvc ledger` eq "$(j -n store pvc ledger '{.spec.storageClassName}')" keep && eq "$(j -n store pvc ledger '{.spec.resources.requests.storage}')" 100Mi \ && grep -qx 'WaitForFirstConsumer' <(tr -d ' \r' < "$WS/lesson-1.1/why.txt" 2>/dev/null)""", hint=r"""The events at the bottom of `describe` have a Reason column. A claim is namespaced; the class is not.""", - show=r"""kubectl -n store apply -f - <<'EOF' + show=[hand(r"""vim ledger.yaml""", + r"""No `kubectl create` for a claim: paste the example from the docs page *Persistent Volumes*, and make it: +``` apiVersion: v1 kind: PersistentVolumeClaim metadata: @@ -69,9 +84,22 @@ spec: resources: requests: storage: 100Mi -EOF -kubectl -n store describe pvc ledger | tail -4 -echo WaitForFirstConsumer > {WS}/lesson-1.1/why.txt"""), +```""", + run=r"""cat > ledger.yaml <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ledger +spec: + storageClassName: keep + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 100Mi +EOF"""), + cmd(r"""k -n store apply -f ledger.yaml +k -n store describe pvc ledger | tail -4""", r"""The event's reason: `WaitForFirstConsumer`."""), + cmd(r"""echo WaitForFirstConsumer > {WS}/lesson-1.1/why.txt""")]), step("Give it a consumer", r"""Run Pod `bookkeeper` in `store` (image `busybox:1.36`) that mounts claim `ledger` at `/data` and runs `sh -c 'echo first > /data/entry.txt && sleep 86400'`. @@ -81,27 +109,22 @@ eq "$(j -n store pvc ledger '{.status.phase}')" Bound || exit 1 pv=$(j -n store pvc ledger '{.spec.volumeName}'); eq "$(j pv "$pv" '{.spec.persistentVolumeReclaimPolicy}')" Retain || exit 1 eq "$($K -n store exec bookkeeper -- cat /data/entry.txt 2>/dev/null)" first""", hint=r"""`kubectl run bookkeeper --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'` gives you the Pod; add `volumes:` (with `persistentVolumeClaim.claimName`) at Pod level and `volumeMounts:` in the container.""", - show=r"""kubectl -n store apply -f - <<'EOF' -apiVersion: v1 -kind: Pod -metadata: - name: bookkeeper -spec: - containers: - - name: bookkeeper - image: busybox:1.36 - command: [sh, -c, 'echo first > /data/entry.txt && sleep 86400'] + show=[cmd(r"""k -n store run bookkeeper --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo first > /data/entry.txt && sleep 86400' > bk.yaml"""), + hand(r"""vim bk.yaml""", + r"""Add the mount to the container, below `resources: {}`: +``` volumeMounts: - - name: data - mountPath: /data + - {name: data, mountPath: /data} +``` +and the volume to the Pod's `spec:`, beside `restartPolicy:`: +``` volumes: - - name: data - persistentVolumeClaim: - claimName: ledger -EOF -kubectl -n store wait --for=condition=Ready pod/bookkeeper --timeout=120s -kubectl -n store get pvc ledger -kubectl get pv"""), + - {name: data, persistentVolumeClaim: {claimName: ledger}} +```""", + run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}] | .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "ledger"}}]' bk.yaml"""), + cmd(r"""k -n store apply -f bk.yaml +k -n store get pvc ledger +k get pv""", r"""Again once the Pod runs: the claim is `Bound`, and its volume has `RECLAIM POLICY` `Retain`.""")]), step("Delete the claim, keep the volume", r"""Delete Pod `bookkeeper` and claim `ledger`. Because of `Retain`, the volume stays, with status `Released`: its data is still on disk, and no new claim can bind it until an admin says so. @@ -111,19 +134,22 @@ Write the PersistentVolume's name to `{WS}/lesson-1.1/released.txt`.""", pv=$(tr -d ' \r\n' < "$WS/lesson-1.1/released.txt" 2>/dev/null); [ -n "$pv" ] || exit 1 eq "$(j pv "$pv" '{.status.phase} {.spec.claimRef.namespace}/{.spec.claimRef.name}')" "Released store/ledger" """, hint=r"""Read the volume name off the claim before you delete it (`kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}'`), or find it afterwards in `kubectl get pv`: the CLAIM column still names `store/ledger`.""", - show=r"""kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}' > {WS}/lesson-1.1/released.txt -kubectl -n store delete pod bookkeeper --grace-period=1 -kubectl -n store delete pvc ledger -kubectl get pv"""), + show=[cmd(r"""k -n store get pvc ledger""", r"""`VOLUME` is the PersistentVolume's name."""), + hand(r"""echo PV > {WS}/lesson-1.1/released.txt""", + r"""`PV`: that name.""", + run=r"""kubectl -n store get pvc ledger -o jsonpath='{.spec.volumeName}' > {WS}/lesson-1.1/released.txt"""), + cmd(r"""k -n store delete po bookkeeper --now +k -n store delete pvc ledger +k get pv""", r"""The volume stays, `Released`.""")]), step("Change the default class", r"""Make `keep` the cluster's default class, and `standard` no longer the default. Two defaults would be ambiguous: set the annotation to `"true"` on one and `"false"` on the other.""", check=r""" eq "$(j sc keep '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" true \ && [ "$(j sc standard '{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]""", hint=r"""`kubectl annotate sc NAME storageclass.kubernetes.io/is-default-class=true --overwrite`, or `kubectl patch` with a `metadata.annotations` map.""", - show=r"""kubectl annotate sc standard storageclass.kubernetes.io/is-default-class=false --overwrite -kubectl annotate sc keep storageclass.kubernetes.io/is-default-class=true --overwrite -kubectl get sc"""), + show=[cmd(r"""k annotate sc standard storageclass.kubernetes.io/is-default-class=false --overwrite +k annotate sc keep storageclass.kubernetes.io/is-default-class=true --overwrite +k get sc""")]), ], "outro": r"""You read a class, made one that keeps its volumes, watched a claim wait for its Pod, kept a volume past its claim, and moved the default. @@ -172,29 +198,29 @@ The reader sees what the writer wrote: an emptyDir is shared by every container check=r"""eq "$($K -n vols exec pair -c reader -- cat /shared/msg 2>/dev/null)" hello \ && [ -n "$(j -n vols pod pair '{.spec.volumes[?(@.name=="shared")].emptyDir}')" ]""", hint=r"""Both containers list the same `volumeMounts` entry (`name: shared`); the Pod lists the volume once, as `emptyDir: {}`.""", - show=r"""kubectl -n vols apply -f - <<'EOF' -apiVersion: v1 -kind: Pod -metadata: - name: pair -spec: + show=[cmd(r"""k -n vols run pair --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo hello > /shared/msg && sleep 86400' > pair.yaml"""), + hand(r"""vim pair.yaml""", + r"""Name the container `writer` and give it the mount; add `reader` below it; add the volume: +``` containers: - name: writer image: busybox:1.36 - command: [sh, -c, 'echo hello > /shared/msg && sleep 86400'] + args: [sh, -c, 'echo hello > /shared/msg && sleep 86400'] volumeMounts: - {name: shared, mountPath: /shared} - name: reader image: busybox:1.36 - command: [sleep, '86400'] + args: [sleep, '86400'] volumeMounts: - {name: shared, mountPath: /shared} volumes: - - name: shared - emptyDir: {} -EOF -kubectl -n vols wait --for=condition=Ready pod/pair --timeout=120s -kubectl -n vols exec pair -c reader -- cat /shared/msg"""), + - {name: shared, emptyDir: {}} +```""", + run=r"""yq -i '.spec.containers[0].name = "writer" | .spec.containers[0].volumeMounts = [{"name": "shared", "mountPath": "/shared"}] + | .spec.containers += [{"name": "reader", "image": "busybox:1.36", "args": ["sleep", "86400"], "volumeMounts": [{"name": "shared", "mountPath": "/shared"}]}] + | .spec.volumes = [{"name": "shared", "emptyDir": {}}]' pair.yaml"""), + cmd(r"""k -n vols apply -f pair.yaml +k -n vols exec pair -c reader -- cat /shared/msg""", r"""Once it runs: `hello`.""")]), step("Scratch space in memory", r"""Create Pod `scratch` in `vols` (`busybox:1.36`, `sleep 86400`) with an emptyDir at `/scratch` that is held in **memory**, at most `64Mi`. @@ -203,24 +229,21 @@ Inside the Pod, `/proc/mounts` shows the difference: `/scratch` is a `tmpfs`. Wh eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.medium}')" Memory && eq "$(j -n vols pod scratch '{.spec.volumes[0].emptyDir.sizeLimit}')" 64Mi \ && $K -n vols exec scratch -- grep -q ' /scratch tmpfs ' /proc/mounts 2>/dev/null""", hint=r"""`emptyDir: {medium: Memory, sizeLimit: 64Mi}`.""", - show=r"""kubectl -n vols apply -f - <<'EOF' -apiVersion: v1 -kind: Pod -metadata: - name: scratch -spec: - containers: - - name: scratch - image: busybox:1.36 - command: [sleep, '86400'] + show=[cmd(r"""k -n vols run scratch --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > scratch.yaml"""), + hand(r"""vim scratch.yaml""", + r"""Add the mount to the container, below `resources: {}`: +``` volumeMounts: - {name: tmp, mountPath: /scratch} +``` +and the volume to the Pod's `spec:`, beside `restartPolicy:`: +``` volumes: - - name: tmp - emptyDir: {medium: Memory, sizeLimit: 64Mi} -EOF -kubectl -n vols wait --for=condition=Ready pod/scratch --timeout=120s -kubectl -n vols exec scratch -- grep /scratch /proc/mounts"""), + - {name: tmp, emptyDir: {medium: Memory, sizeLimit: 64Mi}} +```""", + run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "tmp", "mountPath": "/scratch"}] | .spec.volumes = [{"name": "tmp", "emptyDir": {"medium": "Memory", "sizeLimit": "64Mi"}}]' scratch.yaml"""), + cmd(r"""k -n vols apply -f scratch.yaml +k -n vols exec scratch -- grep /scratch /proc/mounts""", r"""Once it runs: a `tmpfs`.""")]), step("One directory from three sources", r"""Create Pod `bundle` in `vols` (`busybox:1.36`, `sleep 86400`) with **one** `projected` volume mounted at `/etc/bundle`, made of: @@ -233,18 +256,15 @@ Separate volumes would need three mount points; a projected volume puts them sid x() { $K -n vols exec bundle -- cat "/etc/bundle/$1" 2>/dev/null; } [ -n "$(j -n vols pod bundle '{.spec.volumes[0].projected}')" ] && eq "$(x mode)" prod && eq "$(x password)" s3cr3t && eq "$(x podname)" bundle""", hint=r"""`projected: {sources: [...]}` - each source is `configMap: {name: ...}`, `secret: {name: ...}` or `downwardAPI: {items: [{path: podname, fieldRef: {fieldPath: metadata.name}}]}`. `kubectl explain pod.spec.volumes.projected.sources` lists them.""", - show=r"""kubectl -n vols apply -f - <<'EOF' -apiVersion: v1 -kind: Pod -metadata: - name: bundle -spec: - containers: - - name: app - image: busybox:1.36 - command: [sleep, '86400'] + show=[cmd(r"""k -n vols run bundle --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > bundle.yaml"""), + hand(r"""vim bundle.yaml""", + r"""Add the mount to the container, below `resources: {}`: +``` volumeMounts: - {name: bundle, mountPath: /etc/bundle, readOnly: true} +``` +and the volume to the Pod's `spec:`, beside `restartPolicy:` (the docs page *Projected Volumes* has an example): +``` volumes: - name: bundle projected: @@ -255,9 +275,11 @@ spec: items: - path: podname fieldRef: {fieldPath: metadata.name} -EOF -kubectl -n vols wait --for=condition=Ready pod/bundle --timeout=120s -kubectl -n vols exec bundle -- ls /etc/bundle"""), +```""", + run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "bundle", "mountPath": "/etc/bundle", "readOnly": true}] + | .spec.volumes = [{"name": "bundle", "projected": {"sources": [{"configMap": {"name": "app-config"}}, {"secret": {"name": "app-secret"}}, {"downwardAPI": {"items": [{"path": "podname", "fieldRef": {"fieldPath": "metadata.name"}}]}}]}}]' bundle.yaml"""), + cmd(r"""k -n vols apply -f bundle.yaml +k -n vols exec bundle -- ls /etc/bundle""", r"""Once it runs: `mode`, `password` and `podname`.""")]), step("A volume for exactly one Pod", r"""Create PersistentVolume `archive`: `hostPath` `/srv/archive` (type `DirectoryOrCreate`), `200Mi`, access mode **`ReadWriteOncePod`**, reclaim policy `Retain`, and `storageClassName: ""`. @@ -268,7 +290,9 @@ The empty class matters on both sides. A claim with no class at all gets the def eq "$(j pv archive '{.spec.accessModes[0]} {.spec.persistentVolumeReclaimPolicy} {.spec.hostPath.path}')" "ReadWriteOncePod Retain /srv/archive" \ && eq "$(j -n vols pvc archive-claim '{.status.phase} {.spec.volumeName}')" "Bound archive" """, hint=r"""`storageClassName: ""` is spelled out, with the quotes, under `spec` of both objects. A PersistentVolume is cluster-wide: no namespace.""", - show=r"""kubectl apply -f - <<'EOF' + show=[hand(r"""vim archive.yaml""", + r"""No `kubectl create` for these: paste the PersistentVolume and the claim from the docs page *Configure a Pod to Use a PersistentVolume for Storage*, and make them: +``` apiVersion: v1 kind: PersistentVolume metadata: @@ -290,9 +314,33 @@ spec: accessModes: [ReadWriteOncePod] resources: requests: {storage: 200Mi} -EOF -kubectl get pv archive -kubectl -n vols get pvc archive-claim"""), +```""", + run=r"""cat > archive.yaml <<'EOF' +apiVersion: v1 +kind: PersistentVolume +metadata: + name: archive +spec: + capacity: {storage: 200Mi} + accessModes: [ReadWriteOncePod] + persistentVolumeReclaimPolicy: Retain + storageClassName: "" + hostPath: {path: /srv/archive, type: DirectoryOrCreate} +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: archive-claim + namespace: vols +spec: + storageClassName: "" + accessModes: [ReadWriteOncePod] + resources: + requests: {storage: 200Mi} +EOF"""), + cmd(r"""k apply -f archive.yaml +k get pv archive +k -n vols get pvc archive-claim""", r"""Both `Bound`, to each other.""")]), step("Hand a Released volume to a new claim", r"""Delete claim `archive-claim`. The volume becomes `Released`: it still remembers its old claim in `spec.claimRef`, and nothing binds it while it does. @@ -300,10 +348,14 @@ Make it `Available` again by removing `spec.claimRef` from the PV, then create c check=r""" [ -z "$(j -n vols pvc archive-claim '{.metadata.name}')" ] && eq "$(j -n vols pvc archive-claim-2 '{.status.phase} {.spec.volumeName}')" "Bound archive" """, hint=r"""`kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'` - or `kubectl edit pv archive` and delete the `claimRef:` block.""", - show=r"""kubectl -n vols delete pvc archive-claim -kubectl get pv archive -kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' -kubectl apply -f - <<'EOF' + show=[cmd(r"""k -n vols delete pvc archive-claim +k get pv archive""", r"""`Released`."""), + hand(r"""k edit pv archive""", + r"""Delete the whole `claimRef:` block, then `:wq`. The volume is `Available` again.""", + run=r"""kubectl patch pv archive --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' """), + hand(r"""vim claim2.yaml""", + r"""The claim of `archive.yaml`, named `archive-claim-2`: +``` apiVersion: v1 kind: PersistentVolumeClaim metadata: @@ -314,8 +366,21 @@ spec: accessModes: [ReadWriteOncePod] resources: requests: {storage: 200Mi} -EOF -kubectl get pv archive"""), +```""", + run=r"""cat > claim2.yaml <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: archive-claim-2 + namespace: vols +spec: + storageClassName: "" + accessModes: [ReadWriteOncePod] + resources: + requests: {storage: 200Mi} +EOF"""), + cmd(r"""k apply -f claim2.yaml +k get pv archive""")]), ], "outro": r"""You shared an emptyDir between containers, put one in memory, combined three sources in one projected volume, and took a `Retain` volume through `Bound`, `Released` and back. @@ -401,7 +466,9 @@ $K -n claims wait --for=condition=Ready pod/grower --timeout=180s >/dev/null Without the selector it would bind `pv-small`, the smallest volume that fits. With it, it binds `pv-large`.""", check=r"""eq "$(j -n claims pvc gold-claim '{.status.phase} {.spec.volumeName}')" "Bound pv-large" """, hint=r"""`spec.selector.matchLabels: {tier: gold}` on the claim.""", - show=r"""kubectl -n claims apply -f - <<'EOF' + show=[hand(r"""vim gold.yaml""", + r"""A claim with a `selector` (the docs page *Persistent Volumes* has one): +``` apiVersion: v1 kind: PersistentVolumeClaim metadata: @@ -413,14 +480,30 @@ spec: requests: {storage: 100Mi} selector: matchLabels: {tier: gold} -EOF -kubectl get pv"""), +```""", + run=r"""cat > gold.yaml <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: gold-claim +spec: + storageClassName: manual + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 100Mi} + selector: + matchLabels: {tier: gold} +EOF"""), + cmd(r"""k -n claims apply -f gold.yaml +k get pv""", r"""`gold-claim` bound to `pv-large`.""")]), step("Bind by name, and use it", r"""Create claim `pinned` in `claims` that binds `pv-small` **by name** (class `manual`, `ReadWriteOnce`, `100Mi`), and Pod `user` (`busybox:1.36`, `sleep 86400`) that mounts it at `/data`.""", check=r"""eq "$(j -n claims pvc pinned '{.status.phase} {.spec.volumeName}')" "Bound pv-small" && eq "$(j -n claims pod user '{.status.phase}')" Running \ && eq "$(j -n claims pod user '{.spec.volumes[0].persistentVolumeClaim.claimName}')" pinned""", hint=r"""`spec.volumeName: pv-small` on the claim. The volume's class and modes must still match the claim's.""", - show=r"""kubectl -n claims apply -f - <<'EOF' + show=[hand(r"""vim pinned.yaml""", + r"""A claim that names its volume: +``` apiVersion: v1 kind: PersistentVolumeClaim metadata: @@ -431,24 +514,35 @@ spec: accessModes: [ReadWriteOnce] resources: requests: {storage: 100Mi} ---- +```""", + run=r"""cat > pinned.yaml <<'EOF' apiVersion: v1 -kind: Pod +kind: PersistentVolumeClaim metadata: - name: user + name: pinned spec: - containers: - - name: user - image: busybox:1.36 - command: [sleep, '86400'] + storageClassName: manual + volumeName: pv-small + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 100Mi} +EOF"""), + cmd(r"""k -n claims apply -f pinned.yaml +k -n claims run user --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > user.yaml"""), + hand(r"""vim user.yaml""", + r"""Add the mount to the container, below `resources: {}`: +``` volumeMounts: - {name: data, mountPath: /data} +``` +and the volume to the Pod's `spec:`, beside `restartPolicy:`: +``` volumes: - - name: data - persistentVolumeClaim: {claimName: pinned} -EOF -kubectl -n claims wait --for=condition=Ready pod/user --timeout=120s -kubectl -n claims get pvc"""), + - {name: data, persistentVolumeClaim: {claimName: pinned}} +```""", + run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "data", "mountPath": "/data"}] | .spec.volumes = [{"name": "data", "persistentVolumeClaim": {"claimName": "pinned"}}]' user.yaml"""), + cmd(r"""k -n claims apply -f user.yaml +k -n claims get pvc,po""")]), step("A claim in use does not go", r"""Delete claim `pinned` without waiting (`--wait=false`). It does not disappear: `kubectl -n claims get pvc` shows it `Terminating`, held by a **finalizer**. @@ -457,17 +551,19 @@ Write that finalizer's name to `{WS}/lesson-1.3/finalizer.txt`.""", [ -n "$(j -n claims pvc pinned '{.metadata.deletionTimestamp}')" ] \ && eq "$(tr -d ' \r\n' < "$WS/lesson-1.3/finalizer.txt" 2>/dev/null)" kubernetes.io/pvc-protection""", hint=r"""`kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers}'`. Without `--wait=false`, `kubectl delete` would sit there until the claim is really gone.""", - show=r"""kubectl -n claims delete pvc pinned --wait=false -kubectl -n claims get pvc pinned -kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers[0]}' > {WS}/lesson-1.3/finalizer.txt -cat {WS}/lesson-1.3/finalizer.txt"""), + show=[cmd(r"""k -n claims delete pvc pinned --wait=false +k -n claims get pvc pinned +k -n claims get pvc pinned -o yaml | grep -A1 finalizers"""), + hand(r"""echo FINALIZER > {WS}/lesson-1.3/finalizer.txt""", + r"""`FINALIZER`: the name under `finalizers:`.""", + run=r"""kubectl -n claims get pvc pinned -o jsonpath='{.metadata.finalizers[0]}' > {WS}/lesson-1.3/finalizer.txt""")]), step("Let it go", r"""Delete Pod `user`. With no Pod using it any more, the protection lifts and claim `pinned` is removed. `pv-small` (reclaim policy `Retain`) is now `Released`.""", check=r"""[ -z "$(j -n claims pod user '{.metadata.name}')" ] && [ -z "$(j -n claims pvc pinned '{.metadata.name}')" ] && eq "$(j pv pv-small '{.status.phase}')" Released""", hint=r"""`kubectl -n claims delete pod user`, then `kubectl -n claims get pvc` and `kubectl get pv`.""", - show=r"""kubectl -n claims delete pod user --grace-period=1 -kubectl -n claims get pvc -kubectl get pv pv-small"""), + show=[cmd(r"""k -n claims delete po user --now +k -n claims get pvc +k get pv pv-small""", r"""The claim is gone; `pv-small` is `Released`.""")]), step("Claims made by a StatefulSet", r"""Create a StatefulSet `db` in `claims` with 2 replicas (`busybox:1.36`, `sleep 86400`) and its headless Service `db`. Give it a `volumeClaimTemplates` entry `data` (`ReadWriteOnce`, `50Mi`, class `standard`) mounted at `/var/lib/db`, and a `persistentVolumeClaimRetentionPolicy` that **deletes** a replica's claim when the StatefulSet is scaled down and **retains** the claims when the StatefulSet is deleted. @@ -476,7 +572,9 @@ Each replica gets a claim of its own: `data-db-0`, `data-db-1`.""", eq "$(j -n claims sts db '{.spec.persistentVolumeClaimRetentionPolicy.whenScaled} {.spec.persistentVolumeClaimRetentionPolicy.whenDeleted}')" "Delete Retain" || exit 1 eq "$(j -n claims sts db '{.status.readyReplicas}')" 2 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound && eq "$(j -n claims pvc data-db-1 '{.status.phase}')" Bound""", hint=r"""`kubectl create deploy db --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400` is a start: change the kind, add `serviceName: db`, drop `strategy`, add `volumeClaimTemplates` beside `template`. The Service: `clusterIP: None`. `kubectl explain sts.spec.persistentVolumeClaimRetentionPolicy`.""", - show=r"""kubectl -n claims apply -f - <<'EOF' + show=[hand(r"""vim db.yaml""", + r"""No `kubectl create` for a StatefulSet: paste the example from the docs page *StatefulSets* (it comes with its headless Service), and make it: +``` apiVersion: v1 kind: Service metadata: @@ -517,9 +615,52 @@ spec: accessModes: [ReadWriteOnce] resources: requests: {storage: 50Mi} -EOF -kubectl -n claims rollout status sts db --timeout=180s -kubectl -n claims get pvc"""), +```""", + run=r"""cat > db.yaml <<'EOF' +apiVersion: v1 +kind: Service +metadata: + name: db +spec: + clusterIP: None + selector: {app: db} + ports: + - port: 5432 +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: db +spec: + serviceName: db + replicas: 2 + selector: + matchLabels: {app: db} + persistentVolumeClaimRetentionPolicy: + whenScaled: Delete + whenDeleted: Retain + template: + metadata: + labels: {app: db} + spec: + containers: + - name: db + image: busybox:1.36 + command: [sleep, '86400'] + volumeMounts: + - {name: data, mountPath: /var/lib/db} + volumeClaimTemplates: + - metadata: + name: data + spec: + storageClassName: standard + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 50Mi} +EOF"""), + cmd(r"""k -n claims apply -f db.yaml +k -n claims rollout status sts db +k -n claims get pvc""", r"""`data-db-0` and `data-db-1`.""")]), step("Scale down, and the claim goes with the replica", r"""Scale `db` to 1 replica. Pod `db-1` goes, and because `whenScaled` is `Delete`, so does `data-db-1`. `data-db-0` stays. @@ -528,23 +669,26 @@ The default for both fields is `Retain`: without the policy, `data-db-1` would w eq "$(j -n claims sts db '{.spec.replicas}')" 1 && eq "$(j -n claims pvc data-db-0 '{.status.phase}')" Bound || exit 1 for i in $(seq 1 10); do [ -z "$(j -n claims pvc data-db-1 '{.metadata.name}')" ] && exit 0; sleep 3; done; exit 1""", hint=r"""`kubectl -n claims scale sts db --replicas=1`, then `kubectl -n claims get pvc -w`.""", - show=r"""kubectl -n claims scale sts db --replicas=1 -kubectl -n claims rollout status sts db --timeout=120s -kubectl -n claims get pvc"""), + show=[cmd(r"""k -n claims scale sts db --replicas=1 +k -n claims rollout status sts db +k -n claims get pvc""", r"""Again in a moment: `data-db-1` is gone, `data-db-0` stays.""")]), step("Grow a claim in use", r"""Claim `growing` is full. Its class allows expansion (`kubectl get sc csi-hostpath-sc`: `ALLOWVOLUMEEXPANSION true`), so make it **50Mi** - while Pod `grower` keeps running. Raise the request; the driver does the rest, and `status.capacity` follows.""", check=r"""eq "$(j -n claims pvc growing '{.spec.resources.requests.storage} {.status.capacity.storage}')" "50Mi 50Mi" && eq "$(j -n claims pod grower '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """, hint=r"""`kubectl -n claims edit pvc growing`, or `kubectl patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}'`. A claim never shrinks; with `standard` (local-path) the same patch is refused.""", - show=r"""kubectl get sc csi-hostpath-sc -kubectl -n claims patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}' -sleep 10 -kubectl -n claims get pvc growing"""), + show=[cmd(r"""k get sc csi-hostpath-sc"""), + hand(r"""k -n claims edit pvc growing""", + r"""Make `storage:` under `spec.resources.requests` read `50Mi`, then `:wq`.""", + run=r"""kubectl -n claims patch pvc growing -p '{"spec":{"resources":{"requests":{"storage":"50Mi"}}}}' """), + cmd(r"""k -n claims get pvc growing""", r"""Again until `CAPACITY` shows `50Mi`.""")]), step("Snapshot it, and restore the snapshot", r"""Take VolumeSnapshot `growing-snap` of claim `growing` (snapshot class `csi-hostpath-snapclass`), wait until it is ready to use, then create claim `restored` (`csi-hostpath-sc`, 50Mi) **from** the snapshot: its `dataSource` names the VolumeSnapshot. The new volume holds what `growing` held when the snapshot was taken.""", check=r""" eq "$(j -n claims volumesnapshot growing-snap '{.spec.source.persistentVolumeClaimName} {.status.readyToUse}')" "growing true" && eq "$(j -n claims pvc restored '{.spec.dataSource.kind}/{.spec.dataSource.name} {.status.phase}')" "VolumeSnapshot/growing-snap Bound" """, hint=r"""`apiVersion: snapshot.storage.k8s.io/v1`, `kind: VolumeSnapshot`, `spec: {volumeSnapshotClassName: ..., source: {persistentVolumeClaimName: growing}}`. The claim: `dataSource: {apiGroup: snapshot.storage.k8s.io, kind: VolumeSnapshot, name: growing-snap}`. A clone is the same with `dataSource: {kind: PersistentVolumeClaim, name: growing}`.""", - show=r"""kubectl -n claims apply -f - <<'EOF' + show=[hand(r"""vim snap.yaml""", + r"""Paste a VolumeSnapshot from the docs page *Volume Snapshots*, and make it: +``` apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: @@ -553,9 +697,24 @@ spec: volumeSnapshotClassName: csi-hostpath-snapclass source: persistentVolumeClaimName: growing -EOF -kubectl -n claims wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/growing-snap --timeout=120s -kubectl -n claims apply -f - <<'EOF' +```""", + run=r"""cat > snap.yaml <<'EOF' +apiVersion: snapshot.storage.k8s.io/v1 +kind: VolumeSnapshot +metadata: + name: growing-snap +spec: + volumeSnapshotClassName: csi-hostpath-snapclass + source: + persistentVolumeClaimName: growing +EOF"""), + cmd(r"""k -n claims apply -f snap.yaml"""), + hand(r"""k -n claims get volumesnapshot""", + r"""Again until `READYTOUSE` is `true`.""", + run=r"""kubectl -n claims wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/growing-snap --timeout=120s"""), + hand(r"""vim restored.yaml""", + r"""A claim whose `dataSource` is the snapshot (the same docs page, *Provisioning Volumes from Snapshots*): +``` apiVersion: v1 kind: PersistentVolumeClaim metadata: @@ -569,8 +728,24 @@ spec: apiGroup: snapshot.storage.k8s.io kind: VolumeSnapshot name: growing-snap -EOF -kubectl -n claims get volumesnapshot,pvc"""), +```""", + run=r"""cat > restored.yaml <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: restored +spec: + storageClassName: csi-hostpath-sc + accessModes: [ReadWriteOnce] + resources: + requests: {storage: 50Mi} + dataSource: + apiGroup: snapshot.storage.k8s.io + kind: VolumeSnapshot + name: growing-snap +EOF"""), + cmd(r"""k -n claims apply -f restored.yaml +k -n claims get volumesnapshot,pvc""")]), ], "outro": r"""You steered binding with a selector and with a volume's name, watched claim protection hold a claim until its Pod went, let a StatefulSet's claims follow its replicas, and grew, snapshotted and restored a claim on a CSI class. diff --git a/cka/lessons_tshoot.py b/cka/lessons_tshoot.py index 2b2380b..28338c2 100644 --- a/cka/lessons_tshoot.py +++ b/cka/lessons_tshoot.py @@ -1,5 +1,5 @@ """CKA lessons of the Troubleshooting domain (2.1-2.5). See cka/lessons.py for what a lesson is.""" -from sim.bank import NS, step +from sim.bank import NS, step, cmd, hand # ---- 2.1 clusters and nodes ------------------------------------------------------------------- L21 = { @@ -35,9 +35,11 @@ echo "$W1 is still Ready a minute after its kubelet was stopped" >&2; exit 1 Then look at `kubectl describe node` for it: under **Conditions** every row says `Kubelet stopped posting node status` - the control plane has not heard from the kubelet, so it knows nothing more. The answer is on the node.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/node.txt" 2>/dev/null)" "$W1" """, hint=r"""`kubectl get nodes`. `--no-headers` and `awk '$2 != "Ready" {print $1}'` give just the name.""", - show=r"""kubectl get nodes -kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt -kubectl describe node {W1} | grep -A8 Conditions"""), + show=[cmd(r"""k get no +k describe no {W1} | grep -A8 Conditions"""), + hand(r"""echo NODE > {WS}/lesson-2.1/node.txt""", + r"""`NODE`: the one that is not `Ready`.""", + run=r"""kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt""")]), step("Bring the kubelet back - for good", r"""On `{W1}` (`ssh {W1}`, then `sudo -i`), look at the kubelet: `systemctl status kubelet` says `inactive (dead)` and `disabled`. Start it, and enable it so that it survives a reboot - a fix that is gone after the next reboot is not a fix. @@ -46,12 +48,15 @@ Within a few seconds the node is `Ready` again.""", nx "$W1" 'systemctl is-active -q kubelet && systemctl is-enabled -q kubelet' /dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["cgroupDriver"])') [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/cgroup.txt" 2>/dev/null)" "$want" """, hint=r"""`configz` returns JSON: `| jq -r .kubeletconfig.cgroupDriver`. It must match containerd's `SystemdCgroup` setting, or Pods fail to start.""", - show=r"""ssh -n {W1} 'ps -o args= -C kubelet | tr " " "\n" | grep -- --config; sudo grep -n cgroupDriver /var/lib/kubelet/config.yaml' -kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt -cat {WS}/lesson-2.1/cgroup.txt"""), + show=[hand(r"""ssh {W1} +sudo -i +ps -o args= -C kubelet +grep cgroupDriver /var/lib/kubelet/config.yaml""", + r"""`--config=` names the kubelet's file. Then `exit` twice.""", + run=r"""ssh -n {W1} 'ps -o args= -C kubelet; sudo grep cgroupDriver /var/lib/kubelet/config.yaml' """), + cmd(r"""k get --raw /api/v1/nodes/{W1}/proxy/configz | tr , '\n' | grep cgroupDriver""", r"""The live value."""), + hand(r"""echo VALUE > {WS}/lesson-2.1/cgroup.txt""", + r"""`VALUE`: `systemd` or `cgroupfs`.""", + run=r"""kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt""")]), step("Set the eviction thresholds", r"""Look at `evictionHard` in `/var/lib/kubelet/config.yaml` on `{W1}`: kind sets every threshold to `0%`, which means "never evict" - fine for a test cluster, not for a real one. @@ -74,15 +86,26 @@ e = json.load(sys.stdin)["kubeletconfig"].get("evictionHard") or {} sys.exit(0 if e.get("memory.available") == "100Mi" and e.get("nodefs.available") == "10%" else 1)' || exit 1 eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""", hint=r"""Edit the file with `vim`, or `sed`: change the `nodefs.available` line and add a `memory.available` line inside the `evictionHard:` block (two spaces of indent). A kubelet that does not come back after the restart: `journalctl -u kubelet -n 20` shows the line it could not parse.""", - show=r"""ssh {W1} sudo bash <<'EOF' + show=[hand(r"""ssh {W1} +sudo -i +vim /var/lib/kubelet/config.yaml""", + r"""Under `evictionHard:`, set `nodefs.available` to `10%` and add `memory.available`, indented like the others: +``` +evictionHard: + memory.available: 100Mi + nodefs.available: 10% +``` +then restart the kubelet, and `exit` twice: +``` +systemctl restart kubelet +```""", + run=r"""ssh {W1} sudo bash <<'EOF' f=/var/lib/kubelet/config.yaml -grep -n -A4 '^evictionHard:' $f sed -i 's/^ nodefs.available: .*/ nodefs.available: 10%/' $f sed -i '/^evictionHard:/a\ memory.available: 100Mi' $f -grep -n -A5 '^evictionHard:' $f systemctl restart kubelet -EOF -kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq .kubeletconfig.evictionHard"""), +EOF"""), + cmd(r"""k get --raw /api/v1/nodes/{W1}/proxy/configz | tr , '\n' | grep -A4 evictionHard""", r"""Again if the kubelet is not back yet.""")]), step("Read the pressure conditions", r"""The kubelet reports the result of those thresholds as node conditions. Write the status of `{W1}`'s three pressure conditions to `{WS}/lesson-2.1/pressure.txt`, one per line, as `TYPE STATUS` - for example `MemoryPressure False`. @@ -91,8 +114,15 @@ kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq .kubeletconfig.evictionH want=$(for c in MemoryPressure DiskPressure PIDPressure; do echo "$c $(j node "$W1" "{.status.conditions[?(@.type==\"$c\")].status}")"; done | sort) eq "$(tr -s ' \t' ' ' < "$WS/lesson-2.1/pressure.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """, hint=r"""`kubectl describe node {W1}` lists them under Conditions. As a list: `-o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}'`, then keep the three `*Pressure` lines.""", - show=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt -cat {WS}/lesson-2.1/pressure.txt"""), + show=[cmd(r"""k describe no {W1} | grep Pressure"""), + hand(r"""vim {WS}/lesson-2.1/pressure.txt""", + r"""Three lines, `TYPE STATUS`, with the statuses you read: +``` +MemoryPressure False +DiskPressure False +PIDPressure False +```""", + run=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt""")]), ], "outro": r"""You found a `NotReady` node, brought its kubelet back for good, read its configuration from the file and from the running kubelet, set eviction thresholds and read what the kubelet reports about them. @@ -144,10 +174,12 @@ Two things in Namespace `ops` do not work: Deployment `queue` has no Pods, and P `kubectl -n kube-system get pods` shows them; on the control-plane node, `sudo crictl ps -a` shows their containers, and `ls /etc/kubernetes/manifests` what the kubelet is told to run.""", check=r"""eq "$(tr -d ' \r' < "$WS/lesson-2.2/down.txt" 2>/dev/null | grep . | sort | tr '\n' ' ')" "kube-controller-manager kube-scheduler " """, hint=r"""A Pod that is not `Running` with all its containers ready. The Pod names end in the node's name; the component is the part before it.""", - show=r"""kubectl -n kube-system get pods + show=[cmd(r"""k -n kube-system get po ls /etc/kubernetes/manifests -sudo crictl ps -a | grep kube- -printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt"""), +sudo crictl ps -a | grep kube-"""), + hand(r"""vim {WS}/lesson-2.2/down.txt""", + r"""The components that do not run, one per line.""", + run=r"""printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt""")]), step("Repair the scheduler", r"""Find out why `kube-scheduler` does not start, and fix its manifest. `kubectl -n kube-system describe pod kube-scheduler-{CP}` shows the error in its events, and so do `sudo crictl ps -a --name kube-scheduler` and `sudo journalctl -u kubelet`. @@ -156,30 +188,33 @@ The kubelet notices the saved file on its own: no restart, no `kubectl apply`. O nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml && [ -n "$(crictl ps --name kube-scheduler -q)" ]' /dev/null && grep -q 'readyz check passed' "$WS/lesson-2.2/readyz.txt" """, hint=r"""`kubectl get --raw` sends a GET to any API path, with your credentials. Quote the path: `?` means something to the shell.""", - show=r"""kubectl get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt | tail -3 -kubectl get --raw '/livez'; echo"""), + show=[cmd(r"""k get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt +k get --raw /livez""")]), ], "outro": r"""You read the control plane from both sides - `kubectl` and the node's container runtime - and repaired two static Pods by editing their manifests in place. @@ -261,38 +296,46 @@ echo "kubectl top shows no metrics for the lesson's Pods" >&2; exit 1 r"""Write the name of the Pod in `metrics` that uses the most memory to `{WS}/lesson-2.3/memory.txt`. `kubectl top` can sort for you.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/memory.txt" 2>/dev/null)" hoarder""", hint=r"""`kubectl top pod -n metrics --sort-by=memory`. `--no-headers | head -1 | awk '{print $1}'` keeps just the name.""", - show=r"""kubectl top pod -n metrics --sort-by=memory -kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt"""), + show=[cmd(r"""k top po -n metrics --sort-by=memory"""), + hand(r"""echo POD > {WS}/lesson-2.3/memory.txt""", + r"""`POD`: the first one.""", + run=r"""kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt""")]), step("The busiest Pod in the cluster", r"""Across **all** Namespaces, which Pod uses the most CPU? Write it as `NAMESPACE/NAME` to `{WS}/lesson-2.3/cpu.txt`.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/cpu.txt" 2>/dev/null)" metrics/burner""", hint=r"""`kubectl top pod -A --sort-by=cpu`; with `-A` the first column is the Namespace.""", - show=r"""kubectl top pod -A --sort-by=cpu | head -5 -kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt"""), + show=[cmd(r"""k top po -A --sort-by=cpu | head -5"""), + hand(r"""echo NAMESPACE/POD > {WS}/lesson-2.3/cpu.txt""", + r"""From the first line.""", + run=r"""kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt""")]), step("Inside one Pod", r"""`kubectl top pod` adds up a Pod's containers. Which container of Pod `duo` uses the memory? Write its name to `{WS}/lesson-2.3/container.txt`.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/container.txt" 2>/dev/null)" cache""", hint=r"""`--containers` splits the Pod into its containers.""", - show=r"""kubectl top pod duo -n metrics --containers -kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt"""), + show=[cmd(r"""k top po duo -n metrics --containers"""), + hand(r"""echo CONTAINER > {WS}/lesson-2.3/container.txt""", + r"""The container with the most `MEMORY(bytes)`.""", + run=r"""kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt""")]), step("Usage against what is reserved", r"""`kubectl top node` shows use; `kubectl describe node {W1}`, under **Allocated resources**, shows what Pods have **requested** - which is what decides whether another Pod fits. Write the memory that `{W1}` can give to Pods - its **allocatable** memory, as the API reports it (for example `8012345Ki`) - to `{WS}/lesson-2.3/allocatable.txt`.""", check=r"""want=$(j node "$W1" '{.status.allocatable.memory}'); [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/allocatable.txt" 2>/dev/null)" "$want" """, hint=r"""Allocatable is capacity minus what the node keeps for itself and for the eviction threshold: `.status.allocatable.memory`.""", - show=r"""kubectl top node -kubectl describe node {W1} | grep -A9 'Allocated resources' -kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt"""), + show=[cmd(r"""k top no +k describe no {W1} | grep -A6 -e Allocatable: -e 'Allocated resources' """, r"""`memory:` under `Allocatable:` is what `{W1}` can give to Pods, for example `8012345Ki`."""), + hand(r"""echo MEMORY > {WS}/lesson-2.3/allocatable.txt""", + r"""`MEMORY`: that value, as it stands.""", + run=r"""kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt""")]), step("The metrics API itself", r"""`kubectl top` only reads an API. Read it yourself: save what `/apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder` returns in `{WS}/lesson-2.3/raw.json`. If this path answers `NotFound` or `ServiceUnavailable` on some cluster, `kubectl top` fails there too: look at the `metrics-server` Deployment in `kube-system`, and at `kubectl get apiservice v1beta1.metrics.k8s.io`.""", check=r"""python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); sys.exit(0 if d.get("kind")=="PodMetrics" and d["metadata"]["name"]=="hoarder" and d.get("containers") else 1)' "$WS/lesson-2.3/raw.json" 2>/dev/null""", hint=r"""`kubectl get --raw PATH`.""", - show=r"""kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json -jq '.containers' {WS}/lesson-2.3/raw.json -kubectl get apiservice v1beta1.metrics.k8s.io"""), + show=[cmd(r"""k get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json +cat {WS}/lesson-2.3/raw.json +k get apiservice v1beta1.metrics.k8s.io""")]), ], "outro": r"""You ranked Pods by memory and CPU, looked inside a Pod, compared use with what a node can give, and read the metrics API that `kubectl top` uses. @@ -361,16 +404,17 @@ echo "flaky did not restart" >&2; exit 1 Save those in `{WS}/lesson-2.4/previous.log`.""", check=r"""grep -q 'FATAL config /etc/flaky.conf not found' "$WS/lesson-2.4/previous.log" 2>/dev/null && ! grep -q 'second attempt' "$WS/lesson-2.4/previous.log" """, hint=r"""`kubectl logs --previous` (or `-p`).""", - show=r"""kubectl -n logs logs flaky -kubectl -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log -cat {WS}/lesson-2.4/previous.log"""), + show=[cmd(r"""k -n logs logs flaky +k -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log +cat {WS}/lesson-2.4/previous.log""")]), step("The one replica that complains", r"""Pods `api-1`, `api-2` and `api-3` all carry `app=api`. One of them logs errors. Read all three at once, find it, and write its name to `{WS}/lesson-2.4/erroring.txt`.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/erroring.txt" 2>/dev/null)" api-2""", hint=r"""`kubectl logs -l app=api --prefix` puts `[pod/NAME/CONTAINER]` before each line. With a selector, kubectl shows only the last 10 lines of each Pod unless you say `--tail`.""", - show=r"""kubectl -n logs logs -l app=api --prefix --tail=2 -kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt -cat {WS}/lesson-2.4/erroring.txt"""), + show=[cmd(r"""k -n logs logs -l app=api --prefix --tail=20 | grep ERROR""", r"""`[pod/NAME/...]` at the start of the line names the Pod."""), + hand(r"""echo POD > {WS}/lesson-2.4/erroring.txt""", + r"""`POD`: that name.""", + run=r"""kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt""")]), step("The log file on the node", r"""The node keeps one file per container instance. Write the full path of the file that holds the output of `flaky`'s **current** instance to `{WS}/lesson-2.4/logfile.txt`. @@ -380,11 +424,15 @@ p=$(tr -d ' \r\n' < "$WS/lesson-2.4/logfile.txt" 2>/dev/null) uid=$(j -n logs pod flaky '{.metadata.uid}'); n=$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}'); node=$(j -n logs pod flaky '{.spec.nodeName}') eq "$p" "/var/log/pods/logs_flaky_$uid/flaky/$n.log" && nx "$node" "grep -q 'second attempt' '$p'" {WS}/lesson-2.4/logfile.txt -ssh -n {W1} "sudo cat $(cat {WS}/lesson-2.4/logfile.txt)"""), + show=[cmd(r"""k -n logs get po flaky -o wide +k -n logs get po flaky -o yaml | grep uid:""", r"""Its node, and its UID."""), + hand(r"""ssh {W1} sudo ls /var/log/pods/""", + r"""`logs_flaky_UID` holds `flaky/`, one file per instance: `0.log` the first, `1.log` the current one.""", + run=r"""ssh -n {W1} sudo ls /var/log/pods/"""), + hand(r"""echo /var/log/pods/logs_flaky_UID/flaky/1.log > {WS}/lesson-2.4/logfile.txt""", + r"""With the Pod's UID.""", + run=r"""uid=$(kubectl -n logs get pod flaky -o jsonpath='{.metadata.uid}') +echo "/var/log/pods/logs_flaky_$uid/flaky/1.log" > {WS}/lesson-2.4/logfile.txt""")]), step("Debug a container that has no shell", r"""`kubectl -n logs exec quiet -- sh` fails: the `pause` image has nothing but its one program. Add an ephemeral container to the running Pod instead: name `dbg`, image `busybox:1.36`, **targeting** container `quiet` so that it sees its processes, running `sleep 3600`. @@ -393,10 +441,13 @@ Then run `ps` in `dbg` and write the command of the target's process - the first eq "$(j -n logs pod quiet '{.spec.ephemeralContainers[0].name} {.spec.ephemeralContainers[0].targetContainerName}')" "dbg quiet" || exit 1 grep -qx '/pause' <(tr -d ' \r' < "$WS/lesson-2.4/target.txt" 2>/dev/null)""", hint=r"""`kubectl debug POD --image=IMAGE --target=CONTAINER -c NAME -- COMMAND`. Interactively you would add `-it` and use the shell straight away; without it, `kubectl exec POD -c dbg -- ps` reaches it once it runs. Ephemeral containers cannot be removed - the Pod keeps them until it goes.""", - show=r"""kubectl -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600 -for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done -kubectl -n logs exec quiet -c dbg -- ps -kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt"""), + show=[cmd(r"""k -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600"""), + hand(r"""k -n logs exec quiet -c dbg -- ps""", + r"""Again if `dbg` is not up yet. PID 1 is the target's process.""", + run=r"""for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done"""), + hand(r"""echo COMMAND > {WS}/lesson-2.4/target.txt""", + r"""`COMMAND`: PID 1's command, as `ps` shows it.""", + run=r"""kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt""")]), step("Debug the node itself", r"""`kubectl debug node/{W1} --image=busybox:1.36` starts a Pod on that node with the node's whole filesystem mounted at `/host` - useful when you cannot `ssh` to a node, but can reach the API. @@ -406,11 +457,16 @@ want=$(nx "$W1" '. /etc/os-release; echo "$PRETTY_NAME"' /dev/null | sed 's/^PRETTY_NAME=//; s/ *$//')" "$want" \ && [ -n "$($K get pods -A -o name 2>/dev/null | grep node-debugger)" ]""", hint=r"""Without `-it`, give it a command and read its output with `kubectl logs`: the Pod is called `node-debugger-NODE-xxxxx`, in your current Namespace. Delete it afterwards.""", - show=r"""kubectl debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release -pod=$(kubectl get pods -o name | grep node-debugger | head -1) -kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s -kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt -cat {WS}/lesson-2.4/os.txt"""), + show=[cmd(r"""k debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release +k get po""", r"""The `node-debugger-...` Pod: once it is `Completed`, its log is the file."""), + hand(r"""k logs node-debugger-NAME""", + r"""`node-debugger-NAME`: its full name, from `get po`.""", + run=r"""pod=$(kubectl get pods -o name | grep node-debugger | head -1) +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s"""), + hand(r"""echo 'VALUE' > {WS}/lesson-2.4/os.txt""", + r"""`VALUE`: what `PRETTY_NAME=` says, without the double quotes.""", + run=r"""pod=$(kubectl get pods -o name | grep node-debugger | head -1) +kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt""")]), ], "outro": r"""You read a crashed instance's logs, searched several Pods at once, found the log file on the node, and debugged both a Pod without a shell and a node. @@ -477,19 +533,20 @@ eq "$(j -n shop svc web '{.spec.selector.app}')" web || exit 1 n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' | grep -c .) [ "$n" -eq 2 ]""", hint=r"""Compare `kubectl -n shop get svc web -o wide` (SELECTOR) with `kubectl -n shop get pods --show-labels`. `kubectl set selector` or `kubectl edit svc`.""", - show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web -kubectl -n shop get svc web -o wide -kubectl -n shop get pods --show-labels -kubectl -n shop set selector svc web app=web -kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""), + show=[cmd(r"""k -n shop get endpointslices -l kubernetes.io/service-name=web +k -n shop get svc web -o wide +k -n shop get po --show-labels +k -n shop set selector svc web app=web +k -n shop get endpointslices -l kubernetes.io/service-name=web""")]), step("The wrong port", r"""Now the Service has endpoints, but they point at port `8080` - and nginx listens on `80`. Make the Service send traffic to the port the container serves.""", check=r""" eq "$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{.items[0].ports[0].port}')" 80 && eq "$(j -n shop svc web '{.spec.ports[0].port}')" 80""", hint=r"""`targetPort` is the container's port; `port` is the Service's own. `kubectl -n shop edit svc web`, or a JSON patch on `/spec/ports/0/targetPort`.""", - show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web -kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]' -kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""), + show=[hand(r"""k -n shop edit svc web""", + r"""Make `targetPort: 8080` read `targetPort: 80`, then `:wq`.""", + run=r"""kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]' """), + cmd(r"""k -n shop get endpointslices -l kubernetes.io/service-name=web""")]), step("Let the right client in", r"""Still nothing: NetworkPolicy `lockdown` selects the `web` Pods and allows no ingress at all. @@ -500,7 +557,17 @@ ok() { $K -n shop exec "$1" -- wget -qO- -T 3 web >/dev/null 2>&1; } for i in 1 2 3 4 5; do ok client && break; sleep 3; done; ok client || exit 1 ! ok stranger""", hint=r"""Add an `ingress` rule to `lockdown`: `from: [{podSelector: {matchLabels: {app: client}}}]` with `ports: [{protocol: TCP, port: 80}]`. A NetworkPolicy's ports are the Pod's, not the Service's.""", - show=r"""kubectl -n shop apply -f - <<'EOF' + show=[hand(r"""k -n shop edit netpol lockdown""", + r"""Add the rule to its `spec:`, beside `policyTypes:`: +``` + ingress: + - from: + - podSelector: + matchLabels: {app: client} + ports: + - {protocol: TCP, port: 80} +```""", + run=r"""kubectl -n shop apply -f - <<'EOF' apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: @@ -515,9 +582,9 @@ spec: matchLabels: {app: client} ports: - {protocol: TCP, port: 80} -EOF -kubectl -n shop exec client -- wget -qO- -T3 web | head -4 -kubectl -n shop exec stranger -- wget -qO- -T3 web || echo refused"""), +EOF"""), + cmd(r"""k -n shop exec client -- wget -qO- -T3 web | head -4 +k -n shop exec stranger -- wget -qO- -T3 web""", r"""`client` gets the page; `stranger` times out.""")]), step("How kube-proxy routes it", r"""kube-proxy turned Service `web` into rules on every node. Find out how: its mode is in ConfigMap `kube-proxy` in `kube-system`. Write the mode to `{WS}/lesson-2.5/mode.txt`. @@ -529,11 +596,18 @@ ip=$(j -n shop svc web '{.spec.clusterIP}') want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" /dev/null)" "$want" """, hint=r"""`kubectl -n kube-system get cm kube-proxy -o yaml | grep mode`. On the node: `sudo iptables-save -t nat | grep CLUSTER-IP` - the line for the Service jumps (`-j`) to its chain. In `nftables` mode the rules are in `sudo nft list ruleset` instead.""", - show=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt -ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}') -sudo iptables-save -t nat | grep -- "-d $ip/32" -sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt -sudo iptables-save -t nat | grep "^-A $(cat {WS}/lesson-2.5/chain.txt)" """), + show=[cmd(r"""k -n kube-system get cm kube-proxy -o yaml | grep mode:"""), + hand(r"""echo MODE > {WS}/lesson-2.5/mode.txt""", + r"""`MODE`: its value.""", + run=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt"""), + cmd(r"""k -n shop get svc web""", r"""Its `CLUSTER-IP`:"""), + hand(r"""sudo iptables-save -t nat | grep IP""", + r"""`IP`: that address. The rule for it jumps (`-j`) to a `KUBE-SVC-...` chain.""", + run='ip=$(kubectl -n shop get svc web -o jsonpath=\'{.spec.clusterIP}\')\nsudo iptables-save -t nat | grep -- "-d $ip/32"'), + hand(r"""echo KUBE-SVC-NAME > {WS}/lesson-2.5/chain.txt""", + r"""The chain's full name.""", + run=r"""ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}') +sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt""")]), ], "outro": r"""You took a Service from "times out" to answering by fixing its selector, its target port and a NetworkPolicy, and followed it into kube-proxy's rules. diff --git a/cka/lessons_work.py b/cka/lessons_work.py index d77387c..e6e82c3 100644 --- a/cka/lessons_work.py +++ b/cka/lessons_work.py @@ -1,5 +1,5 @@ """CKA lessons of the Workloads and Scheduling domain (3.1-3.5). See cka/lessons.py for what a lesson is.""" -from sim.bank import NS, step +from sim.bank import NS, step, cmd, hand # ---- 3.1 rolling updates and rollbacks -------------------------------------------------------- L31 = { @@ -27,8 +27,10 @@ $K -n rollout rollout status deploy api --timeout=180s >/dev/null r"""Change the update strategy of `api` so that an update first adds **one** Pod and never takes one away before its replacement is ready: `maxSurge: 1`, `maxUnavailable: 0`.""", check=r"""eq "$(j -n rollout deploy api '{.spec.strategy.type} {.spec.strategy.rollingUpdate.maxSurge} {.spec.strategy.rollingUpdate.maxUnavailable}')" "RollingUpdate 1 0" """, hint=r"""`kubectl -n rollout edit deploy api`, or `kubectl patch` with `{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}`.""", - show=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' -kubectl -n rollout get deploy api -o jsonpath='{.spec.strategy}'; echo"""), + show=[hand(r"""k -n rollout edit deploy api""", + r"""Under `strategy.rollingUpdate`, set `maxSurge: 1` and `maxUnavailable: 0`, then save and quit (`:wq`).""", + run=r"""kubectl -n rollout patch deploy api -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' """), + cmd(r"""k -n rollout describe deploy api | grep -i strategy""")]), step("A release with a reason", r"""Update `api` to `nginx:1.27`, and record why: the annotation `kubernetes.io/change-cause` on the Deployment, set to `upgrade to 1.27`. `kubectl rollout history` shows it beside the revision.""", check=r""" @@ -36,10 +38,10 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin eq "$(j -n rollout deploy api '{.metadata.annotations.kubernetes\.io/change-cause}')" "upgrade to 1.27" || exit 1 eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas}')" "4 4" """, hint=r"""`kubectl set image deploy/api nginx=nginx:1.27` (the container is called `nginx`), then `kubectl annotate deploy api kubernetes.io/change-cause='...'`. The `--record` flag is gone.""", - show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27 -kubectl -n rollout annotate deploy api kubernetes.io/change-cause='upgrade to 1.27' --overwrite -kubectl -n rollout rollout status deploy api --timeout=180s -kubectl -n rollout rollout history deploy api"""), + show=[cmd(r"""k -n rollout set image deploy/api nginx=nginx:1.27 +k -n rollout annotate deploy api kubernetes.io/change-cause='upgrade to 1.27' --overwrite +k -n rollout rollout status deploy api +k -n rollout rollout history deploy api""")]), step("A release that cannot start", r"""Now a mistake: set the image of `api` to `nginx:1.27-broken`, a tag that does not exist. @@ -49,29 +51,27 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin [ "$(j -n rollout deploy api '{.status.availableReplicas}')" = 4 ] || exit 1 $K -n rollout get pods -l app=api --no-headers 2>/dev/null | grep -Eq 'ErrImagePull|ImagePullBackOff' """, hint=r"""`kubectl set image` again. `kubectl -n rollout get pods` shows the Pod that cannot pull; `kubectl -n rollout get rs` shows the two ReplicaSets side by side.""", - show=r"""kubectl -n rollout set image deploy/api nginx=nginx:1.27-broken -sleep 10 -kubectl -n rollout get pods -kubectl -n rollout get rs"""), + show=[cmd(r"""k -n rollout set image deploy/api nginx=nginx:1.27-broken +k -n rollout get po,rs""", r"""Run `get` again after a few seconds: one new Pod is stuck in `ErrImagePull`, and the four old ones still run.""")]), step("Undo", r"""Roll `api` back to the revision before - `nginx:1.27` - and let the rollout finish.""", check=r""" eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1 eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """, hint=r"""`kubectl rollout undo deploy/api`. It does not create a ReplicaSet: it scales the old one back up, and that revision gets a new number.""", - show=r"""kubectl -n rollout rollout undo deploy/api -kubectl -n rollout rollout status deploy api --timeout=180s -kubectl -n rollout rollout history deploy api"""), + show=[cmd(r"""k -n rollout rollout undo deploy/api +k -n rollout rollout status deploy api +k -n rollout rollout history deploy api""")]), step("Back to a revision by number", r"""Look at the history, then go back to the revision that ran **`nginx:1.26`**, by its number. `kubectl rollout history deploy api --revision=N` shows what a revision's template was.""", check=r""" eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" nginx:1.26 || exit 1 eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """, hint=r"""Revision 1 is where the Deployment started. `kubectl rollout undo deploy/api --to-revision=1`.""", - show=r"""kubectl -n rollout rollout history deploy api -kubectl -n rollout rollout history deploy api --revision=1 | grep Image -kubectl -n rollout rollout undo deploy/api --to-revision=1 -kubectl -n rollout rollout status deploy api --timeout=180s"""), + show=[cmd(r"""k -n rollout rollout history deploy api +k -n rollout rollout history deploy api --revision=1""", r"""Find the revision whose `Image:` is `nginx:1.26` - revision 1 here."""), + cmd(r"""k -n rollout rollout undo deploy/api --to-revision=1 +k -n rollout rollout status deploy api""")]), step("Two changes, one rollout", r"""Each `set` command starts a rollout of its own. To make two changes at once, **pause** the Deployment, make them - image `nginx:1.27`, and an environment variable `MODE=prod` in the container - then **resume**: one new revision, one rollout.""", check=r""" @@ -80,12 +80,12 @@ eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].image}')" ngin eq "$(j -n rollout deploy api '{.spec.template.spec.containers[0].env[?(@.name=="MODE")].value}')" prod || exit 1 eq "$(j -n rollout deploy api '{.status.updatedReplicas} {.status.availableReplicas} {.status.replicas}')" "4 4 4" """, hint=r"""`kubectl rollout pause deploy/api`, `kubectl set image ...`, `kubectl set env deploy/api MODE=prod`, `kubectl rollout resume deploy/api`.""", - show=r"""kubectl -n rollout rollout pause deploy/api -kubectl -n rollout set image deploy/api nginx=nginx:1.27 -kubectl -n rollout set env deploy/api MODE=prod -kubectl -n rollout rollout resume deploy/api -kubectl -n rollout rollout status deploy api --timeout=180s -kubectl -n rollout rollout history deploy api"""), + show=[cmd(r"""k -n rollout rollout pause deploy/api +k -n rollout set image deploy/api nginx=nginx:1.27 +k -n rollout set env deploy/api MODE=prod +k -n rollout rollout resume deploy/api +k -n rollout rollout status deploy api +k -n rollout rollout history deploy api""")]), ], "outro": r"""You tuned the strategy, released with a recorded cause, watched a broken release stall safely, undid it, went back to a numbered revision, and batched two changes into one rollout. @@ -132,15 +132,15 @@ printf 'DB_USER=ledger\nDB_PASS=t0ps3cret\n' > "$WS/lesson-3.2/db.env" check=r""" eq "$(j -n conf cm app-config '{.data.MODE}')" prod && $K -n conf get cm app-config -o jsonpath='{.data.app\.properties}' | grep -q '^cache.size=128$'""", hint=r"""`kubectl create configmap NAME --from-file=PATH --from-literal=KEY=VALUE`. With `--from-file`, the key is the file's name.""", - show=r"""kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod -kubectl -n conf describe cm app-config"""), + show=[cmd(r"""k -n conf create cm app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod +k -n conf describe cm app-config""")]), step("A Secret from an env file", r"""Create Secret `db` (type `Opaque`) in `conf` from `{WS}/lesson-3.2/db.env`, one key per line of the file: `DB_USER` and `DB_PASS`. Then read `DB_PASS` back, decoded.""", check=r""" eq "$(j -n conf secret db '{.type}')" Opaque && eq "$(j -n conf secret db '{.data.DB_USER}' | base64 -d)" ledger && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""", hint=r"""`kubectl create secret generic NAME --from-env-file=PATH`. Reading one key: `-o jsonpath='{.data.DB_PASS}' | base64 -d`.""", - show=r"""kubectl -n conf create secret generic db --from-env-file={WS}/lesson-3.2/db.env -kubectl -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d; echo"""), + show=[cmd(r"""k -n conf create secret generic db --from-env-file={WS}/lesson-3.2/db.env +k -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d""")]), step("Consume both", r"""Run Pod `app` in `conf` (`busybox:1.36`, `sleep 86400`) that gets: @@ -151,16 +151,10 @@ kubectl -n conf get secret db -o jsonpath='{.data.DB_PASS}' | base64 -d; echo""" x() { $K -n conf exec app -- "$@" 2>/dev/null; } eq "$(x printenv DB_PASS)" t0ps3cret && eq "$(x printenv APP_MODE)" prod && x cat /etc/app/app.properties | grep -q '^log.level=info$'""", hint=r"""`envFrom: [{secretRef: {name: db}}]`, `env: [{name: APP_MODE, valueFrom: {configMapKeyRef: {name: app-config, key: MODE}}}]`, and a `configMap` volume.""", - show=r"""kubectl -n conf apply -f - <<'EOF' -apiVersion: v1 -kind: Pod -metadata: - name: app -spec: - containers: - - name: app - image: busybox:1.36 - command: [sleep, '86400'] + show=[cmd(r"""k -n conf run app --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > app.yaml"""), + hand(r"""vim app.yaml""", + r"""Add to the container, below `resources: {}` (the docs page *Configure a Pod to Use a ConfigMap* has each of these): +``` envFrom: - secretRef: {name: db} env: @@ -169,33 +163,38 @@ spec: configMapKeyRef: {name: app-config, key: MODE} volumeMounts: - {name: config, mountPath: /etc/app} +``` +and to the Pod's `spec:`, beside `restartPolicy: Always` and indented like it: +``` volumes: - - name: config - configMap: {name: app-config} -EOF -kubectl -n conf wait --for=condition=Ready pod/app --timeout=120s -kubectl -n conf exec app -- sh -c 'printenv DB_USER APP_MODE; cat /etc/app/app.properties'"""), + - {name: config, configMap: {name: app-config}} +```""", + run=r"""yq -i '.spec.containers[0].envFrom = [{"secretRef": {"name": "db"}}] + | .spec.containers[0].env = [{"name": "APP_MODE", "valueFrom": {"configMapKeyRef": {"name": "app-config", "key": "MODE"}}}] + | .spec.containers[0].volumeMounts = [{"name": "config", "mountPath": "/etc/app"}] + | .spec.volumes = [{"name": "config", "configMap": {"name": "app-config"}}]' app.yaml"""), + cmd(r"""k -n conf apply -f app.yaml +k -n conf exec app -- env | grep -e DB_ -e APP_""")]), step("Change the configuration, live", r"""Change `cache.size` to `256` in ConfigMap `app-config` - edit the file and re-create the ConfigMap's key from it, or `kubectl edit`. Then watch `/etc/app/app.properties` inside Pod `app`: within a minute the kubelet replaces the file. `APP_MODE` would not change, even if you edited `MODE`: the environment is fixed at start.""", check=r"""$K -n conf exec app -- cat /etc/app/app.properties 2>/dev/null | grep -q '^cache.size=256$'""", hint=r"""Regenerate and apply: `kubectl create configmap app-config --from-file=... --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f -` - the usual way to update a ConfigMap from files.""", - show=r"""sed -i 's/^cache.size=.*/cache.size=256/' {WS}/lesson-3.2/app.properties -kubectl -n conf create configmap app-config --from-file={WS}/lesson-3.2/app.properties --from-literal=MODE=prod --dry-run=client -o yaml | kubectl apply -f - -sleep 30 -kubectl -n conf exec app -- cat /etc/app/app.properties"""), + show=[hand(r"""k -n conf edit cm app-config""", + r"""In `app.properties`, change the line to `cache.size=256`, then `:wq`.""", + run=r"""kubectl -n conf get cm app-config -o yaml | sed 's/cache.size=.*/cache.size=256/' | kubectl replace -f -"""), + cmd(r"""k -n conf exec app -- cat /etc/app/app.properties""", r"""Again until it shows `cache.size=256`: the kubelet rewrites the file within a minute.""")]), step("A TLS Secret", r"""Make a self-signed certificate for `shop.example.com` (`openssl` is on this host), and store it as Secret `shop-tls` in `conf`, of type `kubernetes.io/tls` - the type an Ingress or Gateway expects. Its keys are `tls.crt` and `tls.key`.""", check=r""" eq "$(j -n conf secret shop-tls '{.type}')" kubernetes.io/tls || exit 1 j -n conf secret shop-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'shop.example.com'""", hint=r"""`openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com`, then `kubectl create secret tls NAME --cert=tls.crt --key=tls.key`.""", - show=r"""cd {WS}/lesson-3.2 -openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -days 30 -subj /CN=shop.example.com 2>/dev/null -kubectl -n conf create secret tls shop-tls --cert=tls.crt --key=tls.key -kubectl -n conf get secret shop-tls -cd"""), + show=[cmd(r"""cd {WS}/lesson-3.2 +openssl req -x509 -newkey rsa:2048 -nodes -keyout tls.key -out tls.crt -subj /CN=shop.example.com +k -n conf create secret tls shop-tls --cert=tls.crt --key=tls.key +cd""")]), step("Encrypt Secrets at rest", r"""Read Secret `db` straight out of etcd on `{CP}`, and you see `t0ps3cret` in plain text. Make the API server encrypt Secrets before it stores them: @@ -206,7 +205,23 @@ cd"""), Edit a copy of the manifest outside `/etc/kubernetes/manifests` and copy it back: the kubelet runs every file in that directory, backups too.""", check=_ENC_CHECK + r""" && eq "$(j -n conf secret db '{.data.DB_PASS}' | base64 -d)" t0ps3cret""", hint=r"""The docs page "Encrypting Confidential Data at Rest" has the file. Stored values of encrypted Secrets start with `k8s:enc:aescbc:v1:key1:`: `sudo ETCDCTL_API=3 etcdctl ... get /registry/secrets/conf/db | od -c | head` (or `hexdump -C`). Rewriting: `kubectl get secret db -o json | kubectl replace -f -`. If the API server does not come back, `sudo crictl ps -a` and the logs under `/var/log/pods/kube-system_kube-apiserver-*` say why.""", - show=r"""sudo mkdir -p /etc/kubernetes/enc + show=[cmd(r"""head -c 32 /dev/urandom | base64""", r"""Copy the output: that is the key."""), + hand(r"""sudo mkdir -p /etc/kubernetes/enc +sudo vim /etc/kubernetes/enc/enc.yaml""", + r"""Paste the example from the docs page *Encrypting Confidential Data at Rest* (search for `aescbc`) and cut it down to: +``` +apiVersion: apiserver.config.k8s.io/v1 +kind: EncryptionConfiguration +resources: +- resources: [secrets] + providers: + - aescbc: + keys: + - name: key1 + secret: + - identity: {} +```""", + run=r"""sudo mkdir -p /etc/kubernetes/enc sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null </dev/null 2>&1 && break; sleep 3; done -kubectl -n conf get secret db -o json | kubectl replace -f - -sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \ - --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \ - get /registry/secrets/conf/db | od -c | head -4"""), + | .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' ~/kube-apiserver.yaml"""), + cmd(r"""sudo cp ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""), + hand(r"""k get po -n kube-system""", + r"""Again until it answers: the API server is back in a minute or so.""", + run=r"""sleep 20; for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""), + cmd(r"""k -n conf get secret db -o json | k replace -f -""", r"""Written again, the Secret is stored encrypted. Reading it from etcd would now show `k8s:enc:aescbc:v1:key1` and no plain text.""")]), ], "outro": r"""You built a ConfigMap and Secrets from files, consumed them three ways, updated configuration in a running Pod, and turned on encryption at rest for Secrets. @@ -303,9 +329,8 @@ $K -n scale rollout status deploy api --timeout=180s >/dev/null && $K -n scale r eq "$(j -n scale hpa api '{.spec.minReplicas} {.spec.maxReplicas} {.spec.metrics[0].resource.name} {.spec.metrics[0].resource.target.averageUtilization}')" "2 6 cpu 50" || exit 1 [ -n "$(j -n scale hpa api '{.status.currentMetrics[0].resource.current.averageUtilization}')" ] && [ "$(j -n scale deploy api '{.status.readyReplicas}')" -ge 2 ] 2>/dev/null""", hint=r"""`kubectl autoscale deploy api --min=2 --max=6 --cpu=50%` (kubectl before 1.34: `--cpu-percent=50`). `` in the TARGETS column means no metrics yet - or a Pod without a CPU request.""", - show=r"""kubectl -n scale autoscale deploy api --min=2 --max=6 --cpu=50% -sleep 45 -kubectl -n scale get hpa api"""), + show=[cmd(r"""k -n scale autoscale deploy api --min=2 --max=6 --cpu=50% +k -n scale get hpa api""", r"""Again in a minute: `TARGETS` shows a value once metrics arrive.""")]), step("Scale down calmly", r"""Give the HPA of `api` a `behavior`: when scaling **down**, wait **120** seconds of stable recommendations (`stabilizationWindowSeconds`), and remove at most **1** Pod per **60** seconds. @@ -314,36 +339,27 @@ kubectl -n scale get hpa api"""), eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.stabilizationWindowSeconds}')" 120 || exit 1 eq "$(j -n scale hpa api '{.spec.behavior.scaleDown.policies[0].type} {.spec.behavior.scaleDown.policies[0].value} {.spec.behavior.scaleDown.policies[0].periodSeconds}')" "Pods 1 60" """, hint=r"""Under `spec`: `behavior: {scaleDown: {stabilizationWindowSeconds: 120, policies: [{type: Pods, value: 1, periodSeconds: 60}]}}`. `kubectl explain hpa.spec.behavior.scaleDown`.""", - show=r"""kubectl -n scale patch hpa api --type merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":120,"policies":[{"type":"Pods","value":1,"periodSeconds":60}]}}}}' -kubectl -n scale describe hpa api | grep -A4 Behavior"""), + show=[hand(r"""k -n scale edit hpa api""", + r"""Add under `spec:` (the docs page *Horizontal Pod Autoscaling* has it under *Configurable scaling behavior*): +``` + behavior: + scaleDown: + stabilizationWindowSeconds: 120 + policies: + - {type: Pods, value: 1, periodSeconds: 60} +```""", + run=r"""kubectl -n scale patch hpa api --type merge -p '{"spec":{"behavior":{"scaleDown":{"stabilizationWindowSeconds":120,"policies":[{"type":"Pods","value":1,"periodSeconds":60}]}}}}' """), + cmd(r"""k -n scale describe hpa api""")]), step("Watch it scale up", r"""Write an HPA for `cruncher` as a manifest (`autoscaling/v2`): 1 to **4** replicas at **60%** average CPU. `cruncher` uses about three times its request, so the HPA soon adds Pods: wait until it wants more than one.""", check=r""" eq "$(j -n scale hpa cruncher '{.spec.maxReplicas} {.spec.metrics[0].resource.target.averageUtilization}')" "4 60" || exit 1 [ "$(j -n scale hpa cruncher '{.status.desiredReplicas}')" -gt 1 ] 2>/dev/null""", hint=r"""`kubectl autoscale ... --dry-run=client -o yaml` writes the manifest for you. `kubectl -n scale get hpa cruncher -w` shows it react; `describe hpa` lists its decisions as events.""", - show=r"""kubectl -n scale apply -f - <<'EOF' -apiVersion: autoscaling/v2 -kind: HorizontalPodAutoscaler -metadata: - name: cruncher -spec: - scaleTargetRef: - apiVersion: apps/v1 - kind: Deployment - name: cruncher - minReplicas: 1 - maxReplicas: 4 - metrics: - - type: Resource - resource: - name: cpu - target: - type: Utilization - averageUtilization: 60 -EOF -sleep 60 -kubectl -n scale get hpa cruncher"""), + show=[cmd(r"""k -n scale autoscale deploy cruncher --min=1 --max=4 --cpu=60% --dry-run=client -o yaml > hpa.yaml +cat hpa.yaml""", r"""That is the manifest the step asks for: `autoscaling/v2`, `minReplicas`, `maxReplicas` and a `Resource` metric for CPU."""), + cmd(r"""k -n scale apply -f hpa.yaml +k -n scale get hpa cruncher""", r"""Again in a minute or two: `REPLICAS` goes above 1.""")]), step("Resize a running Pod", r"""Run Pod `sizer` in `scale` (`busybox:1.36`, `sleep 86400`) with CPU request `100m` and limit `200m`, memory request and limit `64Mi`. @@ -352,11 +368,23 @@ Then give it more CPU **without replacing it**: request `150m`, limit `300m`, th eq "$(j -n scale pod sizer '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.limits.cpu}')" "150m 300m" || exit 1 eq "$(j -n scale pod sizer '{.status.containerStatuses[0].resources.limits.cpu} {.status.containerStatuses[0].restartCount}')" "300m 0" """, hint=r"""`kubectl patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{...}}]}}'`. A plain `kubectl edit` of a Pod's resources is refused; the subresource is the way in. A container's `resizePolicy` decides whether a change needs a restart.""", - show=r"""kubectl -n scale run sizer --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"sizer","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"200m","memory":"64Mi"}}}]}}' -kubectl -n scale wait --for=condition=Ready pod/sizer --timeout=120s -kubectl -n scale patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{"requests":{"cpu":"150m"},"limits":{"cpu":"300m"}}}]}}' -sleep 5 -kubectl -n scale get pod sizer -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}{.status.containerStatuses[0].restartCount}{"\n"}'"""), + show=[cmd(r"""k -n scale run sizer --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > sizer.yaml"""), + hand(r"""vim sizer.yaml""", + r"""Make `resources: {}`: +``` + resources: + requests: {cpu: 100m, memory: 64Mi} + limits: {cpu: 200m, memory: 64Mi} +```""", + run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "100m", "memory": "64Mi"}, "limits": {"cpu": "200m", "memory": "64Mi"}}' sizer.yaml"""), + cmd(r"""k -n scale apply -f sizer.yaml"""), + hand(r"""k -n scale get po sizer""", + r"""Once it is `Running`:""", + run=r"""kubectl -n scale wait --for=condition=Ready pod/sizer --timeout=120s"""), + hand(r"""k -n scale edit po sizer --subresource resize""", + r"""Set the CPU request to `150m` and the CPU limit to `300m`, then `:wq`.""", + run=r"""kubectl -n scale patch pod sizer --subresource resize --patch '{"spec":{"containers":[{"name":"sizer","resources":{"requests":{"cpu":"150m"},"limits":{"cpu":"300m"}}}]}}' """), + cmd(r"""k -n scale describe po sizer | grep -A6 Limits""", r"""The new values, and `Restart Count: 0`.""")]), ], "outro": r"""You made HPAs both ways, slowed down their scale-down, watched one scale up, and resized a running Pod without replacing it. @@ -421,17 +449,24 @@ rs=$(tr -d ' \r\n' < "$WS/lesson-3.4/rs.txt" 2>/dev/null); [ -n "$rs" ] || exit eq "$(j -n heal rs "$rs" '{.metadata.ownerReferences[0].name} {.status.readyReplicas}')" "web 3" || exit 1 now=$(j -n heal pods -l app=web '{.items[*].metadata.uid}'); for u in $now; do grep -q "$u" "$PRIV/lesson-3.4.uids" || exit 0; done; exit 1""", hint=r"""`kubectl -n heal delete pod $(kubectl -n heal get pod -l app=web -o name | head -1)`. Pods of a Deployment are named DEPLOYMENT-RSHASH-RANDOM; the middle part is the ReplicaSet's suffix.""", - show=r"""kubectl -n heal delete $(kubectl -n heal get pod -l app=web -o name | head -1) -kubectl -n heal get pods -l app=web -kubectl -n heal get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].name}' > {WS}/lesson-3.4/rs.txt -kubectl -n heal get rs"""), + show=[cmd(r"""k -n heal get po -l app=web"""), + hand(r"""k -n heal delete po POD""", + r"""`POD`: any one of the three names.""", + run=r"""kubectl -n heal delete $(kubectl -n heal get pod -l app=web -o name | head -1)"""), + cmd(r"""k -n heal get po -l app=web +k -n heal get rs""", r"""A new Pod already, and one ReplicaSet behind them all: `web-` and a hash."""), + hand(r"""echo RS > {WS}/lesson-3.4/rs.txt""", + r"""`RS`: that ReplicaSet's name.""", + run=r"""kubectl -n heal get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0].name}' > {WS}/lesson-3.4/rs.txt""")]), step("A readiness probe that asks the wrong thing", r"""The Pods of `api` run, but none is `Ready`: `kubectl -n heal describe pod -l app=api` shows the readiness probe failing with `404`. nginx has no `/healthz`. Make the probe ask for `/` instead, and both Pods become ready.""", check=r"""eq "$(j -n heal deploy api '{.spec.template.spec.containers[0].readinessProbe.httpGet.path}')" / && eq "$(j -n heal deploy api '{.status.readyReplicas}')" 2""", hint=r"""A JSON patch on `/spec/template/spec/containers/0/readinessProbe/httpGet/path`, or `kubectl edit deploy api`. The fix is a change of template: a rollout.""", - show=r"""kubectl -n heal describe pod -l app=api | grep -m2 Readiness -kubectl -n heal patch deploy api --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]' -kubectl -n heal rollout status deploy api --timeout=120s"""), + show=[cmd(r"""k -n heal describe po -l app=api | grep -i readiness"""), + hand(r"""k -n heal edit deploy api""", + r"""In the `readinessProbe`, change `path: /healthz` to `path: /`, then `:wq`.""", + run=r"""kubectl -n heal patch deploy api --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]' """), + cmd(r"""k -n heal rollout status deploy api""")]), step("A liveness probe that restarts", r"""Run Pod `fragile` in `heal` (`busybox:1.36`) with the command `sh -c 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400'` and a **liveness** probe that runs `cat /tmp/healthy` every 5 seconds, starting after 5. @@ -440,24 +475,19 @@ After 20 seconds the file is gone, the probe fails three times, and the kubelet eq "$(j -n heal pod fragile '{.spec.containers[0].livenessProbe.exec.command[1]}')" /tmp/healthy || exit 1 [ "$(j -n heal pod fragile '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null""", hint=r"""`livenessProbe: {exec: {command: [cat, /tmp/healthy]}, initialDelaySeconds: 5, periodSeconds: 5}`. `kubectl -n heal get pod fragile -w` shows RESTARTS go up; `describe` shows `Liveness probe failed`.""", - show=r"""kubectl -n heal apply -f - <<'EOF' -apiVersion: v1 -kind: Pod -metadata: - name: fragile -spec: - containers: - - name: fragile - image: busybox:1.36 - command: [sh, -c, 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400'] + show=[cmd(r"""k -n heal run fragile --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'touch /tmp/healthy; sleep 20; rm /tmp/healthy; sleep 86400' > fragile.yaml"""), + hand(r"""vim fragile.yaml""", + r"""Add to the container, below `resources: {}` (the docs page *Configure Liveness, Readiness and Startup Probes* has it): +``` livenessProbe: exec: command: [cat, /tmp/healthy] initialDelaySeconds: 5 periodSeconds: 5 -EOF -sleep 50 -kubectl -n heal get pod fragile"""), +```""", + run=r"""yq -i '.spec.containers[0].livenessProbe = {"exec": {"command": ["cat", "/tmp/healthy"]}, "initialDelaySeconds": 5, "periodSeconds": 5}' fragile.yaml"""), + cmd(r"""k -n heal apply -f fragile.yaml +k -n heal get po fragile""", r"""Again after about 40 seconds: `RESTARTS` is 1.""")]), step("A budget for disruptions", r"""Create PodDisruptionBudget `web-pdb` in `heal`: at least **2** Pods of `app=web` must stay available. @@ -468,20 +498,25 @@ Then try to drain `{W1}`, where all three `web` Pods run: `kubectl drain {W1} -- eq "$(j -n heal pdb web-pdb '{.spec.minAvailable} {.spec.selector.matchLabels.app}')" "2 web" || exit 1 grep -qi 'disruption budget' "$WS/lesson-3.4/drain.txt" 2>/dev/null && [ "$(j node "$W1" '{.spec.unschedulable}')" != true ]""", hint=r"""`kubectl create pdb web-pdb --selector=app=web --min-available=2`. Drain prints the refusals on stderr: `2>&1 | tee FILE`. `kubectl uncordon {W1}` afterwards - drain cordons first.""", - show=r"""kubectl -n heal create pdb web-pdb --selector=app=web --min-available=2 -kubectl -n heal get pdb web-pdb -kubectl -n heal delete pod fragile --grace-period=1 -kubectl drain {W1} --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 | tee {WS}/lesson-3.4/drain.txt | tail -3 -kubectl uncordon {W1}"""), + show=[cmd(r"""k -n heal create pdb web-pdb --selector=app=web --min-available=2 +k -n heal delete po fragile --now +k drain {W1} --ignore-daemonsets --delete-emptydir-data --timeout=30s 2>&1 | tee {WS}/lesson-3.4/drain.txt +k uncordon {W1}""", r"""The drain evicts one `web` Pod, then refuses the others: "Cannot evict pod as it would violate the pod's disruption budget".""")]), step("A Job that runs to completion", r"""Create Job `batch` in `heal`: `busybox:1.36` running `sh -c 'echo processed; sleep 2'`, **3** completions, **2** at a time, giving up after **2** retries (`backoffLimit`). Wait until it is complete.""", check=r"""eq "$(j -n heal job batch '{.spec.completions} {.spec.parallelism} {.spec.backoffLimit} {.status.succeeded}')" "3 2 2 3" """, hint=r"""`kubectl create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c '...'`, then add `completions`, `parallelism` and `backoffLimit` under `spec`. `kubectl wait --for=condition=complete job/batch`.""", - show=r"""kubectl -n heal create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo processed; sleep 2' > {WS}/lesson-3.4/batch.yaml -yq -i '.spec.completions = 3 | .spec.parallelism = 2 | .spec.backoffLimit = 2' {WS}/lesson-3.4/batch.yaml -kubectl -n heal apply -f {WS}/lesson-3.4/batch.yaml -kubectl -n heal wait --for=condition=complete job/batch --timeout=120s -kubectl -n heal get pods -l job-name=batch"""), + show=[cmd(r"""k -n heal create job batch --image=busybox:1.36 --dry-run=client -o yaml -- sh -c 'echo processed; sleep 2' > batch.yaml"""), + hand(r"""vim batch.yaml""", + r"""Add to the Job's own `spec:`, above `template:`: +``` + completions: 3 + parallelism: 2 + backoffLimit: 2 +```""", + run=r"""yq -i '.spec.completions = 3 | .spec.parallelism = 2 | .spec.backoffLimit = 2' batch.yaml"""), + cmd(r"""k apply -f batch.yaml +k -n heal get job,po""", r"""Again until `COMPLETIONS` is `3/3`.""")]), step("A CronJob, and a run on demand", r"""Create CronJob `report` in `heal`: every 30 minutes (`*/30 * * * *`), `busybox:1.36` running `date`, never two runs at once (`concurrencyPolicy: Forbid`), keeping the last **2** successful Jobs. @@ -490,11 +525,17 @@ Do not wait half an hour: start a Job `report-now` from the CronJob yourself, an eq "$(j -n heal cronjob report '{.spec.schedule} {.spec.concurrencyPolicy} {.spec.successfulJobsHistoryLimit}')" "*/30 * * * * Forbid 2" || exit 1 eq "$(j -n heal job report-now '{.metadata.annotations.cronjob\.kubernetes\.io/instantiate} {.status.succeeded}')" "manual 1" """, hint=r"""`kubectl create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date`, then the two fields. `kubectl create job report-now --from=cronjob/report`.""", - show=r"""kubectl -n heal create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date > {WS}/lesson-3.4/report.yaml -yq -i '.spec.concurrencyPolicy = "Forbid" | .spec.successfulJobsHistoryLimit = 2' {WS}/lesson-3.4/report.yaml -kubectl -n heal apply -f {WS}/lesson-3.4/report.yaml -kubectl -n heal create job report-now --from=cronjob/report -kubectl -n heal wait --for=condition=complete job/report-now --timeout=120s"""), + show=[cmd(r"""k -n heal create cronjob report --image=busybox:1.36 --schedule='*/30 * * * *' --dry-run=client -o yaml -- date > report.yaml"""), + hand(r"""vim report.yaml""", + r"""Add to the CronJob's own `spec:`, beside `schedule:`: +``` + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 2 +```""", + run=r"""yq -i '.spec.concurrencyPolicy = "Forbid" | .spec.successfulJobsHistoryLimit = 2' report.yaml"""), + cmd(r"""k apply -f report.yaml +k -n heal create job report-now --from=cronjob/report +k -n heal get job,po""", r"""Again until `report-now` shows `1/1`.""")]), ], "outro": r"""You watched a ReplicaSet replace a Pod, fixed a readiness probe, saw a liveness probe restart a container, stopped a drain with a disruption budget, and ran a Job and a CronJob. @@ -537,35 +578,48 @@ mkns sched check=r""" for p in gold:Guaranteed silver:Burstable bronze:BestEffort; do eq "$(j -n sched pod "${p%%:*}" '{.status.qosClass}')" "${p##*:}" || exit 1; done""", hint=r"""`kubectl run NAME --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400`, then `resources:` in the container. `.status.qosClass` shows the result.""", - show=r"""kubectl -n sched run bronze --image=busybox:1.36 -- sleep 86400 -kubectl -n sched run silver --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"silver","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"memory":"32Mi"}}}]}}' -kubectl -n sched run gold --image=busybox:1.36 --overrides='{"spec":{"containers":[{"name":"gold","image":"busybox:1.36","command":["sleep","86400"],"resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"64Mi"}}}]}}' -kubectl -n sched get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass"""), + show=[cmd(r"""k -n sched run bronze --image=busybox:1.36 -- sleep 86400 +k -n sched run silver --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > silver.yaml"""), + hand(r"""vim silver.yaml""", + r"""Make `resources: {}`: +``` + resources: + requests: {memory: 32Mi} +```""", + run=r"""yq -i '.spec.containers[0].resources = {"requests": {"memory": "32Mi"}}' silver.yaml"""), + cmd(r"""k -n sched apply -f silver.yaml +k -n sched run gold --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > gold.yaml"""), + hand(r"""vim gold.yaml""", + r"""Make `resources: {}`: +``` + resources: + requests: {cpu: 100m, memory: 64Mi} + limits: {cpu: 100m, memory: 64Mi} +```""", + run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "100m", "memory": "64Mi"}, "limits": {"cpu": "100m", "memory": "64Mi"}}' gold.yaml"""), + cmd(r"""k -n sched apply -f gold.yaml +k -n sched describe po | grep -e ^Name: -e QoS""")]), step("Only on SSD nodes", r"""Label `{W2}` with `disk=ssd`. Then run Pod `fast` in `sched` (`busybox:1.36`, `sleep 86400`) with a **required** node affinity for `disk In (ssd)`. It lands on `{W2}` - and if no node had the label, it would stay `Pending`.""", check=r""" eq "$(j node "$W2" '{.metadata.labels.disk}')" ssd && eq "$(j -n sched pod fast '{.spec.nodeName}')" "$W2" \ && eq "$(j -n sched pod fast '{.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key}')" disk""", hint=r"""`kubectl label node {W2} disk=ssd`. The affinity block: `affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[].matchExpressions[]` with `key`, `operator: In`, `values`. The docs page "Assign Pods to Nodes using Node Affinity" has it to copy.""", - show=r"""kubectl label node {W2} disk=ssd -kubectl -n sched apply -f - <<'EOF' -apiVersion: v1 -kind: Pod -metadata: - name: fast -spec: + show=[cmd(r"""k label node {W2} disk=ssd +k -n sched run fast --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > fast.yaml"""), + hand(r"""vim fast.yaml""", + r"""Add to the Pod's `spec:`, above `containers:` (the docs page *Assign Pods to Nodes using Node Affinity* has it): +``` affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - {key: disk, operator: In, values: [ssd]} - containers: - - name: fast - image: busybox:1.36 - command: [sleep, '86400'] -EOF -kubectl -n sched get pod fast -o wide"""), +```""", + run=r"""yq -i '.spec.affinity = {"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "disk", "operator": "In", "values": ["ssd"]}]}]}}}' fast.yaml"""), + cmd(r"""k -n sched apply -f fast.yaml +k -n sched get po fast -o wide""")]), step("One replica per node", r"""Create Deployment `spread` in `sched` (2 replicas, `busybox:1.36`, `sleep 86400`, label `app=spread`) whose Pods **refuse** to share a node with each other: a required pod anti-affinity on `app=spread`, with `topologyKey: kubernetes.io/hostname`. One Pod lands on each worker; a third replica would stay `Pending`.""", check=r""" @@ -573,32 +627,20 @@ eq "$(j -n sched deploy spread '{.status.readyReplicas}')" 2 || exit 1 eq "$(j -n sched deploy spread '{.spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey}')" kubernetes.io/hostname || exit 1 [ "$($K -n sched get pods -l app=spread -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 2 ]""", hint=r"""`podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution: [{labelSelector: {matchLabels: {app: spread}}, topologyKey: kubernetes.io/hostname}]` in the Pod template. `topologySpreadConstraints` is the softer, counting way to spread.""", - show=r"""kubectl -n sched apply -f - <<'EOF' -apiVersion: apps/v1 -kind: Deployment -metadata: - name: spread -spec: - replicas: 2 - selector: - matchLabels: {app: spread} - template: - metadata: - labels: {app: spread} - spec: + show=[cmd(r"""k -n sched create deploy spread --image=busybox:1.36 --replicas=2 --dry-run=client -o yaml -- sleep 86400 > spread.yaml"""), + hand(r"""vim spread.yaml""", + r"""Add to the Pod template's `spec:`, above `containers:` (the docs page *Assigning Pods to Nodes* has an example): +``` affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: {app: spread} topologyKey: kubernetes.io/hostname - containers: - - name: spread - image: busybox:1.36 - command: [sleep, '86400'] -EOF -kubectl -n sched rollout status deploy spread --timeout=120s -kubectl -n sched get pods -l app=spread -o wide"""), +```""", + run=r"""yq -i '.spec.template.spec.affinity = {"podAntiAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": [{"labelSelector": {"matchLabels": {"app": "spread"}}, "topologyKey": "kubernetes.io/hostname"}]}}' spread.yaml"""), + cmd(r"""k apply -f spread.yaml +k -n sched get po -l app=spread -o wide""")]), step("A node for batch work only", r"""Taint `{W1}` with `dedicated=batch:NoSchedule`: from now on only Pods that tolerate it are scheduled there (Pods already running stay - `NoExecute` would evict them). @@ -607,23 +649,19 @@ Then run Pod `nightly` in `sched` (`busybox:1.36`, `sleep 86400`) that tolerates [ -n "$($K get node "$W1" -o jsonpath='{range .spec.taints[*]}{.key}={.value}:{.effect}{"\n"}{end}' | grep -x 'dedicated=batch:NoSchedule')" ] || exit 1 eq "$(j -n sched pod nightly '{.spec.nodeName} {.status.phase}')" "$W1 Running" && eq "$(j -n sched pod nightly '{.spec.tolerations[?(@.key=="dedicated")].value}')" batch""", hint=r"""`kubectl taint node {W1} dedicated=batch:NoSchedule`. The toleration: `{key: dedicated, operator: Equal, value: batch, effect: NoSchedule}`.""", - show=r"""kubectl taint node {W1} dedicated=batch:NoSchedule -kubectl -n sched apply -f - <<'EOF' -apiVersion: v1 -kind: Pod -metadata: - name: nightly -spec: + show=[cmd(r"""k taint node {W1} dedicated=batch:NoSchedule +k -n sched run nightly --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > nightly.yaml"""), + hand(r"""vim nightly.yaml""", + r"""Add to the Pod's `spec:`, above `containers:`: +``` nodeSelector: kubernetes.io/hostname: {W1} tolerations: - {key: dedicated, operator: Equal, value: batch, effect: NoSchedule} - containers: - - name: nightly - image: busybox:1.36 - command: [sleep, '86400'] -EOF -kubectl -n sched get pod nightly -o wide"""), +```""", + run=r"""yq -i '.spec.nodeSelector = {"kubernetes.io/hostname": "{W1}"} | .spec.tolerations = [{"key": "dedicated", "operator": "Equal", "value": "batch", "effect": "NoSchedule"}]' nightly.yaml"""), + cmd(r"""k -n sched apply -f nightly.yaml +k -n sched get po nightly -o wide""")]), step("Defaults for Pods that bring none", r"""Create LimitRange `defaults` in `sched`: containers that say nothing get a **request** of `50m` CPU and `32Mi` memory and a **limit** of `200m` and `128Mi`. @@ -632,7 +670,9 @@ Then run Pod `plain` (`busybox:1.36`, `sleep 86400`, no resources) and look at w eq "$(j -n sched limitrange defaults '{.spec.limits[0].type}')" Container || exit 1 eq "$(j -n sched pod plain '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.requests.memory} {.spec.containers[0].resources.limits.cpu} {.spec.containers[0].resources.limits.memory}')" "50m 32Mi 200m 128Mi" """, hint=r"""`spec.limits: [{type: Container, defaultRequest: {cpu: 50m, memory: 32Mi}, default: {cpu: 200m, memory: 128Mi}}]` - `default` is the limit.""", - show=r"""kubectl -n sched apply -f - <<'EOF' + show=[hand(r"""vim lr.yaml""", + r"""No `kubectl create` for a LimitRange: paste the example from the docs page *Configure Default CPU Requests and Limits for a Namespace* and make it: +``` apiVersion: v1 kind: LimitRange metadata: @@ -642,19 +682,37 @@ spec: - type: Container defaultRequest: {cpu: 50m, memory: 32Mi} default: {cpu: 200m, memory: 128Mi} -EOF -kubectl -n sched run plain --image=busybox:1.36 -- sleep 86400 -kubectl -n sched get pod plain -o jsonpath='{.spec.containers[0].resources}'; echo"""), +```""", + run=r"""cat > lr.yaml <<'EOF' +apiVersion: v1 +kind: LimitRange +metadata: + name: defaults +spec: + limits: + - type: Container + defaultRequest: {cpu: 50m, memory: 32Mi} + default: {cpu: 200m, memory: 128Mi} +EOF"""), + cmd(r"""k -n sched apply -f lr.yaml +k -n sched run plain --image=busybox:1.36 -- sleep 86400 +k -n sched describe po plain | grep -A6 Limits""")]), step("Past the scheduler", r"""Cordon `{W2}`. Then create Pod `manual` in `sched` (`busybox:1.36`, `sleep 86400`) with **`nodeName: {W2}`**. It runs there anyway: `nodeName` hands the Pod straight to that node's kubelet, and cordoning only tells the *scheduler* to stay away. Leave `{W2}` cordoned.""", check=r"""eq "$(j node "$W2" '{.spec.unschedulable}')" true && eq "$(j -n sched pod manual '{.spec.nodeName} {.status.phase}')" "$W2 Running" """, hint=r"""`kubectl cordon {W2}`, then a Pod manifest with `spec.nodeName`. Without a running scheduler, this is also how you place a Pod by hand.""", - show=r"""kubectl cordon {W2} -kubectl -n sched run manual --image=busybox:1.36 --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 86400 -kubectl -n sched get pod manual -o wide -kubectl get nodes"""), + show=[cmd(r"""k cordon {W2} +k -n sched run manual --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > manual.yaml"""), + hand(r"""vim manual.yaml""", + r"""Add to the Pod's `spec:`, above `containers:`: +``` + nodeName: {W2} +```""", + run=r"""yq -i '.spec.nodeName = "{W2}"' manual.yaml"""), + cmd(r"""k -n sched apply -f manual.yaml +k -n sched get po manual -o wide""")]), ], "outro": r"""You produced all three QoS classes, steered Pods with node affinity, kept replicas apart, reserved a node with a taint, set defaults with a LimitRange, and bypassed the scheduler. diff --git a/ksim b/ksim index c943284..ff1de49 100755 --- a/ksim +++ b/ksim @@ -124,7 +124,24 @@ def lesson_task(l, objectives): o = next((o for o in objectives if o["id"] == l["objective"]), None) return dict(l, lesson=True, domain=o["domain"] if o else "", difficulty="normal", set=None, checks=[{"name": s["title"], "pts": 1, "script": s["check"]} for s in l["steps"]], weight=len(l["steps"]), - task=l["intro"], solution="\n\n".join(s["show"] for s in l["steps"] if s["show"])) + task=l["intro"], solution="\n\n".join(show_md(s) for s in l["steps"] if show_parts(s))) + + +def show_parts(s): + """How to do a step, as parts (sim/bank.py: cmd, hand): commands typed as they stand, and what + is done by hand - an edit, a value read off the screen - with the script the audit runs for it.""" + sh = s.get("show") or "" + return sh if isinstance(sh, list) else ([{"type": sh, "how": "", "run": sh}] if sh.strip() else []) + +def show_md(s): + """How to do a step, as you read it: each thing to type in a block of its own (a click copies + it), and beneath it what to do there by hand.""" + return "\n\n".join(f"```\n{p['type'].strip(chr(10))}\n```" + (f"\n\n{p['how'].strip()}" if p["how"] else "") + for p in show_parts(s)) + +def show_script(s): + """What the audit runs for a step: what you type, and for each part done by hand its script.""" + return "\n".join(p["run"] for p in show_parts(s)) def set_questions(name): @@ -1566,7 +1583,9 @@ def begin_session(qs, mode, duration, force, log: Callable[..., Any] = _log_msg, st = {"mode": mode, "track": TRACK_ID, "hard": hard, "ids": [q["id"] for q in qs], "params": params_by_id, "duration": duration, "started_at": time.time(), "ended_at": None, "k8s": k8s_version(), "results": {}, "flags": [], "notes": "", - "objective": objective} # the curriculum objective a drill or a lesson was started for, if any + "objective": objective, # the curriculum objective a drill or a lesson was started for, if any + # the tasks or lessons whose host failed its setup: the page and ./ksim lesson say so, and how to rebuild it + "broken": [q["id"] for q in qs if any(p.startswith(host_of(q) + ":") for p in problems)]} lesson = mode == "lesson" if lesson: st["steps"] = {} # lesson id -> step index -> its latest check (a pass sticks) @@ -1742,6 +1761,9 @@ def cli_md(text): def print_step(l, params, i, st=None): s = l["steps"][i] done = bool(st and steps_of(st, l).get(str(i), {}).get("pass")) + if st and l["id"] in st.get("broken", []): + print(red(f"This lesson's host failed its setup when the session started: it is not as the steps describe.\n" + f" Rebuild it (about a minute): ./ksim reset {l['id']}")) print(bold(f"Step {i + 1} of {len(l['steps'])}: {render(s['title'], params, l)}") + (green(" ✓ checked") if done else "")) print(cli_md(render(s["text"], params, l))) print(dim(" ./ksim lesson check when it is done ./ksim lesson hint / show if you are stuck")) @@ -1770,6 +1792,9 @@ def cmd_lesson(args): sub = args[0] if args else "" if sub == "audit": return lesson_audit(args[1:]) + if sub == "lint": # no cluster needed: what each step has you type, against TYPING_TRAPS + names = [a for a in args[1:] if not a.startswith("--")] + return print_lint(lesson_lint(LESSONS if not names else [x for x in LESSONS if x["objective"] in names or x["domain"] in names])) st = load_state() live = lesson_of(st) if st and not st.get("ended_at") else None if sub == "next": @@ -1794,9 +1819,9 @@ def cmd_lesson(args): if sub == "step": return print_step(live, params, i, st) if sub in ("hint", "show"): - print(bold(f"{'Hint' if sub == 'hint' else 'The commands'} - step {i + 1}: {render(s['title'], params, live)}")) - text = render(s[sub], params, live) - print((cli_md(text) if sub == "hint" else text) or dim("Nothing for this step.")) + print(bold(f"{'Hint' if sub == 'hint' else 'How to do it'} - step {i + 1}: {render(s['title'], params, live)}")) + text = render(s["hint"] if sub == "hint" else show_md(s), params, live) + print(cli_md(text) or dim("Nothing for this step.")) return r, why = lesson_check(st, i) if why: @@ -1840,6 +1865,38 @@ def cmd_lesson(args): print(dim(f"\n ./ksim lesson {LESSONS[0]['objective']} starts that lesson ./ksim lesson audit proves them on real clusters")) print(dim(f" ./ksim lesson {LESSONS[0]['domain']} a domain's lessons not done yet, back to back ({runs} done)")) +# What a step must not ask you to type: in the exam you have a keyboard, vim and the docs, and no +# time to compose scripts. Such things belong in the script of a part done by hand (sim/bank.py). +TYPING_TRAPS = [(re.compile(p, re.M), why) for p, why in ( + (r"\b(sed|awk|yq|jq|perl|python3?)\b", "a text tool - edit in vim, or read the value and echo it"), + (r"<<", "a heredoc - generate the YAML, or paste the docs' example into vim"), + (r"^\s*(sleep|for|while|until)\b", "a pause or a loop - look again yourself"), + (r"\b(k|kubectl)\b[^|\n]* wait\b", "kubectl wait - look with get"), + (r"--overrides", "--overrides - generate the YAML and edit it"), + (r"\bpatch\b[^\n]*(-p|--patch)[ =]'[\[{]", "a JSON patch - kubectl edit"), + (r"jsonpath[^\n|]*>", "jsonpath into a file - read the value, then echo it"), + (r"\$\(", "command substitution - read the value, then type it"), + (r"(^|[|;&(]\s*)(sudo\s+)?kubectl\b", "kubectl in full - the exam has k"))] # the command, not the package + +def lesson_lint(ls): + """What each step has you type, against TYPING_TRAPS. Returns [(lesson, step number, why, line)].""" + out = [] + for l in ls: + for i, s in enumerate(l["steps"]): + for p in show_parts(s): + for line in p["type"].splitlines(): + for rx, why in TYPING_TRAPS: + if rx.search(line): + out.append((l, i + 1, why, line.strip())) + return out + +def print_lint(problems): + for l, n, why, line in problems: + print(f" {yellow(l['objective'] + ' step ' + str(n))} {why}\n {dim(line[:150])}") + steps = len({(l["id"], n) for l, n, _, _ in problems}) + print(bold(f"{steps} step(s) ask you to type what you would not type in the exam.") if steps + else green("Nothing a step asks you to type is a script: every step is done as in the exam.")) + def lesson_audit(args): """Prove lessons on real clusters: each is seeded on a host of its own; then every step must fail its check, have its commands run as `candidate`, and pass its check - so a lesson can be @@ -1862,6 +1919,8 @@ def lesson_audit(args): open(lock, "w").close() import atexit atexit.register(lambda: os.path.exists(lock) and os.remove(lock)) + print(bold(f"What the {len(ls)} lesson(s) have you type:")) + print_lint(lesson_lint(ls)) print(bold(f"Lesson audit of {len(ls)} {TRACK_ID.upper()} lesson(s), {PARALLEL} at a time - each step must fail, be done by its own commands, then pass.")) def one(l): params = make_params(l, random.Random(l["id"])); host = host_of(l); t0 = time.time() @@ -1883,7 +1942,7 @@ def lesson_audit(args): if check_step(l, params, i)["pass"]: res.update(verdict=f"step {i + 1} passes before its commands") break - _, sout = solve_on_host(l, params, script=s["show"]) + _, sout = solve_on_host(l, params, script=show_script(s)) # what the commands created may still be starting: a learner would watch it come up r, waited = check_step(l, params, i), 0 while not r["pass"] and waited < 90: @@ -2155,13 +2214,24 @@ def cmd_history(args): cells.append(f"{round(100 * de / dt):>3}%" if dt else " -") print(f" {d:<7} " + " → ".join(cells)) +def rebuild(st, q, log: Callable[..., Any] = _log_msg): + """A task's or a lesson's host again from scratch, scenario included. A lesson's steps start over + on it; a host that comes up whole is no longer listed as broken. Returns what went wrong, or None.""" + problem = provision(q, session_params(st, q), log=log) + st = load_state() or st + if q.get("lesson"): + st.get("steps", {}).pop(str(q["id"]), None) + st["broken"] = [i for i in st.get("broken", []) if i != q["id"]] + ([q["id"]] if problem else []) + save_state(st) + return problem + def cmd_reset(args): """Throw a task's cluster away and build it again, scenario included.""" if not args: die("usage: ./ksim reset N (rebuild one task's host from scratch: new cluster, scenario seeded again)") st = require_started(); q = q_by_id(int(args[0])) print(dim(f" rebuilding {host_of(q)} - about a minute...")) - problem = provision(q, session_params(st, q)) + problem = rebuild(st, q) print(("rebuilt " if not problem else "problem rebuilding ") + f"Q{q['id']:02d}: {q['title']}") # --------------------------------------------------------------- web ---- @@ -2330,7 +2400,9 @@ class Api(http.server.BaseHTTPRequestHandler): return self._json({"error": down_message(down)}, 503) finish_session(st, log=lambda *_: None) elif p.path == "/api/reset": - q = q_by_id(int(body["id"])); provision(q, session_params(st, q), log=lambda *_: None) + problem = rebuild(st, q_by_id(int(body["id"])), log=lambda *_: None) + if problem: + return self._json({"error": f"Rebuilding failed again ({problem}) - try ./ksim reset {body['id']} in a terminal to see why."}, 500) else: return self._json({"error": "not found"}, 404) return self._json(self.session_payload()) @@ -2368,6 +2440,7 @@ class Api(http.server.BaseHTTPRequestHandler): "paused": bool(st.get("paused_at")), "paused_total": paused_total(st), "notes": st.get("notes", ""), "objective": st.get("objective"), "milestones": st.get("milestones", []), "aborted": bool(st.get("aborted")), + "broken": st.get("broken", []), "results": st["results"], "score": {"earned": earned, "total": total, "pct": pct, "pass_pct": PASS_PCT}, "domains": self.domain_scores(st), "server_time": time.time()} l = lesson_of(st) @@ -2389,7 +2462,7 @@ class Api(http.server.BaseHTTPRequestHandler): r = lambda t: render(t, params, l).strip("\n") return {"id": l["id"], "objective": l["objective"], "title": l["title"], "host": host_of(l), "nodes": 1 + workers_of(l), "intro": r(l["intro"]), "outro": r(l["outro"]), - "steps": [{"title": r(s["title"]), "text": r(s["text"]), "hint": r(s["hint"]), "show": r(s["show"])} for s in l["steps"]]} + "steps": [{"title": r(s["title"]), "text": r(s["text"]), "hint": r(s["hint"]), "show": r(show_md(s))} for s in l["steps"]]} def path_payload(self) -> dict[str, Any]: """The curriculum as the UI shows it: every objective with its notes, tasks and state, how @@ -2699,7 +2772,8 @@ def cmd_help(args): " lesson DOMAIN [--all] teach a domain: its lessons not done yet (--all: every one) back to back, hosts made up front\n" " lesson next | OBJ in a domain's run: on to its next lesson, or to any lesson of it\n" " lesson step|check|hint|show [N] the step you are on, check it, a hint, the commands; ./ksim end finishes\n" - " lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after them\n") + " lesson audit [OBJ..] prove lessons: every step fails before its commands and passes after them\n" + " lesson lint [OBJ|DOMAIN..] what the steps have you type: no scripts, only what you would type in the exam\n") print(f"""usage: ./ksim [--track cka|ckad|cks] (current track: {TRACK_ID}, {len(QUESTIONS)} tasks) Every task runs on its own cluster at the exam's Kubernetes version ({TRACK_ID}: {k8s_version()}), reached with `ssh ` from the remote desktop. diff --git a/sim/bank.py b/sim/bank.py index 901cade..5d7a5a1 100644 --- a/sim/bank.py +++ b/sim/bank.py @@ -21,10 +21,22 @@ def C(name, pts, script): def step(title, text, check, hint="", show=""): """One step of a lesson (/lessons.py): what to do (Markdown), the check that proves it - (bash, exit 0 = done, run like a task's checks), a hint, and the commands that do it - shown on - request, and run by ./ksim lesson audit to prove the step can be done as shown.""" + (bash, exit 0 = done, run like a task's checks), a hint, and how to do it as you would in the + exam - shown on request, and done by ./ksim lesson audit to prove the step works as shown. + `show` is commands you type as they stand, or a list of cmd() and hand() parts.""" return {"title": title, "text": text, "check": check, "hint": hint, "show": show} +def cmd(text, how=""): + """Commands you type as they stand, and what to look for (Markdown); the audit runs them as + they stand.""" + return {"type": text, "how": how, "run": text} + +def hand(type_, how, run): + """What you do by hand: what you type to begin (`vim f`, `k edit ...`, `echo NAME > f`), what + to do there (Markdown), and a script that does the same, which the audit runs instead - an + editor needs a person, and so does a value read off the screen.""" + return {"type": type_, "how": how, "run": run} + def ns_pick(r, base): """Namespace name with a random suffix so the same task never lands on the same namespace twice.""" return f"{base}-{r.choice(['alpha', 'bravo', 'delta', 'echo', 'kilo', 'nova', 'omega', 'zulu'])}" diff --git a/web/index.html b/web/index.html index 7bf5bc7..63286ed 100644 --- a/web/index.html +++ b/web/index.html @@ -607,8 +607,10 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' // minimal markdown: paragraphs, **bold**, `code` (click to copy), ```fenced blocks``` (kept as // written, click to copy), and a paragraph made only of "- " lines as a list. Other lists stay as // literal lines. - const inlineMd = s => esc(s).replace(/\*\*(.+?)\*\*/g, '$1') - .replace(/`([^`]+)`/g, (m, c) => `${c}`); + // `code` first, so that the stars of a cron schedule in it are not read as **bold** or *italics* + const inlineMd = s => s.split(/(`[^`]+`)/).map((p, i) => i % 2 + ? `${esc(p.slice(1, -1))}` + : esc(p).replace(/\*\*(.+?)\*\*/g, '$1').replace(/(^|[^\w*])\*(?!\s)([^*]+?)\*(?![\w*])/g, '$1$2')).join(''); const render = md => md.split(/^```[^\n]*\n([\s\S]*?)\n?^```[ \t]*$/m).map((part, i) => i % 2 ? `
${esc(part)}
` : part.split(/\n\s*\n/).filter(p => p.trim()).map(p => /^(- .*(\n|$))+$/.test(p.trim() + '\n') @@ -925,7 +927,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' $('#startBody').innerHTML = `
No clock. Take your time: the lessons teach, the drills test.
  • every lesson has a host of its own, and all ${n} are up already: the top of each lesson says which to ssh to;
    • -
  • a lesson is an introduction, then steps - each with a Check step button, a hint, and the commands if you are stuck; steps count in order;
    • +
  • a lesson is an introduction, then steps - each with a Check step button, a hint, and Show me how: the exam way, if you are stuck; steps count in order;
  • a lesson's wrap-up leads to the next one, and the list at the top goes to any lesson of the run;
  • a lesson counts as done the moment its last step is checked; nothing goes into your history or scores;
  • finish the run from Controls, then drill ${esc(C.domain)} against the clock.
    • @@ -939,7 +941,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' $('#startBody').innerHTML = `
    No clock. Take your time: a lesson teaches, the drills test.
    • it runs on a host of its own: ${esc('ssh ' + L.host)} in the desktop's terminal;
      • -
    • read the introduction, then do the ${L.steps.length} steps - each has a Check step button, a hint, and the commands if you are stuck;
      • +
    • read the introduction, then do the ${L.steps.length} steps - each has a Check step button, a hint, and Show me how: the exam way, if you are stuck;
    • steps count in order, and nothing goes into your history or scores;
    • the wrap-up sends you to the objective's timed drill.
    `; @@ -1015,7 +1017,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' const r = sess.results[q.id], show = r && (sess.mode === 'drill' || sess.ended_at); const oids = q.objectives || [], oid = oids.includes(sess.objective) ? sess.objective : oids[0]; const obj = oid && (S.objectives || []).find(o => o.id === oid); // which objective a task belongs to is a hint: it follows the solution's rule - $('#qContent').innerHTML = ` + $('#qContent').innerHTML = brokenHost(q.id) + `
    Solve this question on instance: ssh ${esc(q.host)}
    ${q.nodes}-node cluster · task files in ${esc(S.workspace)} on that host
    @@ -1031,6 +1033,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' ${q.solution ? `
    Reference solution
    ${esc(q.solution)}
    ` : ''} ${obj ? `
    Objective ${obj.id} · ${esc(obj.text)}
    ${notesHtml(obj)}
    ` : ''}`; copyable($('#qContent')); + wireRebuild(); const bc = $('#btnCheck'); if (bc) bc.onclick = checkTask; const fl = sess.flags.includes(q.id); $('#btnFlag').className = 'btn ' + (fl ? 'btn-warning' : 'btn-outline-warning') + (on ? '' : ' hidden'); @@ -1038,6 +1041,24 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' $('#btnSatisfied').disabled = i >= n - 1; paintDesk(); } + // a host whose setup failed when the session started: say so where it is used, and offer to rebuild it + const brokenHost = id => live() && (S.session.broken || []).includes(id) + ? `
    This host's setup failed when the session started, so it is not as the text describes. +
    + about a minute - or ./ksim reset ${id} in a terminal
    ` : ''; + function wireRebuild() { + const b = $('#btnRebuild'); if (!b) return; + b.onclick = async () => { + $('#busyTitle').textContent = 'Rebuilding the host'; + $('#busyBody').innerHTML = '

    A new cluster, and its setup run again - about a minute.

    '; + openModal('#mBusy'); + try { S = await api('/api/reset', {id: +b.dataset.id}); } + catch (e) { alert(e.message); } + finally { closeModal('#mBusy'); } + if (isLesson()) lpage = null; + paintExam(); + }; + } // what the exam and a lesson share: the notepad, the remote desktop, the clock function paintDesk() { if (document.activeElement !== $('#notes')) $('#notes').value = S.session.notes || ''; @@ -1101,7 +1122,7 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' + C.lessons.map(x => { const c = x.steps.filter(s => s.pass).length; return ``; }).join('') + '
    '; - let body = `
    ${pick}
    This lesson runs on: ${esc('ssh ' + L.host)}
    + let body = brokenHost(L.id) + `
    ${pick}
    This lesson runs on: ${esc('ssh ' + L.host)}
    ${L.nodes}-node cluster · files in ${esc(S.workspace)} on that host · no clock
    `; if (lpage === 0) { body += `
    Lesson ${esc(L.objective)} · ${n} steps

    ${esc(L.title)}

    ${render(L.intro)}
    `; @@ -1120,10 +1141,11 @@ try { const t = localStorage.getItem('ksim.theme'); if (t === 'light' || t === ' + (r && !r.pass && r.out ? `
    What the check saw
    ${esc(r.out)}
    ` : ''); } else if (r && r.pass) body += '
    ✓ Done
    '; if (s.hint) body += `
    Hint
    ${render(s.hint)}
    `; - if (s.show) body += `
    Show me the commands
    ${esc(s.show)}
    `; + if (s.show) body += `
    Show me how
    ${render(s.show)}
    `; } $('#qContent').innerHTML = body; copyable($('#qContent')); + wireRebuild(); const bs = $('#btnStep'); if (bs) bs.onclick = () => checkStep(lpage - 1); const bf = $('#btnFinish'); if (bf) bf.onclick = finishLesson; const bn = $('#btnNextLesson'); if (bn) bn.onclick = () => goLesson(nx.id);