0e5f7686ee
Add lessons: untimed guided labs, one per curriculum objective, on a host of their own, in the browser and in the terminal. Each step has a check, a hint and its commands; a lesson writes no history or scores, only a done mark on its objective. CKAD 2.1-2.4 (24 steps) and all 27 CKA objectives (135 steps) have one, and `./ksim lesson audit` proves that every step fails, is done by its own commands, then passes. Add CKA tasks 132-147 (cka/bank_course.py) on what a full CKA course teaches beyond the bank: a real upgrade from v1.36, a drain under a PodDisruptionBudget, QoS and eviction, in-place resize, kube-proxy's chains, encryption at rest, an operator, and CSI expansion, snapshots and clones (ensure_csi_hostpath). Move CKA and CKS to Kubernetes v1.37 and rebuild the hosts on Ubuntu 24.04, with kubelet, kubeadm and kubectl as held packages from pkgs.k8s.io; the desktop runs Ubuntu too. Hosts gain man pages, hexdump and xxd. Fix CKS Q4, which gave 3 of 12 points untouched, and CKA Q4 and Q49, whose solutions raced their new CRD. Audited on v1.37: CKA 147/147, CKAD 105/105, CKS 109/110 (Q103, a Falco task, failed to seed once and awaits a re-audit); lessons: 135 of 135 CKA steps and 24 of 24 CKAD steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
552 lines
38 KiB
Python
552 lines
38 KiB
Python
"""CKA lessons of the Troubleshooting domain (2.1-2.5). See cka/lessons.py for what a lesson is."""
|
|
from sim.bank import NS, step
|
|
|
|
# ---- 2.1 clusters and nodes -------------------------------------------------------------------
|
|
L21 = {
|
|
"id": 9201, "objective": "2.1", "workers": 1,
|
|
"title": "A node goes NotReady: the kubelet, its configuration, and pressure",
|
|
"intro": r"""A node is `Ready` while its **kubelet** reports in. When it stops, the control plane waits under a minute (its node monitor grace period) and marks the node `NotReady`; five minutes later - the default toleration for an unreachable node - its Pods are evicted. So for a `NotReady` node the questions go in this order:
|
|
|
|
```
|
|
kubectl get nodes / describe node what the control plane last heard
|
|
ssh NODE; sudo -i
|
|
systemctl status kubelet is it running, is it enabled?
|
|
journalctl -u kubelet -n 50 why it stopped, or why it will not start
|
|
/var/lib/kubelet/config.yaml its configuration (KubeletConfiguration)
|
|
/etc/kubernetes/kubelet.conf how it reaches the API server
|
|
/etc/systemd/system/kubelet.service.d/ its flags (drop-ins), and /var/lib/kubelet/kubeadm-flags.env
|
|
systemctl status containerd the runtime it needs
|
|
```
|
|
|
|
The kubelet also watches the node's resources. Below a **hard eviction threshold** - `memory.available`, `nodefs.available`, ... - it sets a condition (`MemoryPressure`, `DiskPressure`, `PIDPressure`) and evicts Pods to recover.
|
|
|
|
Something has happened to the kubelet on worker `{W1}`.""",
|
|
"setup": r"""
|
|
mkdir -p "$WS/lesson-2.1"
|
|
nx "$W1" 'systemctl disable --now kubelet' </dev/null >/dev/null 2>&1
|
|
# the control plane notices after its grace period; the lesson starts with the node NotReady
|
|
for i in $(seq 1 40); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" != True ] && exit 0; sleep 3; done
|
|
echo "$W1 is still Ready a minute after its kubelet was stopped" >&2; exit 1
|
|
""",
|
|
"steps": [
|
|
step("Which node, and since when",
|
|
r"""Find the node that is not `Ready` and write its name to `{WS}/lesson-2.1/node.txt`.
|
|
|
|
Then look at `kubectl describe node` for it: under **Conditions** every row says `Kubelet stopped posting node status` - the control plane has not heard from the kubelet, so it knows nothing more. The answer is on the node.""",
|
|
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/node.txt" 2>/dev/null)" "$W1" """,
|
|
hint=r"""`kubectl get nodes`. `--no-headers` and `awk '$2 != "Ready" {print $1}'` give just the name.""",
|
|
show=r"""kubectl get nodes
|
|
kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt
|
|
kubectl describe node {W1} | grep -A8 Conditions"""),
|
|
step("Bring the kubelet back - for good",
|
|
r"""On `{W1}` (`ssh {W1}`, then `sudo -i`), look at the kubelet: `systemctl status kubelet` says `inactive (dead)` and `disabled`. Start it, and enable it so that it survives a reboot - a fix that is gone after the next reboot is not a fix.
|
|
|
|
Within a few seconds the node is `Ready` again.""",
|
|
check=r"""
|
|
nx "$W1" 'systemctl is-active -q kubelet && systemctl is-enabled -q kubelet' </dev/null || exit 1
|
|
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
|
|
hint=r"""`systemctl enable --now kubelet` does both at once. `journalctl -u kubelet -n 20` tells you when it is running happily.""",
|
|
show=r"""ssh {W1} sudo bash <<'EOF'
|
|
systemctl status kubelet --no-pager | head -4
|
|
systemctl enable --now kubelet
|
|
systemctl is-active kubelet; systemctl is-enabled kubelet
|
|
EOF
|
|
kubectl get nodes"""),
|
|
step("Where the kubelet's configuration is",
|
|
r"""A kubelet reads its settings from the file named by its `--config` flag, and the kubeadm flags from `/var/lib/kubelet/kubeadm-flags.env`. On `{W1}`, find the file (`ps -o args= -C kubelet`, or the drop-in under `/etc/systemd/system/kubelet.service.d/`).
|
|
|
|
The kubelet also tells the API server what it actually runs with: `kubectl get --raw /api/v1/nodes/{W1}/proxy/configz`. Write the live value of its `cgroupDriver` to `{WS}/lesson-2.1/cgroup.txt`.""",
|
|
check=r"""
|
|
want=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["cgroupDriver"])')
|
|
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/cgroup.txt" 2>/dev/null)" "$want" """,
|
|
hint=r"""`configz` returns JSON: `| jq -r .kubeletconfig.cgroupDriver`. It must match containerd's `SystemdCgroup` setting, or Pods fail to start.""",
|
|
show=r"""ssh -n {W1} 'ps -o args= -C kubelet | tr " " "\n" | grep -- --config; sudo grep -n cgroupDriver /var/lib/kubelet/config.yaml'
|
|
kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt
|
|
cat {WS}/lesson-2.1/cgroup.txt"""),
|
|
step("Set the eviction thresholds",
|
|
r"""Look at `evictionHard` in `/var/lib/kubelet/config.yaml` on `{W1}`: kind sets every threshold to `0%`, which means "never evict" - fine for a test cluster, not for a real one.
|
|
|
|
On `{W1}`, make the kubelet evict when less than **`100Mi`** of memory or **`10%`** of the node filesystem is left: `memory.available: 100Mi` and `nodefs.available: 10%` under `evictionHard`. Restart the kubelet, and confirm it through `configz`.""",
|
|
check=r"""
|
|
$K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c '
|
|
import json, sys
|
|
e = json.load(sys.stdin)["kubeletconfig"].get("evictionHard") or {}
|
|
sys.exit(0 if e.get("memory.available") == "100Mi" and e.get("nodefs.available") == "10%" else 1)' || exit 1
|
|
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
|
|
hint=r"""Edit the file with `vim`, or `sed`: change the `nodefs.available` line and add a `memory.available` line inside the `evictionHard:` block (two spaces of indent). A kubelet that does not come back after the restart: `journalctl -u kubelet -n 20` shows the line it could not parse.""",
|
|
show=r"""ssh {W1} sudo bash <<'EOF'
|
|
f=/var/lib/kubelet/config.yaml
|
|
grep -n -A4 '^evictionHard:' $f
|
|
sed -i 's/^ nodefs.available: .*/ nodefs.available: 10%/' $f
|
|
sed -i '/^evictionHard:/a\ memory.available: 100Mi' $f
|
|
grep -n -A5 '^evictionHard:' $f
|
|
systemctl restart kubelet
|
|
EOF
|
|
kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq .kubeletconfig.evictionHard"""),
|
|
step("Read the pressure conditions",
|
|
r"""The kubelet reports the result of those thresholds as node conditions. Write the status of `{W1}`'s three pressure conditions to `{WS}/lesson-2.1/pressure.txt`, one per line, as `TYPE STATUS` - for example `MemoryPressure False`.
|
|
|
|
`True` on any of them means the kubelet is evicting Pods, and the node gets a taint (`node.kubernetes.io/memory-pressure`, ...) so no new Pods land there.""",
|
|
check=r"""
|
|
want=$(for c in MemoryPressure DiskPressure PIDPressure; do echo "$c $(j node "$W1" "{.status.conditions[?(@.type==\"$c\")].status}")"; done | sort)
|
|
eq "$(tr -s ' \t' ' ' < "$WS/lesson-2.1/pressure.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """,
|
|
hint=r"""`kubectl describe node {W1}` lists them under Conditions. As a list: `-o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}'`, then keep the three `*Pressure` lines.""",
|
|
show=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt
|
|
cat {WS}/lesson-2.1/pressure.txt"""),
|
|
],
|
|
"outro": r"""You found a `NotReady` node, brought its kubelet back for good, read its configuration from the file and from the running kubelet, set eviction thresholds and read what the kubelet reports about them.
|
|
|
|
**For the exam**
|
|
|
|
- `NotReady`: the node is the place to look. `systemctl status kubelet`, then `journalctl -u kubelet`. A kubelet that restarts in a loop prints the reason a moment before each exit.
|
|
- Fixes stay fixed: `systemctl enable`, and edits in the files, not on the command line.
|
|
- Usual faults: kubelet stopped or disabled, a wrong path in `config.yaml` (CA file, static Pod path), a wrong flag in a drop-in or `kubeadm-flags.env` (`systemctl daemon-reload` after editing a unit), containerd down or the wrong runtime endpoint, an expired client certificate.
|
|
- `kubectl get --raw /api/v1/nodes/NODE/proxy/configz` is the kubelet's live configuration.
|
|
- Eviction: `evictionHard` (and `evictionSoft` with a grace period) in `config.yaml`. Pressure conditions taint the node; evicted Pods show `Failed` with reason `Evicted`.
|
|
|
|
Next, prove it against the clock: **drill 2.1**.""",
|
|
}
|
|
|
|
# ---- 2.2 cluster components -------------------------------------------------------------------
|
|
_BREAK_CP = r"""
|
|
mkns ops
|
|
mkdir -p "$WS/lesson-2.2"
|
|
gone() { local i; for i in $(seq 1 45); do [ -z "$(nx "$CP" "crictl ps --name $1 -q" </dev/null)" ] && return 0; sleep 2; done; return 1; }
|
|
# the controller manager first: it must be down before the Deployment exists, or it makes its Pods
|
|
nx "$CP" 'sed -i "s|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)$|\1-broken|" /etc/kubernetes/manifests/kube-controller-manager.yaml' </dev/null
|
|
gone kube-controller-manager || { echo "kube-controller-manager is still running" >&2; exit 1; }
|
|
$K -n ops create deploy queue --image=busybox:1.36 --replicas=2 -- sleep 86400 >/dev/null
|
|
nx "$CP" 'sed -i "s|^\( *\)- kube-scheduler$|\1- kube-schedulerr|" /etc/kubernetes/manifests/kube-scheduler.yaml' </dev/null
|
|
gone kube-scheduler || { echo "kube-scheduler is still running" >&2; exit 1; }
|
|
$K -n ops run waiting --image=busybox:1.36 -- sleep 86400 >/dev/null
|
|
"""
|
|
|
|
L22 = {
|
|
"id": 9202, "objective": "2.2", "workers": 1,
|
|
"title": "The control plane's own Pods: find what is down and why",
|
|
"intro": r"""On a kubeadm cluster the control plane runs as **static Pods**: the kubelet of the control-plane node starts whatever manifests it finds in `/etc/kubernetes/manifests`, and restarts them when a file changes. Each component has its own symptom when it is down:
|
|
|
|
```
|
|
kube-apiserver kubectl itself fails: "connection refused"
|
|
etcd the API server fails, then kubectl
|
|
kube-scheduler new Pods stay Pending, with no node and no event
|
|
kube-controller-manager Deployments get no ReplicaSet, ReplicaSets no Pods; nothing self-heals
|
|
```
|
|
|
|
When the API server is up, `kubectl -n kube-system get pods` and `describe` show the others. When it is not, go to the node: `crictl ps -a`, `crictl logs ID`, the files under `/var/log/pods/`, and `journalctl -u kubelet`.
|
|
|
|
Two things in Namespace `ops` do not work: Deployment `queue` has no Pods, and Pod `waiting` has no node.""",
|
|
"setup": NS + _BREAK_CP,
|
|
"steps": [
|
|
step("Which components are down",
|
|
r"""Find the control-plane components that are not running and write their names to `{WS}/lesson-2.2/down.txt`, one per line (for example `kube-proxy`).
|
|
|
|
`kubectl -n kube-system get pods` shows them; on the control-plane node, `sudo crictl ps -a` shows their containers, and `ls /etc/kubernetes/manifests` what the kubelet is told to run.""",
|
|
check=r"""eq "$(tr -d ' \r' < "$WS/lesson-2.2/down.txt" 2>/dev/null | grep . | sort | tr '\n' ' ')" "kube-controller-manager kube-scheduler " """,
|
|
hint=r"""A Pod that is not `Running` with all its containers ready. The Pod names end in the node's name; the component is the part before it.""",
|
|
show=r"""kubectl -n kube-system get pods
|
|
ls /etc/kubernetes/manifests
|
|
sudo crictl ps -a | grep kube-
|
|
printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt"""),
|
|
step("Repair the scheduler",
|
|
r"""Find out why `kube-scheduler` does not start, and fix its manifest. `kubectl -n kube-system describe pod kube-scheduler-{CP}` shows the error in its events, and so do `sudo crictl ps -a --name kube-scheduler` and `sudo journalctl -u kubelet`.
|
|
|
|
The kubelet notices the saved file on its own: no restart, no `kubectl apply`. Once the scheduler runs, Pod `waiting` gets a node.""",
|
|
check=r"""
|
|
nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml && [ -n "$(crictl ps --name kube-scheduler -q)" ]' </dev/null || exit 1
|
|
[ -n "$(j -n ops pod waiting '{.spec.nodeName}')" ]""",
|
|
hint=r"""Look at the first entry of the container's `command:` - the program the container runs. A static Pod's manifest is edited in place (`sudo vim`), never with `kubectl edit`.""",
|
|
show=r"""kubectl -n kube-system describe pod kube-scheduler-{CP} | tail -5
|
|
sudo grep -n -A2 'command:' /etc/kubernetes/manifests/kube-scheduler.yaml
|
|
sudo sed -i 's/- kube-schedulerr$/- kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml
|
|
sudo crictl ps --name kube-scheduler
|
|
kubectl -n ops get pod waiting -o wide"""),
|
|
step("Repair the controller manager",
|
|
r"""Now `kube-controller-manager`. Its Pod is not crashing - its container never starts. `describe` on it tells you which step fails. Fix the manifest; once it runs, Deployment `queue` gets its ReplicaSet and both Pods.""",
|
|
check=r"""
|
|
nx "$CP" '! grep -q -- "-broken" /etc/kubernetes/manifests/kube-controller-manager.yaml && [ -n "$(crictl ps --name kube-controller-manager -q)" ]' </dev/null || exit 1
|
|
eq "$(j -n ops deploy queue '{.status.availableReplicas}')" 2""",
|
|
hint=r"""`ErrImagePull` / `ImagePullBackOff`: the image tag does not exist. Every control-plane image runs the cluster's version - compare with `kube-apiserver.yaml`.""",
|
|
show=r"""kubectl -n kube-system describe pod kube-controller-manager-{CP} | tail -5
|
|
sudo grep -n 'image:' /etc/kubernetes/manifests/*.yaml
|
|
sudo sed -i 's/-broken$//' /etc/kubernetes/manifests/kube-controller-manager.yaml
|
|
sudo crictl ps --name kube-controller-manager
|
|
kubectl -n ops get deploy queue"""),
|
|
step("Ask the API server how it feels",
|
|
r"""The API server has health endpoints of its own: `/livez` (restart me if this fails) and `/readyz` (send me traffic). With `?verbose` they list every check.
|
|
|
|
Save the output of `kubectl get --raw '/readyz?verbose'` in `{WS}/lesson-2.2/readyz.txt`.""",
|
|
check=r"""grep -q '^\[+\]etcd ok' "$WS/lesson-2.2/readyz.txt" 2>/dev/null && grep -q 'readyz check passed' "$WS/lesson-2.2/readyz.txt" """,
|
|
hint=r"""`kubectl get --raw` sends a GET to any API path, with your credentials. Quote the path: `?` means something to the shell.""",
|
|
show=r"""kubectl get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt | tail -3
|
|
kubectl get --raw '/livez'; echo"""),
|
|
],
|
|
"outro": r"""You read the control plane from both sides - `kubectl` and the node's container runtime - and repaired two static Pods by editing their manifests in place.
|
|
|
|
**For the exam**
|
|
|
|
- Static Pod manifests: `/etc/kubernetes/manifests` (the kubelet's `staticPodPath`). Edit the file; the kubelet restarts the Pod. Keep a copy *outside* that directory - a backup inside it is started too.
|
|
- Symptoms: no scheduler, Pods `Pending` with no events; no controller manager, Deployments without ReplicaSets; no API server, `kubectl` refuses to connect - then `crictl ps -a`, `crictl logs`, `/var/log/pods/kube-system_kube-apiserver-*/`.
|
|
- Typical faults: a wrong image tag, a typo in the command or a flag, a wrong path to a certificate, a volume mount that does not match a `hostPath`.
|
|
- `kubectl get --raw '/readyz?verbose'` and `/livez` show the API server's own checks, etcd's among them.
|
|
|
|
Next, prove it against the clock: **drill 2.2**.""",
|
|
}
|
|
|
|
# ---- 2.3 resource usage -----------------------------------------------------------------------
|
|
L23 = {
|
|
"id": 9203, "objective": "2.3", "workers": 1,
|
|
"title": "What uses the CPU and memory: kubectl top and the metrics pipeline",
|
|
"intro": r"""`kubectl top` shows what Pods and nodes **use** right now. The numbers come from **metrics-server**, which asks every kubelet every 15 seconds and serves the result as an API of its own, `metrics.k8s.io`:
|
|
|
|
```
|
|
kubelet (cAdvisor) ──▶ metrics-server ──▶ /apis/metrics.k8s.io ──▶ kubectl top, HorizontalPodAutoscaler
|
|
```
|
|
|
|
Without metrics-server there is no `kubectl top` and no autoscaling on CPU or memory. Usage is not the same as **requests**: requests are what the scheduler reserves, and `kubectl describe node` shows them under *Allocated resources*.
|
|
|
|
metrics-server runs on this cluster. Namespace `metrics` has a few Pods: one of them burns CPU, one holds a lot of memory, and Pod `duo` has two containers.""",
|
|
"setup": NS + r"""
|
|
ensure_metrics_server || { echo "metrics-server did not become ready" >&2; exit 1; }
|
|
mkns metrics
|
|
mkdir -p "$WS/lesson-2.3"
|
|
$K -n metrics apply -f - >/dev/null <<'EOF'
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: {name: burner}
|
|
spec:
|
|
containers:
|
|
- name: burner
|
|
image: busybox:1.36
|
|
command: [sh, -c, 'while true; do :; done']
|
|
resources: {limits: {cpu: 300m, memory: 32Mi}}
|
|
---
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: {name: hoarder}
|
|
spec:
|
|
containers:
|
|
- name: hoarder
|
|
image: busybox:1.36
|
|
command: [sh, -c, 'dd if=/dev/zero of=/cache/blob bs=1M count=90 2>/dev/null; sleep 86400']
|
|
volumeMounts: [{name: cache, mountPath: /cache}]
|
|
volumes: [{name: cache, emptyDir: {medium: Memory}}]
|
|
---
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: {name: idler}
|
|
spec:
|
|
containers:
|
|
- {name: idler, image: busybox:1.36, command: [sleep, '86400']}
|
|
---
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: {name: duo}
|
|
spec:
|
|
containers:
|
|
- {name: app, image: busybox:1.36, command: [sleep, '86400']}
|
|
- name: cache
|
|
image: busybox:1.36
|
|
command: [sh, -c, 'dd if=/dev/zero of=/cache/blob bs=1M count=40 2>/dev/null; sleep 86400']
|
|
volumeMounts: [{name: cache, mountPath: /cache}]
|
|
volumes: [{name: cache, emptyDir: {medium: Memory}}]
|
|
EOF
|
|
$K -n metrics wait --for=condition=Ready pod --all --timeout=180s >/dev/null
|
|
# until metrics-server has a sample of every Pod
|
|
for i in $(seq 1 40); do [ "$($K top pod -n metrics --no-headers 2>/dev/null | grep -c .)" -ge 4 ] && exit 0; sleep 5; done
|
|
echo "kubectl top shows no metrics for the lesson's Pods" >&2; exit 1
|
|
""",
|
|
"steps": [
|
|
step("The heaviest Pod by memory",
|
|
r"""Write the name of the Pod in `metrics` that uses the most memory to `{WS}/lesson-2.3/memory.txt`. `kubectl top` can sort for you.""",
|
|
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/memory.txt" 2>/dev/null)" hoarder""",
|
|
hint=r"""`kubectl top pod -n metrics --sort-by=memory`. `--no-headers | head -1 | awk '{print $1}'` keeps just the name.""",
|
|
show=r"""kubectl top pod -n metrics --sort-by=memory
|
|
kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt"""),
|
|
step("The busiest Pod in the cluster",
|
|
r"""Across **all** Namespaces, which Pod uses the most CPU? Write it as `NAMESPACE/NAME` to `{WS}/lesson-2.3/cpu.txt`.""",
|
|
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/cpu.txt" 2>/dev/null)" metrics/burner""",
|
|
hint=r"""`kubectl top pod -A --sort-by=cpu`; with `-A` the first column is the Namespace.""",
|
|
show=r"""kubectl top pod -A --sort-by=cpu | head -5
|
|
kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt"""),
|
|
step("Inside one Pod",
|
|
r"""`kubectl top pod` adds up a Pod's containers. Which container of Pod `duo` uses the memory? Write its name to `{WS}/lesson-2.3/container.txt`.""",
|
|
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/container.txt" 2>/dev/null)" cache""",
|
|
hint=r"""`--containers` splits the Pod into its containers.""",
|
|
show=r"""kubectl top pod duo -n metrics --containers
|
|
kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt"""),
|
|
step("Usage against what is reserved",
|
|
r"""`kubectl top node` shows use; `kubectl describe node {W1}`, under **Allocated resources**, shows what Pods have **requested** - which is what decides whether another Pod fits.
|
|
|
|
Write the memory that `{W1}` can give to Pods - its **allocatable** memory, as the API reports it (for example `8012345Ki`) - to `{WS}/lesson-2.3/allocatable.txt`.""",
|
|
check=r"""want=$(j node "$W1" '{.status.allocatable.memory}'); [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/allocatable.txt" 2>/dev/null)" "$want" """,
|
|
hint=r"""Allocatable is capacity minus what the node keeps for itself and for the eviction threshold: `.status.allocatable.memory`.""",
|
|
show=r"""kubectl top node
|
|
kubectl describe node {W1} | grep -A9 'Allocated resources'
|
|
kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt"""),
|
|
step("The metrics API itself",
|
|
r"""`kubectl top` only reads an API. Read it yourself: save what `/apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder` returns in `{WS}/lesson-2.3/raw.json`.
|
|
|
|
If this path answers `NotFound` or `ServiceUnavailable` on some cluster, `kubectl top` fails there too: look at the `metrics-server` Deployment in `kube-system`, and at `kubectl get apiservice v1beta1.metrics.k8s.io`.""",
|
|
check=r"""python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); sys.exit(0 if d.get("kind")=="PodMetrics" and d["metadata"]["name"]=="hoarder" and d.get("containers") else 1)' "$WS/lesson-2.3/raw.json" 2>/dev/null""",
|
|
hint=r"""`kubectl get --raw PATH`.""",
|
|
show=r"""kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json
|
|
jq '.containers' {WS}/lesson-2.3/raw.json
|
|
kubectl get apiservice v1beta1.metrics.k8s.io"""),
|
|
],
|
|
"outro": r"""You ranked Pods by memory and CPU, looked inside a Pod, compared use with what a node can give, and read the metrics API that `kubectl top` uses.
|
|
|
|
**For the exam**
|
|
|
|
- `kubectl top pod [-A] [-n NS] [-l SELECTOR] --sort-by=cpu|memory [--containers]`, `kubectl top node`. Write exactly what is asked - often just the name.
|
|
- No metrics: is metrics-server running, and is `apiservice v1beta1.metrics.k8s.io` `Available`? On kubeadm clusters it often needs `--kubelet-insecure-tls`.
|
|
- `top` is usage; `describe node` → *Allocated resources* is requests and limits. The scheduler goes by requests.
|
|
- Metrics lag by a few seconds and are averages: a Pod that has just started may not appear yet.
|
|
|
|
Next, prove it against the clock: **drill 2.3**.""",
|
|
}
|
|
|
|
# ---- 2.4 container output streams ------------------------------------------------------------
|
|
L24 = {
|
|
"id": 9204, "objective": "2.4", "workers": 1,
|
|
"title": "Logs, the previous container, the node's log files, and debug containers",
|
|
"intro": r"""Everything a container writes to **stdout** and **stderr** is kept by the node's runtime and served by `kubectl logs`:
|
|
|
|
```
|
|
kubectl logs POD [-c CONTAINER] one container (-c when the Pod has several)
|
|
--previous the instance before the last restart - where a crash is explained
|
|
-l app=api --prefix every Pod of a label, each line marked with its Pod
|
|
--since=10m --tail=50 -f a window, the end, follow
|
|
```
|
|
|
|
On the node the same output lives in `/var/log/pods/NAMESPACE_POD_UID/CONTAINER/N.log` (N counts restarts), linked from `/var/log/containers/`. `crictl logs` reads it too.
|
|
|
|
And when a container has no shell to `exec` into, `kubectl debug` adds an **ephemeral container** to the running Pod - your image, your tools, sharing the Pod's network and, with `--target`, its processes. `kubectl debug node/NAME` does the same for a node.
|
|
|
|
Namespace `logs` has Pod `flaky`, which crashed once; Pods `api-1` to `api-3`; and Pod `quiet`, built from an image with no shell at all.""",
|
|
"setup": NS + r"""
|
|
mkns logs
|
|
mkdir -p "$WS/lesson-2.4"
|
|
$K -n logs apply -f - >/dev/null <<'EOF'
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: {name: flaky}
|
|
spec:
|
|
volumes: [{name: state, emptyDir: {}}]
|
|
containers:
|
|
- name: flaky
|
|
image: busybox:1.36
|
|
volumeMounts: [{name: state, mountPath: /state}]
|
|
command: [sh, -c, 'if [ ! -f /state/booted ]; then touch /state/booted; echo "FATAL config /etc/flaky.conf not found"; exit 1; fi; echo "INFO started on second attempt"; exec sleep 86400']
|
|
---
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata: {name: quiet}
|
|
spec:
|
|
containers:
|
|
- {name: quiet, image: registry.k8s.io/pause:3.10.2}
|
|
EOF
|
|
for n in 1 2 3; do
|
|
msg='INFO request served'; [ $n = 2 ] && msg='ERROR upstream refused the request'
|
|
$K -n logs run api-$n --image=busybox:1.36 --labels=app=api -- sh -c "while true; do echo \"$msg\"; sleep 5; done" >/dev/null
|
|
done
|
|
$K -n logs wait --for=condition=Ready pod --all --timeout=180s >/dev/null
|
|
for i in $(seq 1 30); do [ "$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null && exit 0; sleep 3; done
|
|
echo "flaky did not restart" >&2; exit 1
|
|
""",
|
|
"steps": [
|
|
step("Why it crashed",
|
|
r"""Pod `flaky` is `Running` now, with one restart. The logs of the running container only say it started; the reason for the crash is in the logs of the instance **before** it.
|
|
|
|
Save those in `{WS}/lesson-2.4/previous.log`.""",
|
|
check=r"""grep -q 'FATAL config /etc/flaky.conf not found' "$WS/lesson-2.4/previous.log" 2>/dev/null && ! grep -q 'second attempt' "$WS/lesson-2.4/previous.log" """,
|
|
hint=r"""`kubectl logs --previous` (or `-p`).""",
|
|
show=r"""kubectl -n logs logs flaky
|
|
kubectl -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log
|
|
cat {WS}/lesson-2.4/previous.log"""),
|
|
step("The one replica that complains",
|
|
r"""Pods `api-1`, `api-2` and `api-3` all carry `app=api`. One of them logs errors. Read all three at once, find it, and write its name to `{WS}/lesson-2.4/erroring.txt`.""",
|
|
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/erroring.txt" 2>/dev/null)" api-2""",
|
|
hint=r"""`kubectl logs -l app=api --prefix` puts `[pod/NAME/CONTAINER]` before each line. With a selector, kubectl shows only the last 10 lines of each Pod unless you say `--tail`.""",
|
|
show=r"""kubectl -n logs logs -l app=api --prefix --tail=2
|
|
kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt
|
|
cat {WS}/lesson-2.4/erroring.txt"""),
|
|
step("The log file on the node",
|
|
r"""The node keeps one file per container instance. Write the full path of the file that holds the output of `flaky`'s **current** instance to `{WS}/lesson-2.4/logfile.txt`.
|
|
|
|
Find the node with `kubectl -n logs get pod flaky -o wide`, then look under `/var/log/pods/` there. The directory name has the Pod's UID in it; the file name is the restart count.""",
|
|
check=r"""
|
|
p=$(tr -d ' \r\n' < "$WS/lesson-2.4/logfile.txt" 2>/dev/null)
|
|
uid=$(j -n logs pod flaky '{.metadata.uid}'); n=$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}'); node=$(j -n logs pod flaky '{.spec.nodeName}')
|
|
eq "$p" "/var/log/pods/logs_flaky_$uid/flaky/$n.log" && nx "$node" "grep -q 'second attempt' '$p'" </dev/null""",
|
|
hint=r"""`/var/log/pods/NAMESPACE_POD_UID/CONTAINER/RESTARTS.log`. `sudo ls -R /var/log/pods/ | grep flaky` on the node gets you close.""",
|
|
show=r"""kubectl -n logs get pod flaky -o wide
|
|
uid=$(kubectl -n logs get pod flaky -o jsonpath='{.metadata.uid}')
|
|
ssh -n {W1} "sudo ls /var/log/pods/logs_flaky_$uid/flaky/"
|
|
echo "/var/log/pods/logs_flaky_$uid/flaky/1.log" > {WS}/lesson-2.4/logfile.txt
|
|
ssh -n {W1} "sudo cat $(cat {WS}/lesson-2.4/logfile.txt)"""),
|
|
step("Debug a container that has no shell",
|
|
r"""`kubectl -n logs exec quiet -- sh` fails: the `pause` image has nothing but its one program. Add an ephemeral container to the running Pod instead: name `dbg`, image `busybox:1.36`, **targeting** container `quiet` so that it sees its processes, running `sleep 3600`.
|
|
|
|
Then run `ps` in `dbg` and write the command of the target's process - the first one, PID 1 - to `{WS}/lesson-2.4/target.txt`.""",
|
|
check=r"""
|
|
eq "$(j -n logs pod quiet '{.spec.ephemeralContainers[0].name} {.spec.ephemeralContainers[0].targetContainerName}')" "dbg quiet" || exit 1
|
|
grep -qx '/pause' <(tr -d ' \r' < "$WS/lesson-2.4/target.txt" 2>/dev/null)""",
|
|
hint=r"""`kubectl debug POD --image=IMAGE --target=CONTAINER -c NAME -- COMMAND`. Interactively you would add `-it` and use the shell straight away; without it, `kubectl exec POD -c dbg -- ps` reaches it once it runs. Ephemeral containers cannot be removed - the Pod keeps them until it goes.""",
|
|
show=r"""kubectl -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600
|
|
for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done
|
|
kubectl -n logs exec quiet -c dbg -- ps
|
|
kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt"""),
|
|
step("Debug the node itself",
|
|
r"""`kubectl debug node/{W1} --image=busybox:1.36` starts a Pod on that node with the node's whole filesystem mounted at `/host` - useful when you cannot `ssh` to a node, but can reach the API.
|
|
|
|
Use it to read the node's `/etc/os-release`, and write the value of its `PRETTY_NAME` to `{WS}/lesson-2.4/os.txt`.""",
|
|
check=r"""
|
|
want=$(nx "$W1" '. /etc/os-release; echo "$PRETTY_NAME"' </dev/null); [ -n "$want" ] || exit 1
|
|
eq "$(tr -d '\r"' < "$WS/lesson-2.4/os.txt" 2>/dev/null | sed 's/^PRETTY_NAME=//; s/ *$//')" "$want" \
|
|
&& [ -n "$($K get pods -A -o name 2>/dev/null | grep node-debugger)" ]""",
|
|
hint=r"""Without `-it`, give it a command and read its output with `kubectl logs`: the Pod is called `node-debugger-NODE-xxxxx`, in your current Namespace. Delete it afterwards.""",
|
|
show=r"""kubectl debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release
|
|
pod=$(kubectl get pods -o name | grep node-debugger | head -1)
|
|
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s
|
|
kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt
|
|
cat {WS}/lesson-2.4/os.txt"""),
|
|
],
|
|
"outro": r"""You read a crashed instance's logs, searched several Pods at once, found the log file on the node, and debugged both a Pod without a shell and a node.
|
|
|
|
**For the exam**
|
|
|
|
- A crash is explained in `kubectl logs --previous`. `describe pod` adds the exit code and the reason (`Error`, `OOMKilled`, `Completed`).
|
|
- Several containers: `-c NAME`, or `--all-containers`. Several Pods: `-l SELECTOR --prefix`, and mind the default `--tail=10`.
|
|
- Write logs to a file exactly as asked: `> file` for stdout. A container's stderr is in its logs too - `kubectl logs` does not separate them; the node's log file does, with `stdout` / `stderr` on each line.
|
|
- `kubectl debug -it POD --image=busybox --target=CONTAINER` for a container without tools; `kubectl debug node/NAME -it --image=busybox`, then `chroot /host`, for a node.
|
|
|
|
Next, prove it against the clock: **drill 2.4**.""",
|
|
}
|
|
|
|
# ---- 2.5 services and networking --------------------------------------------------------------
|
|
L25 = {
|
|
"id": 9205, "objective": "2.5", "workers": 1,
|
|
"title": "A Service that does not answer: selector, ports, policy, and kube-proxy",
|
|
"intro": r"""A request to a Service goes through several hands, and each can drop it:
|
|
|
|
```
|
|
client ──DNS──▶ ClusterIP:port ──kube-proxy rules──▶ endpoint PodIP:targetPort ──NetworkPolicy──▶ container
|
|
```
|
|
|
|
So check them in order:
|
|
|
|
- **Endpoints**: `kubectl get endpointslices -l kubernetes.io/service-name=SVC`. None? The selector matches no ready Pod.
|
|
- **Ports**: `targetPort` must be the port the container listens on (a number, or the name of a container port).
|
|
- **NetworkPolicy**: once a policy selects a Pod, only what it allows gets in.
|
|
- **kube-proxy** turns each Service into rules on every node - iptables chains named `KUBE-SVC-...` here.
|
|
|
|
In Namespace `shop`, Deployment `web` (nginx) has a Service `web` in front of it, and nothing gets through. Pods `client` and `stranger` are there to send requests from.""",
|
|
"setup": NS + r"""
|
|
mkns shop
|
|
mkdir -p "$WS/lesson-2.5"
|
|
$K -n shop create deploy web --image=nginx:1.27 --replicas=2 --port=80 >/dev/null
|
|
$K -n shop apply -f - >/dev/null <<'EOF'
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata: {name: web}
|
|
spec:
|
|
selector: {app: webapp}
|
|
ports:
|
|
- {port: 80, targetPort: 8080}
|
|
---
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata: {name: lockdown}
|
|
spec:
|
|
podSelector: {matchLabels: {app: web}}
|
|
policyTypes: [Ingress]
|
|
EOF
|
|
$K -n shop run client --image=busybox:1.36 --labels=app=client -- sleep 86400 >/dev/null
|
|
$K -n shop run stranger --image=busybox:1.36 --labels=app=stranger -- sleep 86400 >/dev/null
|
|
$K -n shop rollout status deploy web --timeout=180s >/dev/null
|
|
$K -n shop wait --for=condition=Ready pod --all --timeout=180s >/dev/null
|
|
""",
|
|
"steps": [
|
|
step("No endpoints",
|
|
r"""Ask the Service from `client`: `kubectl -n shop exec client -- wget -qO- -T3 web`. It times out. Look at the Service's endpoints: there are none, because its selector matches no Pod.
|
|
|
|
Fix the **Service** so that it selects the Pods of Deployment `web`.""",
|
|
check=r"""
|
|
eq "$(j -n shop svc web '{.spec.selector.app}')" web || exit 1
|
|
n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' | grep -c .)
|
|
[ "$n" -eq 2 ]""",
|
|
hint=r"""Compare `kubectl -n shop get svc web -o wide` (SELECTOR) with `kubectl -n shop get pods --show-labels`. `kubectl set selector` or `kubectl edit svc`.""",
|
|
show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
|
|
kubectl -n shop get svc web -o wide
|
|
kubectl -n shop get pods --show-labels
|
|
kubectl -n shop set selector svc web app=web
|
|
kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""),
|
|
step("The wrong port",
|
|
r"""Now the Service has endpoints, but they point at port `8080` - and nginx listens on `80`. Make the Service send traffic to the port the container serves.""",
|
|
check=r"""
|
|
eq "$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{.items[0].ports[0].port}')" 80 && eq "$(j -n shop svc web '{.spec.ports[0].port}')" 80""",
|
|
hint=r"""`targetPort` is the container's port; `port` is the Service's own. `kubectl -n shop edit svc web`, or a JSON patch on `/spec/ports/0/targetPort`.""",
|
|
show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web
|
|
kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]'
|
|
kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""),
|
|
step("Let the right client in",
|
|
r"""Still nothing: NetworkPolicy `lockdown` selects the `web` Pods and allows no ingress at all.
|
|
|
|
Keep it - do not delete it - and let **`client`** (label `app=client`) reach the `web` Pods on TCP 80. `stranger` must still be refused.""",
|
|
check=r"""
|
|
[ -n "$(j -n shop netpol lockdown '{.metadata.name}')" ] || exit 1
|
|
ok() { $K -n shop exec "$1" -- wget -qO- -T 3 web >/dev/null 2>&1; }
|
|
for i in 1 2 3 4 5; do ok client && break; sleep 3; done; ok client || exit 1
|
|
! ok stranger""",
|
|
hint=r"""Add an `ingress` rule to `lockdown`: `from: [{podSelector: {matchLabels: {app: client}}}]` with `ports: [{protocol: TCP, port: 80}]`. A NetworkPolicy's ports are the Pod's, not the Service's.""",
|
|
show=r"""kubectl -n shop apply -f - <<'EOF'
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: lockdown
|
|
spec:
|
|
podSelector:
|
|
matchLabels: {app: web}
|
|
policyTypes: [Ingress]
|
|
ingress:
|
|
- from:
|
|
- podSelector:
|
|
matchLabels: {app: client}
|
|
ports:
|
|
- {protocol: TCP, port: 80}
|
|
EOF
|
|
kubectl -n shop exec client -- wget -qO- -T3 web | head -4
|
|
kubectl -n shop exec stranger -- wget -qO- -T3 web || echo refused"""),
|
|
step("How kube-proxy routes it",
|
|
r"""kube-proxy turned Service `web` into rules on every node. Find out how: its mode is in ConfigMap `kube-proxy` in `kube-system`. Write the mode to `{WS}/lesson-2.5/mode.txt`.
|
|
|
|
Then, on the control-plane node, find the iptables chain that kube-proxy made for this Service - the `KUBE-SVC-...` chain that its ClusterIP jumps to - and write the chain's name to `{WS}/lesson-2.5/chain.txt`.""",
|
|
check=r"""
|
|
mode=$($K -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}'); mode=${mode:-iptables}
|
|
eq "$(tr -d ' \r\n"' < "$WS/lesson-2.5/mode.txt" 2>/dev/null)" "$mode" || exit 1
|
|
ip=$(j -n shop svc web '{.spec.clusterIP}')
|
|
want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" </dev/null)
|
|
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.5/chain.txt" 2>/dev/null)" "$want" """,
|
|
hint=r"""`kubectl -n kube-system get cm kube-proxy -o yaml | grep mode`. On the node: `sudo iptables-save -t nat | grep CLUSTER-IP` - the line for the Service jumps (`-j`) to its chain. In `nftables` mode the rules are in `sudo nft list ruleset` instead.""",
|
|
show=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt
|
|
ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}')
|
|
sudo iptables-save -t nat | grep -- "-d $ip/32"
|
|
sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt
|
|
sudo iptables-save -t nat | grep "^-A $(cat {WS}/lesson-2.5/chain.txt)" """),
|
|
],
|
|
"outro": r"""You took a Service from "times out" to answering by fixing its selector, its target port and a NetworkPolicy, and followed it into kube-proxy's rules.
|
|
|
|
**For the exam**
|
|
|
|
- Endpoints first: `kubectl get endpointslices -l kubernetes.io/service-name=SVC` (or `kubectl get endpoints SVC`). Empty means the selector, or Pods that are not Ready.
|
|
- `port` is the Service's, `targetPort` the container's; a named `targetPort` must match a `containerPort` name.
|
|
- Test from a Pod: `kubectl exec POD -- wget -qO- -T3 SVC:PORT`, or a throwaway `kubectl run tmp --rm -it --restart=Never --image=busybox:1.36 -- ...`.
|
|
- A NetworkPolicy never blocks what it does not select; once one selects a Pod, everything not allowed is dropped - and egress policies need DNS (UDP/TCP 53) allowed.
|
|
- kube-proxy: a DaemonSet in `kube-system`, configured by ConfigMap `kube-proxy`; `iptables-save -t nat | grep KUBE-SVC` or `nft list ruleset` on a node.
|
|
|
|
Next, prove it against the clock: **drill 2.5**.""",
|
|
}
|
|
|
|
LESSONS = [L21, L22, L23, L24, L25]
|