vumast3r/kubestronaut-sim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
e761d53a96
kubestronaut-sim/cka/lessons_tshoot.py
vumas3r 226ef19ff9 feat: show every CKA lesson step the way the exam is done 
Show me how gave the commands the audit runs: sed, awk and yq edits,
heredoc manifests, JSON patches, --overrides, sleeps and wait loops -
nothing anyone types in the exam. A step's show is now a list of
parts (sim/bank.py): cmd() is what you type as it stands; hand() is
what you do by hand - vim on a generated skeleton or on a manifest
pasted from the docs page it names, kubectl edit, ssh NODE and sudo
-i, a value you read and echo - with a script that does the same for
the audit. All 135 CKA steps are rewritten so, with k for kubectl;
their checks, titles and texts are unchanged.

Add ./ksim lesson lint, which reads what every step has you type and
flags scripts. The lesson audit runs it first.

Record the hosts whose setup failed when a session started. The page
warns above that task or lesson and offers Rebuild this host; the CLI
names ./ksim reset N on each step.

Render *italics* in lessons and tasks, outside code spans.

Audited on v1.37: CKA lessons 27/27, 135/135 steps, done the exam
way; lint clean. kubectl edit --subresource resize, the one edit the
audit replaces with a patch, was proven on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 20:34:36 -06:00

626 lines
44 KiB
Python
Raw Blame History

"""CKA lessons of the Troubleshooting domain (2.1-2.5). See cka/lessons.py for what a lesson is."""
from sim.bank import NS, step, cmd, hand
# ---- 2.1 clusters and nodes -------------------------------------------------------------------
L21 = {
"id": 9201, "objective": "2.1", "workers": 1,
"title": "A node goes NotReady: the kubelet, its configuration, and pressure",
"intro": r"""A node is `Ready` while its **kubelet** reports in. When it stops, the control plane waits under a minute (its node monitor grace period) and marks the node `NotReady`; five minutes later - the default toleration for an unreachable node - its Pods are evicted. So for a `NotReady` node the questions go in this order:
```
kubectl get nodes / describe node what the control plane last heard
ssh NODE; sudo -i
systemctl status kubelet is it running, is it enabled?
journalctl -u kubelet -n 50 why it stopped, or why it will not start
/var/lib/kubelet/config.yaml its configuration (KubeletConfiguration)
/etc/kubernetes/kubelet.conf how it reaches the API server
/etc/systemd/system/kubelet.service.d/ its flags (drop-ins), and /var/lib/kubelet/kubeadm-flags.env
systemctl status containerd the runtime it needs
```
The kubelet also watches the node's resources. Below a **hard eviction threshold** - `memory.available`, `nodefs.available`, ... - it sets a condition (`MemoryPressure`, `DiskPressure`, `PIDPressure`) and evicts Pods to recover.
Something has happened to the kubelet on worker `{W1}`.""",
"setup": r"""
mkdir -p "$WS/lesson-2.1"
nx "$W1" 'systemctl disable --now kubelet' </dev/null >/dev/null 2>&1
# the control plane notices after its grace period; the lesson starts with the node NotReady
for i in $(seq 1 40); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" != True ] && exit 0; sleep 3; done
echo "$W1 is still Ready a minute after its kubelet was stopped" >&2; exit 1
""",
"steps": [
step("Which node, and since when",
r"""Find the node that is not `Ready` and write its name to `{WS}/lesson-2.1/node.txt`.
Then look at `kubectl describe node` for it: under **Conditions** every row says `Kubelet stopped posting node status` - the control plane has not heard from the kubelet, so it knows nothing more. The answer is on the node.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/node.txt" 2>/dev/null)" "$W1" """,
hint=r"""`kubectl get nodes`. `--no-headers` and `awk '$2 != "Ready" {print $1}'` give just the name.""",
show=[cmd(r"""k get no
k describe no {W1} | grep -A8 Conditions"""),
hand(r"""echo NODE > {WS}/lesson-2.1/node.txt""",
r"""`NODE`: the one that is not `Ready`.""",
run=r"""kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt""")]),
step("Bring the kubelet back - for good",
r"""On `{W1}` (`ssh {W1}`, then `sudo -i`), look at the kubelet: `systemctl status kubelet` says `inactive (dead)` and `disabled`. Start it, and enable it so that it survives a reboot - a fix that is gone after the next reboot is not a fix.
Within a few seconds the node is `Ready` again.""",
check=r"""
nx "$W1" 'systemctl is-active -q kubelet && systemctl is-enabled -q kubelet' </dev/null || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""`systemctl enable --now kubelet` does both at once. `journalctl -u kubelet -n 20` tells you when it is running happily.""",
show=[hand(r"""ssh {W1}
sudo -i
systemctl status kubelet
systemctl enable --now kubelet""",
r"""Then `exit` twice.""",
run=r"""ssh {W1} sudo bash <<'EOF'
systemctl enable --now kubelet
EOF"""),
cmd(r"""k get no""", r"""Again in a few seconds: `Ready`.""")]),
step("Where the kubelet's configuration is",
r"""A kubelet reads its settings from the file named by its `--config` flag, and the kubeadm flags from `/var/lib/kubelet/kubeadm-flags.env`. On `{W1}`, find the file (`ps -o args= -C kubelet`, or the drop-in under `/etc/systemd/system/kubelet.service.d/`).
The kubelet also tells the API server what it actually runs with: `kubectl get --raw /api/v1/nodes/{W1}/proxy/configz`. Write the live value of its `cgroupDriver` to `{WS}/lesson-2.1/cgroup.txt`.""",
check=r"""
want=$($K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["cgroupDriver"])')
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/cgroup.txt" 2>/dev/null)" "$want" """,
hint=r"""`configz` returns JSON: `| jq -r .kubeletconfig.cgroupDriver`. It must match containerd's `SystemdCgroup` setting, or Pods fail to start.""",
show=[hand(r"""ssh {W1}
sudo -i
ps -o args= -C kubelet
grep cgroupDriver /var/lib/kubelet/config.yaml""",
r"""`--config=` names the kubelet's file. Then `exit` twice.""",
run=r"""ssh -n {W1} 'ps -o args= -C kubelet; sudo grep cgroupDriver /var/lib/kubelet/config.yaml' """),
cmd(r"""k get --raw /api/v1/nodes/{W1}/proxy/configz | tr , '\n' | grep cgroupDriver""", r"""The live value."""),
hand(r"""echo VALUE > {WS}/lesson-2.1/cgroup.txt""",
r"""`VALUE`: `systemd` or `cgroupfs`.""",
run=r"""kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt""")]),
step("Set the eviction thresholds",
r"""Look at `evictionHard` in `/var/lib/kubelet/config.yaml` on `{W1}`: kind sets every threshold to `0%`, which means "never evict" - fine for a test cluster, not for a real one.
On `{W1}`, make the kubelet evict when less than **`100Mi`** of memory or **`10%`** of the node filesystem is left: `memory.available: 100Mi` and `nodefs.available: 10%` under `evictionHard`. Restart the kubelet, and confirm it through `configz`.""",
check=r"""
$K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c '
import json, sys
e = json.load(sys.stdin)["kubeletconfig"].get("evictionHard") or {}
sys.exit(0 if e.get("memory.available") == "100Mi" and e.get("nodefs.available") == "10%" else 1)' || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""",
hint=r"""Edit the file with `vim`, or `sed`: change the `nodefs.available` line and add a `memory.available` line inside the `evictionHard:` block (two spaces of indent). A kubelet that does not come back after the restart: `journalctl -u kubelet -n 20` shows the line it could not parse.""",
show=[hand(r"""ssh {W1}
sudo -i
vim /var/lib/kubelet/config.yaml""",
r"""Under `evictionHard:`, set `nodefs.available` to `10%` and add `memory.available`, indented like the others:
```
evictionHard:
memory.available: 100Mi
nodefs.available: 10%
```
then restart the kubelet, and `exit` twice:
```
systemctl restart kubelet
```""",
run=r"""ssh {W1} sudo bash <<'EOF'
f=/var/lib/kubelet/config.yaml
sed -i 's/^ nodefs.available: .*/ nodefs.available: 10%/' $f
sed -i '/^evictionHard:/a\ memory.available: 100Mi' $f
systemctl restart kubelet
EOF"""),
cmd(r"""k get --raw /api/v1/nodes/{W1}/proxy/configz | tr , '\n' | grep -A4 evictionHard""", r"""Again if the kubelet is not back yet.""")]),
step("Read the pressure conditions",
r"""The kubelet reports the result of those thresholds as node conditions. Write the status of `{W1}`'s three pressure conditions to `{WS}/lesson-2.1/pressure.txt`, one per line, as `TYPE STATUS` - for example `MemoryPressure False`.
`True` on any of them means the kubelet is evicting Pods, and the node gets a taint (`node.kubernetes.io/memory-pressure`, ...) so no new Pods land there.""",
check=r"""
want=$(for c in MemoryPressure DiskPressure PIDPressure; do echo "$c $(j node "$W1" "{.status.conditions[?(@.type==\"$c\")].status}")"; done | sort)
eq "$(tr -s ' \t' ' ' < "$WS/lesson-2.1/pressure.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """,
hint=r"""`kubectl describe node {W1}` lists them under Conditions. As a list: `-o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}'`, then keep the three `*Pressure` lines.""",
show=[cmd(r"""k describe no {W1} | grep Pressure"""),
hand(r"""vim {WS}/lesson-2.1/pressure.txt""",
r"""Three lines, `TYPE STATUS`, with the statuses you read:
```
MemoryPressure False
DiskPressure False
PIDPressure False
```""",
run=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt""")]),
],
"outro": r"""You found a `NotReady` node, brought its kubelet back for good, read its configuration from the file and from the running kubelet, set eviction thresholds and read what the kubelet reports about them.
**For the exam**
- `NotReady`: the node is the place to look. `systemctl status kubelet`, then `journalctl -u kubelet`. A kubelet that restarts in a loop prints the reason a moment before each exit.
- Fixes stay fixed: `systemctl enable`, and edits in the files, not on the command line.
- Usual faults: kubelet stopped or disabled, a wrong path in `config.yaml` (CA file, static Pod path), a wrong flag in a drop-in or `kubeadm-flags.env` (`systemctl daemon-reload` after editing a unit), containerd down or the wrong runtime endpoint, an expired client certificate.
- `kubectl get --raw /api/v1/nodes/NODE/proxy/configz` is the kubelet's live configuration.
- Eviction: `evictionHard` (and `evictionSoft` with a grace period) in `config.yaml`. Pressure conditions taint the node; evicted Pods show `Failed` with reason `Evicted`.
Next, prove it against the clock: **drill 2.1**.""",
}
# ---- 2.2 cluster components -------------------------------------------------------------------
_BREAK_CP = r"""
mkns ops
mkdir -p "$WS/lesson-2.2"
gone() { local i; for i in $(seq 1 45); do [ -z "$(nx "$CP" "crictl ps --name $1 -q" </dev/null)" ] && return 0; sleep 2; done; return 1; }
# the controller manager first: it must be down before the Deployment exists, or it makes its Pods
nx "$CP" 'sed -i "s|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)$|\1-broken|" /etc/kubernetes/manifests/kube-controller-manager.yaml' </dev/null
gone kube-controller-manager || { echo "kube-controller-manager is still running" >&2; exit 1; }
$K -n ops create deploy queue --image=busybox:1.36 --replicas=2 -- sleep 86400 >/dev/null
nx "$CP" 'sed -i "s|^\( *\)- kube-scheduler$|\1- kube-schedulerr|" /etc/kubernetes/manifests/kube-scheduler.yaml' </dev/null
gone kube-scheduler || { echo "kube-scheduler is still running" >&2; exit 1; }
$K -n ops run waiting --image=busybox:1.36 -- sleep 86400 >/dev/null
"""
L22 = {
"id": 9202, "objective": "2.2", "workers": 1,
"title": "The control plane's own Pods: find what is down and why",
"intro": r"""On a kubeadm cluster the control plane runs as **static Pods**: the kubelet of the control-plane node starts whatever manifests it finds in `/etc/kubernetes/manifests`, and restarts them when a file changes. Each component has its own symptom when it is down:
```
kube-apiserver kubectl itself fails: "connection refused"
etcd the API server fails, then kubectl
kube-scheduler new Pods stay Pending, with no node and no event
kube-controller-manager Deployments get no ReplicaSet, ReplicaSets no Pods; nothing self-heals
```
When the API server is up, `kubectl -n kube-system get pods` and `describe` show the others. When it is not, go to the node: `crictl ps -a`, `crictl logs ID`, the files under `/var/log/pods/`, and `journalctl -u kubelet`.
Two things in Namespace `ops` do not work: Deployment `queue` has no Pods, and Pod `waiting` has no node.""",
"setup": NS + _BREAK_CP,
"steps": [
step("Which components are down",
r"""Find the control-plane components that are not running and write their names to `{WS}/lesson-2.2/down.txt`, one per line (for example `kube-proxy`).
`kubectl -n kube-system get pods` shows them; on the control-plane node, `sudo crictl ps -a` shows their containers, and `ls /etc/kubernetes/manifests` what the kubelet is told to run.""",
check=r"""eq "$(tr -d ' \r' < "$WS/lesson-2.2/down.txt" 2>/dev/null | grep . | sort | tr '\n' ' ')" "kube-controller-manager kube-scheduler " """,
hint=r"""A Pod that is not `Running` with all its containers ready. The Pod names end in the node's name; the component is the part before it.""",
show=[cmd(r"""k -n kube-system get po
ls /etc/kubernetes/manifests
sudo crictl ps -a | grep kube-"""),
hand(r"""vim {WS}/lesson-2.2/down.txt""",
r"""The components that do not run, one per line.""",
run=r"""printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt""")]),
step("Repair the scheduler",
r"""Find out why `kube-scheduler` does not start, and fix its manifest. `kubectl -n kube-system describe pod kube-scheduler-{CP}` shows the error in its events, and so do `sudo crictl ps -a --name kube-scheduler` and `sudo journalctl -u kubelet`.
The kubelet notices the saved file on its own: no restart, no `kubectl apply`. Once the scheduler runs, Pod `waiting` gets a node.""",
check=r"""
nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml && [ -n "$(crictl ps --name kube-scheduler -q)" ]' </dev/null || exit 1
[ -n "$(j -n ops pod waiting '{.spec.nodeName}')" ]""",
hint=r"""Look at the first entry of the container's `command:` - the program the container runs. A static Pod's manifest is edited in place (`sudo vim`), never with `kubectl edit`.""",
show=[cmd(r"""k -n kube-system describe po kube-scheduler-{CP} | tail -5""", r"""The command it cannot find: a typo."""),
hand(r"""sudo vim /etc/kubernetes/manifests/kube-scheduler.yaml""",
r"""Under `command:`, make `- kube-schedulerr` read `- kube-scheduler`, then `:wq`.""",
run=r"""sudo sed -i 's/- kube-schedulerr$/- kube-scheduler/' /etc/kubernetes/manifests/kube-scheduler.yaml"""),
cmd(r"""sudo crictl ps --name kube-scheduler
k -n ops get po waiting -o wide""", r"""Again in a few seconds: the scheduler runs, and `waiting` has a node.""")]),
step("Repair the controller manager",
r"""Now `kube-controller-manager`. Its Pod is not crashing - its container never starts. `describe` on it tells you which step fails. Fix the manifest; once it runs, Deployment `queue` gets its ReplicaSet and both Pods.""",
check=r"""
nx "$CP" '! grep -q -- "-broken" /etc/kubernetes/manifests/kube-controller-manager.yaml && [ -n "$(crictl ps --name kube-controller-manager -q)" ]' </dev/null || exit 1
eq "$(j -n ops deploy queue '{.status.availableReplicas}')" 2""",
hint=r"""`ErrImagePull` / `ImagePullBackOff`: the image tag does not exist. Every control-plane image runs the cluster's version - compare with `kube-apiserver.yaml`.""",
show=[cmd(r"""k -n kube-system describe po kube-controller-manager-{CP} | tail -5
sudo grep image: /etc/kubernetes/manifests/*.yaml""", r"""Its image tag ends in `-broken`; the other components show the right tag."""),
hand(r"""sudo vim /etc/kubernetes/manifests/kube-controller-manager.yaml""",
r"""Remove `-broken` from the end of the `image:` line, then `:wq`.""",
run=r"""sudo sed -i 's/-broken$//' /etc/kubernetes/manifests/kube-controller-manager.yaml"""),
cmd(r"""sudo crictl ps --name kube-controller-manager
k -n ops get deploy queue""", r"""Again until `queue` shows `2/2`.""")]),
step("Ask the API server how it feels",
r"""The API server has health endpoints of its own: `/livez` (restart me if this fails) and `/readyz` (send me traffic). With `?verbose` they list every check.
Save the output of `kubectl get --raw '/readyz?verbose'` in `{WS}/lesson-2.2/readyz.txt`.""",
check=r"""grep -q '^\[+\]etcd ok' "$WS/lesson-2.2/readyz.txt" 2>/dev/null && grep -q 'readyz check passed' "$WS/lesson-2.2/readyz.txt" """,
hint=r"""`kubectl get --raw` sends a GET to any API path, with your credentials. Quote the path: `?` means something to the shell.""",
show=[cmd(r"""k get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt
k get --raw /livez""")]),
],
"outro": r"""You read the control plane from both sides - `kubectl` and the node's container runtime - and repaired two static Pods by editing their manifests in place.
**For the exam**
- Static Pod manifests: `/etc/kubernetes/manifests` (the kubelet's `staticPodPath`). Edit the file; the kubelet restarts the Pod. Keep a copy *outside* that directory - a backup inside it is started too.
- Symptoms: no scheduler, Pods `Pending` with no events; no controller manager, Deployments without ReplicaSets; no API server, `kubectl` refuses to connect - then `crictl ps -a`, `crictl logs`, `/var/log/pods/kube-system_kube-apiserver-*/`.
- Typical faults: a wrong image tag, a typo in the command or a flag, a wrong path to a certificate, a volume mount that does not match a `hostPath`.
- `kubectl get --raw '/readyz?verbose'` and `/livez` show the API server's own checks, etcd's among them.
Next, prove it against the clock: **drill 2.2**.""",
}
# ---- 2.3 resource usage -----------------------------------------------------------------------
L23 = {
"id": 9203, "objective": "2.3", "workers": 1,
"title": "What uses the CPU and memory: kubectl top and the metrics pipeline",
"intro": r"""`kubectl top` shows what Pods and nodes **use** right now. The numbers come from **metrics-server**, which asks every kubelet every 15 seconds and serves the result as an API of its own, `metrics.k8s.io`:
```
kubelet (cAdvisor) ──▶ metrics-server ──▶ /apis/metrics.k8s.io ──▶ kubectl top, HorizontalPodAutoscaler
```
Without metrics-server there is no `kubectl top` and no autoscaling on CPU or memory. Usage is not the same as **requests**: requests are what the scheduler reserves, and `kubectl describe node` shows them under *Allocated resources*.
metrics-server runs on this cluster. Namespace `metrics` has a few Pods: one of them burns CPU, one holds a lot of memory, and Pod `duo` has two containers.""",
"setup": NS + r"""
ensure_metrics_server || { echo "metrics-server did not become ready" >&2; exit 1; }
mkns metrics
mkdir -p "$WS/lesson-2.3"
$K -n metrics apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: burner}
spec:
containers:
- name: burner
image: busybox:1.36
command: [sh, -c, 'while true; do :; done']
resources: {limits: {cpu: 300m, memory: 32Mi}}
---
apiVersion: v1
kind: Pod
metadata: {name: hoarder}
spec:
containers:
- name: hoarder
image: busybox:1.36
command: [sh, -c, 'dd if=/dev/zero of=/cache/blob bs=1M count=90 2>/dev/null; sleep 86400']
volumeMounts: [{name: cache, mountPath: /cache}]
volumes: [{name: cache, emptyDir: {medium: Memory}}]
---
apiVersion: v1
kind: Pod
metadata: {name: idler}
spec:
containers:
- {name: idler, image: busybox:1.36, command: [sleep, '86400']}
---
apiVersion: v1
kind: Pod
metadata: {name: duo}
spec:
containers:
- {name: app, image: busybox:1.36, command: [sleep, '86400']}
- name: cache
image: busybox:1.36
command: [sh, -c, 'dd if=/dev/zero of=/cache/blob bs=1M count=40 2>/dev/null; sleep 86400']
volumeMounts: [{name: cache, mountPath: /cache}]
volumes: [{name: cache, emptyDir: {medium: Memory}}]
EOF
$K -n metrics wait --for=condition=Ready pod --all --timeout=180s >/dev/null
# until metrics-server has a sample of every Pod
for i in $(seq 1 40); do [ "$($K top pod -n metrics --no-headers 2>/dev/null | grep -c .)" -ge 4 ] && exit 0; sleep 5; done
echo "kubectl top shows no metrics for the lesson's Pods" >&2; exit 1
""",
"steps": [
step("The heaviest Pod by memory",
r"""Write the name of the Pod in `metrics` that uses the most memory to `{WS}/lesson-2.3/memory.txt`. `kubectl top` can sort for you.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/memory.txt" 2>/dev/null)" hoarder""",
hint=r"""`kubectl top pod -n metrics --sort-by=memory`. `--no-headers | head -1 | awk '{print $1}'` keeps just the name.""",
show=[cmd(r"""k top po -n metrics --sort-by=memory"""),
hand(r"""echo POD > {WS}/lesson-2.3/memory.txt""",
r"""`POD`: the first one.""",
run=r"""kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt""")]),
step("The busiest Pod in the cluster",
r"""Across **all** Namespaces, which Pod uses the most CPU? Write it as `NAMESPACE/NAME` to `{WS}/lesson-2.3/cpu.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/cpu.txt" 2>/dev/null)" metrics/burner""",
hint=r"""`kubectl top pod -A --sort-by=cpu`; with `-A` the first column is the Namespace.""",
show=[cmd(r"""k top po -A --sort-by=cpu | head -5"""),
hand(r"""echo NAMESPACE/POD > {WS}/lesson-2.3/cpu.txt""",
r"""From the first line.""",
run=r"""kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt""")]),
step("Inside one Pod",
r"""`kubectl top pod` adds up a Pod's containers. Which container of Pod `duo` uses the memory? Write its name to `{WS}/lesson-2.3/container.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/container.txt" 2>/dev/null)" cache""",
hint=r"""`--containers` splits the Pod into its containers.""",
show=[cmd(r"""k top po duo -n metrics --containers"""),
hand(r"""echo CONTAINER > {WS}/lesson-2.3/container.txt""",
r"""The container with the most `MEMORY(bytes)`.""",
run=r"""kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt""")]),
step("Usage against what is reserved",
r"""`kubectl top node` shows use; `kubectl describe node {W1}`, under **Allocated resources**, shows what Pods have **requested** - which is what decides whether another Pod fits.
Write the memory that `{W1}` can give to Pods - its **allocatable** memory, as the API reports it (for example `8012345Ki`) - to `{WS}/lesson-2.3/allocatable.txt`.""",
check=r"""want=$(j node "$W1" '{.status.allocatable.memory}'); [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/allocatable.txt" 2>/dev/null)" "$want" """,
hint=r"""Allocatable is capacity minus what the node keeps for itself and for the eviction threshold: `.status.allocatable.memory`.""",
show=[cmd(r"""k top no
k describe no {W1} | grep -A6 -e Allocatable: -e 'Allocated resources' """, r"""`memory:` under `Allocatable:` is what `{W1}` can give to Pods, for example `8012345Ki`."""),
hand(r"""echo MEMORY > {WS}/lesson-2.3/allocatable.txt""",
r"""`MEMORY`: that value, as it stands.""",
run=r"""kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt""")]),
step("The metrics API itself",
r"""`kubectl top` only reads an API. Read it yourself: save what `/apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder` returns in `{WS}/lesson-2.3/raw.json`.
If this path answers `NotFound` or `ServiceUnavailable` on some cluster, `kubectl top` fails there too: look at the `metrics-server` Deployment in `kube-system`, and at `kubectl get apiservice v1beta1.metrics.k8s.io`.""",
check=r"""python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); sys.exit(0 if d.get("kind")=="PodMetrics" and d["metadata"]["name"]=="hoarder" and d.get("containers") else 1)' "$WS/lesson-2.3/raw.json" 2>/dev/null""",
hint=r"""`kubectl get --raw PATH`.""",
show=[cmd(r"""k get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json
cat {WS}/lesson-2.3/raw.json
k get apiservice v1beta1.metrics.k8s.io""")]),
],
"outro": r"""You ranked Pods by memory and CPU, looked inside a Pod, compared use with what a node can give, and read the metrics API that `kubectl top` uses.
**For the exam**
- `kubectl top pod [-A] [-n NS] [-l SELECTOR] --sort-by=cpu|memory [--containers]`, `kubectl top node`. Write exactly what is asked - often just the name.
- No metrics: is metrics-server running, and is `apiservice v1beta1.metrics.k8s.io` `Available`? On kubeadm clusters it often needs `--kubelet-insecure-tls`.
- `top` is usage; `describe node` → *Allocated resources* is requests and limits. The scheduler goes by requests.
- Metrics lag by a few seconds and are averages: a Pod that has just started may not appear yet.
Next, prove it against the clock: **drill 2.3**.""",
}
# ---- 2.4 container output streams ------------------------------------------------------------
L24 = {
"id": 9204, "objective": "2.4", "workers": 1,
"title": "Logs, the previous container, the node's log files, and debug containers",
"intro": r"""Everything a container writes to **stdout** and **stderr** is kept by the node's runtime and served by `kubectl logs`:
```
kubectl logs POD [-c CONTAINER] one container (-c when the Pod has several)
--previous the instance before the last restart - where a crash is explained
-l app=api --prefix every Pod of a label, each line marked with its Pod
--since=10m --tail=50 -f a window, the end, follow
```
On the node the same output lives in `/var/log/pods/NAMESPACE_POD_UID/CONTAINER/N.log` (N counts restarts), linked from `/var/log/containers/`. `crictl logs` reads it too.
And when a container has no shell to `exec` into, `kubectl debug` adds an **ephemeral container** to the running Pod - your image, your tools, sharing the Pod's network and, with `--target`, its processes. `kubectl debug node/NAME` does the same for a node.
Namespace `logs` has Pod `flaky`, which crashed once; Pods `api-1` to `api-3`; and Pod `quiet`, built from an image with no shell at all.""",
"setup": NS + r"""
mkns logs
mkdir -p "$WS/lesson-2.4"
$K -n logs apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: flaky}
spec:
volumes: [{name: state, emptyDir: {}}]
containers:
- name: flaky
image: busybox:1.36
volumeMounts: [{name: state, mountPath: /state}]
command: [sh, -c, 'if [ ! -f /state/booted ]; then touch /state/booted; echo "FATAL config /etc/flaky.conf not found"; exit 1; fi; echo "INFO started on second attempt"; exec sleep 86400']
---
apiVersion: v1
kind: Pod
metadata: {name: quiet}
spec:
containers:
- {name: quiet, image: registry.k8s.io/pause:3.10.2}
EOF
for n in 1 2 3; do
msg='INFO request served'; [ $n = 2 ] && msg='ERROR upstream refused the request'
$K -n logs run api-$n --image=busybox:1.36 --labels=app=api -- sh -c "while true; do echo \"$msg\"; sleep 5; done" >/dev/null
done
$K -n logs wait --for=condition=Ready pod --all --timeout=180s >/dev/null
for i in $(seq 1 30); do [ "$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null && exit 0; sleep 3; done
echo "flaky did not restart" >&2; exit 1
""",
"steps": [
step("Why it crashed",
r"""Pod `flaky` is `Running` now, with one restart. The logs of the running container only say it started; the reason for the crash is in the logs of the instance **before** it.
Save those in `{WS}/lesson-2.4/previous.log`.""",
check=r"""grep -q 'FATAL config /etc/flaky.conf not found' "$WS/lesson-2.4/previous.log" 2>/dev/null && ! grep -q 'second attempt' "$WS/lesson-2.4/previous.log" """,
hint=r"""`kubectl logs --previous` (or `-p`).""",
show=[cmd(r"""k -n logs logs flaky
k -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log
cat {WS}/lesson-2.4/previous.log""")]),
step("The one replica that complains",
r"""Pods `api-1`, `api-2` and `api-3` all carry `app=api`. One of them logs errors. Read all three at once, find it, and write its name to `{WS}/lesson-2.4/erroring.txt`.""",
check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/erroring.txt" 2>/dev/null)" api-2""",
hint=r"""`kubectl logs -l app=api --prefix` puts `[pod/NAME/CONTAINER]` before each line. With a selector, kubectl shows only the last 10 lines of each Pod unless you say `--tail`.""",
show=[cmd(r"""k -n logs logs -l app=api --prefix --tail=20 | grep ERROR""", r"""`[pod/NAME/...]` at the start of the line names the Pod."""),
hand(r"""echo POD > {WS}/lesson-2.4/erroring.txt""",
r"""`POD`: that name.""",
run=r"""kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt""")]),
step("The log file on the node",
r"""The node keeps one file per container instance. Write the full path of the file that holds the output of `flaky`'s **current** instance to `{WS}/lesson-2.4/logfile.txt`.
Find the node with `kubectl -n logs get pod flaky -o wide`, then look under `/var/log/pods/` there. The directory name has the Pod's UID in it; the file name is the restart count.""",
check=r"""
p=$(tr -d ' \r\n' < "$WS/lesson-2.4/logfile.txt" 2>/dev/null)
uid=$(j -n logs pod flaky '{.metadata.uid}'); n=$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}'); node=$(j -n logs pod flaky '{.spec.nodeName}')
eq "$p" "/var/log/pods/logs_flaky_$uid/flaky/$n.log" && nx "$node" "grep -q 'second attempt' '$p'" </dev/null""",
hint=r"""`/var/log/pods/NAMESPACE_POD_UID/CONTAINER/RESTARTS.log`. `sudo ls -R /var/log/pods/ | grep flaky` on the node gets you close.""",
show=[cmd(r"""k -n logs get po flaky -o wide
k -n logs get po flaky -o yaml | grep uid:""", r"""Its node, and its UID."""),
hand(r"""ssh {W1} sudo ls /var/log/pods/""",
r"""`logs_flaky_UID` holds `flaky/`, one file per instance: `0.log` the first, `1.log` the current one.""",
run=r"""ssh -n {W1} sudo ls /var/log/pods/"""),
hand(r"""echo /var/log/pods/logs_flaky_UID/flaky/1.log > {WS}/lesson-2.4/logfile.txt""",
r"""With the Pod's UID.""",
run=r"""uid=$(kubectl -n logs get pod flaky -o jsonpath='{.metadata.uid}')
echo "/var/log/pods/logs_flaky_$uid/flaky/1.log" > {WS}/lesson-2.4/logfile.txt""")]),
step("Debug a container that has no shell",
r"""`kubectl -n logs exec quiet -- sh` fails: the `pause` image has nothing but its one program. Add an ephemeral container to the running Pod instead: name `dbg`, image `busybox:1.36`, **targeting** container `quiet` so that it sees its processes, running `sleep 3600`.
Then run `ps` in `dbg` and write the command of the target's process - the first one, PID 1 - to `{WS}/lesson-2.4/target.txt`.""",
check=r"""
eq "$(j -n logs pod quiet '{.spec.ephemeralContainers[0].name} {.spec.ephemeralContainers[0].targetContainerName}')" "dbg quiet" || exit 1
grep -qx '/pause' <(tr -d ' \r' < "$WS/lesson-2.4/target.txt" 2>/dev/null)""",
hint=r"""`kubectl debug POD --image=IMAGE --target=CONTAINER -c NAME -- COMMAND`. Interactively you would add `-it` and use the shell straight away; without it, `kubectl exec POD -c dbg -- ps` reaches it once it runs. Ephemeral containers cannot be removed - the Pod keeps them until it goes.""",
show=[cmd(r"""k -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600"""),
hand(r"""k -n logs exec quiet -c dbg -- ps""",
r"""Again if `dbg` is not up yet. PID 1 is the target's process.""",
run=r"""for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done"""),
hand(r"""echo COMMAND > {WS}/lesson-2.4/target.txt""",
r"""`COMMAND`: PID 1's command, as `ps` shows it.""",
run=r"""kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt""")]),
step("Debug the node itself",
r"""`kubectl debug node/{W1} --image=busybox:1.36` starts a Pod on that node with the node's whole filesystem mounted at `/host` - useful when you cannot `ssh` to a node, but can reach the API.
Use it to read the node's `/etc/os-release`, and write the value of its `PRETTY_NAME` to `{WS}/lesson-2.4/os.txt`.""",
check=r"""
want=$(nx "$W1" '. /etc/os-release; echo "$PRETTY_NAME"' </dev/null); [ -n "$want" ] || exit 1
eq "$(tr -d '\r"' < "$WS/lesson-2.4/os.txt" 2>/dev/null | sed 's/^PRETTY_NAME=//; s/ *$//')" "$want" \
&& [ -n "$($K get pods -A -o name 2>/dev/null | grep node-debugger)" ]""",
hint=r"""Without `-it`, give it a command and read its output with `kubectl logs`: the Pod is called `node-debugger-NODE-xxxxx`, in your current Namespace. Delete it afterwards.""",
show=[cmd(r"""k debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release
k get po""", r"""The `node-debugger-...` Pod: once it is `Completed`, its log is the file."""),
hand(r"""k logs node-debugger-NAME""",
r"""`node-debugger-NAME`: its full name, from `get po`.""",
run=r"""pod=$(kubectl get pods -o name | grep node-debugger | head -1)
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s"""),
hand(r"""echo 'VALUE' > {WS}/lesson-2.4/os.txt""",
r"""`VALUE`: what `PRETTY_NAME=` says, without the double quotes.""",
run=r"""pod=$(kubectl get pods -o name | grep node-debugger | head -1)
kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt""")]),
],
"outro": r"""You read a crashed instance's logs, searched several Pods at once, found the log file on the node, and debugged both a Pod without a shell and a node.
**For the exam**
- A crash is explained in `kubectl logs --previous`. `describe pod` adds the exit code and the reason (`Error`, `OOMKilled`, `Completed`).
- Several containers: `-c NAME`, or `--all-containers`. Several Pods: `-l SELECTOR --prefix`, and mind the default `--tail=10`.
- Write logs to a file exactly as asked: `> file` for stdout. A container's stderr is in its logs too - `kubectl logs` does not separate them; the node's log file does, with `stdout` / `stderr` on each line.
- `kubectl debug -it POD --image=busybox --target=CONTAINER` for a container without tools; `kubectl debug node/NAME -it --image=busybox`, then `chroot /host`, for a node.
Next, prove it against the clock: **drill 2.4**.""",
}
# ---- 2.5 services and networking --------------------------------------------------------------
L25 = {
"id": 9205, "objective": "2.5", "workers": 1,
"title": "A Service that does not answer: selector, ports, policy, and kube-proxy",
"intro": r"""A request to a Service goes through several hands, and each can drop it:
```
client ──DNS──▶ ClusterIP:port ──kube-proxy rules──▶ endpoint PodIP:targetPort ──NetworkPolicy──▶ container
```
So check them in order:
- **Endpoints**: `kubectl get endpointslices -l kubernetes.io/service-name=SVC`. None? The selector matches no ready Pod.
- **Ports**: `targetPort` must be the port the container listens on (a number, or the name of a container port).
- **NetworkPolicy**: once a policy selects a Pod, only what it allows gets in.
- **kube-proxy** turns each Service into rules on every node - iptables chains named `KUBE-SVC-...` here.
In Namespace `shop`, Deployment `web` (nginx) has a Service `web` in front of it, and nothing gets through. Pods `client` and `stranger` are there to send requests from.""",
"setup": NS + r"""
mkns shop
mkdir -p "$WS/lesson-2.5"
$K -n shop create deploy web --image=nginx:1.27 --replicas=2 --port=80 >/dev/null
$K -n shop apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: web}
spec:
selector: {app: webapp}
ports:
- {port: 80, targetPort: 8080}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: lockdown}
spec:
podSelector: {matchLabels: {app: web}}
policyTypes: [Ingress]
EOF
$K -n shop run client --image=busybox:1.36 --labels=app=client -- sleep 86400 >/dev/null
$K -n shop run stranger --image=busybox:1.36 --labels=app=stranger -- sleep 86400 >/dev/null
$K -n shop rollout status deploy web --timeout=180s >/dev/null
$K -n shop wait --for=condition=Ready pod --all --timeout=180s >/dev/null
""",
"steps": [
step("No endpoints",
r"""Ask the Service from `client`: `kubectl -n shop exec client -- wget -qO- -T3 web`. It times out. Look at the Service's endpoints: there are none, because its selector matches no Pod.
Fix the **Service** so that it selects the Pods of Deployment `web`.""",
check=r"""
eq "$(j -n shop svc web '{.spec.selector.app}')" web || exit 1
n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' | grep -c .)
[ "$n" -eq 2 ]""",
hint=r"""Compare `kubectl -n shop get svc web -o wide` (SELECTOR) with `kubectl -n shop get pods --show-labels`. `kubectl set selector` or `kubectl edit svc`.""",
show=[cmd(r"""k -n shop get endpointslices -l kubernetes.io/service-name=web
k -n shop get svc web -o wide
k -n shop get po --show-labels
k -n shop set selector svc web app=web
k -n shop get endpointslices -l kubernetes.io/service-name=web""")]),
step("The wrong port",
r"""Now the Service has endpoints, but they point at port `8080` - and nginx listens on `80`. Make the Service send traffic to the port the container serves.""",
check=r"""
eq "$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{.items[0].ports[0].port}')" 80 && eq "$(j -n shop svc web '{.spec.ports[0].port}')" 80""",
hint=r"""`targetPort` is the container's port; `port` is the Service's own. `kubectl -n shop edit svc web`, or a JSON patch on `/spec/ports/0/targetPort`.""",
show=[hand(r"""k -n shop edit svc web""",
r"""Make `targetPort: 8080` read `targetPort: 80`, then `:wq`.""",
run=r"""kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]' """),
cmd(r"""k -n shop get endpointslices -l kubernetes.io/service-name=web""")]),
step("Let the right client in",
r"""Still nothing: NetworkPolicy `lockdown` selects the `web` Pods and allows no ingress at all.
Keep it - do not delete it - and let **`client`** (label `app=client`) reach the `web` Pods on TCP 80. `stranger` must still be refused.""",
check=r"""
[ -n "$(j -n shop netpol lockdown '{.metadata.name}')" ] || exit 1
ok() { $K -n shop exec "$1" -- wget -qO- -T 3 web >/dev/null 2>&1; }
for i in 1 2 3 4 5; do ok client && break; sleep 3; done; ok client || exit 1
! ok stranger""",
hint=r"""Add an `ingress` rule to `lockdown`: `from: [{podSelector: {matchLabels: {app: client}}}]` with `ports: [{protocol: TCP, port: 80}]`. A NetworkPolicy's ports are the Pod's, not the Service's.""",
show=[hand(r"""k -n shop edit netpol lockdown""",
r"""Add the rule to its `spec:`, beside `policyTypes:`:
```
ingress:
- from:
- podSelector:
matchLabels: {app: client}
ports:
- {protocol: TCP, port: 80}
```""",
run=r"""kubectl -n shop apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: lockdown
spec:
podSelector:
matchLabels: {app: web}
policyTypes: [Ingress]
ingress:
- from:
- podSelector:
matchLabels: {app: client}
ports:
- {protocol: TCP, port: 80}
EOF"""),
cmd(r"""k -n shop exec client -- wget -qO- -T3 web | head -4
k -n shop exec stranger -- wget -qO- -T3 web""", r"""`client` gets the page; `stranger` times out.""")]),
step("How kube-proxy routes it",
r"""kube-proxy turned Service `web` into rules on every node. Find out how: its mode is in ConfigMap `kube-proxy` in `kube-system`. Write the mode to `{WS}/lesson-2.5/mode.txt`.
Then, on the control-plane node, find the iptables chain that kube-proxy made for this Service - the `KUBE-SVC-...` chain that its ClusterIP jumps to - and write the chain's name to `{WS}/lesson-2.5/chain.txt`.""",
check=r"""
mode=$($K -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}'); mode=${mode:-iptables}
eq "$(tr -d ' \r\n"' < "$WS/lesson-2.5/mode.txt" 2>/dev/null)" "$mode" || exit 1
ip=$(j -n shop svc web '{.spec.clusterIP}')
want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" </dev/null)
[ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.5/chain.txt" 2>/dev/null)" "$want" """,
hint=r"""`kubectl -n kube-system get cm kube-proxy -o yaml | grep mode`. On the node: `sudo iptables-save -t nat | grep CLUSTER-IP` - the line for the Service jumps (`-j`) to its chain. In `nftables` mode the rules are in `sudo nft list ruleset` instead.""",
show=[cmd(r"""k -n kube-system get cm kube-proxy -o yaml | grep mode:"""),
hand(r"""echo MODE > {WS}/lesson-2.5/mode.txt""",
r"""`MODE`: its value.""",
run=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt"""),
cmd(r"""k -n shop get svc web""", r"""Its `CLUSTER-IP`:"""),
hand(r"""sudo iptables-save -t nat | grep IP""",
r"""`IP`: that address. The rule for it jumps (`-j`) to a `KUBE-SVC-...` chain.""",
run='ip=$(kubectl -n shop get svc web -o jsonpath=\'{.spec.clusterIP}\')\nsudo iptables-save -t nat | grep -- "-d $ip/32"'),
hand(r"""echo KUBE-SVC-NAME > {WS}/lesson-2.5/chain.txt""",
r"""The chain's full name.""",
run=r"""ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt""")]),
],
"outro": r"""You took a Service from "times out" to answering by fixing its selector, its target port and a NetworkPolicy, and followed it into kube-proxy's rules.
**For the exam**
- Endpoints first: `kubectl get endpointslices -l kubernetes.io/service-name=SVC` (or `kubectl get endpoints SVC`). Empty means the selector, or Pods that are not Ready.
- `port` is the Service's, `targetPort` the container's; a named `targetPort` must match a `containerPort` name.
- Test from a Pod: `kubectl exec POD -- wget -qO- -T3 SVC:PORT`, or a throwaway `kubectl run tmp --rm -it --restart=Never --image=busybox:1.36 -- ...`.
- A NetworkPolicy never blocks what it does not select; once one selects a Pod, everything not allowed is dropped - and egress policies need DNS (UDP/TCP 53) allowed.
- kube-proxy: a DaemonSet in `kube-system`, configured by ConfigMap `kube-proxy`; `iptables-save -t nat | grep KUBE-SVC` or `nft list ruleset` on a node.
Next, prove it against the clock: **drill 2.5**.""",
}
LESSONS = [L21, L22, L23, L24, L25]
Reference in New Issue View Git Blame Copy Permalink