"""CKA lessons of the Troubleshooting domain (2.1-2.5). See cka/lessons.py for what a lesson is.""" from sim.bank import NS, step # ---- 2.1 clusters and nodes ------------------------------------------------------------------- L21 = { "id": 9201, "objective": "2.1", "workers": 1, "title": "A node goes NotReady: the kubelet, its configuration, and pressure", "intro": r"""A node is `Ready` while its **kubelet** reports in. When it stops, the control plane waits under a minute (its node monitor grace period) and marks the node `NotReady`; five minutes later - the default toleration for an unreachable node - its Pods are evicted. So for a `NotReady` node the questions go in this order: ``` kubectl get nodes / describe node what the control plane last heard ssh NODE; sudo -i systemctl status kubelet is it running, is it enabled? journalctl -u kubelet -n 50 why it stopped, or why it will not start /var/lib/kubelet/config.yaml its configuration (KubeletConfiguration) /etc/kubernetes/kubelet.conf how it reaches the API server /etc/systemd/system/kubelet.service.d/ its flags (drop-ins), and /var/lib/kubelet/kubeadm-flags.env systemctl status containerd the runtime it needs ``` The kubelet also watches the node's resources. Below a **hard eviction threshold** - `memory.available`, `nodefs.available`, ... - it sets a condition (`MemoryPressure`, `DiskPressure`, `PIDPressure`) and evicts Pods to recover. Something has happened to the kubelet on worker `{W1}`.""", "setup": r""" mkdir -p "$WS/lesson-2.1" nx "$W1" 'systemctl disable --now kubelet' /dev/null 2>&1 # the control plane notices after its grace period; the lesson starts with the node NotReady for i in $(seq 1 40); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" != True ] && exit 0; sleep 3; done echo "$W1 is still Ready a minute after its kubelet was stopped" >&2; exit 1 """, "steps": [ step("Which node, and since when", r"""Find the node that is not `Ready` and write its name to `{WS}/lesson-2.1/node.txt`. Then look at `kubectl describe node` for it: under **Conditions** every row says `Kubelet stopped posting node status` - the control plane has not heard from the kubelet, so it knows nothing more. The answer is on the node.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/node.txt" 2>/dev/null)" "$W1" """, hint=r"""`kubectl get nodes`. `--no-headers` and `awk '$2 != "Ready" {print $1}'` give just the name.""", show=r"""kubectl get nodes kubectl get nodes --no-headers | awk '$2 != "Ready" {print $1}' > {WS}/lesson-2.1/node.txt kubectl describe node {W1} | grep -A8 Conditions"""), step("Bring the kubelet back - for good", r"""On `{W1}` (`ssh {W1}`, then `sudo -i`), look at the kubelet: `systemctl status kubelet` says `inactive (dead)` and `disabled`. Start it, and enable it so that it survives a reboot - a fix that is gone after the next reboot is not a fix. Within a few seconds the node is `Ready` again.""", check=r""" nx "$W1" 'systemctl is-active -q kubelet && systemctl is-enabled -q kubelet' /dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["kubeletconfig"]["cgroupDriver"])') [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.1/cgroup.txt" 2>/dev/null)" "$want" """, hint=r"""`configz` returns JSON: `| jq -r .kubeletconfig.cgroupDriver`. It must match containerd's `SystemdCgroup` setting, or Pods fail to start.""", show=r"""ssh -n {W1} 'ps -o args= -C kubelet | tr " " "\n" | grep -- --config; sudo grep -n cgroupDriver /var/lib/kubelet/config.yaml' kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq -r .kubeletconfig.cgroupDriver > {WS}/lesson-2.1/cgroup.txt cat {WS}/lesson-2.1/cgroup.txt"""), step("Set the eviction thresholds", r"""Look at `evictionHard` in `/var/lib/kubelet/config.yaml` on `{W1}`: kind sets every threshold to `0%`, which means "never evict" - fine for a test cluster, not for a real one. On `{W1}`, make the kubelet evict when less than **`100Mi`** of memory or **`10%`** of the node filesystem is left: `memory.available: 100Mi` and `nodefs.available: 10%` under `evictionHard`. Restart the kubelet, and confirm it through `configz`.""", check=r""" $K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c ' import json, sys e = json.load(sys.stdin)["kubeletconfig"].get("evictionHard") or {} sys.exit(0 if e.get("memory.available") == "100Mi" and e.get("nodefs.available") == "10%" else 1)' || exit 1 eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True""", hint=r"""Edit the file with `vim`, or `sed`: change the `nodefs.available` line and add a `memory.available` line inside the `evictionHard:` block (two spaces of indent). A kubelet that does not come back after the restart: `journalctl -u kubelet -n 20` shows the line it could not parse.""", show=r"""ssh {W1} sudo bash <<'EOF' f=/var/lib/kubelet/config.yaml grep -n -A4 '^evictionHard:' $f sed -i 's/^ nodefs.available: .*/ nodefs.available: 10%/' $f sed -i '/^evictionHard:/a\ memory.available: 100Mi' $f grep -n -A5 '^evictionHard:' $f systemctl restart kubelet EOF kubectl get --raw /api/v1/nodes/{W1}/proxy/configz | jq .kubeletconfig.evictionHard"""), step("Read the pressure conditions", r"""The kubelet reports the result of those thresholds as node conditions. Write the status of `{W1}`'s three pressure conditions to `{WS}/lesson-2.1/pressure.txt`, one per line, as `TYPE STATUS` - for example `MemoryPressure False`. `True` on any of them means the kubelet is evicting Pods, and the node gets a taint (`node.kubernetes.io/memory-pressure`, ...) so no new Pods land there.""", check=r""" want=$(for c in MemoryPressure DiskPressure PIDPressure; do echo "$c $(j node "$W1" "{.status.conditions[?(@.type==\"$c\")].status}")"; done | sort) eq "$(tr -s ' \t' ' ' < "$WS/lesson-2.1/pressure.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$want" """, hint=r"""`kubectl describe node {W1}` lists them under Conditions. As a list: `-o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}'`, then keep the three `*Pressure` lines.""", show=r"""kubectl get node {W1} -o jsonpath='{range .status.conditions[*]}{.type} {.status}{"\n"}{end}' | grep Pressure > {WS}/lesson-2.1/pressure.txt cat {WS}/lesson-2.1/pressure.txt"""), ], "outro": r"""You found a `NotReady` node, brought its kubelet back for good, read its configuration from the file and from the running kubelet, set eviction thresholds and read what the kubelet reports about them. **For the exam** - `NotReady`: the node is the place to look. `systemctl status kubelet`, then `journalctl -u kubelet`. A kubelet that restarts in a loop prints the reason a moment before each exit. - Fixes stay fixed: `systemctl enable`, and edits in the files, not on the command line. - Usual faults: kubelet stopped or disabled, a wrong path in `config.yaml` (CA file, static Pod path), a wrong flag in a drop-in or `kubeadm-flags.env` (`systemctl daemon-reload` after editing a unit), containerd down or the wrong runtime endpoint, an expired client certificate. - `kubectl get --raw /api/v1/nodes/NODE/proxy/configz` is the kubelet's live configuration. - Eviction: `evictionHard` (and `evictionSoft` with a grace period) in `config.yaml`. Pressure conditions taint the node; evicted Pods show `Failed` with reason `Evicted`. Next, prove it against the clock: **drill 2.1**.""", } # ---- 2.2 cluster components ------------------------------------------------------------------- _BREAK_CP = r""" mkns ops mkdir -p "$WS/lesson-2.2" gone() { local i; for i in $(seq 1 45); do [ -z "$(nx "$CP" "crictl ps --name $1 -q" &2; exit 1; } $K -n ops create deploy queue --image=busybox:1.36 --replicas=2 -- sleep 86400 >/dev/null nx "$CP" 'sed -i "s|^\( *\)- kube-scheduler$|\1- kube-schedulerr|" /etc/kubernetes/manifests/kube-scheduler.yaml' &2; exit 1; } $K -n ops run waiting --image=busybox:1.36 -- sleep 86400 >/dev/null """ L22 = { "id": 9202, "objective": "2.2", "workers": 1, "title": "The control plane's own Pods: find what is down and why", "intro": r"""On a kubeadm cluster the control plane runs as **static Pods**: the kubelet of the control-plane node starts whatever manifests it finds in `/etc/kubernetes/manifests`, and restarts them when a file changes. Each component has its own symptom when it is down: ``` kube-apiserver kubectl itself fails: "connection refused" etcd the API server fails, then kubectl kube-scheduler new Pods stay Pending, with no node and no event kube-controller-manager Deployments get no ReplicaSet, ReplicaSets no Pods; nothing self-heals ``` When the API server is up, `kubectl -n kube-system get pods` and `describe` show the others. When it is not, go to the node: `crictl ps -a`, `crictl logs ID`, the files under `/var/log/pods/`, and `journalctl -u kubelet`. Two things in Namespace `ops` do not work: Deployment `queue` has no Pods, and Pod `waiting` has no node.""", "setup": NS + _BREAK_CP, "steps": [ step("Which components are down", r"""Find the control-plane components that are not running and write their names to `{WS}/lesson-2.2/down.txt`, one per line (for example `kube-proxy`). `kubectl -n kube-system get pods` shows them; on the control-plane node, `sudo crictl ps -a` shows their containers, and `ls /etc/kubernetes/manifests` what the kubelet is told to run.""", check=r"""eq "$(tr -d ' \r' < "$WS/lesson-2.2/down.txt" 2>/dev/null | grep . | sort | tr '\n' ' ')" "kube-controller-manager kube-scheduler " """, hint=r"""A Pod that is not `Running` with all its containers ready. The Pod names end in the node's name; the component is the part before it.""", show=r"""kubectl -n kube-system get pods ls /etc/kubernetes/manifests sudo crictl ps -a | grep kube- printf 'kube-controller-manager\nkube-scheduler\n' > {WS}/lesson-2.2/down.txt"""), step("Repair the scheduler", r"""Find out why `kube-scheduler` does not start, and fix its manifest. `kubectl -n kube-system describe pod kube-scheduler-{CP}` shows the error in its events, and so do `sudo crictl ps -a --name kube-scheduler` and `sudo journalctl -u kubelet`. The kubelet notices the saved file on its own: no restart, no `kubectl apply`. Once the scheduler runs, Pod `waiting` gets a node.""", check=r""" nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml && [ -n "$(crictl ps --name kube-scheduler -q)" ]' /dev/null && grep -q 'readyz check passed' "$WS/lesson-2.2/readyz.txt" """, hint=r"""`kubectl get --raw` sends a GET to any API path, with your credentials. Quote the path: `?` means something to the shell.""", show=r"""kubectl get --raw '/readyz?verbose' | tee {WS}/lesson-2.2/readyz.txt | tail -3 kubectl get --raw '/livez'; echo"""), ], "outro": r"""You read the control plane from both sides - `kubectl` and the node's container runtime - and repaired two static Pods by editing their manifests in place. **For the exam** - Static Pod manifests: `/etc/kubernetes/manifests` (the kubelet's `staticPodPath`). Edit the file; the kubelet restarts the Pod. Keep a copy *outside* that directory - a backup inside it is started too. - Symptoms: no scheduler, Pods `Pending` with no events; no controller manager, Deployments without ReplicaSets; no API server, `kubectl` refuses to connect - then `crictl ps -a`, `crictl logs`, `/var/log/pods/kube-system_kube-apiserver-*/`. - Typical faults: a wrong image tag, a typo in the command or a flag, a wrong path to a certificate, a volume mount that does not match a `hostPath`. - `kubectl get --raw '/readyz?verbose'` and `/livez` show the API server's own checks, etcd's among them. Next, prove it against the clock: **drill 2.2**.""", } # ---- 2.3 resource usage ----------------------------------------------------------------------- L23 = { "id": 9203, "objective": "2.3", "workers": 1, "title": "What uses the CPU and memory: kubectl top and the metrics pipeline", "intro": r"""`kubectl top` shows what Pods and nodes **use** right now. The numbers come from **metrics-server**, which asks every kubelet every 15 seconds and serves the result as an API of its own, `metrics.k8s.io`: ``` kubelet (cAdvisor) ──▶ metrics-server ──▶ /apis/metrics.k8s.io ──▶ kubectl top, HorizontalPodAutoscaler ``` Without metrics-server there is no `kubectl top` and no autoscaling on CPU or memory. Usage is not the same as **requests**: requests are what the scheduler reserves, and `kubectl describe node` shows them under *Allocated resources*. metrics-server runs on this cluster. Namespace `metrics` has a few Pods: one of them burns CPU, one holds a lot of memory, and Pod `duo` has two containers.""", "setup": NS + r""" ensure_metrics_server || { echo "metrics-server did not become ready" >&2; exit 1; } mkns metrics mkdir -p "$WS/lesson-2.3" $K -n metrics apply -f - >/dev/null <<'EOF' apiVersion: v1 kind: Pod metadata: {name: burner} spec: containers: - name: burner image: busybox:1.36 command: [sh, -c, 'while true; do :; done'] resources: {limits: {cpu: 300m, memory: 32Mi}} --- apiVersion: v1 kind: Pod metadata: {name: hoarder} spec: containers: - name: hoarder image: busybox:1.36 command: [sh, -c, 'dd if=/dev/zero of=/cache/blob bs=1M count=90 2>/dev/null; sleep 86400'] volumeMounts: [{name: cache, mountPath: /cache}] volumes: [{name: cache, emptyDir: {medium: Memory}}] --- apiVersion: v1 kind: Pod metadata: {name: idler} spec: containers: - {name: idler, image: busybox:1.36, command: [sleep, '86400']} --- apiVersion: v1 kind: Pod metadata: {name: duo} spec: containers: - {name: app, image: busybox:1.36, command: [sleep, '86400']} - name: cache image: busybox:1.36 command: [sh, -c, 'dd if=/dev/zero of=/cache/blob bs=1M count=40 2>/dev/null; sleep 86400'] volumeMounts: [{name: cache, mountPath: /cache}] volumes: [{name: cache, emptyDir: {medium: Memory}}] EOF $K -n metrics wait --for=condition=Ready pod --all --timeout=180s >/dev/null # until metrics-server has a sample of every Pod for i in $(seq 1 40); do [ "$($K top pod -n metrics --no-headers 2>/dev/null | grep -c .)" -ge 4 ] && exit 0; sleep 5; done echo "kubectl top shows no metrics for the lesson's Pods" >&2; exit 1 """, "steps": [ step("The heaviest Pod by memory", r"""Write the name of the Pod in `metrics` that uses the most memory to `{WS}/lesson-2.3/memory.txt`. `kubectl top` can sort for you.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/memory.txt" 2>/dev/null)" hoarder""", hint=r"""`kubectl top pod -n metrics --sort-by=memory`. `--no-headers | head -1 | awk '{print $1}'` keeps just the name.""", show=r"""kubectl top pod -n metrics --sort-by=memory kubectl top pod -n metrics --sort-by=memory --no-headers | head -1 | awk '{print $1}' > {WS}/lesson-2.3/memory.txt"""), step("The busiest Pod in the cluster", r"""Across **all** Namespaces, which Pod uses the most CPU? Write it as `NAMESPACE/NAME` to `{WS}/lesson-2.3/cpu.txt`.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/cpu.txt" 2>/dev/null)" metrics/burner""", hint=r"""`kubectl top pod -A --sort-by=cpu`; with `-A` the first column is the Namespace.""", show=r"""kubectl top pod -A --sort-by=cpu | head -5 kubectl top pod -A --sort-by=cpu --no-headers | head -1 | awk '{print $1 "/" $2}' > {WS}/lesson-2.3/cpu.txt"""), step("Inside one Pod", r"""`kubectl top pod` adds up a Pod's containers. Which container of Pod `duo` uses the memory? Write its name to `{WS}/lesson-2.3/container.txt`.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/container.txt" 2>/dev/null)" cache""", hint=r"""`--containers` splits the Pod into its containers.""", show=r"""kubectl top pod duo -n metrics --containers kubectl top pod duo -n metrics --containers --no-headers | awk '{sub(/Mi$/, "", $4); print $4, $2}' | sort -n | tail -1 | cut -d' ' -f2 > {WS}/lesson-2.3/container.txt"""), step("Usage against what is reserved", r"""`kubectl top node` shows use; `kubectl describe node {W1}`, under **Allocated resources**, shows what Pods have **requested** - which is what decides whether another Pod fits. Write the memory that `{W1}` can give to Pods - its **allocatable** memory, as the API reports it (for example `8012345Ki`) - to `{WS}/lesson-2.3/allocatable.txt`.""", check=r"""want=$(j node "$W1" '{.status.allocatable.memory}'); [ -n "$want" ] && eq "$(tr -d ' \r\n' < "$WS/lesson-2.3/allocatable.txt" 2>/dev/null)" "$want" """, hint=r"""Allocatable is capacity minus what the node keeps for itself and for the eviction threshold: `.status.allocatable.memory`.""", show=r"""kubectl top node kubectl describe node {W1} | grep -A9 'Allocated resources' kubectl get node {W1} -o jsonpath='{.status.allocatable.memory}' > {WS}/lesson-2.3/allocatable.txt"""), step("The metrics API itself", r"""`kubectl top` only reads an API. Read it yourself: save what `/apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder` returns in `{WS}/lesson-2.3/raw.json`. If this path answers `NotFound` or `ServiceUnavailable` on some cluster, `kubectl top` fails there too: look at the `metrics-server` Deployment in `kube-system`, and at `kubectl get apiservice v1beta1.metrics.k8s.io`.""", check=r"""python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); sys.exit(0 if d.get("kind")=="PodMetrics" and d["metadata"]["name"]=="hoarder" and d.get("containers") else 1)' "$WS/lesson-2.3/raw.json" 2>/dev/null""", hint=r"""`kubectl get --raw PATH`.""", show=r"""kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/metrics/pods/hoarder > {WS}/lesson-2.3/raw.json jq '.containers' {WS}/lesson-2.3/raw.json kubectl get apiservice v1beta1.metrics.k8s.io"""), ], "outro": r"""You ranked Pods by memory and CPU, looked inside a Pod, compared use with what a node can give, and read the metrics API that `kubectl top` uses. **For the exam** - `kubectl top pod [-A] [-n NS] [-l SELECTOR] --sort-by=cpu|memory [--containers]`, `kubectl top node`. Write exactly what is asked - often just the name. - No metrics: is metrics-server running, and is `apiservice v1beta1.metrics.k8s.io` `Available`? On kubeadm clusters it often needs `--kubelet-insecure-tls`. - `top` is usage; `describe node` → *Allocated resources* is requests and limits. The scheduler goes by requests. - Metrics lag by a few seconds and are averages: a Pod that has just started may not appear yet. Next, prove it against the clock: **drill 2.3**.""", } # ---- 2.4 container output streams ------------------------------------------------------------ L24 = { "id": 9204, "objective": "2.4", "workers": 1, "title": "Logs, the previous container, the node's log files, and debug containers", "intro": r"""Everything a container writes to **stdout** and **stderr** is kept by the node's runtime and served by `kubectl logs`: ``` kubectl logs POD [-c CONTAINER] one container (-c when the Pod has several) --previous the instance before the last restart - where a crash is explained -l app=api --prefix every Pod of a label, each line marked with its Pod --since=10m --tail=50 -f a window, the end, follow ``` On the node the same output lives in `/var/log/pods/NAMESPACE_POD_UID/CONTAINER/N.log` (N counts restarts), linked from `/var/log/containers/`. `crictl logs` reads it too. And when a container has no shell to `exec` into, `kubectl debug` adds an **ephemeral container** to the running Pod - your image, your tools, sharing the Pod's network and, with `--target`, its processes. `kubectl debug node/NAME` does the same for a node. Namespace `logs` has Pod `flaky`, which crashed once; Pods `api-1` to `api-3`; and Pod `quiet`, built from an image with no shell at all.""", "setup": NS + r""" mkns logs mkdir -p "$WS/lesson-2.4" $K -n logs apply -f - >/dev/null <<'EOF' apiVersion: v1 kind: Pod metadata: {name: flaky} spec: volumes: [{name: state, emptyDir: {}}] containers: - name: flaky image: busybox:1.36 volumeMounts: [{name: state, mountPath: /state}] command: [sh, -c, 'if [ ! -f /state/booted ]; then touch /state/booted; echo "FATAL config /etc/flaky.conf not found"; exit 1; fi; echo "INFO started on second attempt"; exec sleep 86400'] --- apiVersion: v1 kind: Pod metadata: {name: quiet} spec: containers: - {name: quiet, image: registry.k8s.io/pause:3.10.2} EOF for n in 1 2 3; do msg='INFO request served'; [ $n = 2 ] && msg='ERROR upstream refused the request' $K -n logs run api-$n --image=busybox:1.36 --labels=app=api -- sh -c "while true; do echo \"$msg\"; sleep 5; done" >/dev/null done $K -n logs wait --for=condition=Ready pod --all --timeout=180s >/dev/null for i in $(seq 1 30); do [ "$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null && exit 0; sleep 3; done echo "flaky did not restart" >&2; exit 1 """, "steps": [ step("Why it crashed", r"""Pod `flaky` is `Running` now, with one restart. The logs of the running container only say it started; the reason for the crash is in the logs of the instance **before** it. Save those in `{WS}/lesson-2.4/previous.log`.""", check=r"""grep -q 'FATAL config /etc/flaky.conf not found' "$WS/lesson-2.4/previous.log" 2>/dev/null && ! grep -q 'second attempt' "$WS/lesson-2.4/previous.log" """, hint=r"""`kubectl logs --previous` (or `-p`).""", show=r"""kubectl -n logs logs flaky kubectl -n logs logs flaky --previous > {WS}/lesson-2.4/previous.log cat {WS}/lesson-2.4/previous.log"""), step("The one replica that complains", r"""Pods `api-1`, `api-2` and `api-3` all carry `app=api`. One of them logs errors. Read all three at once, find it, and write its name to `{WS}/lesson-2.4/erroring.txt`.""", check=r"""eq "$(tr -d ' \r\n' < "$WS/lesson-2.4/erroring.txt" 2>/dev/null)" api-2""", hint=r"""`kubectl logs -l app=api --prefix` puts `[pod/NAME/CONTAINER]` before each line. With a selector, kubectl shows only the last 10 lines of each Pod unless you say `--tail`.""", show=r"""kubectl -n logs logs -l app=api --prefix --tail=2 kubectl -n logs logs -l app=api --prefix --tail=20 | grep ERROR | head -1 | sed 's|^\[pod/\([^/]*\)/.*|\1|' > {WS}/lesson-2.4/erroring.txt cat {WS}/lesson-2.4/erroring.txt"""), step("The log file on the node", r"""The node keeps one file per container instance. Write the full path of the file that holds the output of `flaky`'s **current** instance to `{WS}/lesson-2.4/logfile.txt`. Find the node with `kubectl -n logs get pod flaky -o wide`, then look under `/var/log/pods/` there. The directory name has the Pod's UID in it; the file name is the restart count.""", check=r""" p=$(tr -d ' \r\n' < "$WS/lesson-2.4/logfile.txt" 2>/dev/null) uid=$(j -n logs pod flaky '{.metadata.uid}'); n=$(j -n logs pod flaky '{.status.containerStatuses[0].restartCount}'); node=$(j -n logs pod flaky '{.spec.nodeName}') eq "$p" "/var/log/pods/logs_flaky_$uid/flaky/$n.log" && nx "$node" "grep -q 'second attempt' '$p'" {WS}/lesson-2.4/logfile.txt ssh -n {W1} "sudo cat $(cat {WS}/lesson-2.4/logfile.txt)"""), step("Debug a container that has no shell", r"""`kubectl -n logs exec quiet -- sh` fails: the `pause` image has nothing but its one program. Add an ephemeral container to the running Pod instead: name `dbg`, image `busybox:1.36`, **targeting** container `quiet` so that it sees its processes, running `sleep 3600`. Then run `ps` in `dbg` and write the command of the target's process - the first one, PID 1 - to `{WS}/lesson-2.4/target.txt`.""", check=r""" eq "$(j -n logs pod quiet '{.spec.ephemeralContainers[0].name} {.spec.ephemeralContainers[0].targetContainerName}')" "dbg quiet" || exit 1 grep -qx '/pause' <(tr -d ' \r' < "$WS/lesson-2.4/target.txt" 2>/dev/null)""", hint=r"""`kubectl debug POD --image=IMAGE --target=CONTAINER -c NAME -- COMMAND`. Interactively you would add `-it` and use the shell straight away; without it, `kubectl exec POD -c dbg -- ps` reaches it once it runs. Ephemeral containers cannot be removed - the Pod keeps them until it goes.""", show=r"""kubectl -n logs debug quiet --image=busybox:1.36 --target=quiet -c dbg -- sleep 3600 for i in $(seq 1 30); do kubectl -n logs exec quiet -c dbg -- true 2>/dev/null && break; sleep 2; done kubectl -n logs exec quiet -c dbg -- ps kubectl -n logs exec quiet -c dbg -- ps -o pid,args | awk '$1 == 1 {print $2}' > {WS}/lesson-2.4/target.txt"""), step("Debug the node itself", r"""`kubectl debug node/{W1} --image=busybox:1.36` starts a Pod on that node with the node's whole filesystem mounted at `/host` - useful when you cannot `ssh` to a node, but can reach the API. Use it to read the node's `/etc/os-release`, and write the value of its `PRETTY_NAME` to `{WS}/lesson-2.4/os.txt`.""", check=r""" want=$(nx "$W1" '. /etc/os-release; echo "$PRETTY_NAME"' /dev/null | sed 's/^PRETTY_NAME=//; s/ *$//')" "$want" \ && [ -n "$($K get pods -A -o name 2>/dev/null | grep node-debugger)" ]""", hint=r"""Without `-it`, give it a command and read its output with `kubectl logs`: the Pod is called `node-debugger-NODE-xxxxx`, in your current Namespace. Delete it afterwards.""", show=r"""kubectl debug node/{W1} --image=busybox:1.36 -- cat /host/etc/os-release pod=$(kubectl get pods -o name | grep node-debugger | head -1) kubectl wait --for=jsonpath='{.status.phase}'=Succeeded $pod --timeout=90s kubectl logs $pod | sed -n 's/^PRETTY_NAME="\(.*\)"$/\1/p' > {WS}/lesson-2.4/os.txt cat {WS}/lesson-2.4/os.txt"""), ], "outro": r"""You read a crashed instance's logs, searched several Pods at once, found the log file on the node, and debugged both a Pod without a shell and a node. **For the exam** - A crash is explained in `kubectl logs --previous`. `describe pod` adds the exit code and the reason (`Error`, `OOMKilled`, `Completed`). - Several containers: `-c NAME`, or `--all-containers`. Several Pods: `-l SELECTOR --prefix`, and mind the default `--tail=10`. - Write logs to a file exactly as asked: `> file` for stdout. A container's stderr is in its logs too - `kubectl logs` does not separate them; the node's log file does, with `stdout` / `stderr` on each line. - `kubectl debug -it POD --image=busybox --target=CONTAINER` for a container without tools; `kubectl debug node/NAME -it --image=busybox`, then `chroot /host`, for a node. Next, prove it against the clock: **drill 2.4**.""", } # ---- 2.5 services and networking -------------------------------------------------------------- L25 = { "id": 9205, "objective": "2.5", "workers": 1, "title": "A Service that does not answer: selector, ports, policy, and kube-proxy", "intro": r"""A request to a Service goes through several hands, and each can drop it: ``` client ──DNS──▶ ClusterIP:port ──kube-proxy rules──▶ endpoint PodIP:targetPort ──NetworkPolicy──▶ container ``` So check them in order: - **Endpoints**: `kubectl get endpointslices -l kubernetes.io/service-name=SVC`. None? The selector matches no ready Pod. - **Ports**: `targetPort` must be the port the container listens on (a number, or the name of a container port). - **NetworkPolicy**: once a policy selects a Pod, only what it allows gets in. - **kube-proxy** turns each Service into rules on every node - iptables chains named `KUBE-SVC-...` here. In Namespace `shop`, Deployment `web` (nginx) has a Service `web` in front of it, and nothing gets through. Pods `client` and `stranger` are there to send requests from.""", "setup": NS + r""" mkns shop mkdir -p "$WS/lesson-2.5" $K -n shop create deploy web --image=nginx:1.27 --replicas=2 --port=80 >/dev/null $K -n shop apply -f - >/dev/null <<'EOF' apiVersion: v1 kind: Service metadata: {name: web} spec: selector: {app: webapp} ports: - {port: 80, targetPort: 8080} --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: {name: lockdown} spec: podSelector: {matchLabels: {app: web}} policyTypes: [Ingress] EOF $K -n shop run client --image=busybox:1.36 --labels=app=client -- sleep 86400 >/dev/null $K -n shop run stranger --image=busybox:1.36 --labels=app=stranger -- sleep 86400 >/dev/null $K -n shop rollout status deploy web --timeout=180s >/dev/null $K -n shop wait --for=condition=Ready pod --all --timeout=180s >/dev/null """, "steps": [ step("No endpoints", r"""Ask the Service from `client`: `kubectl -n shop exec client -- wget -qO- -T3 web`. It times out. Look at the Service's endpoints: there are none, because its selector matches no Pod. Fix the **Service** so that it selects the Pods of Deployment `web`.""", check=r""" eq "$(j -n shop svc web '{.spec.selector.app}')" web || exit 1 n=$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' | grep -c .) [ "$n" -eq 2 ]""", hint=r"""Compare `kubectl -n shop get svc web -o wide` (SELECTOR) with `kubectl -n shop get pods --show-labels`. `kubectl set selector` or `kubectl edit svc`.""", show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web kubectl -n shop get svc web -o wide kubectl -n shop get pods --show-labels kubectl -n shop set selector svc web app=web kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""), step("The wrong port", r"""Now the Service has endpoints, but they point at port `8080` - and nginx listens on `80`. Make the Service send traffic to the port the container serves.""", check=r""" eq "$($K -n shop get endpointslices -l kubernetes.io/service-name=web -o jsonpath='{.items[0].ports[0].port}')" 80 && eq "$(j -n shop svc web '{.spec.ports[0].port}')" 80""", hint=r"""`targetPort` is the container's port; `port` is the Service's own. `kubectl -n shop edit svc web`, or a JSON patch on `/spec/ports/0/targetPort`.""", show=r"""kubectl -n shop get endpointslices -l kubernetes.io/service-name=web kubectl -n shop patch svc web --type json -p '[{"op":"replace","path":"/spec/ports/0/targetPort","value":80}]' kubectl -n shop get endpointslices -l kubernetes.io/service-name=web"""), step("Let the right client in", r"""Still nothing: NetworkPolicy `lockdown` selects the `web` Pods and allows no ingress at all. Keep it - do not delete it - and let **`client`** (label `app=client`) reach the `web` Pods on TCP 80. `stranger` must still be refused.""", check=r""" [ -n "$(j -n shop netpol lockdown '{.metadata.name}')" ] || exit 1 ok() { $K -n shop exec "$1" -- wget -qO- -T 3 web >/dev/null 2>&1; } for i in 1 2 3 4 5; do ok client && break; sleep 3; done; ok client || exit 1 ! ok stranger""", hint=r"""Add an `ingress` rule to `lockdown`: `from: [{podSelector: {matchLabels: {app: client}}}]` with `ports: [{protocol: TCP, port: 80}]`. A NetworkPolicy's ports are the Pod's, not the Service's.""", show=r"""kubectl -n shop apply -f - <<'EOF' apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: lockdown spec: podSelector: matchLabels: {app: web} policyTypes: [Ingress] ingress: - from: - podSelector: matchLabels: {app: client} ports: - {protocol: TCP, port: 80} EOF kubectl -n shop exec client -- wget -qO- -T3 web | head -4 kubectl -n shop exec stranger -- wget -qO- -T3 web || echo refused"""), step("How kube-proxy routes it", r"""kube-proxy turned Service `web` into rules on every node. Find out how: its mode is in ConfigMap `kube-proxy` in `kube-system`. Write the mode to `{WS}/lesson-2.5/mode.txt`. Then, on the control-plane node, find the iptables chain that kube-proxy made for this Service - the `KUBE-SVC-...` chain that its ClusterIP jumps to - and write the chain's name to `{WS}/lesson-2.5/chain.txt`.""", check=r""" mode=$($K -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}'); mode=${mode:-iptables} eq "$(tr -d ' \r\n"' < "$WS/lesson-2.5/mode.txt" 2>/dev/null)" "$mode" || exit 1 ip=$(j -n shop svc web '{.spec.clusterIP}') want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" /dev/null)" "$want" """, hint=r"""`kubectl -n kube-system get cm kube-proxy -o yaml | grep mode`. On the node: `sudo iptables-save -t nat | grep CLUSTER-IP` - the line for the Service jumps (`-j`) to its chain. In `nftables` mode the rules are in `sudo nft list ruleset` instead.""", show=r"""kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' > {WS}/lesson-2.5/mode.txt ip=$(kubectl -n shop get svc web -o jsonpath='{.spec.clusterIP}') sudo iptables-save -t nat | grep -- "-d $ip/32" sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/lesson-2.5/chain.txt sudo iptables-save -t nat | grep "^-A $(cat {WS}/lesson-2.5/chain.txt)" """), ], "outro": r"""You took a Service from "times out" to answering by fixing its selector, its target port and a NetworkPolicy, and followed it into kube-proxy's rules. **For the exam** - Endpoints first: `kubectl get endpointslices -l kubernetes.io/service-name=SVC` (or `kubectl get endpoints SVC`). Empty means the selector, or Pods that are not Ready. - `port` is the Service's, `targetPort` the container's; a named `targetPort` must match a `containerPort` name. - Test from a Pod: `kubectl exec POD -- wget -qO- -T3 SVC:PORT`, or a throwaway `kubectl run tmp --rm -it --restart=Never --image=busybox:1.36 -- ...`. - A NetworkPolicy never blocks what it does not select; once one selects a Pod, everything not allowed is dropped - and egress policies need DNS (UDP/TCP 53) allowed. - kube-proxy: a DaemonSet in `kube-system`, configured by ConfigMap `kube-proxy`; `iptables-save -t nat | grep KUBE-SVC` or `nft list ruleset` on a node. Next, prove it against the clock: **drill 2.5**.""", } LESSONS = [L21, L22, L23, L24, L25]