vumast3r/kubestronaut-sim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6eb0e02c31
kubestronaut-sim/cka/bank_course.py
vumas3r e761d53a96 feat: show every CKA reference solution the way the exam is done 
105 of the 147 CKA reference solutions had you type what nobody types
in the exam: heredoc manifests, sed, JSON patches, kubectl wait, loops,
command substitution, jsonpath into files. Each is rewritten as the
lessons are: a generator and the lines to add in vim, a manifest pasted
from the docs page it names, kubectl edit, ssh NODE and sudo -i, or a
value read off the screen and echoed. The other 42 were commands typed
as they stand already.

A solution can now be parts (sim/bank.py cmd, hand) as a lesson step's
show is. The page and ./ksim solution N render it as Markdown, each
thing to type a block of its own that a click copies, with the edit
beneath it. A solution in parts is its own audit script: load_track
runs its parts, and for each edit by hand the script beside it, so the
solution you read is the one proven. Their separate scripts in
cka/solve.py are dropped, and a task that has both is refused.

./ksim solution lint [N|DOMAIN..] reads what the solutions have you
type against TYPING_TRAPS: the CKA bank is clean. Q49's params gain
the plural stem and shortName the solution spells out (no new draws);
Q103, ported from CKAD Q23, takes a solution of its own.

Audited from a copy with renamed hosts: Q3-Q67 (52 tasks) all solved by
their parts, after two waits were added to the audit scripts (Q43's
DNS record, Q53's Pods); Q68-Q147 still running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNSZLLsGXTMBsw17kDunkQ
2026-10-27 20:56:51 -06:00

1274 lines
67 KiB
Python
Raw Blame History

"""CKA pool tasks 132-147: what a full CKA course teaches beyond the bank as it was - checked on
2026-10-19 against the topics of KubeDojo's CKA course, and written here from scratch. Each has a
lesson that teaches it (cka/lessons_*.py); these are the timed tasks that test it.
132 upgrades its cluster from v1.36 to v1.37 for real ("k8s": "v1.36"). 133 drains a node under a
PodDisruptionBudget, 134 debugs a container without a shell, 135 issues and withholds
ServiceAccount tokens, 136 builds a projected volume, 137 reads QoS classes, 138 resizes a Pod in
place, 139 sets the kubelet's eviction thresholds, 140 follows a Service into kube-proxy's rules,
141 backs a Service with a hand-made EndpointSlice, 142 encrypts Secrets at rest, 143 spreads
replicas with anti-affinity, 144 rolls a StatefulSet out behind a partition, 145 makes an
operator with broken RBAC do its job, and 146-147 grow, snapshot, restore and clone volumes on
the CSI hostpath driver's class `csi-hostpath-sc`, as the real exam's clusters have it
(ensure_csi_hostpath).
"""
from sim.bank import cmd, hand
from cka.common import C, CAN, NET, NS, ONE, backup_operator, ns_pick, name_pick
_ETCD = ("ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt "
"--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key")
# packages of a node, and whether they are held
APT = r"""
pkgs() { nx "$1" "dpkg-query -W -f='\${Version} ' kubelet kubeadm kubectl" </dev/null; }
held() { [ "$(nx "$1" 'apt-mark showhold | grep -c -x -e kubelet -e kubeadm -e kubectl' </dev/null)" = 3 ]; }
"""
QUESTIONS = [
# ------------------------------------------------------------------ 132 ----
{
"id": 132, "domain": "ARCH", "difficulty": "hard", "workers": 1, "k8s": "v1.36", "title": "Upgrade the cluster to v1.37",
"params": lambda r: {"ns": ns_pick(r, "shop")},
"task": """
The cluster runs Kubernetes **v1.36**. Upgrade it to **v1.37.1**: first the control-plane node
`{CP}` (this host), then worker `{W1}`, the way kubeadm documents it.
- `{CP}`: the kubeadm package, `kubeadm upgrade apply`, then the kubelet and kubectl packages.
- `{W1}`: the kubeadm package, `kubeadm upgrade node`, then the kubelet and kubectl packages.
Drain each node before its kubelet is upgraded and make it schedulable again afterwards. The three
packages are held (`apt-mark`) and must be held again when you are done. Deployment `web` in
`%{ns}` must be available at the end.
Reach the worker with `{SHELL_W1}`.
""",
"setup": NS + r"""
mkns %{ns}
$K -n %{ns} create deploy web --image=nginx:1.27 --replicas=2 >/dev/null
$K -n %{ns} rollout status deploy web --timeout=180s >/dev/null
""",
"checks": [
C("the API server runs v1.37.1", 3, r"""
eq "$($K version -o json 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["serverVersion"]["gitVersion"])')" v1.37.1"""),
C("{CP}: kubelet v1.37.1; kubelet, kubeadm, kubectl packages 1.37.1 and held", 2, APT + r"""
eq "$(j node "$CP" '{.status.nodeInfo.kubeletVersion}')" v1.37.1 && eq "$(pkgs "$CP")" "1.37.1-1.1 1.37.1-1.1 1.37.1-1.1 " && held "$CP" """),
C("{W1}: kubelet v1.37.1; kubelet, kubeadm, kubectl packages 1.37.1 and held", 3, APT + r"""
eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" v1.37.1 && eq "$(pkgs "$W1")" "1.37.1-1.1 1.37.1-1.1 1.37.1-1.1 " && held "$W1" """),
C("both nodes Ready and schedulable, Deployment web available", 2, r"""
for n in "$CP" "$W1"; do
eq "$(j node "$n" '{.status.conditions[?(@.type=="Ready")].status}')" True && [ "$(j node "$n" '{.spec.unschedulable}')" != true ] || exit 1
done
eq "$(j node "$W1" '{.status.nodeInfo.kubeletVersion}')" v1.37.1 && eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" 2"""),
],
"solution": [hand(r"""sudo vim /etc/apt/sources.list.d/kubernetes.list""",
r"""The docs page *Upgrading kubeadm clusters* starts here: pkgs.k8s.io has one repository per minor, so 1.37 packages exist for apt only once the line says `/v1.37/` instead of `/v1.36/`.""",
run=r"""apt_get() { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq -y -o Dpkg::Options::=--force-confold "$@" </dev/null >/dev/null; }
sudo sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list"""),
hand(r"""sudo apt-get update
apt-cache madison kubeadm | head -3
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.37.1-1.1
sudo apt-mark hold kubeadm""",
r"""The control plane - this host - first, and kubeadm first: it drives the upgrade.""",
run=r"""apt_get update
sudo apt-mark unhold kubeadm >/dev/null && apt_get install kubeadm=1.37.1-1.1 && sudo apt-mark hold kubeadm >/dev/null"""),
hand(r"""sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.1""",
r"""Answer `y`; it takes a few minutes, and the API server restarts on the new version.""",
run=r"""sudo kubeadm upgrade apply v1.37.1 -y </dev/null | tail -2"""),
cmd(r"""k drain {CP} --ignore-daemonsets --delete-emptydir-data"""),
hand(r"""sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet""",
r"""Then the node's kubelet and kubectl, held again.""",
run=r"""sudo apt-mark unhold kubelet kubectl >/dev/null && apt_get install kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 && sudo apt-mark hold kubelet kubectl >/dev/null
sudo systemctl daemon-reload && sudo systemctl restart kubelet"""),
hand(r"""k uncordon {CP}
k get nodes""",
r"""Again until `k` answers - the kubelet is restarting. `{CP}` shows `v1.37.1`.""",
run=r"""for i in $(seq 1 30); do kubectl uncordon {CP} >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""k drain {W1} --ignore-daemonsets --delete-emptydir-data"""),
hand(r"""{SHELL_W1}
sudo -i
vim /etc/apt/sources.list.d/kubernetes.list""",
r"""`/v1.36/` to `/v1.37/` here too.""",
run=r""""""),
hand(r"""apt-get update
apt-mark unhold kubeadm kubelet kubectl
apt-get install -y kubeadm=1.37.1-1.1
kubeadm upgrade node
apt-get install -y kubelet=1.37.1-1.1 kubectl=1.37.1-1.1
apt-mark hold kubeadm kubelet kubectl
systemctl daemon-reload
systemctl restart kubelet
exit
exit""",
r"""A worker runs `kubeadm upgrade node`, not `apply`: it hosts no control plane, only its kubelet's configuration to update.""",
run=r"""ssh {W1} sudo bash <<'EOF'
export DEBIAN_FRONTEND=noninteractive
sed -i 's|/v1.36/|/v1.37/|' /etc/apt/sources.list.d/kubernetes.list
apt-get update -qq </dev/null
apt-mark unhold kubeadm kubelet kubectl >/dev/null
apt-get install -y -qq kubeadm=1.37.1-1.1 </dev/null >/dev/null
kubeadm upgrade node </dev/null | tail -1
apt-get install -y -qq kubelet=1.37.1-1.1 kubectl=1.37.1-1.1 </dev/null >/dev/null
apt-mark hold kubeadm kubelet kubectl >/dev/null
systemctl daemon-reload && systemctl restart kubelet
EOF"""),
cmd(r"""k uncordon {W1}
k get nodes""", r"""Both `Ready`, both `v1.37.1` - `VERSION` is each kubelet's.""")],
},
# ------------------------------------------------------------------ 133 ----
{
"id": 133, "domain": "WORK", "workers": 2, "title": "Maintenance under a disruption budget",
"params": lambda r: {"ns": ns_pick(r, "orders"), "dep": name_pick(r, "checkout")},
"task": """
Node `{W1}` goes down for maintenance tonight. Deployment `%{dep}` in `%{ns}` (3 replicas) must
never have fewer than **2** Pods available while that happens - not for a drain, and not for any
other voluntary disruption.
1. Create a PodDisruptionBudget `%{dep}` in `%{ns}` that guarantees it.
2. Drain `{W1}`. DaemonSet Pods stay where they are; Pod `scratch` in `%{ns}` is disposable.
3. Leave `{W1}` unschedulable. All three Pods of `%{dep}` must be running, on other nodes.
""",
"setup": NS + r"""
mkns %{ns}
$K uncordon "$W1" >/dev/null 2>&1
$K -n %{ns} apply -f - >/dev/null <<EOF
apiVersion: apps/v1
kind: Deployment
metadata: {name: %{dep}}
spec:
replicas: 3
selector: {matchLabels: {app: %{dep}}}
template:
metadata: {labels: {app: %{dep}}}
spec:
topologySpreadConstraints:
- {maxSkew: 1, topologyKey: kubernetes.io/hostname, whenUnsatisfiable: ScheduleAnyway, labelSelector: {matchLabels: {app: %{dep}}}}
containers: [{name: web, image: nginx:1.27}]
---
apiVersion: v1
kind: Pod
metadata: {name: scratch}
spec:
nodeName: $W1
terminationGracePeriodSeconds: 1
containers: [{name: scratch, image: busybox:1.36, command: [sleep, '86400']}]
EOF
$K -n %{ns} rollout status deploy %{dep} --timeout=180s >/dev/null && $K -n %{ns} wait --for=condition=Ready pod/scratch --timeout=120s >/dev/null
""",
"checks": [
C("PDB %{dep} keeps 2 of the 3 Pods available", 2, r"""
eq "$(j -n %{ns} pdb %{dep} '{.spec.selector.matchLabels.app}')" %{dep} || exit 1
m=$(j -n %{ns} pdb %{dep} '{.spec.minAvailable}'); u=$(j -n %{ns} pdb %{dep} '{.spec.maxUnavailable}')
[ "$m" = 2 ] || [ "$u" = 1 ]"""),
C("{W1} is drained and stays unschedulable", 2, r"""
eq "$(j node "$W1" '{.spec.unschedulable}')" true || exit 1
[ -z "$($K -n %{ns} get pods -o jsonpath="{range .items[?(@.spec.nodeName==\"$W1\")]}{.metadata.name}{end}" 2>/dev/null)" ]"""),
C("all 3 Pods of %{dep} are available on other nodes", 3, r"""
eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 || exit 1
for n in $($K -n %{ns} get pods -l app=%{dep} -o jsonpath='{.items[*].spec.nodeName}'); do [ "$n" != "$W1" ] || exit 1; done
eq "$(j node "$W1" '{.spec.unschedulable}')" true"""),
],
"solution": """
k -n %{ns} create pdb %{dep} --selector=app=%{dep} --min-available=2
k -n %{ns} get pods -o wide # which of them run on {W1}
k drain {W1} --ignore-daemonsets --delete-emptydir-data --force
k -n %{ns} get pods -o wide
# drain = cordon + evict. Evictions respect the PDB: with 3 available and minAvailable 2, one Pod
# may go at a time, and the drain retries the others until their replacements are ready. --force
# is needed for `scratch`, which no controller would re-create - it is deleted for good. Leave the
# node cordoned: `k uncordon {W1}` is for after the maintenance.
""",
},
# ------------------------------------------------------------------ 134 ----
{
"id": 134, "domain": "TSHOOT", "workers": 1, "title": "A container you cannot exec into",
"params": lambda r: {"ns": ns_pick(r, "vault"), "pod": name_pick(r, "sealer"), "mode": r.choice(["sealed-3", "sealed-7", "sealed-9"])},
"task": """
Pod `%{pod}` in `%{ns}` runs an image that holds nothing but its own program: `kubectl exec` finds
no shell. Its container `app` takes its environment from ConfigMap `%{pod}-env` - which someone has
edited since the container started.
Without restarting or replacing the Pod:
1. Attach an ephemeral container named `debugger` (image `busybox:1.36`) that sees the processes
of container `app`.
2. Find out which value of `MODE` the running process of `app` actually has, and write it to
`{WS}/q134/mode.txt`.
""",
"setup": NS + r"""
mkns %{ns}; rm -rf "$WS/q134"; mkdir -p "$WS/q134"
$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true --grace-period=1 >/dev/null 2>&1
$K -n %{ns} create configmap %{pod}-env --from-literal=MODE=%{mode} --dry-run=client -o yaml | $K apply -f - >/dev/null
$K -n %{ns} apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: %{pod}}
spec:
containers:
- name: app
image: registry.k8s.io/pause:3.10.2
envFrom: [{configMapRef: {name: %{pod}-env}}]
EOF
$K -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=120s >/dev/null || exit 1
$K -n %{ns} create configmap %{pod}-env --from-literal=MODE=open --dry-run=client -o yaml | $K apply -f - >/dev/null
j -n %{ns} pod %{pod} '{.metadata.uid}' > "$PRIV/q134.uid"
""",
"checks": [
C("ephemeral container debugger (busybox:1.36) targets app - in the same Pod, never restarted", 3, r"""
eq "$(j -n %{ns} pod %{pod} '{.spec.ephemeralContainers[?(@.name=="debugger")].targetContainerName} {.spec.ephemeralContainers[?(@.name=="debugger")].image}')" "app busybox:1.36" \
&& eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q134.uid") 0" """),
C("mode.txt holds the value the process runs with", 3, ONE + r"""eq "$(one "$WS/q134/mode.txt")" %{mode}"""),
],
"solution": [cmd(r"""k -n %{ns} exec %{pod} -- sh
k -n %{ns} get cm %{pod}-env -o yaml""", r"""No shell in the image. And the ConfigMap says `MODE=open` - now; the environment was read once, when the container started."""),
cmd(r"""k -n %{ns} debug %{pod} --image=busybox:1.36 --target=app -c debugger --profile=general -- sleep 3600""", r"""`--target` puts the debug container into `app`'s process namespace; the `general` profile gives it `SYS_PTRACE`, which reading another process's environment needs. Interactively: `k debug -it ... -- sh`."""),
hand(r"""k -n %{ns} exec %{pod} -c debugger -- ps""",
r"""Again until it answers: PID 1 is `/pause`, `app`'s process.""",
run=r"""for i in $(seq 1 30); do kubectl -n %{ns} exec %{pod} -c debugger -- true 2>/dev/null && break; sleep 2; done"""),
cmd(r"""k -n %{ns} exec %{pod} -c debugger -- cat /proc/1/environ | tr '\0' '\n'""", r"""`MODE=%{mode}`: what the process runs with."""),
hand(r"""echo %{mode} > {WS}/q134/mode.txt""",
r"""With the value after `MODE=`.""",
run=r"""kubectl -n %{ns} exec %{pod} -c debugger -- sh -c 'tr "\0" "\n" < /proc/1/environ' | sed -n 's/^MODE=//p' > {WS}/q134/mode.txt""")],
},
# ------------------------------------------------------------------ 135 ----
{
"id": 135, "domain": "ARCH", "title": "A token for the pipeline, none for the app",
"params": lambda r: {"ns": ns_pick(r, "ci"), "h": r.choice([2, 3, 4])},
"task": """
In Namespace `%{ns}`:
1. Create ServiceAccount `pipeline`, and allow it - through a Role and a RoleBinding, both named
`pipeline` - to `get`, `list`, `create`, `update` and `patch` Deployments, and nothing else.
2. Issue a token for `pipeline` that is valid for **%{h} hours**, and save just the token in
`{WS}/q135/pipeline.token`.
3. Deployment `frontend` never talks to the API server. Change it so that its Pods get **no**
ServiceAccount token at all. It keeps running as the `default` ServiceAccount.
""",
"setup": NS + r"""
mkns %{ns}; rm -rf "$WS/q135"; mkdir -p "$WS/q135"
$K -n %{ns} create deploy frontend --image=nginx:1.27 --replicas=2 >/dev/null
$K -n %{ns} rollout status deploy frontend --timeout=180s >/dev/null
""",
"checks": [
C("pipeline may get, list, create, update, patch Deployments - nothing more", 2, CAN + r"""
S=system:serviceaccount:%{ns}:pipeline
for v in get list create update patch; do can --as=$S -- $v deployments.apps -n %{ns} || exit 1; done
cannot --as=$S -- delete deployments.apps -n %{ns} && cannot --as=$S -- get secrets -n %{ns} && cannot --as=$S -- list pods -n %{ns} \
&& eq "$(j -n %{ns} rolebinding pipeline '{.roleRef.kind}/{.roleRef.name}')" Role/pipeline"""),
C("pipeline.token authenticates as pipeline and lasts %{h} hours", 3, r"""
t=$(tr -d ' \r\n' < "$WS/q135/pipeline.token" 2>/dev/null); [ -n "$t" ] || exit 1
srv=$($K config view --minify -o jsonpath='{.clusters[0].cluster.server}')
eq "$($K --kubeconfig=/dev/null --server="$srv" --insecure-skip-tls-verify --token="$t" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)" system:serviceaccount:%{ns}:pipeline || exit 1
echo "$t" | cut -d. -f2 | tr '_-' '/+' | python3 -c '
import base64, json, sys
s = sys.stdin.read().strip(); c = json.loads(base64.b64decode(s + "=" * (-len(s) % 4)))
sys.exit(0 if c["exp"] - c["iat"] == %{h} * 3600 else 1)'"""),
C("frontend's Pods run with no ServiceAccount token, as default", 3, r"""
eq "$(j -n %{ns} deploy frontend '{.status.availableReplicas} {.status.updatedReplicas}')" "2 2" || exit 1
sa=$(j -n %{ns} deploy frontend '{.spec.template.spec.serviceAccountName}'); [ -z "$sa" ] || [ "$sa" = default ] || exit 1
for p in $($K -n %{ns} get pods -l app=frontend -o name); do
$K -n %{ns} get "$p" -o json | python3 -c '
import json, sys; s = json.load(sys.stdin)["spec"]
tok = [v for v in s.get("volumes", []) if any("serviceAccountToken" in src for src in v.get("projected", {}).get("sources", []))]
sys.exit(1 if tok else 0)' || exit 1
done"""),
],
"solution": [cmd(r"""k -n %{ns} create serviceaccount pipeline
k -n %{ns} create role pipeline --verb=get,list,create,update,patch --resource=deployments.apps
k -n %{ns} create rolebinding pipeline --role=pipeline --serviceaccount=%{ns}:pipeline
k auth can-i delete deployments -n %{ns} --as=system:serviceaccount:%{ns}:pipeline""", r"""`no`."""),
cmd(r"""k -n %{ns} create token pipeline --duration=%{h}h > {WS}/q135/pipeline.token""", r"""The TokenRequest API issues a short-lived, bound token; nothing is stored in a Secret."""),
hand(r"""k -n %{ns} edit deploy frontend""",
r"""In the Pod template's `spec:`, beside `containers:` and indented like it, then `:wq`:
```
automountServiceAccountToken: false
```
It leaves out the projected volume `kube-api-access-*`. (On the ServiceAccount it works too; the Pod's setting wins.)""",
run=r"""kubectl -n %{ns} patch deploy frontend -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":false}}}}'"""),
cmd(r"""k -n %{ns} rollout status deploy frontend""")],
},
# ------------------------------------------------------------------ 136 ----
{
"id": 136, "domain": "STOR", "title": "One mount, four sources",
"params": lambda r: {"ns": ns_pick(r, "portal"), "pod": name_pick(r, "portal"), "mem": r.choice([96, 128, 160])},
"task": """
Create Pod `%{pod}` in `%{ns}`: image `busybox:1.36`, command `sleep 86400`, label `app=%{pod}`,
memory limit `%{mem}Mi`. Mount a **single** volume, read-only, at `/etc/portal`, that holds:
- every key of ConfigMap `portal-config`, each as a file of its name;
- key `password` of Secret `portal-db`, as the file `db/password`;
- the Pod's labels, as the file `labels`;
- the container's memory limit in MiB, as the file `mem-limit` (just the number).
""",
"setup": NS + r"""
mkns %{ns}
$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true --grace-period=1 >/dev/null 2>&1
$K -n %{ns} create configmap portal-config --from-literal=theme=dark --from-literal=locale=en_GB --dry-run=client -o yaml | $K apply -f - >/dev/null
$K -n %{ns} create secret generic portal-db --from-literal=password=pw-%{mem}-x --from-literal=user=portal --dry-run=client -o yaml | $K apply -f - >/dev/null
""",
"checks": [
C("one projected volume, read-only at /etc/portal", 2, r"""
eq "$(j -n %{ns} pod %{pod} '{.spec.containers[0].volumeMounts[?(@.mountPath=="/etc/portal")].readOnly}')" true || exit 1
v=$(j -n %{ns} pod %{pod} '{.spec.containers[0].volumeMounts[?(@.mountPath=="/etc/portal")].name}')
[ -n "$(j -n %{ns} pod %{pod} "{.spec.volumes[?(@.name==\"$v\")].projected.sources}")" ]"""),
C("the ConfigMap's keys and db/password are files there", 2, r"""
x() { $K -n %{ns} exec %{pod} -- cat "/etc/portal/$1" 2>/dev/null; }
eq "$(x theme)" dark && eq "$(x locale)" en_GB && eq "$(x db/password)" pw-%{mem}-x && ! $K -n %{ns} exec %{pod} -- test -e /etc/portal/user"""),
C("labels and mem-limit come from the downward API", 3, r"""
x() { $K -n %{ns} exec %{pod} -- cat "/etc/portal/$1" 2>/dev/null; }
x labels | grep -qx 'app="%{pod}"' && eq "$(x mem-limit)" %{mem}"""),
],
"solution": [cmd(r"""k -n %{ns} run %{pod} --image=busybox:1.36 --labels=app=%{pod} --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Replace the container's `resources: {}` with the limit and the mount:
```
resources:
limits:
memory: %{mem}Mi
volumeMounts:
- name: portal
mountPath: /etc/portal
readOnly: true
```
and add to the Pod's `spec:`, right below `spec:` (the docs page *Projected Volumes* has an example with all three kinds):
```
volumes:
- name: portal
projected:
sources:
- configMap:
name: portal-config
- secret:
name: portal-db
items:
- key: password
path: db/password
- downwardAPI:
items:
- path: labels
fieldRef:
fieldPath: metadata.labels
- path: mem-limit
resourceFieldRef:
containerName: %{pod}
resource: limits.memory
divisor: 1Mi
```
`items` picks keys and renames them; in a volume a `resourceFieldRef` needs the container's name, and the divisor turns bytes into MiB.""",
run=r"""yq -i '.spec.containers[0].resources = {"limits": {"memory": "%{mem}Mi"}}
| .spec.containers[0].volumeMounts = [{"name": "portal", "mountPath": "/etc/portal", "readOnly": true}]
| .spec.volumes = [{"name": "portal", "projected": {"sources": [
{"configMap": {"name": "portal-config"}},
{"secret": {"name": "portal-db", "items": [{"key": "password", "path": "db/password"}]}},
{"downwardAPI": {"items": [{"path": "labels", "fieldRef": {"fieldPath": "metadata.labels"}},
{"path": "mem-limit", "resourceFieldRef": {"containerName": "%{pod}", "resource": "limits.memory", "divisor": "1Mi"}}]}}]}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod %{pod}""",
r"""Again until it is `Running`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=120s"""),
cmd(r"""k -n %{ns} exec %{pod} -- ls -R /etc/portal
k -n %{ns} exec %{pod} -- cat /etc/portal/labels /etc/portal/mem-limit""")],
},
# ------------------------------------------------------------------ 137 ----
{
"id": 137, "domain": "WORK", "title": "Who is evicted first",
"params": lambda r: {"ns": ns_pick(r, "batch")},
"task": """
Namespace `%{ns}` runs Pods `alpha`, `beta`, `gamma` and `delta`, and Deployment `critical`.
1. Write the QoS class of each of the four Pods to `{WS}/q137/qos.txt`, one per line, as
`POD CLASS`.
2. When the node runs short of memory, which of the four does the kubelet evict **first**? Write
its name to `{WS}/q137/first.txt`.
3. Change Deployment `critical` so that its Pods are of class `Guaranteed`, with `200m` CPU and
`128Mi` memory.
""",
"setup": NS + r"""
mkns %{ns}; rm -rf "$WS/q137"; mkdir -p "$WS/q137"
$K -n %{ns} delete pod alpha beta gamma delta --ignore-not-found --wait=true --grace-period=1 >/dev/null 2>&1
$K -n %{ns} apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: alpha}
spec:
containers: [{name: c, image: busybox:1.36, command: [sleep, '86400'], resources: {requests: {cpu: 50m, memory: 32Mi}, limits: {cpu: 50m, memory: 32Mi}}}]
---
apiVersion: v1
kind: Pod
metadata: {name: beta}
spec:
containers: [{name: c, image: busybox:1.36, command: [sleep, '86400'], resources: {requests: {memory: 32Mi}}}]
---
apiVersion: v1
kind: Pod
metadata: {name: gamma}
spec:
containers: [{name: c, image: busybox:1.36, command: [sleep, '86400']}]
---
apiVersion: v1
kind: Pod
metadata: {name: delta}
spec:
containers:
- {name: main, image: busybox:1.36, command: [sleep, '86400'], resources: {requests: {cpu: 50m, memory: 32Mi}, limits: {cpu: 50m, memory: 32Mi}}}
- {name: helper, image: busybox:1.36, command: [sleep, '86400']}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: critical}
spec:
replicas: 1
selector: {matchLabels: {app: critical}}
template:
metadata: {labels: {app: critical}}
spec:
containers: [{name: app, image: nginx:1.27, resources: {requests: {cpu: 100m, memory: 64Mi}}}]
EOF
$K -n %{ns} wait --for=condition=Ready pod --all --timeout=180s >/dev/null
""",
"checks": [
C("qos.txt names each Pod's class", 3, r"""
eq "$(tr -s ' \t' ' ' < "$WS/q137/qos.txt" 2>/dev/null | tr -d '\r' | sed 's/ *$//' | grep . | sort)" "$(printf 'alpha Guaranteed\nbeta Burstable\ndelta Burstable\ngamma BestEffort')" """),
C("first.txt names the Pod evicted first", 2, ONE + r"""eq "$(one "$WS/q137/first.txt")" gamma"""),
C("critical's Pods are Guaranteed with 200m and 128Mi", 3, r"""
eq "$(j -n %{ns} deploy critical '{.status.availableReplicas} {.status.updatedReplicas}')" "1 1" || exit 1
eq "$(j -n %{ns} deploy critical '{.spec.template.spec.containers[0].resources.requests.cpu} {.spec.template.spec.containers[0].resources.limits.cpu} {.spec.template.spec.containers[0].resources.requests.memory} {.spec.template.spec.containers[0].resources.limits.memory}')" "200m 200m 128Mi 128Mi" || exit 1
for q in $($K -n %{ns} get pods -l app=critical -o jsonpath='{.items[*].status.qosClass}'); do [ "$q" = Guaranteed ] || exit 1; done"""),
],
"solution": [cmd(r"""k -n %{ns} get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass""", r"""Guaranteed: every container has requests equal to limits, CPU and memory. BestEffort: no container has any. Everything else is Burstable - `delta` too, as one of its containers sets nothing."""),
hand(r"""vim {WS}/q137/qos.txt""",
r"""```
alpha Guaranteed
beta Burstable
gamma BestEffort
delta Burstable
```""",
run=r"""printf 'alpha Guaranteed\nbeta Burstable\ngamma BestEffort\ndelta Burstable\n' > {WS}/q137/qos.txt"""),
cmd(r"""echo gamma > {WS}/q137/first.txt""", r"""Under memory pressure BestEffort goes first, then Burstable Pods using more than they requested, Guaranteed last."""),
cmd(r"""k -n %{ns} set resources deploy critical --requests=cpu=200m,memory=128Mi --limits=cpu=200m,memory=128Mi
k -n %{ns} rollout status deploy critical
k -n %{ns} get pods -l app=critical -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass""")],
},
# ------------------------------------------------------------------ 138 ----
{
"id": 138, "domain": "WORK", "title": "More CPU, same Pod",
"params": lambda r: {"ns": ns_pick(r, "render"), "pod": name_pick(r, "encoder"), "req": r.choice(["250m", "300m"]), "lim": r.choice(["500m", "600m"])},
"task": """
Pod `%{pod}` in `%{ns}` is throttled at its CPU limit. Give its container `encoder` a CPU request
of `%{req}` and a CPU limit of `%{lim}` - **without** replacing the Pod and without restarting the
container. Its memory request and limit stay as they are.
""",
"setup": NS + r"""
mkns %{ns}
$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true --grace-period=1 >/dev/null 2>&1
$K -n %{ns} apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Pod
metadata: {name: %{pod}}
spec:
containers:
- name: encoder
image: busybox:1.36
command: [sh, -c, 'while true; do :; done']
resources: {requests: {cpu: 100m, memory: 48Mi}, limits: {cpu: 200m, memory: 48Mi}}
EOF
$K -n %{ns} wait --for=condition=Ready pod/%{pod} --timeout=120s >/dev/null || exit 1
j -n %{ns} pod %{pod} '{.metadata.uid}' > "$PRIV/q138.uid"
""",
"checks": [
C("its spec asks for %{req} / %{lim} CPU, memory unchanged", 2, r"""
eq "$(j -n %{ns} pod %{pod} '{.spec.containers[0].resources.requests.cpu} {.spec.containers[0].resources.limits.cpu} {.spec.containers[0].resources.requests.memory} {.spec.containers[0].resources.limits.memory}')" "%{req} %{lim} 48Mi 48Mi" """),
C("the running container has the new CPU limit", 2, r"""
eq "$(j -n %{ns} pod %{pod} '{.status.containerStatuses[0].resources.requests.cpu} {.status.containerStatuses[0].resources.limits.cpu}')" "%{req} %{lim}" """),
C("in the same Pod, and its container never restarted", 2, r"""
eq "$(j -n %{ns} pod %{pod} '{.spec.containers[0].resources.limits.cpu}')" %{lim} \
&& eq "$(j -n %{ns} pod %{pod} '{.metadata.uid} {.status.containerStatuses[0].restartCount}')" "$(cat "$PRIV/q138.uid") 0" """),
],
"solution": [cmd(r"""k -n %{ns} get pod %{pod} -o yaml | grep -A6 resources:"""),
hand(r"""k -n %{ns} edit pod %{pod} --subresource resize""",
r"""Through the `resize` subresource - a plain `k edit` of a Pod's resources is refused. Set the CPU, leave the memory, then `:wq`:
```
resources:
limits:
cpu: %{lim}
memory: 48Mi
requests:
cpu: %{req}
memory: 48Mi
```""",
run=r"""kubectl -n %{ns} patch pod %{pod} --subresource resize --patch '{"spec":{"containers":[{"name":"encoder","resources":{"requests":{"cpu":"%{req}"},"limits":{"cpu":"%{lim}"}}}]}}'"""),
cmd(r"""k -n %{ns} get pod %{pod} -o yaml | grep -A6 resources:
k -n %{ns} get pod %{pod}""", r"""The spec says what you asked for, `status.containerStatuses[].resources` what the kubelet applied; `RESTARTS 0` - a container's `resizePolicy` decides, and CPU needs no restart.""")],
},
# ------------------------------------------------------------------ 139 ----
{
"id": 139, "domain": "TSHOOT", "workers": 1, "title": "Evict before the node runs dry",
"params": lambda r: {"mem": r.choice([200, 250, 300]), "fs": r.choice([10, 12, 15]), "soft": r.choice([400, 500])},
"task": """
Node `{W1}` ran out of memory last week and froze before its kubelet did anything about it.
Configure the kubelet on `{W1}` - in its configuration file, so that it lasts - to:
- evict Pods at once when less than `%{mem}Mi` of memory, or less than `%{fs}%` of the node
filesystem, is available;
- evict Pods when less than `%{soft}Mi` of memory has been available for **90 seconds**.
Leave its other settings as they are. The node must be `Ready` afterwards. Reach it with
`{SHELL_W1}`.
""",
"setup": r"""
nx "$W1" 'cp -n /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.ksim 2>/dev/null; true' </dev/null
""",
"checks": [
C("hard thresholds: memory.available %{mem}Mi, nodefs.available %{fs}%", 3, r"""
$K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c '
import json, sys; c = json.load(sys.stdin)["kubeletconfig"]; e = c.get("evictionHard") or {}
sys.exit(0 if e.get("memory.available") == "%{mem}Mi" and e.get("nodefs.available") == "%{fs}%" else 1)'"""),
C("soft threshold memory.available %{soft}Mi with a 90 s grace period", 3, r"""
$K get --raw "/api/v1/nodes/$W1/proxy/configz" 2>/dev/null | python3 -c '
import json, re, sys; c = json.load(sys.stdin)["kubeletconfig"]
s = (c.get("evictionSoft") or {}).get("memory.available"); g = (c.get("evictionSoftGracePeriod") or {}).get("memory.available", "")
m = re.fullmatch(r"(?:(\d+)m)?(?:(\d+)s)?", g); secs = int(m.group(1) or 0) * 60 + int(m.group(2) or 0) if m and g else -1
sys.exit(0 if s == "%{soft}Mi" and secs == 90 else 1)'"""),
C("set in the file, the other settings kept, the node Ready", 2, r"""
nx "$W1" 'grep -q "%{soft}Mi" /var/lib/kubelet/config.yaml && grep -q "%{mem}Mi" /var/lib/kubelet/config.yaml && grep -q "^staticPodPath:" /var/lib/kubelet/config.yaml && systemctl is-active -q kubelet' </dev/null || exit 1
eq "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" True"""),
],
"solution": [hand(r"""{SHELL_W1}
sudo -i
grep -n -A4 ^eviction /var/lib/kubelet/config.yaml
vim /var/lib/kubelet/config.yaml""",
r"""Under `evictionHard:` set the two hard thresholds, and add the soft one with its grace period at the top level - a soft threshold without a grace period for the same signal, and the kubelet refuses to start:
```
evictionHard:
memory.available: "%{mem}Mi"
nodefs.available: "%{fs}%"
evictionSoft:
memory.available: "%{soft}Mi"
evictionSoftGracePeriod:
memory.available: "1m30s"
```
Keep the other `evictionHard` entries as they are.""",
run=r"""ssh {W1} sudo bash <<'EOF'
yq -i '.evictionHard."memory.available" = "%{mem}Mi" | .evictionHard."nodefs.available" = "%{fs}%" | .evictionSoft."memory.available" = "%{soft}Mi" | .evictionSoftGracePeriod."memory.available" = "1m30s"' /var/lib/kubelet/config.yaml
systemctl restart kubelet
EOF"""),
hand(r"""systemctl restart kubelet
journalctl -u kubelet -n 5 --no-pager
exit
exit""",
r"""No complaint about the configuration; then back on the host.""",
run=r"""for i in $(seq 1 30); do [ "$(kubectl get node {W1} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" = True ] && break; sleep 3; done"""),
cmd(r"""k get node {W1}
k get --raw /api/v1/nodes/{W1}/proxy/configz | tr ',' '\n' | grep -i eviction""", r"""`Ready`; `configz` shows the configuration the running kubelet uses.""")],
},
# ------------------------------------------------------------------ 140 ----
{
"id": 140, "domain": "TSHOOT", "title": "Follow a Service into kube-proxy's rules",
"params": lambda r: {"ns": ns_pick(r, "pay"), "svc": name_pick(r, "ledger"), "n": r.choice([2, 3, 4])},
"task": """
Service `%{svc}` in `%{ns}` works through the rules kube-proxy writes on every node. On `{CP}`
(this host):
1. Write the mode kube-proxy runs in to `{WS}/q140/mode.txt`.
2. Write the name of the chain kube-proxy made for Service `%{svc}` - the one traffic to its
ClusterIP is sent to - to `{WS}/q140/chain.txt`.
3. Write the number of endpoint chains that chain spreads the traffic over to
`{WS}/q140/endpoints.txt`.
""",
"setup": NS + r"""
mkns %{ns}; rm -rf "$WS/q140"; mkdir -p "$WS/q140"
$K -n %{ns} create deploy %{svc} --image=nginx:1.27 --replicas=%{n} --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
$K -n %{ns} expose deploy %{svc} --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
$K -n %{ns} rollout status deploy %{svc} --timeout=180s >/dev/null
""",
"checks": [
C("mode.txt: kube-proxy's mode", 2, ONE + r"""
m=$($K -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | awk '/^mode:/ {print $2}' | tr -d '"'); m=${m:-iptables}
eq "$(one "$WS/q140/mode.txt" | tr -d '"')" "$m" """),
C("chain.txt: the Service's KUBE-SVC chain", 3, ONE + r"""
ip=$(j -n %{ns} svc %{svc} '{.spec.clusterIP}')
want=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" </dev/null)
[ -n "$want" ] && eq "$(one "$WS/q140/chain.txt")" "$want" """),
C("endpoints.txt: the number of KUBE-SEP chains behind it", 2, ONE + r"""
ip=$(j -n %{ns} svc %{svc} '{.spec.clusterIP}')
ch=$(nx "$CP" "iptables-save -t nat | grep -- '-d $ip/32' | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1" </dev/null)
n=$(nx "$CP" "iptables-save -t nat | grep -- '^-A $ch ' | grep -c -- '-j KUBE-SEP-'" </dev/null)
eq "$n" %{n} && eq "$(one "$WS/q140/endpoints.txt")" "$n" """),
],
"solution": [cmd(r"""k -n kube-system get cm kube-proxy -o yaml | grep mode""", r"""`mode: iptables` - an empty mode means the default, iptables too."""),
hand(r"""echo iptables > {WS}/q140/mode.txt""",
r"""The mode it printed.""",
run=r"""m=$(kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"')
echo ${m:-iptables} > {WS}/q140/mode.txt"""),
cmd(r"""k -n %{ns} get svc %{svc}""", r"""Note its `CLUSTER-IP`."""),
hand(r"""sudo iptables-save -t nat | grep 10.96.12.34""",
r"""With that IP: `KUBE-SERVICES` matches the ClusterIP and port and jumps (`-j`) to the Service's own `KUBE-SVC-...` chain.""",
run=r""""""),
hand(r"""echo KUBE-SVC-ABCDEFGHIJKLMNOP > {WS}/q140/chain.txt
sudo iptables-save -t nat | grep -- '-A KUBE-SVC-ABCDEFGHIJKLMNOP '""",
r"""With the chain's name. Its rules pick one endpoint chain at random (`statistic mode random`); each `KUBE-SEP-...` DNATs to one Pod - count them. In nftables mode: `sudo nft list ruleset | grep -A5 %{svc}`.""",
run=r"""ip=$(kubectl -n %{ns} get svc %{svc} -o jsonpath='{.spec.clusterIP}')
sudo iptables-save -t nat | grep -- "-d $ip/32" | grep -o 'KUBE-SVC-[A-Z0-9]*' | head -1 > {WS}/q140/chain.txt"""),
hand(r"""echo 3 > {WS}/q140/endpoints.txt""",
r"""With the number of `-j KUBE-SEP-` rules: one per ready endpoint.""",
run=r"""chain=$(cat {WS}/q140/chain.txt)
sudo iptables-save -t nat | grep -- "^-A $chain " | grep -c -- '-j KUBE-SEP-' > {WS}/q140/endpoints.txt""")],
},
# ------------------------------------------------------------------ 141 ----
{
"id": 141, "domain": "NET", "workers": 1, "title": "A Service for an API outside the cluster",
"params": lambda r: {"ns": ns_pick(r, "apps"), "port": r.choice([8080, 9090])},
"task": """
The billing API is not part of the cluster: it answers at the address and port written in
`{WS}/q141/billing.txt`. Give the applications in `%{ns}` a stable name for it:
1. Create Service `billing` in `%{ns}`, port `80`, **without** a selector.
2. Its endpoints are yours to maintain: create EndpointSlice `billing-1` that sends its traffic
to that address and port.
`wget -qO- billing` from Pod `client` in `%{ns}` must return the API's answer.
""",
"setup": NS + r"""
mkns %{ns}; mkns billing-host; rm -rf "$WS/q141"; mkdir -p "$WS/q141"
$K -n %{ns} delete svc billing --ignore-not-found >/dev/null 2>&1
$K -n %{ns} run client --image=busybox:1.36 -- sleep 86400 >/dev/null 2>&1
$K -n billing-host run api --image=busybox:1.36 --overrides="{\"spec\":{\"hostNetwork\":true,\"nodeName\":\"$W1\"}}" --command -- \
sh -c 'mkdir -p /www && echo billing-api-%{port} > /www/index.html && exec httpd -f -p %{port} -h /www' >/dev/null 2>&1
$K -n %{ns} wait --for=condition=Ready pod/client --timeout=120s >/dev/null && $K -n billing-host wait --for=condition=Ready pod/api --timeout=120s >/dev/null || exit 1
echo "$(j node "$W1" '{.status.addresses[?(@.type=="InternalIP")].address}'):%{port}" > "$WS/q141/billing.txt"
""",
"checks": [
C("Service billing, port 80, has no selector", 1, r"""eq "$(j -n %{ns} svc billing '{.spec.ports[0].port}')" 80 && [ -z "$(j -n %{ns} svc billing '{.spec.selector}')" ]"""),
C("EndpointSlice billing-1 belongs to it and holds the API's address and port", 3, r"""
a=$(cut -d: -f1 "$WS/q141/billing.txt")
eq "$(j -n %{ns} endpointslice billing-1 '{.metadata.labels.kubernetes\.io/service-name} {.addressType} {.endpoints[0].addresses[0]} {.ports[0].port}')" "billing IPv4 $a %{port}" """),
C("wget billing from client returns the API's answer", 3, r"""
for i in 1 2 3 4 5; do [ "$($K -n %{ns} exec client -- wget -qO- -T 3 billing 2>/dev/null)" = billing-api-%{port} ] && exit 0; sleep 3; done; exit 1"""),
],
"solution": [cmd(r"""cat {WS}/q141/billing.txt""", r"""`ADDRESS:PORT`."""),
hand(r"""vim billing.yaml""",
r"""A Service with no selector - no controller writes its endpoints - and the EndpointSlice you maintain, tied to it by the label `kubernetes.io/service-name` (the docs page *Service* has both under *Services without selectors*). With the address and port from the file:
```
apiVersion: v1
kind: Service
metadata:
name: billing
namespace: %{ns}
spec:
ports:
- port: 80
protocol: TCP
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: billing-1
namespace: %{ns}
labels:
kubernetes.io/service-name: billing
addressType: IPv4
endpoints:
- addresses:
- 172.18.0.3
ports:
- port: %{port}
protocol: TCP
```
The slice's port is where traffic goes - the Service has no `targetPort` to speak of - and its name must match the Service port's: here both unnamed.""",
run=r"""addr=$(cut -d: -f1 {WS}/q141/billing.txt); port=$(cut -d: -f2 {WS}/q141/billing.txt)
cat > billing.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
name: billing
namespace: %{ns}
spec:
ports:
- port: 80
protocol: TCP
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: billing-1
namespace: %{ns}
labels:
kubernetes.io/service-name: billing
addressType: IPv4
endpoints:
- addresses:
- "$addr"
ports:
- port: $port
protocol: TCP
EOF"""),
cmd(r"""k apply -f billing.yaml
k -n %{ns} exec client -- wget -qO- -T3 billing""", r"""`billing-api-%{port}`. (`ExternalName` would need a DNS name; an address needs this.)""")],
},
# ------------------------------------------------------------------ 142 ----
{
"id": 142, "domain": "WORK", "difficulty": "hard", "title": "Secrets encrypted at rest",
"params": lambda r: {"ns": ns_pick(r, "bank"), "sec": name_pick(r, "vault-key")},
"task": """
This cluster stores Secrets in etcd in plain text. Change that:
1. Write an `EncryptionConfiguration` to `/etc/kubernetes/enc/enc.yaml` on `{CP}` (this host) that
encrypts `secrets` with the `aescbc` provider, with one key named `key1`, and can still read
Secrets stored unencrypted.
2. Make the API server use it.
3. Secret `%{sec}` in `%{ns}` exists already: make sure it, too, is stored encrypted.
""",
"setup": NS + r"""
mkns %{ns}
$K -n %{ns} create secret generic %{sec} --from-literal=key=k-%{sec} --dry-run=client -o yaml | $K apply -f - >/dev/null
""",
"checks": [
C("enc.yaml encrypts secrets with aescbc key1 first, identity after", 1, r"""
nx "$CP" 'cat /etc/kubernetes/enc/enc.yaml' </dev/null 2>/dev/null | python3 -c '
import sys, yaml
c = yaml.safe_load(sys.stdin); r = [x for x in c["resources"] if "secrets" in x["resources"]][0]["providers"]
sys.exit(0 if "aescbc" in r[0] and r[0]["aescbc"]["keys"][0]["name"] == "key1" and any("identity" in p for p in r[1:]) else 1)' 2>/dev/null \
|| nx "$CP" 'grep -q aescbc /etc/kubernetes/enc/enc.yaml && grep -q "name: key1" /etc/kubernetes/enc/enc.yaml && grep -q identity /etc/kubernetes/enc/enc.yaml' </dev/null"""),
C("the API server runs with --encryption-provider-config", 2, r"""
nx "$CP" 'grep -q -- "--encryption-provider-config=/etc/kubernetes/enc/enc.yaml" /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null || exit 1
$K -n kube-system get pod -l component=kube-apiserver -o jsonpath='{.items[0].spec.containers[0].command}' | grep -q encryption-provider-config"""),
C("Secret %{sec} is stored encrypted, and still reads back", 3, r"""
nx "$CP" '""" + _ETCD + r""" get /registry/secrets/%{ns}/%{sec} --print-value-only 2>/dev/null | head -c 64' </dev/null | grep -aq '^k8s:enc:aescbc:v1:key1:' \
&& eq "$(j -n %{ns} secret %{sec} '{.data.key}' | base64 -d)" k-%{sec}"""),
C("a new Secret is stored encrypted too", 2, r"""
$K -n %{ns} delete secret ksim-probe --ignore-not-found >/dev/null 2>&1
$K -n %{ns} create secret generic ksim-probe --from-literal=x=y >/dev/null 2>&1 || exit 1
nx "$CP" '""" + _ETCD + r""" get /registry/secrets/%{ns}/ksim-probe --print-value-only 2>/dev/null | head -c 64' </dev/null | grep -aq '^k8s:enc:aescbc:v1:'; r=$?
$K -n %{ns} delete secret ksim-probe --ignore-not-found >/dev/null 2>&1; exit $r"""),
],
"solution": [cmd(r"""head -c 32 /dev/urandom | base64""", r"""The host is the control-plane node. Copy the output: that is the key."""),
hand(r"""sudo mkdir -p /etc/kubernetes/enc
sudo vim /etc/kubernetes/enc/enc.yaml""",
r"""Paste the example from the docs page *Encrypting Confidential Data at Rest* (search for `aescbc`) and cut it down to:
```
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
providers:
- aescbc:
keys:
- name: key1
secret: <your key>
- identity: {}
```
The first provider encrypts; `identity` after it still reads what was stored before.""",
run=r"""sudo mkdir -p /etc/kubernetes/enc
sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
providers:
- aescbc:
keys:
- name: key1
secret: $(head -c 32 /dev/urandom | base64)
- identity: {}
EOF"""),
cmd(r"""sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/""", r"""Edit a copy: the kubelet acts on every save in `manifests/`, half-done edits too."""),
hand(r"""sudo vim ~/kube-apiserver.yaml""",
r"""Add three lines, each beside its kind (`yy` and `p` on an existing `k8s-certs` mount or volume, then change it):
```
- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
```
in the `command:` list,
```
- {name: enc, mountPath: /etc/kubernetes/enc, readOnly: true}
```
under `volumeMounts:`, and
```
- {name: enc, hostPath: {path: /etc/kubernetes/enc, type: DirectoryOrCreate}}
```
under `volumes:` - the API server runs in a container, and sees only what is mounted.""",
run=r"""sudo yq -i '.spec.containers[0].command += ["--encryption-provider-config=/etc/kubernetes/enc/enc.yaml"]
| .spec.containers[0].volumeMounts += [{"name": "enc", "mountPath": "/etc/kubernetes/enc", "readOnly": true}]
| .spec.volumes += [{"name": "enc", "hostPath": {"path": "/etc/kubernetes/enc", "type": "DirectoryOrCreate"}}]' ~/kube-apiserver.yaml"""),
cmd(r"""sudo cp ~/kube-apiserver.yaml /etc/kubernetes/manifests/"""),
hand(r"""k -n kube-system get pods""",
r"""Again until it answers: the API server is back in a minute or so.""",
run=r"""sleep 20; for i in $(seq 1 40); do kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 3; done"""),
cmd(r"""k -n %{ns} get secret %{sec} -o yaml | k replace -f -""", r"""Written again, the Secret is stored encrypted; every Secret at once: `k get secrets -A -o json | k replace -f -`."""),
cmd(r"""sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get /registry/secrets/%{ns}/%{sec} | head -c 80""", r"""`k8s:enc:aescbc:v1:key1:` and no plain text.""")],
},
# ------------------------------------------------------------------ 143 ----
{
"id": 143, "domain": "WORK", "workers": 3, "title": "Never two on one node",
"params": lambda r: {"ns": ns_pick(r, "edge"), "dep": name_pick(r, "gateway")},
"task": """
Deployment `%{dep}` in `%{ns}` runs 3 replicas, and nothing stops two of them from landing on the
same node. Change it so that:
- no two of its Pods ever run on the same node - a hard rule, not a preference;
- among the nodes allowed, those labelled `tier=edge` are **preferred**.
Label `{W2}` and `{W3}` with `tier=edge`. All three replicas must be running afterwards.
""",
"setup": NS + r"""
mkns %{ns}
for n in "$W1" "$W2" "$W3"; do $K label node "$n" tier- >/dev/null 2>&1; done
$K -n %{ns} create deploy %{dep} --image=nginx:1.27 --replicas=3 --dry-run=client -o yaml | $K apply -f - >/dev/null
$K -n %{ns} rollout status deploy %{dep} --timeout=180s >/dev/null
""",
"checks": [
C("required pod anti-affinity on kubernetes.io/hostname for its own Pods", 3, r"""
$K -n %{ns} get deploy %{dep} -o json 2>/dev/null | python3 -c '
import json, sys; d = json.load(sys.stdin); t = d["spec"]["template"]
a = t["spec"].get("affinity", {}).get("podAntiAffinity", {}).get("requiredDuringSchedulingIgnoredDuringExecution", [])
lab = t["metadata"]["labels"]
ok = any(x.get("topologyKey") == "kubernetes.io/hostname" and all(lab.get(k) == v for k, v in x.get("labelSelector", {}).get("matchLabels", {}).items()) and x.get("labelSelector", {}).get("matchLabels") for x in a)
sys.exit(0 if ok else 1)'"""),
C("preferred node affinity for tier=edge; {W2} and {W3} labelled", 2, r"""
eq "$(j node "$W2" '{.metadata.labels.tier}') $(j node "$W3" '{.metadata.labels.tier}')" "edge edge" || exit 1
$K -n %{ns} get deploy %{dep} -o json 2>/dev/null | python3 -c '
import json, sys; s = json.load(sys.stdin)["spec"]["template"]["spec"]
p = s.get("affinity", {}).get("nodeAffinity", {}).get("preferredDuringSchedulingIgnoredDuringExecution", [])
ok = any(e.get("key") == "tier" and e.get("operator") == "In" and "edge" in e.get("values", []) for x in p for e in x["preference"].get("matchExpressions", []))
sys.exit(0 if ok else 1)'"""),
C("3 Pods of the new template running on 3 different nodes", 2, r"""
[ -n "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution}')" ] || exit 1
eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas} {.status.updatedReplicas} {.status.replicas}')" "3 3 3" || exit 1
[ "$($K -n %{ns} get pods -l app=%{dep} --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | grep -c .)" = 3 ]"""),
],
"solution": [cmd(r"""k label node {W2} {W3} tier=edge"""),
hand(r"""k -n %{ns} edit deploy %{dep}""",
r"""In the Pod template's `spec:`, above `containers:` (the docs page *Assigning Pods to Nodes* has both kinds):
```
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: %{dep}
topologyKey: kubernetes.io/hostname
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: tier
operator: In
values:
- edge
```
and in the same edit, under `strategy:`, let the rollout replace one Pod at a time:
```
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
```
With 3 workers and one Pod each, a surge Pod has nowhere to go, and 25% of 3 unavailable rounds down to 0: the default rollout would wait for ever. Then `:wq`.""",
run=r"""kubectl -n %{ns} patch deploy %{dep} --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}},"template":{"spec":{"affinity":{
"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app":"%{dep}"}},"topologyKey":"kubernetes.io/hostname"}]},
"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"preference":{"matchExpressions":[{"key":"tier","operator":"In","values":["edge"]}]}}]}}}}}}'"""),
cmd(r"""k -n %{ns} rollout status deploy %{dep}
k -n %{ns} get pods -o wide""", r"""Three nodes, one Pod each. The anti-affinity's selector names the Pods to keep away from - its own - and `topologyKey` per what: one per hostname, one per node.""")],
},
# ------------------------------------------------------------------ 144 ----
{
"id": 144, "domain": "WORK", "title": "A canary for a StatefulSet",
"params": lambda r: {"ns": ns_pick(r, "store"), "sts": name_pick(r, "cache")},
"task": """
StatefulSet `%{sts}` in `%{ns}` runs `nginx:1.26` on 3 replicas. Roll out `nginx:1.27` as a canary:
only the replica with the **highest ordinal** gets the new image now; the other two keep running
`nginx:1.26`, untouched, until someone lowers the partition.
""",
"setup": NS + r"""
mkns %{ns}
$K -n %{ns} apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: %{sts}}
spec: {clusterIP: None, selector: {app: %{sts}}, ports: [{port: 80}]}
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: %{sts}}
spec:
serviceName: %{sts}
replicas: 3
selector: {matchLabels: {app: %{sts}}}
template:
metadata: {labels: {app: %{sts}}}
spec:
containers: [{name: nginx, image: nginx:1.26}]
EOF
$K -n %{ns} rollout status sts %{sts} --timeout=240s >/dev/null || exit 1
for i in 0 1; do j -n %{ns} pod %{sts}-$i '{.metadata.uid}'; echo; done > "$PRIV/q144.uids"
""",
"checks": [
C("updateStrategy RollingUpdate with partition 2", 2, r"""eq "$(j -n %{ns} sts %{sts} '{.spec.updateStrategy.type} {.spec.updateStrategy.rollingUpdate.partition}')" "RollingUpdate 2" """),
C("%{sts}-2 runs nginx:1.27 and is ready", 3, r"""
eq "$(j -n %{ns} sts %{sts} '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1
eq "$(j -n %{ns} pod %{sts}-2 '{.spec.containers[0].image} {.status.containerStatuses[0].ready}')" "nginx:1.27 true" """),
C("%{sts}-0 and %{sts}-1 still run nginx:1.26, never replaced", 2, r"""
eq "$(j -n %{ns} pod %{sts}-0 '{.spec.containers[0].image}') $(j -n %{ns} pod %{sts}-1 '{.spec.containers[0].image}')" "nginx:1.26 nginx:1.26" || exit 1
eq "$(for i in 0 1; do j -n %{ns} pod %{sts}-$i '{.metadata.uid}'; echo; done)" "$(cat "$PRIV/q144.uids")" \
&& eq "$(j -n %{ns} sts %{sts} '{.spec.template.spec.containers[0].image}')" nginx:1.27"""),
],
"solution": [hand(r"""k -n %{ns} edit sts %{sts}""",
r"""The partition first - a StatefulSet updates from the highest ordinal down, and without one the new image would reach all three within moments. Under `updateStrategy:`, then `:wq`:
```
updateStrategy:
rollingUpdate:
partition: 2
type: RollingUpdate
```""",
run=r"""kubectl -n %{ns} patch sts %{sts} -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'"""),
cmd(r"""k -n %{ns} set image sts %{sts} nginx=nginx:1.27
k -n %{ns} rollout status sts %{sts}
k -n %{ns} get pods -l app=%{sts} -o custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[0].image""", r"""Ordinals at or above the partition get the new template: `%{sts}-2` alone. The next steps of the canary: partition 1, then 0.""")],
},
# ------------------------------------------------------------------ 145 ----
{
"id": 145, "domain": "ARCH", "workers": 1, "title": "An operator that cannot finish its job",
"params": lambda r: {"ns": ns_pick(r, "reports"), "name": name_pick(r, "daily")},
"task": """
The manifests of a backup operator are in `{WS}/q145/`: its CustomResourceDefinition, its RBAC and
its controller.
1. Install the operator.
2. Create Namespace `%{ns}` and, in it, Backup `%{name}` with the schedule `30 1 * * *`.
3. The operator should create ConfigMap `backup-%{name}` for it and set its `status.phase` to
`Scheduled`. Make sure it does - its manifests are not quite right. Change only what is wrong.
""",
"setup": r"""
rm -rf "$WS/q145"; mkdir -p "$WS/q145"
cat > "$WS/q145/operator.yaml" <<'EOF'
""" + backup_operator(crd=True, status_rbac=False) + r"""EOF
$K delete crd backups.data.example.com --ignore-not-found >/dev/null 2>&1
$K delete ns backup-operator --ignore-not-found >/dev/null 2>&1
""",
"checks": [
C("the CRD is established and the controller runs", 2, r"""
eq "$(j crd backups.data.example.com '{.status.conditions[?(@.type=="Established")].status}')" True && eq "$(j -n backup-operator deploy controller '{.status.readyReplicas}')" 1"""),
C("Backup %{name} in %{ns} has the schedule", 1, r"""eq "$(j -n %{ns} backup %{name} '{.spec.schedule}')" "30 1 * * *" """),
C("the operator made ConfigMap backup-%{name}", 1, r"""eq "$(j -n %{ns} cm backup-%{name} '{.data.schedule}')" "30 1 * * *" """),
C("Backup %{name} is Scheduled: the controller may update its status", 3, r"""
eq "$(j -n %{ns} backup %{name} '{.status.phase}')" Scheduled && eq "$(j clusterrolebinding backup-operator '{.subjects[0].name}')" controller"""),
],
"solution": [cmd(r"""grep -e ^kind: -e '^ name:' {WS}/q145/operator.yaml
k apply -f {WS}/q145/operator.yaml"""),
hand(r"""k get crd backups.data.example.com
k -n backup-operator get pods""",
r"""Again until the controller is `Running`.""",
run=r"""kubectl wait --for condition=established crd/backups.data.example.com --timeout=60s
kubectl -n backup-operator rollout status deploy controller --timeout=180s"""),
cmd(r"""k create ns %{ns}"""),
hand(r"""vim backup.yaml""",
r"""```
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: %{name}
namespace: %{ns}
spec:
schedule: "30 1 * * *"
```""",
run=r"""cat > backup.yaml <<'EOF'
apiVersion: data.example.com/v1
kind: Backup
metadata:
name: %{name}
namespace: %{ns}
spec:
schedule: "30 1 * * *"
EOF"""),
cmd(r"""k apply -f backup.yaml"""),
hand(r"""k -n %{ns} get cm backup-%{name}
k -n %{ns} get backup %{name} -o yaml | grep -A1 status
k -n backup-operator logs deploy/controller | tail -3""",
r"""The ConfigMap is made, but no `phase`: the log says `cannot patch resource "backups/status"`. The CRD has a status subresource, and for RBAC a subresource is a resource of its own.""",
run=r"""sleep 10"""),
cmd(r"""k auth can-i patch backups.data.example.com --subresource=status --as=system:serviceaccount:backup-operator:controller -A""", r"""`no`."""),
hand(r"""k edit clusterrole backup-operator""",
r"""Add `backups/status` to the first rule's resources, then `:wq` - the one thing wrong:
```
- apiGroups:
- data.example.com
resources:
- backups
- backups/status
```""",
run=r"""kubectl patch clusterrole backup-operator --type json -p '[{"op":"add","path":"/rules/0/resources/-","value":"backups/status"}]'"""),
hand(r"""k -n %{ns} get backup %{name} -o yaml | grep -A1 status""",
r"""Again until `phase: Scheduled` - the controller tries every 5 seconds.""",
run=r"""for i in $(seq 1 20); do [ "$(kubectl -n %{ns} get backup %{name} -o jsonpath='{.status.phase}')" = Scheduled ] && break; sleep 3; done""")],
},
# ------------------------------------------------------------------ 146 ----
{
"id": 146, "domain": "STOR", "workers": 1, "title": "Grow a claim that is in use",
"params": lambda r: {"ns": ns_pick(r, "media"), "size": r.choice(["70Mi", "80Mi", "100Mi"])},
"task": """
In Namespace `%{ns}`:
1. Create PersistentVolumeClaim `pv-volume`: class `csi-hostpath-sc`, access mode `ReadWriteOnce`,
capacity `10Mi`.
2. Create Pod `web-server` (image `nginx:1.27`) that mounts the claim at `/usr/share/nginx/html`.
3. The site needs more room: expand the claim to `%{size}` while `web-server` keeps running.
""",
"setup": NS + r"""
ensure_csi_hostpath || { echo "could not install the CSI hostpath driver (no network?) - the task needs it" >&2; exit 1; }
mkns %{ns}
$K -n %{ns} delete pod web-server --ignore-not-found --wait=true --grace-period=1 >/dev/null 2>&1
$K -n %{ns} delete pvc pv-volume --ignore-not-found --wait=true >/dev/null 2>&1
""",
"checks": [
C("pv-volume uses csi-hostpath-sc, ReadWriteOnce, and is Bound", 1, r"""
eq "$(j -n %{ns} pvc pv-volume '{.spec.storageClassName} {.spec.accessModes[0]} {.status.phase}')" "csi-hostpath-sc ReadWriteOnce Bound" """),
C("web-server runs nginx with the claim at /usr/share/nginx/html", 2, r"""
eq "$(j -n %{ns} pod web-server '{.status.phase} {.spec.containers[0].image}')" "Running nginx:1.27" || exit 1
v=$(j -n %{ns} pod web-server '{.spec.containers[0].volumeMounts[?(@.mountPath=="/usr/share/nginx/html")].name}'); [ -n "$v" ] || exit 1
eq "$(j -n %{ns} pod web-server "{.spec.volumes[?(@.name==\"$v\")].persistentVolumeClaim.claimName}")" pv-volume"""),
C("the claim has grown to %{size}, and the Pod kept running", 3, r"""
eq "$(j -n %{ns} pvc pv-volume '{.spec.resources.requests.storage} {.status.capacity.storage}')" "%{size} %{size}" || exit 1
eq "$(j -n %{ns} pod web-server '{.status.phase} {.status.containerStatuses[0].restartCount}')" "Running 0" """),
],
"solution": [cmd(r"""k get sc csi-hostpath-sc""", r"""`ALLOWVOLUMEEXPANSION true`: a claim of this class may grow."""),
hand(r"""vim pvc.yaml""",
r"""```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pv-volume
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Mi
```""",
run=r"""cat > pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pv-volume
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Mi
EOF"""),
cmd(r"""k apply -f pvc.yaml
k -n %{ns} run web-server --image=nginx:1.27 --dry-run=client -o yaml > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: web-server`:
```
volumeMounts:
- name: html
mountPath: /usr/share/nginx/html
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: html
persistentVolumeClaim:
claimName: pv-volume
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "html", "mountPath": "/usr/share/nginx/html"}]
| .spec.volumes = [{"name": "html", "persistentVolumeClaim": {"claimName": "pv-volume"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pod web-server""",
r"""Again until it is `Running`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/web-server --timeout=180s"""),
hand(r"""k -n %{ns} edit pvc pv-volume""",
r"""Under `spec.resources.requests`, then `:wq` - a claim only grows:
```
storage: %{size}
```""",
run=r"""kubectl -n %{ns} patch pvc pv-volume -p '{"spec":{"resources":{"requests":{"storage":"%{size}"}}}}'"""),
cmd(r"""k -n %{ns} get pvc pv-volume
k -n %{ns} get pod web-server""", r"""Again until `CAPACITY %{size}`: `spec.resources.requests` is what you asked for, `status.capacity` what the driver has done. This driver grows filesystems online - the Pod keeps running.""")],
},
# ------------------------------------------------------------------ 147 ----
{
"id": 147, "domain": "STOR", "workers": 1, "title": "Snapshot, restore, clone",
"params": lambda r: {"ns": ns_pick(r, "ledger"), "n": r.randint(100, 999)},
"task": """
Claim `ledger-data` in `%{ns}` (class `csi-hostpath-sc`, `50Mi`) holds `ledger.txt`, written by Pod
`writer`.
1. Take a VolumeSnapshot `ledger-snap` of the claim, with snapshot class `csi-hostpath-snapclass`,
and wait until it is ready to use.
2. Create claim `ledger-restore` (`50Mi`, same class) **from that snapshot**.
3. Create claim `ledger-copy` (`50Mi`, same class) as a **clone of `ledger-data`** itself.
4. Run Pod `auditor` (image `busybox:1.36`, command `sleep 86400`) that mounts `ledger-restore` at
`/restore` and `ledger-copy` at `/copy`.
""",
"setup": NS + r"""
ensure_csi_hostpath || { echo "could not install the CSI hostpath driver (no network?) - the task needs it" >&2; exit 1; }
mkns %{ns}
$K -n %{ns} apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: ledger-data}
spec: {storageClassName: csi-hostpath-sc, accessModes: [ReadWriteOnce], resources: {requests: {storage: 50Mi}}}
---
apiVersion: v1
kind: Pod
metadata: {name: writer}
spec:
containers:
- {name: writer, image: busybox:1.36, command: [sh, -c, 'echo entries-%{n} > /data/ledger.txt; sync; sleep 86400'], volumeMounts: [{name: d, mountPath: /data}]}
volumes: [{name: d, persistentVolumeClaim: {claimName: ledger-data}}]
EOF
$K -n %{ns} wait --for=condition=Ready pod/writer --timeout=180s >/dev/null
""",
"checks": [
C("VolumeSnapshot ledger-snap of ledger-data is ready to use", 2, r"""
eq "$(j -n %{ns} volumesnapshot ledger-snap '{.spec.source.persistentVolumeClaimName} {.spec.volumeSnapshotClassName} {.status.readyToUse}')" "ledger-data csi-hostpath-snapclass true" """),
C("ledger-restore is Bound, made from the snapshot", 2, r"""
eq "$(j -n %{ns} pvc ledger-restore '{.spec.dataSource.kind}/{.spec.dataSource.name} {.status.phase}')" "VolumeSnapshot/ledger-snap Bound" """),
C("ledger-copy is Bound, a clone of ledger-data", 2, r"""
eq "$(j -n %{ns} pvc ledger-copy '{.spec.dataSource.kind}/{.spec.dataSource.name} {.status.phase}')" "PersistentVolumeClaim/ledger-data Bound" """),
C("auditor reads the ledger from both", 2, r"""
x() { $K -n %{ns} exec auditor -- cat "$1/ledger.txt" 2>/dev/null; }
eq "$(x /restore)" entries-%{n} && eq "$(x /copy)" entries-%{n} \
&& eq "$(j -n %{ns} pod auditor '{.spec.volumes[*].persistentVolumeClaim.claimName}' | tr ' ' '\n' | sort | tr '\n' ' ')" "ledger-copy ledger-restore " """),
],
"solution": [cmd(r"""k get volumesnapshotclass""", r"""`csi-hostpath-snapclass`, driver `hostpath.csi.k8s.io`."""),
hand(r"""vim snap.yaml""",
r"""The docs page *Volume Snapshots* has the VolumeSnapshot:
```
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: ledger-snap
namespace: %{ns}
spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: ledger-data
```""",
run=r"""cat > snap.yaml <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: ledger-snap
namespace: %{ns}
spec:
volumeSnapshotClassName: csi-hostpath-snapclass
source:
persistentVolumeClaimName: ledger-data
EOF"""),
cmd(r"""k apply -f snap.yaml"""),
hand(r"""k -n %{ns} get volumesnapshot ledger-snap""",
r"""Again until `READYTOUSE true`.""",
run=r"""kubectl -n %{ns} wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/ledger-snap --timeout=120s"""),
hand(r"""vim claims.yaml""",
r"""`dataSource` makes a claim from a snapshot (*Volume Snapshots*) or from another claim of the same class and namespace - a clone (*CSI Volume Cloning*; no `apiGroup`, claims are core). Both at least as big as the source:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger-restore
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: ledger-snap
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger-copy
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
dataSource:
kind: PersistentVolumeClaim
name: ledger-data
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
```""",
run=r"""cat > claims.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger-restore
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: ledger-snap
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ledger-copy
namespace: %{ns}
spec:
storageClassName: csi-hostpath-sc
dataSource:
kind: PersistentVolumeClaim
name: ledger-data
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi
EOF"""),
cmd(r"""k apply -f claims.yaml
k -n %{ns} run auditor --image=busybox:1.36 --dry-run=client -o yaml -- sleep 86400 > pod.yaml"""),
hand(r"""vim pod.yaml""",
r"""Add to the container, below `name: auditor`:
```
volumeMounts:
- name: restore
mountPath: /restore
- name: copy
mountPath: /copy
```
and to the Pod's `spec:`, right below `spec:`:
```
volumes:
- name: restore
persistentVolumeClaim:
claimName: ledger-restore
- name: copy
persistentVolumeClaim:
claimName: ledger-copy
```""",
run=r"""yq -i '.spec.containers[0].volumeMounts = [{"name": "restore", "mountPath": "/restore"}, {"name": "copy", "mountPath": "/copy"}]
| .spec.volumes = [{"name": "restore", "persistentVolumeClaim": {"claimName": "ledger-restore"}}, {"name": "copy", "persistentVolumeClaim": {"claimName": "ledger-copy"}}]' pod.yaml"""),
cmd(r"""k apply -f pod.yaml"""),
hand(r"""k -n %{ns} get pvc,pod""",
r"""Again until both claims are `Bound` and `auditor` `Running`.""",
run=r"""kubectl -n %{ns} wait --for=condition=Ready pod/auditor --timeout=180s"""),
cmd(r"""k -n %{ns} exec auditor -- cat /restore/ledger.txt /copy/ledger.txt""", r"""`entries-%{n}`, twice.""")],
},
]
Reference in New Issue View Git Blame Copy Permalink