e761d53a96
105 of the 147 CKA reference solutions had you type what nobody types in the exam: heredoc manifests, sed, JSON patches, kubectl wait, loops, command substitution, jsonpath into files. Each is rewritten as the lessons are: a generator and the lines to add in vim, a manifest pasted from the docs page it names, kubectl edit, ssh NODE and sudo -i, or a value read off the screen and echoed. The other 42 were commands typed as they stand already. A solution can now be parts (sim/bank.py cmd, hand) as a lesson step's show is. The page and ./ksim solution N render it as Markdown, each thing to type a block of its own that a click copies, with the edit beneath it. A solution in parts is its own audit script: load_track runs its parts, and for each edit by hand the script beside it, so the solution you read is the one proven. Their separate scripts in cka/solve.py are dropped, and a task that has both is refused. ./ksim solution lint [N|DOMAIN..] reads what the solutions have you type against TYPING_TRAPS: the CKA bank is clean. Q49's params gain the plural stem and shortName the solution spells out (no new draws); Q103, ported from CKAD Q23, takes a solution of its own. Audited from a copy with renamed hosts: Q3-Q67 (52 tasks) all solved by their parts, after two waits were added to the audit scripts (Q43's DNS record, Q53's Pods); Q68-Q147 still running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNSZLLsGXTMBsw17kDunkQ
1112 lines
60 KiB
Python
1112 lines
60 KiB
Python
"""Simulator Set D - 17 tasks, fixed set, killer.sh-style.
|
|
|
|
Original scenarios, not copies of any commercial simulator. They exercise the techniques the
|
|
killer.sh CKA environment is known for that the rest of the bank did not cover: saying how every
|
|
control-plane piece is run, cluster facts to an exact file, readiness gated on a Service,
|
|
`--sort-by` scripts, event forensics around kube-proxy with crictl, API-resource and RBAC census,
|
|
a kubelet whose systemd unit points at the wrong binary, Secrets mounted read-only and renamed into
|
|
env vars, a static Pod on the control plane behind a Service, kubelet client/serving certificate
|
|
forensics, control-plane-only scheduling, stopping the scheduler and scheduling by hand, Helm
|
|
release management (including a release stuck in pending-install), StatefulSet storage, the
|
|
namespaceSelector-AND-podSelector NetworkPolicy trap (enforced here - kindnet enforces), a
|
|
dnsPolicy that silently breaks name resolution, and a claim Pending on an access mode.
|
|
|
|
Every value a grader compares against is derived from the live cluster, never hardcoded, and every
|
|
task was seeded, solved and graded on an isolated kind cluster before it went in.
|
|
|
|
Run with: ./ksim sim d
|
|
"""
|
|
from sim.bank import C, NS, cmd, hand
|
|
|
|
Q = []
|
|
# Tasks written for the pre-2025 exam opened by switching kubectl context. Every task now has a host
|
|
# of its own, named above the task, so the line is empty; the ctx parameter is still drawn and unused.
|
|
CTX = ""
|
|
|
|
|
|
def task(**kw):
|
|
kw.setdefault("difficulty", "hard")
|
|
kw.setdefault("params", lambda r: {"ctx": "k8s-c1-H"})
|
|
kw["task"] = CTX + kw["task"].lstrip("\n")
|
|
kw["set"] = "d"
|
|
Q.append(kw)
|
|
|
|
|
|
# ------------------------------------------------------------------ 86 ----
|
|
task(
|
|
id=86, domain="ARCH", title="How the control plane is run",
|
|
task="""
|
|
The operations handbook has to say how each part of this cluster's control plane is run. Write
|
|
`{WS}/q86/controlplane.txt` with exactly these six lines, each value one of `process`,
|
|
`static-pod`, `pod`, `deployment` or `daemonset`:
|
|
|
|
```
|
|
kubelet: <value>
|
|
kube-apiserver: <value>
|
|
kube-scheduler: <value>
|
|
kube-controller-manager: <value>
|
|
etcd: <value>
|
|
dns: <value> <name of that object>
|
|
```
|
|
|
|
`dns` is whatever serves cluster DNS, for example `dns: daemonset some-name`.
|
|
""",
|
|
setup=r"""rm -rf "$WS/q86"; mkdir -p "$WS/q86" """,
|
|
checks=[
|
|
C("kubelet is a process", 1, r"""
|
|
f="$WS/q86/controlplane.txt"; [ -s "$f" ] || exit 1
|
|
want=process; [ "$(nx "$CP" 'systemctl is-active kubelet' </dev/null)" = active ] || want=unknown
|
|
eq "$(sed -n 's/^ *kubelet *: *//p' "$f" | head -1 | tr -d '[:space:]' | tr A-Z a-z)" "$want" """),
|
|
C("apiserver, scheduler, controller-manager and etcd are static pods", 3, r"""
|
|
f="$WS/q86/controlplane.txt"
|
|
# a kubeadm control plane: the manifests live in the kubelet's staticPodPath on the node
|
|
nx "$CP" 'ls /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null >/dev/null 2>&1 || exit 1
|
|
for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do
|
|
eq "$(sed -n "s/^ *$c *: *//p" "$f" | head -1 | tr -d '[:space:]' | tr A-Z a-z)" static-pod || exit 1
|
|
done"""),
|
|
C("dns names the right kind of object and its name", 2, r"""
|
|
f="$WS/q86/controlplane.txt"
|
|
sel=$(j -n kube-system svc kube-dns '{.spec.selector.k8s-app}')
|
|
own=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].kind}')
|
|
name=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].name}')
|
|
if [ "$own" = ReplicaSet ]; then want="deployment $(j -n kube-system rs "$name" '{.metadata.ownerReferences[0].name}')"
|
|
else want="$(echo "$own" | tr A-Z a-z) $name"; fi
|
|
eq "$(sed -n 's/^ *dns *: *//p' "$f" | head -1 | tr -s ' ' | tr A-Z a-z | sed 's/ *$//')" "$want" """),
|
|
],
|
|
solution=[cmd(r"""systemctl status kubelet | head -3
|
|
sudo ls /etc/kubernetes/manifests""", r"""The host is the control-plane node. The kubelet is a systemd service - a process; etcd and the three control-plane components are manifests in the static Pod path."""),
|
|
cmd(r"""k -n kube-system get pods -o wide
|
|
k -n kube-system get deploy,ds""", r"""Static Pods carry the node's name as a suffix. What answers DNS is the `coredns` Deployment, though its Service is called `kube-dns`; `kube-proxy` and `kindnet` are DaemonSets."""),
|
|
hand(r"""vim {WS}/q86/controlplane.txt""",
|
|
r"""```
|
|
kubelet: process
|
|
kube-apiserver: static-pod
|
|
kube-scheduler: static-pod
|
|
kube-controller-manager: static-pod
|
|
etcd: static-pod
|
|
dns: deployment coredns
|
|
```""",
|
|
run=r"""cat > {WS}/q86/controlplane.txt <<'EOF'
|
|
kubelet: process
|
|
kube-apiserver: static-pod
|
|
kube-scheduler: static-pod
|
|
kube-controller-manager: static-pod
|
|
etcd: static-pod
|
|
dns: deployment coredns
|
|
EOF""")],
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 87 ----
|
|
task(
|
|
id=87, domain="TSHOOT", title="Cluster facts to an exact file",
|
|
task="""
|
|
Write `{WS}/q87/cluster-info.txt` — answers only, exactly this format:
|
|
|
|
```
|
|
1: <number of control-plane nodes>
|
|
2: <number of worker nodes>
|
|
3: <the Service CIDR>
|
|
4: <name of the CNI plugin>, <full path of its configuration file on the nodes>
|
|
5: <the suffix the names of static Pods running on node {W3} end with>
|
|
```
|
|
""",
|
|
setup=r"""rm -rf "$WS/q87"; mkdir -p "$WS/q87" """,
|
|
checks=[
|
|
C("node counts (lines 1 and 2)", 2, r"""
|
|
f="$WS/q87/cluster-info.txt"; [ -s "$f" ] || exit 1
|
|
cp=$($K get nodes -l node-role.kubernetes.io/control-plane --no-headers 2>/dev/null | wc -l); all=$($K get nodes --no-headers | wc -l)
|
|
eq "$(sed -n 's/^ *1 *: *//p' "$f" | tr -d '[:space:]')" "$cp" && eq "$(sed -n 's/^ *2 *: *//p' "$f" | tr -d '[:space:]')" "$((all - cp))" """),
|
|
C("Service CIDR (line 3)", 2, r"""
|
|
sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2)
|
|
[ -n "$sc" ] && eq "$(sed -n 's/^ *3 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]')" "$sc" """),
|
|
C("CNI plugin and its config file (line 4)", 2, r"""
|
|
l=$(sed -n 's/^ *4 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]')
|
|
cni=$($K -n kube-system get ds -o name | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1)
|
|
cfg=$(nx "$W1" 'ls /etc/cni/net.d/*.conflist /etc/cni/net.d/*.conf 2>/dev/null | head -1' </dev/null)
|
|
[ -n "$cni" ] && [ -n "$cfg" ] && echo "$l" | grep -qi "$cni" && echo "$l" | grep -qF "$cfg" """),
|
|
C("static Pod name suffix on {W3} (line 5)", 1, r"""
|
|
v=$(sed -n 's/^ *5 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]'); [ "$v" = "-$W3" ] || [ "$v" = "$W3" ]"""),
|
|
],
|
|
solution=[cmd(r"""k get nodes
|
|
k -n kube-system describe pod -l component=kube-apiserver | grep service-cluster-ip-range
|
|
k -n kube-system get ds""", r"""`ROLES control-plane` on one node, the rest workers; the API server's `--service-cluster-ip-range`; and the CNI's DaemonSet, `kindnet`."""),
|
|
hand(r"""ssh {W1} ls /etc/cni/net.d""",
|
|
r"""The CNI's configuration on a node: `10-kindnet.conflist`.""",
|
|
run=r""""""),
|
|
hand(r"""vim {WS}/q87/cluster-info.txt""",
|
|
r"""The values you read - here:
|
|
```
|
|
1: 1
|
|
2: 3
|
|
3: 10.96.0.0/16
|
|
4: kindnet, /etc/cni/net.d/10-kindnet.conflist
|
|
5: -{W3}
|
|
```
|
|
A static Pod's mirror is named `<pod>-<node>`, e.g. `kube-apiserver-{CP}`.""",
|
|
run=r"""C=$(kubectl get nodes --no-headers -l node-role.kubernetes.io/control-plane | wc -l)
|
|
W=$(kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' | wc -l)
|
|
CIDR=$(kubectl -n kube-system get pod -l component=kube-apiserver -o yaml | grep -o 'service-cluster-ip-range=[^ ]*' | head -1 | cut -d= -f2)
|
|
F=$(ssh -n {W1} "sudo sh -c 'ls /etc/cni/net.d/*.conflist | sort | head -1'")
|
|
printf '1: %s\n2: %s\n3: %s\n4: kindnet, %s\n5: -{W3}\n' "$C" "$W" "$CIDR" "$F" > {WS}/q87/cluster-info.txt""")],
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 88 ----
|
|
task(
|
|
id=88, domain="NET", title="Ready only when its dependency answers",
|
|
task="""
|
|
In Namespace `oslo-web`:
|
|
|
|
1. Create Pod `gatekeeper` (image `busybox:1.36`, command `sleep 86400`) with an **exec readiness
|
|
probe** that runs `wget -T2 -qO- http://ledger-api:8080`, initial delay 5 seconds, period 5
|
|
seconds. For now it must stay **not ready** — confirm that it does.
|
|
2. Deployment `ledger` serves HTTP on port 80, but nothing exposes it. Create the Service
|
|
`ledger-api` (ClusterIP, port **8080**) for it, so that `gatekeeper` turns Ready without you
|
|
touching the Pod again.
|
|
""",
|
|
setup=NS + r"""
|
|
mkns oslo-web
|
|
$K -n oslo-web delete pod gatekeeper --ignore-not-found --force --grace-period=0 >/dev/null 2>&1
|
|
$K -n oslo-web delete svc ledger-api --ignore-not-found >/dev/null
|
|
$K -n oslo-web create deploy ledger --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
|
|
$K -n oslo-web rollout status deploy ledger --timeout=120s >/dev/null 2>&1
|
|
""",
|
|
checks=[
|
|
C("gatekeeper has the exec readiness probe (delay 5, period 5)", 2, r"""
|
|
p='{.spec.containers[0].readinessProbe'
|
|
j -n oslo-web pod gatekeeper "$p.exec.command}" | grep -q 'ledger-api:8080' \
|
|
&& eq "$(j -n oslo-web pod gatekeeper "$p.initialDelaySeconds}")" 5 && eq "$(j -n oslo-web pod gatekeeper "$p.periodSeconds}")" 5"""),
|
|
C("Service ledger-api: 8080 -> 80 with ready endpoints", 2, r"""
|
|
eq "$(j -n oslo-web svc ledger-api '{.spec.ports[0].port}')" 8080 && eq "$(j -n oslo-web svc ledger-api '{.spec.ports[0].targetPort}')" 80 \
|
|
&& [ -n "$(j -n oslo-web endpointslices -l kubernetes.io/service-name=ledger-api '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""),
|
|
C("gatekeeper is Ready", 3, r"""
|
|
for i in $(seq 1 20); do [ "$(j -n oslo-web pod gatekeeper '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 3; done; exit 1"""),
|
|
],
|
|
solution="""
|
|
k -n oslo-web run gatekeeper --image=busybox:1.36 $do -- sleep 86400 > gk.yaml
|
|
# add to the container:
|
|
# readinessProbe:
|
|
# exec:
|
|
# command: ["sh", "-c", "wget -T2 -qO- http://ledger-api:8080"]
|
|
# initialDelaySeconds: 5
|
|
# periodSeconds: 5
|
|
k apply -f gk.yaml
|
|
k -n oslo-web get pod gatekeeper # 0/1 Running - the probe cannot resolve ledger-api
|
|
k -n oslo-web describe pod gatekeeper | tail -3 # Readiness probe failed: wget: bad address 'ledger-api:8080'
|
|
k -n oslo-web expose deploy ledger --name=ledger-api --port=8080 --target-port=80
|
|
k -n oslo-web get pod gatekeeper -w # 1/1 after the next probe
|
|
# Readiness removes a Pod from Service endpoints and blocks rollouts; liveness restarts it.
|
|
# A dependency check belongs in readiness - failing liveness here would just restart-loop the Pod.
|
|
""",
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 89 ----
|
|
task(
|
|
id=89, domain="WORK", title="Sorting done by kubectl",
|
|
task="""
|
|
Write two executable scripts that call `kubectl` (spelled out — aliases do not exist in a fresh
|
|
shell) and let **kubectl itself** do the sorting with `--sort-by`:
|
|
|
|
1. `{WS}/q89/pods-by-age.sh` — every Pod in every Namespace, oldest first (by creation time).
|
|
2. `{WS}/q89/pods-by-uid.sh` — the Pods in Namespace `kube-system`, sorted by `metadata.uid`.
|
|
""",
|
|
setup=r"""rm -rf "$WS/q89"; mkdir -p "$WS/q89" """,
|
|
checks=[
|
|
C("pods-by-age.sh: all namespaces, --sort-by creationTimestamp, runs", 2, r"""
|
|
f="$WS/q89/pods-by-age.sh"; [ -x "$f" ] || exit 1
|
|
grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \
|
|
&& grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.creationTimestamp" "$f" || exit 1
|
|
out=$(timeout 60 bash "$f" 2>/dev/null); [ "$(echo "$out" | wc -l)" -ge 5 ] && echo "$out" | grep -q kube-system"""),
|
|
C("pods-by-uid.sh: kube-system, --sort-by metadata.uid, runs", 2, r"""
|
|
f="$WS/q89/pods-by-uid.sh"; [ -x "$f" ] || exit 1
|
|
grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-n|--namespace)[= ]*kube-system' "$f" \
|
|
&& grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.uid" "$f" || exit 1
|
|
timeout 60 bash "$f" 2>/dev/null | grep -q etcd"""),
|
|
C("no aliases in either script", 1, r"""[ -s "$WS/q89/pods-by-age.sh" ] && [ -s "$WS/q89/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q89/pods-by-age.sh" "$WS/q89/pods-by-uid.sh" """),
|
|
],
|
|
solution=[hand(r"""vim {WS}/q89/pods-by-age.sh""",
|
|
r"""`kubectl` in full: a script runs in a fresh shell, where the `k` alias does not exist.
|
|
```
|
|
kubectl get pods -A --sort-by=.metadata.creationTimestamp
|
|
```""",
|
|
run=r"""echo 'kubectl get pods -A --sort-by=.metadata.creationTimestamp' > {WS}/q89/pods-by-age.sh"""),
|
|
hand(r"""vim {WS}/q89/pods-by-uid.sh""",
|
|
r"""```
|
|
kubectl -n kube-system get pods --sort-by=.metadata.uid
|
|
```""",
|
|
run=r"""echo 'kubectl -n kube-system get pods --sort-by=.metadata.uid' > {WS}/q89/pods-by-uid.sh"""),
|
|
cmd(r"""chmod +x {WS}/q89/*.sh
|
|
{WS}/q89/pods-by-age.sh | head -3""", r"""The control-plane Pods first: they are the oldest. `--sort-by` takes a JSONPath into each item, ascending only.""")],
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 90 ----
|
|
task(
|
|
id=90, domain="TSHOOT", title="Event forensics around kube-proxy", breaks="w2-kube-proxy",
|
|
task="""
|
|
1. Write `{WS}/q90/cluster-events.sh`: a script that prints the latest events of the **whole**
|
|
cluster (all Namespaces) sorted by creation time, using `kubectl`. Make it executable.
|
|
2. Delete the kube-proxy Pod that runs on node `{W2}`. Write the events this caused to
|
|
`{WS}/q90/pod-deleted.log`.
|
|
3. On node `{W2}`, stop the kube-proxy **container** with `crictl stop` — leave the Pod object
|
|
alone. Write the events this caused to `{WS}/q90/container-killed.log`.
|
|
4. Which of the two actions made the Pod's `RESTARTS` counter go up? Write `pod` or `container` to
|
|
`{WS}/q90/restarts.txt`.
|
|
|
|
Do step 2 before step 3.
|
|
""",
|
|
setup=r"""
|
|
rm -rf "$WS/q90"; mkdir -p "$WS/q90"
|
|
$K -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 -o jsonpath='{.items[0].metadata.uid}' > "$WS/.q90-uid"
|
|
""",
|
|
checks=[
|
|
C("cluster-events.sh: all namespaces, sorted by time, runs", 2, r"""
|
|
f="$WS/q90/cluster-events.sh"; [ -x "$f" ] || exit 1
|
|
grep -Eq 'kubectl\b.*\bget\b.*\b(ev|events?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \
|
|
&& grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?(metadata\.creationTimestamp|lastTimestamp|eventTime|firstTimestamp)" "$f" || exit 1
|
|
timeout 60 bash "$f" >/dev/null 2>&1"""),
|
|
C("the kube-proxy Pod on {W2} was deleted and replaced", 2, r"""
|
|
old=$(cat "$WS/.q90-uid" 2>/dev/null); [ -n "$old" ] || exit 1
|
|
now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}')
|
|
[ -n "$now" ] && [ "$now" != "$old" ] && eq "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.phase}')" Running"""),
|
|
# counted on the REPLACEMENT pod, which starts at 0 - after a node reboot the original pod
|
|
# often already shows a restart, which would otherwise hand these points out for free
|
|
C("then its container was killed (restart counter >= 1)", 2, r"""
|
|
old=$(cat "$WS/.q90-uid" 2>/dev/null); now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}')
|
|
[ -n "$old" ] && [ -n "$now" ] && [ "$now" != "$old" ] || exit 1
|
|
[ "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null"""),
|
|
C("pod-deleted.log and container-killed.log hold the events", 2, r"""
|
|
for f in pod-deleted.log container-killed.log; do
|
|
grep -q 'kube-proxy' "$WS/q90/$f" 2>/dev/null && grep -Eq 'Killing|SuccessfulCreate|SuccessfulDelete|Scheduled|Pulled|Created|Started|BackOff' "$WS/q90/$f" || exit 1
|
|
done"""),
|
|
C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q90/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""),
|
|
],
|
|
solution=[hand(r"""vim {WS}/q90/cluster-events.sh""",
|
|
r"""```
|
|
kubectl get events -A --sort-by=.metadata.creationTimestamp
|
|
```""",
|
|
run=r"""echo 'kubectl get events -A --sort-by=.metadata.creationTimestamp' > {WS}/q90/cluster-events.sh"""),
|
|
cmd(r"""chmod +x {WS}/q90/cluster-events.sh
|
|
k -n kube-system get pods -l k8s-app=kube-proxy -o wide""", r"""Note the name of the one on `{W2}`."""),
|
|
hand(r"""k -n kube-system delete pod kube-proxy-xxxxx""",
|
|
r"""With that name. The DaemonSet replaces it with a new Pod at once.""",
|
|
run=r"""P=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].metadata.name}')
|
|
kubectl -n kube-system delete pod $P
|
|
for i in $(seq 1 40); do
|
|
[ "$(kubectl -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName={W2} -o jsonpath='{.items[0].status.phase}' 2>/dev/null)" = Running ] && break; sleep 3
|
|
done
|
|
sleep 5"""),
|
|
cmd(r"""k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
|
|
cat {WS}/q90/pod-deleted.log""", r"""`Killing` the old Pod, `SuccessfulCreate` by the DaemonSet, then `Scheduled`, `Pulled`, `Created`, `Started` for the new one."""),
|
|
hand(r"""{SHELL_W2}
|
|
sudo -i
|
|
crictl ps --name kube-proxy
|
|
crictl stop 1a2b3c4d5e6f
|
|
exit
|
|
exit""",
|
|
r"""With the container ID `crictl ps` printed. Only the container goes: the kubelet starts a new one in the same Pod.""",
|
|
run=r"""ssh -n {W2} 'sudo crictl stop $(sudo crictl ps --name kube-proxy -q)'
|
|
sleep 25"""),
|
|
cmd(r"""k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
|
|
k -n kube-system get pods -l k8s-app=kube-proxy -o wide
|
|
echo container > {WS}/q90/restarts.txt""", r"""`RESTARTS 1` on `{W2}`: a new container in the same Pod is a restart; a new Pod starts again at 0.""")],
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 91 ----
|
|
def _census(r):
|
|
n = r.sample(range(2, 10), 4)
|
|
return {"ctx": "k8s-c1-H", "north": n[0], "south": n[1], "east": n[2], "west": n[3]}
|
|
|
|
|
|
task(
|
|
id=91, domain="ARCH", title="API resources and a Role census", params=_census,
|
|
task="""
|
|
1. Write the names of every **namespaced** resource type this API server offers to
|
|
`{WS}/q91/namespaced.txt`, one per line — plural names, as `kubectl api-resources` prints them.
|
|
2. Of the Namespaces `census-north`, `census-south`, `census-east` and `census-west`, which one holds
|
|
the most Roles? Write `<namespace> <number of Roles>` to `{WS}/q91/most-roles.txt`.
|
|
""",
|
|
setup=NS + r"""
|
|
rm -rf "$WS/q91"; mkdir -p "$WS/q91"
|
|
for pair in north:%{north} south:%{south} east:%{east} west:%{west}; do
|
|
ns=census-${pair%%:*}; n=${pair##*:}
|
|
mkns $ns
|
|
$K -n $ns delete role --all >/dev/null 2>&1
|
|
for i in $(seq 1 $n); do $K -n $ns create role reader-$i --verb=get,list --resource=configmaps >/dev/null; done
|
|
done
|
|
""",
|
|
checks=[
|
|
C("namespaced.txt lists exactly the namespaced resource types", 3, r"""
|
|
f="$WS/q91/namespaced.txt"; [ -s "$f" ] || exit 1
|
|
want=$($K api-resources --namespaced=true -o name 2>/dev/null | sed 's/\..*//' | sort -u)
|
|
got=$(awk '{print $1}' "$f" | sed 's/\..*//' | grep . | sort -u)
|
|
[ -n "$want" ] && [ "$got" = "$want" ]"""),
|
|
C("most-roles.txt names the namespace and its Role count", 3, r"""
|
|
best=$(for ns in census-north census-south census-east census-west; do echo "$ns $($K -n $ns get role --no-headers 2>/dev/null | wc -l)"; done | sort -k2 -n | tail -1)
|
|
eq "$(tr -s ' ' < "$WS/q91/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """),
|
|
],
|
|
solution=[cmd(r"""k api-resources --namespaced=true -o name > {WS}/q91/namespaced.txt
|
|
head -3 {WS}/q91/namespaced.txt""", r"""`-o name` prints the plural names alone (`deployments.apps`: name and group); `--namespaced=false` would give the cluster-scoped half."""),
|
|
cmd(r"""k -n census-north get role --no-headers | wc -l
|
|
k -n census-south get role --no-headers | wc -l
|
|
k -n census-east get role --no-headers | wc -l
|
|
k -n census-west get role --no-headers | wc -l""", r"""Or all at once: `k get role -A | grep census`."""),
|
|
hand(r"""echo "census-east 9" > {WS}/q91/most-roles.txt""",
|
|
r"""With the namespace that has the most, and its count.""",
|
|
run=r"""for ns in census-north census-south census-east census-west; do echo "$ns $(kubectl -n $ns get role --no-headers | wc -l)"; done | sort -k2 -n | tail -1 > {WS}/q91/most-roles.txt""")],
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 92 ----
|
|
task(
|
|
id=92, domain="TSHOOT", title="Kubelet unit points at nothing", breaks="w1-kubelet", disruptive=True,
|
|
task="""
|
|
Node `{W1}` went `NotReady` after someone "tidied up" its kubelet installation. Bring it back to
|
|
`Ready`; the kubelet has to run as the systemd service it always was. Then write the full path of the
|
|
file you had to fix to `{WS}/q92/fixed-file.txt`.
|
|
|
|
Reach the node with `{SHELL_W1}`. Pods on a node that stays NotReady are evicted after five minutes.
|
|
""",
|
|
setup=r"""
|
|
rm -rf "$WS/q92"; mkdir -p "$WS/q92"
|
|
nx "$W1" '
|
|
d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
|
|
grep -q "^ExecStart=/usr/bin/kubelet " $d && sed -i "s|^ExecStart=/usr/bin/kubelet |ExecStart=/usr/local/bin/kubelet |" $d
|
|
systemctl daemon-reload; systemctl restart kubelet' </dev/null >/dev/null 2>&1
|
|
exit 0
|
|
""",
|
|
checks=[
|
|
C("{W1} is Ready", 4, r"""
|
|
nx "$W1" 'systemctl is-active --quiet kubelet' </dev/null || exit 1
|
|
for i in $(seq 1 15); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 4; done; exit 1"""),
|
|
C("kubelet active, and its unit runs a kubelet binary that exists", 2, r"""
|
|
nx "$W1" 'systemctl is-active --quiet kubelet && b=$(sed -n "s|^ExecStart=\([^ ]*kubelet\) .*|\1|p" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | tail -1) && [ -x "$b" ]' </dev/null"""),
|
|
C("fixed-file.txt names the drop-in", 2, r"""grep -q '/etc/systemd/system/kubelet.service.d/10-kubeadm.conf' "$WS/q92/fixed-file.txt" 2>/dev/null"""),
|
|
],
|
|
solution="""
|
|
k get nodes # {W1} NotReady
|
|
{SHELL_W1}
|
|
sudo -i
|
|
systemctl status kubelet
|
|
# Process: ... ExecStart=/usr/local/bin/kubelet ... (code=exited, status=203/EXEC)
|
|
journalctl -u kubelet -n 20 --no-pager # Unable to locate executable '/usr/local/bin/kubelet'
|
|
which kubelet # /usr/bin/kubelet - the binary was never moved
|
|
systemctl cat kubelet # shows the unit AND every drop-in, with file names
|
|
vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf # ExecStart=/usr/bin/kubelet ...
|
|
systemctl daemon-reload # systemd caches unit files - without this nothing changes
|
|
systemctl restart kubelet
|
|
systemctl status kubelet # active (running)
|
|
exit; exit
|
|
echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q92/fixed-file.txt
|
|
k get nodes -w
|
|
# status=203/EXEC means systemd could not exec the binary at all - a path problem, not a kubelet
|
|
# configuration problem. `ExecStart=` (empty) in a drop-in clears the unit's own ExecStart first.
|
|
""",
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 93 ----
|
|
task(
|
|
id=93, domain="WORK", title="Secrets as a read-only volume and renamed env vars",
|
|
params=lambda r: {"ctx": "k8s-c1-H", "pw": "ledger-" + "".join(r.choice("abcdefghjkmnpqrstuvwxyz23456789") for _ in range(10))},
|
|
task="""
|
|
In Namespace `oslo-secure`:
|
|
|
|
1. Create Secret `bundle` from the file `{WS}/q93/tls-bundle.pem`; the key must be the file's name.
|
|
2. Create Secret `db-login` with `user=svc-ledger` and `pass=%{pw}`.
|
|
3. Create Pod `vault-reader` (image `busybox:1.36`, command `sleep 86400`) that
|
|
- mounts Secret `bundle` **read-only** at `/etc/bundle`,
|
|
- receives `db-login`'s keys as environment variables named **`APP_USER`** and **`APP_PASS`**.
|
|
""",
|
|
setup=NS + r"""
|
|
mkns oslo-secure
|
|
rm -rf "$WS/q93"; mkdir -p "$WS/q93"
|
|
openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=ledger.oslo.internal" -keyout /dev/null -out "$WS/q93/tls-bundle.pem" >/dev/null 2>&1
|
|
$K -n oslo-secure delete pod vault-reader --ignore-not-found --force --grace-period=0 >/dev/null 2>&1
|
|
$K -n oslo-secure delete secret bundle db-login --ignore-not-found >/dev/null
|
|
""",
|
|
checks=[
|
|
C("Secrets bundle (key tls-bundle.pem, the file's content) and db-login", 2, r"""
|
|
[ "$(j -n oslo-secure secret bundle '{.data.tls-bundle\.pem}' | base64 -d 2>/dev/null)" = "$(cat "$WS/q93/tls-bundle.pem")" ] \
|
|
&& eq "$(j -n oslo-secure secret db-login '{.data.user}' | base64 -d)" svc-ledger && eq "$(j -n oslo-secure secret db-login '{.data.pass}' | base64 -d)" '%{pw}'"""),
|
|
C("vault-reader Running with bundle mounted read-only at /etc/bundle", 3, r"""
|
|
p="$K -n oslo-secure get pod vault-reader"
|
|
eq "$($p -o jsonpath='{.status.phase}' 2>/dev/null)" Running || exit 1
|
|
v=$($p -o jsonpath='{.spec.volumes[?(@.secret.secretName=="bundle")].name}'); [ -n "$v" ] || exit 1
|
|
eq "$($p -o jsonpath="{.spec.containers[0].volumeMounts[?(@.name==\"$v\")].mountPath}")" /etc/bundle \
|
|
&& eq "$($p -o jsonpath="{.spec.containers[0].volumeMounts[?(@.name==\"$v\")].readOnly}")" true \
|
|
&& $K -n oslo-secure exec vault-reader -- cat /etc/bundle/tls-bundle.pem 2>/dev/null | grep -q 'BEGIN CERTIFICATE'"""),
|
|
C("APP_USER / APP_PASS come from db-login", 3, r"""
|
|
e=$(j -n oslo-secure pod vault-reader '{.spec.containers[0].env}')
|
|
echo "$e" | grep -q '"name":"APP_USER"' && echo "$e" | grep -q '"name":"APP_PASS"' && echo "$e" | grep -q 'db-login' || exit 1
|
|
env=$($K -n oslo-secure exec vault-reader -- env 2>/dev/null)
|
|
echo "$env" | grep -qx 'APP_USER=svc-ledger' && echo "$env" | grep -qx 'APP_PASS=%{pw}'"""),
|
|
],
|
|
solution="""
|
|
k -n oslo-secure create secret generic bundle --from-file={WS}/q93/tls-bundle.pem # key = file name
|
|
k -n oslo-secure create secret generic db-login --from-literal=user=svc-ledger --from-literal=pass='%{pw}'
|
|
k -n oslo-secure run vault-reader --image=busybox:1.36 $do -- sleep 86400 > vr.yaml
|
|
# in vr.yaml, pod level:
|
|
# volumes: [{name: bundle, secret: {secretName: bundle}}]
|
|
# in the container:
|
|
# volumeMounts: [{name: bundle, mountPath: /etc/bundle, readOnly: true}]
|
|
# env:
|
|
# - name: APP_USER
|
|
# valueFrom: {secretKeyRef: {name: db-login, key: user}}
|
|
# - name: APP_PASS
|
|
# valueFrom: {secretKeyRef: {name: db-login, key: pass}}
|
|
k apply -f vr.yaml
|
|
k -n oslo-secure exec vault-reader -- env | grep APP_
|
|
k -n oslo-secure exec vault-reader -- ls /etc/bundle
|
|
# envFrom would inject the keys under their own names (user, pass); renaming needs env + secretKeyRef.
|
|
# A Secret volume updates in place when the Secret changes; env vars never do.
|
|
""",
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 94 ----
|
|
task(
|
|
id=94, domain="ARCH", title="Static Pod on the control plane behind a Service",
|
|
task="""
|
|
1. On the control-plane node (`{SHELL_CP}`) create a **static Pod** `ops-beacon` in Namespace
|
|
`default`: image `nginx:1.27`, requests `cpu: 10m` and `memory: 20Mi`, label `app=ops-beacon`.
|
|
2. Expose it with a Service `ops-beacon-svc` of type **NodePort** (port 80 → 80) whose endpoints
|
|
contain that static Pod.
|
|
3. Write the node port the Service was given to `{WS}/q94/nodeport.txt`.
|
|
""",
|
|
setup=r"""
|
|
rm -rf "$WS/q94"; mkdir -p "$WS/q94"
|
|
nx "$CP" 'rm -f /etc/kubernetes/manifests/ops-beacon*.yaml' </dev/null
|
|
$K -n default delete svc ops-beacon-svc --ignore-not-found >/dev/null
|
|
# the kubelet drops the mirror pod on its next sync of the manifest directory
|
|
for i in $(seq 1 30); do $K -n default get pod "ops-beacon-$CP" >/dev/null 2>&1 || break; sleep 2; done
|
|
""",
|
|
checks=[
|
|
C("manifest in the control-plane node's staticPodPath", 1, r"""
|
|
nx "$CP" 'd=$(awk "/^staticPodPath:/{print \$2}" /var/lib/kubelet/config.yaml); grep -ls "ops-beacon" "$d"/*.yaml' </dev/null >/dev/null 2>&1"""),
|
|
C("mirror Pod ops-beacon-{CP} Running with the requests and label", 3, r"""
|
|
p="$K -n default get pod ops-beacon-$CP"
|
|
eq "$($p -o jsonpath='{.status.phase}' 2>/dev/null)" Running && eq "$($p -o jsonpath='{.metadata.labels.app}')" ops-beacon \
|
|
&& eq "$($p -o jsonpath='{.spec.containers[0].resources.requests.cpu}')" 10m && eq "$($p -o jsonpath='{.spec.containers[0].resources.requests.memory}')" 20Mi"""),
|
|
C("NodePort Service ops-beacon-svc 80 -> 80 with the static Pod as endpoint", 3, r"""
|
|
s="$K -n default get svc ops-beacon-svc"
|
|
eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" NodePort && eq "$($s -o jsonpath='{.spec.ports[0].port}')" 80 && eq "$($s -o jsonpath='{.spec.ports[0].targetPort}')" 80 || exit 1
|
|
ip=$(j -n default pod "ops-beacon-$CP" '{.status.podIP}'); [ -n "$ip" ] || exit 1
|
|
j -n default endpointslices -l kubernetes.io/service-name=ops-beacon-svc '{.items[*].endpoints[*].addresses[*]}' | tr ' ' '\n' | grep -qx "$ip" """),
|
|
C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q94/nodeport.txt" 2>/dev/null)" "$np" """),
|
|
],
|
|
solution=[cmd(r"""k -n default run ops-beacon --image=nginx:1.27 --labels=app=ops-beacon --dry-run=client -o yaml > beacon.yaml"""),
|
|
hand(r"""vim beacon.yaml""",
|
|
r"""Replace the container's `resources: {}` with:
|
|
```
|
|
resources:
|
|
requests:
|
|
cpu: 10m
|
|
memory: 20Mi
|
|
```""",
|
|
run=r"""yq -i '.spec.containers[0].resources = {"requests": {"cpu": "10m", "memory": "20Mi"}}' beacon.yaml"""),
|
|
cmd(r"""sudo grep staticPodPath /var/lib/kubelet/config.yaml
|
|
sudo cp beacon.yaml /etc/kubernetes/manifests/ops-beacon.yaml""", r"""The host is the control-plane node. Its taint keeps the scheduler away, not the kubelet: static Pods run regardless."""),
|
|
hand(r"""k -n default get pod ops-beacon-{CP} -o wide""",
|
|
r"""Again until the mirror Pod is `Running`.""",
|
|
run=r"""for i in $(seq 1 60); do [ "$(kubectl -n default get pod ops-beacon-{CP} -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && break; sleep 3; done"""),
|
|
cmd(r"""k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
|
|
k -n default get svc ops-beacon-svc
|
|
k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc""", r"""`expose pod` makes the selector from the Pod's labels: the mirror Pod's IP is the endpoint. Note the node port, after `80:`."""),
|
|
hand(r"""echo 31234 > {WS}/q94/nodeport.txt""",
|
|
r"""With the node port `get svc` showed.""",
|
|
run=r"""kubectl -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt""")],
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 95 ----
|
|
task(
|
|
id=95, domain="ARCH", title="The kubelet's two certificates",
|
|
task="""
|
|
On node `{W2}` find the certificate the kubelet uses as a **client** (to talk to the API server)
|
|
and the one it **serves** its own API with (port 10250). Write exactly this to
|
|
`{WS}/q95/kubelet-certs.txt`:
|
|
|
|
```
|
|
client-issuer: <issuer, as openssl prints it>
|
|
client-eku: <the extended key usage>
|
|
server-issuer: <issuer>
|
|
server-eku: <the extended key usage>
|
|
```
|
|
|
|
Reach the node with `{SHELL_W2}`.
|
|
""",
|
|
setup=r"""rm -rf "$WS/q95"; mkdir -p "$WS/q95" """,
|
|
checks=[
|
|
C("client certificate: issuer and extended key usage", 3, r"""
|
|
f="$WS/q95/kubelet-certs.txt"; [ -s "$f" ] || exit 1
|
|
iss=$(nx "$W2" 'openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -issuer' </dev/null | sed 's/^issuer= *//' | tr -d ' ' | tr A-Z a-z)
|
|
[ -n "$iss" ] && sed -n 's/^ *client-issuer *: *//p' "$f" | tr -d ' ' | tr A-Z a-z | grep -qF "$iss" \
|
|
&& sed -n 's/^ *client-eku *: *//p' "$f" | grep -qi 'client auth'"""),
|
|
C("serving certificate: issuer and extended key usage", 3, r"""
|
|
f="$WS/q95/kubelet-certs.txt"
|
|
iss=$(nx "$W2" 'openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -issuer' </dev/null | sed 's/^issuer= *//' | tr -d ' ' | tr A-Z a-z)
|
|
[ -n "$iss" ] && sed -n 's/^ *server-issuer *: *//p' "$f" | tr -d ' ' | tr A-Z a-z | grep -qF "$iss" \
|
|
&& sed -n 's/^ *server-eku *: *//p' "$f" | grep -qi 'server auth'"""),
|
|
],
|
|
solution="""
|
|
{SHELL_W2}
|
|
sudo -i
|
|
grep -A3 users: /etc/kubernetes/kubelet.conf # client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
|
|
ls /var/lib/kubelet/pki/ # kubelet.crt / kubelet.key = the serving pair
|
|
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text | grep -E 'Issuer|Extended Key Usage' -A1
|
|
# Issuer: CN = kubernetes Extended Key Usage: TLS Web Client Authentication
|
|
openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -text | grep -E 'Issuer|Extended Key Usage' -A1
|
|
# Issuer: CN = {W2}-ca@<timestamp> Extended Key Usage: TLS Web Server Authentication
|
|
exit; exit
|
|
# write the four lines with the values YOUR openssl printed:
|
|
# client-issuer: CN=kubernetes
|
|
# client-eku: TLS Web Client Authentication
|
|
# server-issuer: CN={W2}-ca@<timestamp>
|
|
# server-eku: TLS Web Server Authentication
|
|
# The client cert is signed by the cluster CA and rotated by the kubelet through CSRs. The serving
|
|
# cert here is self-signed by the kubelet at startup (no serverTLSBootstrap) - which is exactly why
|
|
# metrics-server needs --kubelet-insecure-tls on this cluster.
|
|
""",
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 96 ----
|
|
task(
|
|
id=96, domain="WORK", title="A Pod for the control plane only",
|
|
task="""
|
|
Create Pod `cp-watch` in Namespace `default`: image `httpd:2-alpine`, container name `watch`.
|
|
|
|
It must be able to run **only** on control-plane nodes. Select them by the node label kubeadm puts
|
|
on control-plane nodes — not by host name, and not with `nodeName` — and tolerate whatever keeps
|
|
ordinary Pods off them. It must be Running.
|
|
""",
|
|
setup=r"""$K -n default delete pod cp-watch --ignore-not-found --force --grace-period=0 >/dev/null 2>&1""",
|
|
checks=[
|
|
C("cp-watch Running on the control-plane node, container named watch", 2, r"""
|
|
eq "$(j -n default pod cp-watch '{.status.phase}')" Running && eq "$(j -n default pod cp-watch '{.spec.nodeName}')" "$CP" && eq "$(j -n default pod cp-watch '{.spec.containers[0].name}')" watch"""),
|
|
C("selected by the node-role.kubernetes.io/control-plane label", 2, r"""
|
|
s=$(j -n default pod cp-watch '{.spec.nodeSelector}{.spec.affinity.nodeAffinity}')
|
|
echo "$s" | grep -q 'node-role.kubernetes.io/control-plane' && ! echo "$s" | grep -q 'kubernetes.io/hostname'"""),
|
|
C("tolerates the control-plane taint, no nodeName in the manifest", 2, r"""
|
|
$K -n default get pod cp-watch -o json 2>/dev/null | python3 -c '
|
|
import sys,json; t=json.load(sys.stdin)["spec"].get("tolerations") or []
|
|
sys.exit(0 if any("control-plane" in (x.get("key") or "") or (x.get("operator")=="Exists" and not x.get("key")) for x in t) else 1)' || exit 1
|
|
$K -n default get pod cp-watch -o json --show-managed-fields | python3 -c '
|
|
import sys,json
|
|
mf=json.load(sys.stdin)["metadata"].get("managedFields",[])
|
|
sys.exit(1 if any("f:nodeName" in json.dumps(m.get("fieldsV1",{})) for m in mf) else 0)'"""),
|
|
],
|
|
solution="""
|
|
k get node {CP} --show-labels | tr ',' '\\n' | grep node-role # node-role.kubernetes.io/control-plane=
|
|
k describe node {CP} | grep Taints # node-role.kubernetes.io/control-plane:NoSchedule
|
|
k -n default run cp-watch --image=httpd:2-alpine $do > cp.yaml
|
|
# rename the container to watch, then add at pod level:
|
|
# nodeSelector:
|
|
# node-role.kubernetes.io/control-plane: ""
|
|
# tolerations:
|
|
# - {key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule}
|
|
k apply -f cp.yaml
|
|
k -n default get pod cp-watch -o wide
|
|
# The toleration only ALLOWS the control plane; the nodeSelector is what REQUIRES it. You need both.
|
|
""",
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 97 ----
|
|
task(
|
|
id=97, domain="TSHOOT", title="Stop the scheduler, schedule by hand", breaks="scheduler",
|
|
task="""
|
|
1. Stop the kube-scheduler **temporarily** — it runs as a static Pod on the control-plane node — and
|
|
confirm it is gone.
|
|
2. Create Pod `manual-1` (image `busybox:1.36`, command `sleep 86400`) in Namespace `default` and
|
|
confirm it stays `Pending`.
|
|
3. Schedule `manual-1` **by hand** onto the control-plane node and get it Running.
|
|
4. Start the kube-scheduler again, then create Pod `manual-2` (same image and command) in `default`
|
|
and confirm that the scheduler places it.
|
|
""",
|
|
setup=r"""
|
|
# a pristine copy outside the static pod directory, so the pre-session baseline can put the
|
|
# scheduler back if this task is abandoned half-done
|
|
nx "$CP" 'mkdir -p /etc/kubernetes/.cka-sim && [ -f /etc/kubernetes/manifests/kube-scheduler.yaml ] && cp /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/.cka-sim/kube-scheduler.yaml' </dev/null
|
|
$K -n default delete pod manual-1 manual-2 --ignore-not-found --force --grace-period=0 >/dev/null 2>&1
|
|
exit 0
|
|
""",
|
|
checks=[
|
|
C("kube-scheduler is back and Running", 2, r"""
|
|
$K -n default get pod manual-1 >/dev/null 2>&1 || exit 1 # only once the task was actually done
|
|
nx "$CP" 'test -f /etc/kubernetes/manifests/kube-scheduler.yaml' </dev/null || exit 1
|
|
for i in $(seq 1 15); do [ "$(j -n kube-system pod -l component=kube-scheduler '{.items[0].status.containerStatuses[0].ready}')" = true ] && exit 0; sleep 4; done; exit 1"""),
|
|
C("manual-1 Running on the control-plane node", 3, r"""
|
|
eq "$(j -n default pod manual-1 '{.status.phase}')" Running && eq "$(j -n default pod manual-1 '{.spec.nodeName}')" "$CP" """),
|
|
C("manual-2 Running, placed by the scheduler (no nodeName in its manifest)", 3, r"""
|
|
eq "$(j -n default pod manual-2 '{.status.phase}')" Running && [ -n "$(j -n default pod manual-2 '{.spec.nodeName}')" ] || exit 1
|
|
$K -n default get pod manual-2 -o json --show-managed-fields | python3 -c '
|
|
import sys,json
|
|
mf=json.load(sys.stdin)["metadata"].get("managedFields",[])
|
|
sys.exit(1 if any("f:nodeName" in json.dumps(m.get("fieldsV1",{})) for m in mf) else 0)'"""),
|
|
],
|
|
solution="""
|
|
sudo -i # the task's host is the control-plane node: no hop needed
|
|
mv /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/ # the kubelet stops what it no longer sees
|
|
exit
|
|
k -n kube-system get pod -l component=kube-scheduler # gone (give it ~20s)
|
|
k -n default run manual-1 --image=busybox:1.36 -- sleep 86400
|
|
k -n default get pod manual-1 # Pending, no node
|
|
# spec.nodeName is only settable on creation, so recreate it with the node filled in:
|
|
k -n default get pod manual-1 -o yaml > m1.yaml
|
|
# add under spec: nodeName: {CP}
|
|
k replace --force -f m1.yaml
|
|
k -n default get pod manual-1 -o wide # Running on {CP}
|
|
sudo -i # the task's host is the control-plane node: no hop needed
|
|
mv /etc/kubernetes/kube-scheduler.yaml /etc/kubernetes/manifests/
|
|
exit
|
|
k -n kube-system get pod -l component=kube-scheduler -w # Running again
|
|
k -n default run manual-2 --image=busybox:1.36 -- sleep 86400
|
|
k -n default get pod manual-2 -o wide # a worker, chosen by the scheduler
|
|
# nodeName skips the scheduler entirely - including its taint checks, which is why manual-1 can
|
|
# sit on the tainted control plane. (The Binding API does the same without recreating the pod.)
|
|
""",
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 98 ----
|
|
_CHART = r'''
|
|
mkchart() { # dir name version image replicas
|
|
mkdir -p "$1/templates"
|
|
printf 'apiVersion: v2\nname: %s\nversion: %s\nappVersion: "%s"\n' "$2" "$3" "${4##*:}" > "$1/Chart.yaml"
|
|
printf 'replicaCount: %s\nimage: %s\n' "$5" "$4" > "$1/values.yaml"
|
|
cat > "$1/templates/deployment.yaml" <<'YAML'
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata: {name: {{ .Release.Name }}, labels: {app: {{ .Release.Name }}}}
|
|
spec:
|
|
replicas: {{ .Values.replicaCount }}
|
|
selector: {matchLabels: {app: {{ .Release.Name }}}}
|
|
template:
|
|
metadata: {labels: {app: {{ .Release.Name }}}}
|
|
spec:
|
|
containers: [{name: main, image: "{{ .Values.image }}"}]
|
|
YAML
|
|
}
|
|
'''
|
|
|
|
task(
|
|
id=98, domain="ARCH", title="Helm release housekeeping",
|
|
task="""
|
|
Namespace `oslo-helm` holds several Helm releases; local charts are under `{WS}/q98/charts`.
|
|
|
|
1. Delete the release `web-legacy`.
|
|
2. Upgrade the release `web-main` to chart version **1.1.0** (`{WS}/q98/charts/nginx-lite-1.1.0`).
|
|
3. Install a new release `cache` from `{WS}/q98/charts/redis-lite` with **2** replicas (the chart's
|
|
`replicaCount` value).
|
|
4. One release in the Namespace never finished installing. Find it and remove it.
|
|
""",
|
|
setup=NS + _CHART + r"""
|
|
mkns oslo-helm
|
|
C="$WS/q98/charts"; rm -rf "$WS/q98"; mkdir -p "$C"
|
|
mkchart "$C/nginx-lite-1.0.0" nginx-lite 1.0.0 nginx:1.25 1
|
|
mkchart "$C/nginx-lite-1.1.0" nginx-lite 1.1.0 nginx:1.27 1
|
|
mkchart "$C/redis-lite" redis-lite 0.3.0 redis:7-alpine 1
|
|
mkchart /tmp/q98-stuck cache-edge 0.1.0 busybox:0.0-does-not-exist 1
|
|
for r in $(helm -n oslo-helm list -q 2>/dev/null); do helm -n oslo-helm uninstall "$r" >/dev/null 2>&1; done
|
|
helm -n oslo-helm install web-legacy "$C/nginx-lite-1.0.0" >/dev/null
|
|
helm -n oslo-helm install web-main "$C/nginx-lite-1.0.0" >/dev/null
|
|
# helm records the release as pending-install BEFORE it starts waiting; kill it mid-wait and it
|
|
# stays that way - the classic result of an interrupted install or a CI job that timed out
|
|
timeout -s KILL 6 helm -n oslo-helm install cache-broken /tmp/q98-stuck --wait --timeout 120s >/dev/null 2>&1
|
|
rm -rf /tmp/q98-stuck
|
|
helm -n oslo-helm status cache-broken -o json 2>/dev/null | grep -q '"status":"pending-install"' || echo "warn: cache-broken did not end up pending-install"
|
|
""",
|
|
checks=[
|
|
C("web-legacy is gone", 1, r"""! helm -n oslo-helm status web-legacy >/dev/null 2>&1 && helm -n oslo-helm status web-main >/dev/null 2>&1"""),
|
|
C("web-main deployed at chart version 1.1.0", 2, r"""
|
|
helm -n oslo-helm list -o json 2>/dev/null | python3 -c '
|
|
import sys,json
|
|
r={x["name"]:x for x in json.load(sys.stdin)}.get("web-main") or {}
|
|
sys.exit(0 if r.get("chart")=="nginx-lite-1.1.0" and r.get("status")=="deployed" and int(r.get("revision",0))>=2 else 1)'"""),
|
|
C("cache deployed from redis-lite with 2 available replicas", 2, r"""
|
|
helm -n oslo-helm list -o json 2>/dev/null | python3 -c '
|
|
import sys,json
|
|
r={x["name"]:x for x in json.load(sys.stdin)}.get("cache") or {}
|
|
sys.exit(0 if r.get("chart","").startswith("redis-lite-") and r.get("status")=="deployed" else 1)' || exit 1
|
|
for i in $(seq 1 20); do [ "$(j -n oslo-helm deploy cache '{.status.availableReplicas}')" = 2 ] && exit 0; sleep 3; done; exit 1"""),
|
|
C("the stuck release is removed; nothing pending remains", 2, r"""
|
|
! helm -n oslo-helm status cache-broken >/dev/null 2>&1 && [ -z "$(helm -n oslo-helm list --pending -q 2>/dev/null)" ] && ! $K -n oslo-helm get deploy cache-broken >/dev/null 2>&1"""),
|
|
],
|
|
solution="""
|
|
helm -n oslo-helm list # Helm 4 shows every status; Helm 3 hides pending ones
|
|
helm -n oslo-helm list --pending # works in both: cache-broken pending-install
|
|
helm -n oslo-helm uninstall web-legacy
|
|
helm -n oslo-helm upgrade web-main {WS}/q98/charts/nginx-lite-1.1.0
|
|
helm -n oslo-helm install cache {WS}/q98/charts/redis-lite --set replicaCount=2
|
|
helm -n oslo-helm uninstall cache-broken # also removes the Deployment it half-created
|
|
helm -n oslo-helm list # web-main (1.1.0, rev 2) and cache, both deployed
|
|
# A pending-install release blocks a plain re-install under the same name ("cannot re-use a name
|
|
# that is still in use"). `helm history <rel>` shows when it got stuck; uninstall clears it.
|
|
""",
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 99 ----
|
|
task(
|
|
id=99, domain="STOR", title="Per-replica storage for a StatefulSet",
|
|
task="""
|
|
In Namespace `lima-data`:
|
|
|
|
1. Create a headless Service `ledger-db` (port 5432) and a StatefulSet `ledger-db` with **3**
|
|
replicas: image `busybox:1.36`, command `sh -c 'hostname > /data/id.txt && sleep 86400'`, and a
|
|
`volumeClaimTemplates` entry `data` — 100Mi, ReadWriteOnce, StorageClass `standard` — mounted at
|
|
`/data`.
|
|
2. Once all 3 are Ready, scale the StatefulSet down to **2**.
|
|
3. Write the names of the PersistentVolumeClaims that belong to the StatefulSet and still exist
|
|
after the scale-down to `{WS}/q99/pvcs.txt`, one per line, sorted.
|
|
""",
|
|
setup=NS + r"""
|
|
mkns lima-data
|
|
rm -rf "$WS/q99"; mkdir -p "$WS/q99"
|
|
$K -n lima-data delete sts ledger-db --ignore-not-found --wait=true >/dev/null 2>&1
|
|
$K -n lima-data delete svc ledger-db --ignore-not-found >/dev/null
|
|
$K -n lima-data delete pvc -l app=ledger-db --ignore-not-found >/dev/null 2>&1
|
|
for i in 0 1 2 3 4; do $K -n lima-data delete pvc data-ledger-db-$i --ignore-not-found --wait=false >/dev/null 2>&1; done
|
|
""",
|
|
checks=[
|
|
C("headless Service and a StatefulSet with the right claim template", 2, r"""
|
|
eq "$(j -n lima-data svc ledger-db '{.spec.clusterIP}')" None && eq "$(j -n lima-data sts ledger-db '{.spec.serviceName}')" ledger-db || exit 1
|
|
t='{.spec.volumeClaimTemplates[?(@.metadata.name=="data")]'
|
|
eq "$(j -n lima-data sts ledger-db "$t.spec.resources.requests.storage}")" 100Mi && eq "$(j -n lima-data sts ledger-db "$t.spec.storageClassName}")" standard \
|
|
&& j -n lima-data sts ledger-db "$t.spec.accessModes}" | grep -q ReadWriteOnce"""),
|
|
C("scaled to 2, both Ready", 2, r"""eq "$(j -n lima-data sts ledger-db '{.spec.replicas}')" 2 && eq "$(j -n lima-data sts ledger-db '{.status.readyReplicas}')" 2"""),
|
|
C("each replica wrote its own identity to its own volume", 2, r"""
|
|
for i in 0 1; do eq "$($K -n lima-data exec ledger-db-$i -- cat /data/id.txt 2>/dev/null | tr -d '[:space:]')" "ledger-db-$i" || exit 1; done
|
|
[ "$(j -n lima-data pod ledger-db-0 '{.spec.volumes[?(@.name=="data")].persistentVolumeClaim.claimName}')" = data-ledger-db-0 ]"""),
|
|
C("pvcs.txt lists every claim that survived the scale-down", 2, r"""
|
|
want=$($K -n lima-data get pvc --no-headers -o custom-columns=N:.metadata.name 2>/dev/null | grep '^data-ledger-db-' | sort)
|
|
[ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q99/pvcs.txt" 2>/dev/null)" = "$want" ]"""),
|
|
],
|
|
solution=[hand(r"""vim sts.yaml""",
|
|
r"""The docs page *StatefulSets* has a headless Service and a StatefulSet with `volumeClaimTemplates` together; change them to:
|
|
```
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: ledger-db
|
|
namespace: lima-data
|
|
spec:
|
|
clusterIP: None
|
|
selector:
|
|
app: ledger-db
|
|
ports:
|
|
- port: 5432
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: StatefulSet
|
|
metadata:
|
|
name: ledger-db
|
|
namespace: lima-data
|
|
spec:
|
|
serviceName: ledger-db
|
|
replicas: 3
|
|
selector:
|
|
matchLabels:
|
|
app: ledger-db
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: ledger-db
|
|
spec:
|
|
containers:
|
|
- name: db
|
|
image: busybox:1.36
|
|
command: ["sh", "-c", "hostname > /data/id.txt && sleep 86400"]
|
|
volumeMounts:
|
|
- name: data
|
|
mountPath: /data
|
|
volumeClaimTemplates:
|
|
- metadata:
|
|
name: data
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: standard
|
|
resources:
|
|
requests:
|
|
storage: 100Mi
|
|
```""",
|
|
run=r"""cat > sts.yaml <<'EOF'
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: ledger-db
|
|
namespace: lima-data
|
|
spec:
|
|
clusterIP: None
|
|
selector:
|
|
app: ledger-db
|
|
ports:
|
|
- port: 5432
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: StatefulSet
|
|
metadata:
|
|
name: ledger-db
|
|
namespace: lima-data
|
|
spec:
|
|
serviceName: ledger-db
|
|
replicas: 3
|
|
selector:
|
|
matchLabels:
|
|
app: ledger-db
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: ledger-db
|
|
spec:
|
|
containers:
|
|
- name: db
|
|
image: busybox:1.36
|
|
command: ["sh", "-c", "hostname > /data/id.txt && sleep 86400"]
|
|
volumeMounts:
|
|
- name: data
|
|
mountPath: /data
|
|
volumeClaimTemplates:
|
|
- metadata:
|
|
name: data
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: standard
|
|
resources:
|
|
requests:
|
|
storage: 100Mi
|
|
EOF"""),
|
|
cmd(r"""k apply -f sts.yaml
|
|
k -n lima-data rollout status sts ledger-db
|
|
k -n lima-data scale sts ledger-db --replicas=2"""),
|
|
cmd(r"""k -n lima-data get pvc -o custom-columns=NAME:.metadata.name --no-headers | sort > {WS}/q99/pvcs.txt
|
|
cat {WS}/q99/pvcs.txt""", r"""All three, `data-ledger-db-0` to `-2`: claims are named `<template>-<statefulset>-<ordinal>` and outlive their Pods, so scaling up again gives `ledger-db-2` its old data back.""")],
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 100 ----
|
|
task(
|
|
id=100, domain="NET", title="Namespace AND pod selector, enforced",
|
|
task="""
|
|
Pods labelled `app=db` in Namespace `oslo-db` must accept connections on TCP port 80 **only** from
|
|
Pods labelled `role=backend` that run in Namespaces labelled `env=prod`. Every other source must be
|
|
refused. Do not restrict egress.
|
|
|
|
Create the NetworkPolicy `db-ingress` in `oslo-db`. This cluster's CNI enforces NetworkPolicy, so
|
|
test what you built: the `api` Pods in `oslo-api` must get through to Service `db`, the `debug`
|
|
Pods in `oslo-api` must not, and neither must the `api` Pods in `lima-api`.
|
|
""",
|
|
setup=NS + r"""
|
|
mkns oslo-db; mkns oslo-api; mkns lima-api
|
|
$K label ns oslo-api env=prod --overwrite >/dev/null; $K label ns lima-api env=dev --overwrite >/dev/null
|
|
$K -n oslo-db delete netpol --all >/dev/null 2>&1
|
|
$K -n oslo-db create deploy db --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
|
|
$K -n oslo-db expose deploy db --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
|
|
mk() { # ns name role
|
|
cat <<EOF | $K apply -f - >/dev/null
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata: {name: $2, namespace: $1}
|
|
spec:
|
|
replicas: 1
|
|
selector: {matchLabels: {app: $2}}
|
|
template:
|
|
metadata: {labels: {app: $2, role: $3}}
|
|
spec: {containers: [{name: c, image: busybox:1.36, command: [sleep, "86400"]}]}
|
|
EOF
|
|
}
|
|
mk oslo-api api backend; mk oslo-api debug debug; mk lima-api api backend
|
|
for d in oslo-db/db oslo-api/api oslo-api/debug lima-api/api; do $K -n ${d%%/*} rollout status deploy ${d##*/} --timeout=120s >/dev/null 2>&1; done
|
|
""",
|
|
checks=[
|
|
C("selects app=db, Ingress only", 1, r"""
|
|
n="$K -n oslo-db get netpol db-ingress"
|
|
eq "$($n -o jsonpath='{.spec.podSelector.matchLabels.app}' 2>/dev/null)" db && eq "$($n -o jsonpath='{.spec.policyTypes}')" '["Ingress"]'"""),
|
|
C("one peer holding BOTH namespaceSelector env=prod AND podSelector role=backend, on TCP 80", 3, r"""
|
|
$K -n oslo-db get netpol db-ingress -o json 2>/dev/null | python3 -c '
|
|
import sys,json
|
|
ing=json.load(sys.stdin)["spec"].get("ingress",[])
|
|
ok=len(ing)==1 and len(ing[0].get("from",[]))==1
|
|
if ok:
|
|
f=ing[0]["from"][0]
|
|
ok=f.get("namespaceSelector",{}).get("matchLabels",{}).get("env")=="prod" and f.get("podSelector",{}).get("matchLabels",{}).get("role")=="backend"
|
|
ok=ok and [(str(p.get("port")),p.get("protocol","TCP")) for p in ing[0].get("ports",[])]==[("80","TCP")]
|
|
sys.exit(0 if ok else 1)'"""),
|
|
C("oslo-api/api reaches db", 2, r"""
|
|
$K -n oslo-db get netpol db-ingress >/dev/null 2>&1 || exit 1 # no policy yet: nothing earned
|
|
for i in 1 2 3 4 5; do $K -n oslo-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1 && exit 0; sleep 2; done; exit 1"""),
|
|
C("oslo-api/debug and lima-api/api are refused", 2, r"""
|
|
$K -n oslo-db get netpol db-ingress >/dev/null 2>&1 || exit 1
|
|
! $K -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1 \
|
|
&& ! $K -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db >/dev/null 2>&1"""),
|
|
],
|
|
solution=[cmd(r"""k get ns --show-labels | grep -e oslo-api -e lima-api
|
|
k -n oslo-api get pods --show-labels""", r"""`env=prod` and `env=dev`; `api` is `role=backend`, `debug` is not."""),
|
|
hand(r"""vim netpol.yaml""",
|
|
r"""One `from` entry holding both selectors - both must match:
|
|
```
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: db-ingress
|
|
namespace: oslo-db
|
|
spec:
|
|
podSelector:
|
|
matchLabels:
|
|
app: db
|
|
policyTypes:
|
|
- Ingress
|
|
ingress:
|
|
- from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
env: prod
|
|
podSelector:
|
|
matchLabels:
|
|
role: backend
|
|
ports:
|
|
- protocol: TCP
|
|
port: 80
|
|
```
|
|
The trap is one dash: `- podSelector:` would make it a second entry, and the rule an OR - any Pod in a prod namespace, or any backend Pod in `oslo-db`.""",
|
|
run=r"""cat > netpol.yaml <<'EOF'
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: db-ingress
|
|
namespace: oslo-db
|
|
spec:
|
|
podSelector:
|
|
matchLabels:
|
|
app: db
|
|
policyTypes:
|
|
- Ingress
|
|
ingress:
|
|
- from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
env: prod
|
|
podSelector:
|
|
matchLabels:
|
|
role: backend
|
|
ports:
|
|
- protocol: TCP
|
|
port: 80
|
|
EOF"""),
|
|
cmd(r"""k apply -f netpol.yaml
|
|
k -n oslo-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db
|
|
k -n oslo-api exec deploy/debug -- wget -qO- -T3 http://db.oslo-db
|
|
k -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db""", r"""nginx's page, then `download timed out` twice.""")],
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 101 ----
|
|
task(
|
|
id=101, domain="NET", title="Resolves everything except the cluster",
|
|
task="""
|
|
Deployment `reporter` in Namespace `lima-client` cannot reach Service `ledger` — its log says
|
|
`bad address`. The Service, its endpoints and cluster DNS are all fine: other Pods in the Namespace
|
|
resolve `ledger` without trouble.
|
|
|
|
Fix the `reporter` Deployment (not the Service), without changing its image or command, so that its
|
|
requests succeed. Write the field you changed and its new value, as `<field>=<value>`, to
|
|
`{WS}/q101/fix.txt`.
|
|
""",
|
|
setup=NS + r"""
|
|
mkns lima-client
|
|
rm -rf "$WS/q101"; mkdir -p "$WS/q101"
|
|
$K -n lima-client create deploy ledger --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
|
|
$K -n lima-client expose deploy ledger --port=8080 --target-port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
|
|
$K -n lima-client delete deploy reporter --ignore-not-found --wait=true >/dev/null 2>&1
|
|
$K apply -f - >/dev/null <<'EOF'
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata: {name: reporter, namespace: lima-client}
|
|
spec:
|
|
replicas: 1
|
|
selector: {matchLabels: {app: reporter}}
|
|
template:
|
|
metadata: {labels: {app: reporter}}
|
|
spec:
|
|
dnsPolicy: Default
|
|
containers:
|
|
- name: reporter
|
|
image: busybox:1.36
|
|
command: [sh, -c, 'while true; do if wget -qO- -T2 http://ledger:8080/ >/dev/null 2>&1; then echo "$(date +%T) ledger ok"; else wget -qO- -T2 http://ledger:8080/ 2>&1 | head -1; fi; sleep 3; done']
|
|
EOF
|
|
$K -n lima-client rollout status deploy ledger --timeout=120s >/dev/null 2>&1
|
|
""",
|
|
checks=[
|
|
C("reporter uses the cluster DNS policy again", 3, r"""
|
|
p=$(j -n lima-client deploy reporter '{.spec.template.spec.dnsPolicy}'); [ -z "$p" ] || [ "$p" = ClusterFirst ] || [ "$p" = ClusterFirstWithHostNet ]"""),
|
|
C("reporter's requests now succeed", 3, r"""
|
|
for i in $(seq 1 15); do
|
|
p=$($K -n lima-client get pod -l app=reporter --field-selector status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
|
|
[ -n "$p" ] && $K -n lima-client logs "$p" --tail=3 2>/dev/null | grep -q 'ledger ok' && exit 0; sleep 3
|
|
done; exit 1"""),
|
|
C("fix.txt names dnsPolicy and its new value", 1, r"""grep -Eqi '^ *dnsPolicy *= *ClusterFirst *$' "$WS/q101/fix.txt" 2>/dev/null"""),
|
|
],
|
|
solution=[cmd(r"""k -n lima-client logs deploy/reporter --tail=3
|
|
k -n lima-client get endpointslices
|
|
k -n lima-client get deploy reporter -o yaml | grep dnsPolicy""", r"""`bad address 'ledger:8080'`, though the Service has endpoints. `dnsPolicy: Default` uses the *node's* resolver, not cluster DNS - the default policy is `ClusterFirst`, despite the name."""),
|
|
hand(r"""k -n lima-client edit deploy reporter""",
|
|
r"""In the Pod template's `spec:`, then `:wq`:
|
|
```
|
|
dnsPolicy: ClusterFirst
|
|
```""",
|
|
run=r"""kubectl -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'"""),
|
|
cmd(r"""echo dnsPolicy=ClusterFirst > {WS}/q101/fix.txt
|
|
k -n lima-client rollout status deploy reporter
|
|
k -n lima-client logs deploy/reporter --tail=3""", r"""`ledger ok`.""")],
|
|
)
|
|
|
|
# ------------------------------------------------------------------ 102 ----
|
|
task(
|
|
id=102, domain="STOR", title="A claim that can never bind",
|
|
task="""
|
|
Deployment `uploader` in Namespace `oslo-files` has no running Pod. Find out why and fix it, so its
|
|
Pod runs with the claim mounted at `/files`. Keep the claim's name (`shared-files`), its size and its
|
|
StorageClass. Write the root cause, in one line, to `{WS}/q102/reason.txt`.
|
|
""",
|
|
setup=NS + r"""
|
|
mkns oslo-files
|
|
rm -rf "$WS/q102"; mkdir -p "$WS/q102"
|
|
$K -n oslo-files delete deploy uploader --ignore-not-found --wait=true >/dev/null 2>&1
|
|
$K -n oslo-files delete pvc shared-files --ignore-not-found --wait=true >/dev/null 2>&1
|
|
$K apply -f - >/dev/null <<'EOF'
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata: {name: shared-files, namespace: oslo-files}
|
|
spec: {accessModes: [ReadWriteMany], storageClassName: standard, resources: {requests: {storage: 1Gi}}}
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata: {name: uploader, namespace: oslo-files}
|
|
spec:
|
|
replicas: 1
|
|
selector: {matchLabels: {app: uploader}}
|
|
template:
|
|
metadata: {labels: {app: uploader}}
|
|
spec:
|
|
volumes: [{name: files, persistentVolumeClaim: {claimName: shared-files}}]
|
|
containers: [{name: up, image: busybox:1.36, command: [sleep, "86400"], volumeMounts: [{name: files, mountPath: /files}]}]
|
|
EOF
|
|
""",
|
|
checks=[
|
|
C("shared-files Bound: 1Gi, class standard, an access mode local-path supports", 3, r"""
|
|
eq "$(j -n oslo-files pvc shared-files '{.status.phase}')" Bound && eq "$(j -n oslo-files pvc shared-files '{.spec.resources.requests.storage}')" 1Gi \
|
|
&& eq "$(j -n oslo-files pvc shared-files '{.spec.storageClassName}')" standard \
|
|
&& ! j -n oslo-files pvc shared-files '{.spec.accessModes}' | grep -q ReadWriteMany"""),
|
|
C("uploader available with the claim at /files", 2, r"""
|
|
for i in $(seq 1 20); do [ "$(j -n oslo-files deploy uploader '{.status.availableReplicas}')" = 1 ] && break; sleep 3; done
|
|
eq "$(j -n oslo-files deploy uploader '{.status.availableReplicas}')" 1 \
|
|
&& eq "$(j -n oslo-files deploy uploader '{.spec.template.spec.volumes[?(@.persistentVolumeClaim.claimName=="shared-files")].name}')" files \
|
|
&& j -n oslo-files deploy uploader '{.spec.template.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /files"""),
|
|
C("reason.txt names the unsupported access mode", 2, r"""grep -Eqi 'ReadWriteMany|RWX|access ?mode' "$WS/q102/reason.txt" 2>/dev/null"""),
|
|
],
|
|
solution=[cmd(r"""k -n oslo-files get pod,pvc
|
|
k -n oslo-files describe pvc shared-files | tail -4""", r"""`ProvisioningFailed ... NodePath only supports ReadWriteOnce and ReadWriteOncePod access modes`: the claim asks for `ReadWriteMany`, which local-path cannot give."""),
|
|
cmd(r"""echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
|
|
k -n oslo-files get pvc shared-files -o yaml > pvc.yaml"""),
|
|
hand(r"""vim pvc.yaml""",
|
|
r"""Under `accessModes:`, change `ReadWriteMany` to `ReadWriteOnce` - name, size and class stay.""",
|
|
run=r"""yq -i '.spec.accessModes = ["ReadWriteOnce"]' pvc.yaml"""),
|
|
cmd(r"""k replace --force -f pvc.yaml
|
|
k -n oslo-files rollout restart deploy uploader
|
|
k -n oslo-files get pvc,pod""", r"""Access modes cannot change on a claim: `--force` deletes it and creates it again. The Pending Pod was never scheduled, so it does not hold the claim. Again until it is `Bound` and the Pod `Running`.""")],
|
|
)
|
|
|
|
QUESTIONS = Q
|