kubestronaut-sim/ckad/bank_more.py
vumas3r f3d0e536b6 Exam simulator for CKA, CKAD and CKS with per-task hosts and a remote desktop
kubestronaut-sim runs each exam task on a host of its own - a kind (kubeadm)
cluster in rootless podman, or a QEMU/KVM Ubuntu VM for the CKS tasks that
need their own kernel (Falco, AppArmor) - and puts a browser desktop in front
of them, the way killer.sh and sailor.sh do.

- ksim, sim/: the engine (sessions, drills, study mode, grading, coverage,
  audit), host and VM lifecycle, the desktop
- images/: exam host images (common, CKS variant with its tools, VM guest)
- cka/ (131 tasks), ckad/ (105), cks/ (110): task banks, curricula with
  study notes, reference solutions and scripted solutions

Every domain of every track has at least 17 pool tasks. `./ksim audit` proves
a task three ways: it seeds cleanly on its own host, scores nothing before
anyone has worked on it, and reaches full marks when its scripted solution is
run on the host as `candidate`. All 346 tasks pass (CKA audited 2026-10-13,
CKAD and CKS 2026-10-12).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JZPd3GGWyxTgAkL6a4gAJz
2026-10-13 16:29:33 -06:00

742 lines
37 KiB
Python

"""CKAD tasks 58-64: a second pool task for every curriculum objective that had only one.
curriculum.py turns the bank into a study path, one objective at a time, and an objective with a
single task is studied once and never drilled on anything new. Each task here takes its objective
from another side than the first one does: API deprecations (a warning, not a removal), CRDs (a
custom resource the API server rejects), images (ENTRYPOINT, CMD and USER), Kustomize (one that
does not build), probes (checks that are wrong, not missing), kubectl output (custom columns and
selectors), logs (a selector, a prefix and an init container).
Same format as questions.py; every grader reads the live cluster or the files the task asks for.
"""
from sim.bank import C, NS, ns_pick, name_pick
QUESTIONS = [
# ------------------------------------------------------------------- OBS ----
{
"id": 58, "domain": "OBS", "title": "Removed, and deprecated but still served",
"params": lambda r: {"ns": ns_pick(r, "migrate"), "app": name_pick(r, "catalog"), "pct": r.choice([10, 25, 50])},
"task": """
`{WS}/q58` holds two manifests written for an older cluster.
1. `deployment.yaml` no longer applies. Correct the file - same name, Namespace, replicas, labels
and container - so that it applies to this cluster, and apply it.
2. `flag.yaml` still applies, but the API server answers with a deprecation warning. Write the
warning to `{WS}/q58/warning.txt`, change the file to the version the warning points to, and
apply it.
3. Write the API version this cluster serves for each of the kinds `CronJob`,
`HorizontalPodAutoscaler`, `EndpointSlice` and `PriorityClass` to `{WS}/q58/served.txt`, one
per line as `<Kind> <group/version>`.
""",
"setup": NS + r"""
mkns %{ns}; rm -rf "$WS/q58"; mkdir -p "$WS/q58"
$K -n %{ns} delete deploy %{app} --ignore-not-found >/dev/null 2>&1
$K apply -f - >/dev/null <<'EOF'
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: {name: featureflags.rollout.example.com}
spec:
group: rollout.example.com
scope: Namespaced
names: {plural: featureflags, singular: featureflag, kind: FeatureFlag, shortNames: [ff]}
versions:
- name: v1beta1
served: true
storage: false
deprecated: true
deprecationWarning: "rollout.example.com/v1beta1 FeatureFlag is deprecated and will stop being served; use rollout.example.com/v1 FeatureFlag"
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [enabled]
properties:
enabled: {type: boolean}
percent: {type: integer, minimum: 0, maximum: 100}
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [enabled]
properties:
enabled: {type: boolean}
percent: {type: integer, minimum: 0, maximum: 100}
EOF
$K wait --for=condition=Established crd/featureflags.rollout.example.com --timeout=60s >/dev/null
$K -n %{ns} delete featureflags.rollout.example.com --all --wait=true >/dev/null 2>&1
cat > "$WS/q58/deployment.yaml" <<'EOF'
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: %{app}
namespace: %{ns}
labels: {app: %{app}}
spec:
replicas: 2
template:
metadata:
labels: {app: %{app}, tier: web}
spec:
containers:
- name: web
image: nginx:1.27
ports: [{containerPort: 80}]
EOF
cat > "$WS/q58/flag.yaml" <<'EOF'
apiVersion: rollout.example.com/v1beta1
kind: FeatureFlag
metadata:
name: %{app}-checkout
namespace: %{ns}
spec:
enabled: true
percent: %{pct}
EOF
""",
"checks": [
C("deployment.yaml is apps/v1 and applies; Deployment %{app} has 2 available with its labels", 2, r"""
f="$WS/q58/deployment.yaml"
grep -Eq '^apiVersion:[[:space:]]*apps/v1[[:space:]]*$' "$f" 2>/dev/null || exit 1
$K apply --dry-run=server -f "$f" >/dev/null 2>&1 || exit 1
d="$K -n %{ns} get deploy %{app}"
eq "$($d -o jsonpath='{.spec.replicas}/{.spec.template.metadata.labels.app}/{.spec.template.metadata.labels.tier}/{.spec.template.spec.containers[0].name}/{.spec.template.spec.containers[0].image}' 2>/dev/null)" \
"2/%{app}/web/web/nginx:1.27" || exit 1
for i in $(seq 1 30); do [ "$($d -o jsonpath='{.status.availableReplicas}')" = 2 ] && exit 0; sleep 3; done; exit 1"""),
C("warning.txt holds the deprecation warning", 1, r"""
grep -q 'rollout.example.com/v1beta1 FeatureFlag is deprecated' "$WS/q58/warning.txt" 2>/dev/null"""),
C("flag.yaml uses rollout.example.com/v1, applies without a warning, and FeatureFlag %{app}-checkout exists", 2, r"""
f="$WS/q58/flag.yaml"
grep -Eq '^apiVersion:[[:space:]]*rollout\.example\.com/v1[[:space:]]*$' "$f" 2>/dev/null || exit 1
out=$($K apply --dry-run=server -f "$f" 2>&1) || exit 1
echo "$out" | grep -qi deprecated && exit 1
eq "$(j -n %{ns} featureflags.rollout.example.com %{app}-checkout '{.spec.enabled}/{.spec.percent}')" "true/%{pct}" """),
C("served.txt: the four kinds with the versions this cluster serves", 2, r"""
want=$($K api-resources --no-headers 2>/dev/null | awk '{k=$NF; if (k=="CronJob" || k=="HorizontalPodAutoscaler" || k=="EndpointSlice" || k=="PriorityClass") print k, $(NF-2)}' | sort)
[ "$(echo "$want" | grep -c .)" = 4 ] || exit 1
got=$(tr -s ' \t' ' ' < "$WS/q58/served.txt" 2>/dev/null | sed 's/^ *//; s/ *$//' | tr -d '\r' | grep . | sort)
eq "$got" "$want" """),
],
"solution": """
k apply -f {WS}/q58/deployment.yaml --dry-run=server # no matches for kind "Deployment" in version "extensions/v1beta1"
k api-resources | grep -w Deployment # apps/v1
vi {WS}/q58/deployment.yaml
# apiVersion: apps/v1
# spec:
# selector: {matchLabels: {app: %{app}}} <- apps/v1 requires it; the old API made it up from the template labels
k apply -f {WS}/q58/deployment.yaml
k apply -f {WS}/q58/flag.yaml --dry-run=server 2> {WS}/q58/warning.txt
cat {WS}/q58/warning.txt # Warning: rollout.example.com/v1beta1 FeatureFlag is deprecated ...; use rollout.example.com/v1 FeatureFlag
sed -i 's#rollout.example.com/v1beta1#rollout.example.com/v1#' {WS}/q58/flag.yaml
k apply -f {WS}/q58/flag.yaml # no warning any more
k api-resources | grep -w -E 'CronJob|HorizontalPodAutoscaler|EndpointSlice|PriorityClass' # APIVERSION column
cat > {WS}/q58/served.txt <<EOF
CronJob batch/v1
HorizontalPodAutoscaler autoscaling/v2
EndpointSlice discovery.k8s.io/v1
PriorityClass scheduling.k8s.io/v1
EOF
# Removed = "no matches for kind": nothing happens until the manifest is rewritten. Deprecated =
# a Warning on stderr while the object is still created - the time to migrate, before the version
# is removed. The warning is real API machinery (here a CRD version marked deprecated); built-in
# APIs print the same line, e.g. "batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+".
""",
},
{
"id": 62, "domain": "OBS", "title": "Health checks that check the wrong thing",
"params": lambda r: (lambda pair: {"ns": ns_pick(r, "health"), "dep": name_pick(r, "gateway"),
"port": r.choice([8080, 8081, 9090]), "live": pair[0], "ready": pair[1]})(
r.choice([("healthz", "ready"), ("livez", "readyz"), ("alive", "warm")])),
"task": """
Pods of Deployment `%{dep}` in Namespace `%{ns}` are restarted again and again and never become
Ready. The application is healthy - its health checks are wrong. ConfigMap `%{dep}-conf` shows how
the app listens.
1. Write the reason the kubelet puts on its events when a probe fails to `{WS}/q62/reason.txt`.
2. Correct the probes: liveness must request the app's liveness endpoint and readiness its
readiness endpoint, both on the port the app listens on. Keep them HTTP probes with their
timings.
3. Both replicas must be Ready and stay up without restarts. Do not change the ConfigMap.
""",
"setup": NS + r"""
mkns %{ns}; rm -rf "$WS/q62"; mkdir -p "$WS/q62"
$K -n %{ns} delete deploy %{dep} --ignore-not-found --wait=true >/dev/null 2>&1
$K apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata: {name: %{dep}-conf, namespace: %{ns}}
data:
default.conf: |
server {
listen %{port};
# liveness endpoint
location = /%{live} { return 200 "alive\n"; }
# readiness endpoint
location = /%{ready} { return 200 "ready\n"; }
location / { return 404; }
}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: %{dep}, namespace: %{ns}, labels: {app: %{dep}}}
spec:
replicas: 2
selector: {matchLabels: {app: %{dep}}}
template:
metadata: {labels: {app: %{dep}}}
spec:
containers:
- name: gateway
image: nginx:1.27
ports: [{containerPort: %{port}}]
volumeMounts: [{name: conf, mountPath: /etc/nginx/conf.d}]
livenessProbe:
httpGet: {path: /%{live}, port: 80}
periodSeconds: 5
failureThreshold: 2
readinessProbe:
httpGet: {path: /status, port: %{port}}
periodSeconds: 5
volumes: [{name: conf, configMap: {name: %{dep}-conf}}]
EOF
$K -n %{ns} get cm %{dep}-conf -o jsonpath='{.data.default\.conf}' | md5sum | cut -d' ' -f1 > "$WS/.q62-conf.md5"
""",
"checks": [
C("reason.txt holds Unhealthy", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q62/reason.txt" 2>/dev/null)" Unhealthy"""),
C("liveness: GET /%{live} on port %{port}, every 5 s, 2 failures", 2, r"""
$K -n %{ns} get deploy %{dep} -o json 2>/dev/null | python3 -c '
import sys, json
c = json.load(sys.stdin)["spec"]["template"]["spec"]["containers"][0]
p = c.get("livenessProbe") or {}; h = p.get("httpGet") or {}
named = {x.get("name"): x.get("containerPort") for x in c.get("ports", [])}
port = named.get(h.get("port"), h.get("port"))
sys.exit(0 if (h.get("path"), port, p.get("periodSeconds"), p.get("failureThreshold")) == ("/%{live}", %{port}, 5, 2) else 1)'"""),
C("readiness: GET /%{ready} on port %{port}, every 5 s", 2, r"""
$K -n %{ns} get deploy %{dep} -o json 2>/dev/null | python3 -c '
import sys, json
c = json.load(sys.stdin)["spec"]["template"]["spec"]["containers"][0]
p = c.get("readinessProbe") or {}; h = p.get("httpGet") or {}
named = {x.get("name"): x.get("containerPort") for x in c.get("ports", [])}
port = named.get(h.get("port"), h.get("port"))
sys.exit(0 if (h.get("path"), port, p.get("periodSeconds")) == ("/%{ready}", %{port}, 5) else 1)'"""),
C("2/2 Ready and still not restarted 12 s later; ConfigMap untouched", 2, r"""
d="$K -n %{ns} get deploy %{dep}"; ok=
for i in $(seq 1 20); do [ "$($d -o jsonpath='{.status.readyReplicas}/{.status.updatedReplicas}' 2>/dev/null)" = 2/2 ] && { ok=1; break; }; sleep 3; done
[ -n "$ok" ] || exit 1
sleep 12
$K -n %{ns} get pods -l app=%{dep} -o json | python3 -c '
import sys, json
pods = [p for p in json.load(sys.stdin)["items"] if not p["metadata"].get("deletionTimestamp")]
cs = [s for p in pods for s in p.get("status", {}).get("containerStatuses", [])]
sys.exit(0 if len(pods) == 2 and len(cs) == 2 and all(s["ready"] and s["restartCount"] == 0 for s in cs) else 1)' || exit 1
eq "$($K -n %{ns} get cm %{dep}-conf -o jsonpath='{.data.default\.conf}' | md5sum | cut -d' ' -f1)" "$(cat "$WS/.q62-conf.md5" 2>/dev/null)" """),
],
"solution": """
k -n %{ns} get pods # RESTARTS climbing, 0/1 READY
k -n %{ns} describe pod -l app=%{dep} --show-events | grep -E 'Liveness|Readiness|Unhealthy|Killing'
# Liveness: http-get http://:80/%{live} ... Readiness: http-get http://:%{port}/status ...
# Warning Unhealthy Liveness probe failed: Get "http://10.244.x.y:80/%{live}": ... connect: connection refused
# Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 404
# Normal Killing Container gateway failed liveness probe, will be restarted
# (kubectl 1.36 and later leave the Events out when a selector matches several Pods - hence --show-events)
echo Unhealthy > {WS}/q62/reason.txt
k -n %{ns} get cm %{dep}-conf -o yaml # listen %{port}; /%{live} = liveness, /%{ready} = readiness
k -n %{ns} edit deploy %{dep}
# livenessProbe.httpGet: {path: /%{live}, port: %{port}} (was port 80: connection refused -> restart)
# readinessProbe.httpGet: {path: /%{ready}, port: %{port}} (was /status: 404 -> never Ready)
k -n %{ns} rollout status deploy %{dep}
k -n %{ns} get pods # 1/1 Running, RESTARTS 0
# "connection refused" is a port problem, a status code is a path problem. A failing liveness
# probe restarts the container; a failing readiness probe only keeps the Pod out of the Service.
""",
},
{
"id": 63, "domain": "OBS", "title": "An inventory with kubectl alone",
"params": lambda r: {"ns": ns_pick(r, "stock"), "busy": r.choice(["W1", "W2", "W3"]), "tier": r.choice(["web", "api"])},
"task": """
Namespace `%{ns}` runs a handful of Pods; some of them never started.
1. Write a command into `{WS}/q63/inventory.sh` that prints one line per Pod of `%{ns}` in exactly
three columns, under the headers `POD`, `NODE` and `IMAGE`: the Pod's name, the node it runs on
and the image of its first container.
2. Write the names of the Pods labelled `tier=%{tier}` that are **not** in phase `Running` to
`{WS}/q63/not-running.txt`, one per line.
3. Write the name of the node that runs the most Pods of `%{ns}` to `{WS}/q63/busiest.txt`.
""",
"setup": NS + r"""
mkns %{ns}; rm -rf "$WS/q63"; mkdir -p "$WS/q63"
$K -n %{ns} delete pod --all --wait=true --grace-period=1 >/dev/null 2>&1
busy=$%{busy}; others=$(printf '%s\n' "$W1" "$W2" "$W3" | grep -vx "$busy")
o1=$(echo "$others" | sed -n 1p); o2=$(echo "$others" | sed -n 2p)
pod() { # name tier image nodeSelector-key nodeSelector-value
cat <<EOF
---
apiVersion: v1
kind: Pod
metadata: {name: $1, namespace: %{ns}, labels: {app: stock, tier: $2}}
spec:
terminationGracePeriodSeconds: 1
nodeSelector: {"$4": "$5"}
containers: [{name: main, image: "$3", command: [sh, -c, "sleep 86400"]}]
EOF
}
{
pod stock-web-1 web busybox:1.36 kubernetes.io/hostname "$busy"
pod stock-web-2 web nginx:1.27 kubernetes.io/hostname "$busy"
pod stock-api-1 api redis:7-alpine kubernetes.io/hostname "$busy"
pod stock-web-3 web nginx:1.25 kubernetes.io/hostname "$o1"
pod stock-api-2 api busybox:1.36 kubernetes.io/hostname "$o2"
pod stock-web-4 web busybox:1.36 disktype nvme
pod stock-api-3 api nginx:1.27 disktype nvme
pod stock-api-4 api busybox:1.36 disktype nvme
} | $K apply -f - >/dev/null
$K -n %{ns} wait --for=condition=Ready pod/stock-web-1 pod/stock-web-2 pod/stock-api-1 pod/stock-web-3 pod/stock-api-2 --timeout=150s >/dev/null
""",
"checks": [
C("inventory.sh prints POD, NODE and IMAGE for every Pod of %{ns}", 3, r"""
f="$WS/q63/inventory.sh"; [ -s "$f" ] || exit 1
out=$(cd / && bash "$f" 2>/dev/null) || exit 1
want=$($K -n %{ns} get pods --no-headers -o 'custom-columns=POD:.metadata.name,NODE:.spec.nodeName,IMAGE:.spec.containers[0].image' | awk '{print $1, $2, $3}' | sort)
[ "$(echo "$want" | grep -c .)" -ge 8 ] || exit 1
eq "$(echo "$out" | head -1 | awk '{print $1, $2, $3, NF}')" "POD NODE IMAGE 3" || exit 1
echo "$out" | tail -n +2 | awk 'NF && NF != 3 {bad = 1} END {exit bad}' || exit 1
eq "$(echo "$out" | tail -n +2 | awk 'NF {print $1, $2, $3}' | sort)" "$want" """),
C("not-running.txt: the tier=%{tier} Pods that are not Running", 2, r"""
want=$($K -n %{ns} get pods -l tier=%{tier} --field-selector='status.phase!=Running' -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | sort)
[ -n "$want" ] || exit 1
eq "$(tr -d ' \t\r' < "$WS/q63/not-running.txt" 2>/dev/null | sed 's#^pod/##' | grep . | sort)" "$want" """),
C("busiest.txt names the node with the most Pods", 2, r"""
want=$($K -n %{ns} get pods -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | grep . | sort | uniq -c | sort -rn | awk 'NR==1 {print $2}')
[ -n "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q63/busiest.txt" 2>/dev/null)" "$want" """),
],
"solution": """
echo "kubectl -n %{ns} get pods -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,IMAGE:.spec.containers[0].image" > {WS}/q63/inventory.sh
bash {WS}/q63/inventory.sh # a Pod that was never scheduled shows <none> as its node
k -n %{ns} get pods -l tier=%{tier} --field-selector='status.phase!=Running' -o name | cut -d/ -f2 > {WS}/q63/not-running.txt
k -n %{ns} get pods -o wide --no-headers | awk '{print $7}' | sort | uniq -c | sort -rn # NODE is column 7 of -o wide
echo <the node on the first line> > {WS}/q63/busiest.txt
# -l filters on labels, --field-selector on a few fields of the object itself (status.phase,
# spec.nodeName, metadata.name ...), and both run on the server. custom-columns takes
# HEADER:jsonpath pairs without the {} that -o jsonpath needs.
""",
},
{
"id": 64, "domain": "OBS", "title": "One replica is logging errors",
"params": lambda r: {"ns": ns_pick(r, "trace"), "app": name_pick(r, "router"), "bad": r.choice([0, 1, 2]),
"code": r.choice([502, 503, 504]), "ver": r.choice(["2024.11", "2025.03", "2025.07"])},
"task": """
The three Pods of StatefulSet `%{app}` in Namespace `%{ns}` all log the requests they route, and
one of them logs errors as well.
1. Write the name of the Pod that logs `ERROR` lines to `{WS}/q64/bad-pod.txt`.
2. Save that Pod's `ERROR` lines, and nothing else, to `{WS}/q64/errors.log`.
3. Every Pod ran an init container `migrate` before it started. Write the schema version it
reported to `{WS}/q64/schema.txt`.
4. Write a command into `{WS}/q64/tail.sh` that prints the last **2** log lines of every Pod of
the StatefulSet in one go, each line prefixed with the Pod it comes from.
""",
"setup": NS + r"""
mkns %{ns}; rm -rf "$WS/q64"; mkdir -p "$WS/q64"
$K -n %{ns} delete statefulset %{app} --ignore-not-found --wait=true >/dev/null 2>&1
$K -n %{ns} delete pod -l app=%{app} --wait=true --grace-period=1 >/dev/null 2>&1
$K apply -f - >/dev/null <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: %{app}, namespace: %{ns}}
spec: {clusterIP: None, selector: {app: %{app}}, ports: [{port: 80}]}
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: %{app}, namespace: %{ns}}
spec:
serviceName: %{app}
replicas: 3
podManagementPolicy: Parallel
selector: {matchLabels: {app: %{app}}}
template:
metadata: {labels: {app: %{app}}}
spec:
terminationGracePeriodSeconds: 2
initContainers:
- name: migrate
image: busybox:1.36
command: [sh, -c, "echo 'checking schema'; echo 'schema version %{ver} is current'"]
containers:
- name: router
image: busybox:1.36
command:
- sh
- -c
- |
trap 'exit 0' TERM
i=0
while true; do
i=$((i+1))
echo "INFO request $i routed"
if [ "${HOSTNAME##*-}" = "%{bad}" ] && [ $((i % 3)) = 0 ]; then
echo "ERROR upstream timeout code=%{code} request=$i"
fi
sleep 2 & wait $!
done
EOF
$K -n %{ns} rollout status statefulset %{app} --timeout=150s >/dev/null
""",
"checks": [
C("bad-pod.txt names %{app}-%{bad}", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q64/bad-pod.txt" 2>/dev/null | sed 's#^pod/##')" %{app}-%{bad}"""),
C("errors.log holds its ERROR lines and nothing else", 2, r"""
f="$WS/q64/errors.log"; [ "$(grep -c . "$f" 2>/dev/null)" -ge 1 ] || exit 1
[ -z "$(grep . "$f" | grep -Ev 'ERROR upstream timeout code=%{code} request=[0-9]+[[:space:]]*$')" ]"""),
C("schema.txt holds %{ver}", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q64/schema.txt" 2>/dev/null)" %{ver}"""),
C("tail.sh prints the last 2 lines of all three Pods, prefixed with the Pod", 2, r"""
f="$WS/q64/tail.sh"; [ -s "$f" ] || exit 1
out=$(cd / && bash "$f" 2>/dev/null) || exit 1
for n in 0 1 2; do [ "$(echo "$out" | grep -c "^\[pod/%{app}-$n/router\] ")" = 2 ] || exit 1; done"""),
],
"solution": """
k -n %{ns} logs -l app=%{app} --prefix --tail=-1 | grep ERROR | head -3 # [pod/%{app}-%{bad}/router] ERROR ...
echo %{app}-%{bad} > {WS}/q64/bad-pod.txt
k -n %{ns} logs %{app}-%{bad} | grep ERROR > {WS}/q64/errors.log
k -n %{ns} logs %{app}-0 -c migrate # schema version %{ver} is current
echo %{ver} > {WS}/q64/schema.txt
echo "kubectl -n %{ns} logs -l app=%{app} --prefix --tail=2" > {WS}/q64/tail.sh
bash {WS}/q64/tail.sh
# With a selector, kubectl logs shows only the last 10 lines of each Pod unless --tail says
# otherwise (-1 = everything), and --prefix says which Pod and container a line came from.
# An init container's log is read like any other container's: -c NAME.
""",
},
# ------------------------------------------------------------------- ENV ----
{
"id": 59, "domain": "ENV", "title": "A custom resource the API server rejects",
"params": lambda r: (lambda eng, pair: {
"ns": ns_pick(r, "cache"), "name": name_pick(r, "sessions"), "eng": eng, "engcap": eng.capitalize(),
"mem": r.choice([256, 512, 1024]), "maxr": r.choice([3, 5, 7]), "bad1": pair[0], "bad2": pair[1]})(
r.choice(["redis", "valkey", "memcached"]), r.sample(["catalog", "ratelimit", "search", "tokens"], 2)),
"task": """
An operator for cache clusters runs in this cluster. Its CustomResourceDefinition belongs to the
API group `cache.example.com`.
1. Write the short name kubectl accepts for that kind to `{WS}/q59/shortname.txt`.
2. The operator reports the state of every cache cluster in `status.phase`. Write each cache
cluster in the **whole cluster** whose phase is `Degraded` to `{WS}/q59/degraded.txt`, one per
line as `<namespace>/<name>`.
3. `{WS}/q59/cache.yaml` is meant to create cache cluster `%{name}` in Namespace `%{ns}`, but the
API server rejects it. Correct the manifest and apply it: engine `%{eng}`, `%{mem}` MB of
memory per node, and as many replicas as the CRD allows.
""",
"setup": NS + r"""
rm -rf "$WS/q59"; mkdir -p "$WS/q59"
$K apply -f - >/dev/null <<'EOF'
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: {name: cacheclusters.cache.example.com}
spec:
group: cache.example.com
scope: Namespaced
names: {plural: cacheclusters, singular: cachecluster, kind: CacheCluster, shortNames: [cc]}
versions:
- name: v1
served: true
storage: true
subresources: {status: {}}
additionalPrinterColumns:
- {name: Engine, type: string, jsonPath: .spec.engine}
- {name: Replicas, type: integer, jsonPath: .spec.replicas}
- {name: Phase, type: string, jsonPath: .status.phase}
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [engine, replicas, memoryMb]
properties:
engine: {type: string, enum: [redis, valkey, memcached], description: "Cache engine"}
replicas: {type: integer, minimum: 1, maximum: %{maxr}, description: "Number of cache nodes"}
memoryMb: {type: integer, minimum: 64, maximum: 4096, description: "Memory per node, in MB"}
status:
type: object
properties:
phase: {type: string}
EOF
$K wait --for=condition=Established crd/cacheclusters.cache.example.com --timeout=60s >/dev/null
mkns %{ns}; mkns %{ns}-shared
$K delete cacheclusters.cache.example.com --all -A --wait=true >/dev/null 2>&1
mk() { # namespace name engine replicas memoryMb
phase=Ready; case "$2" in %{bad1}|%{bad2}) phase=Degraded ;; esac
printf 'apiVersion: cache.example.com/v1\nkind: CacheCluster\nmetadata: {name: %s, namespace: %s}\nspec: {engine: %s, replicas: %s, memoryMb: %s}\n' \
"$2" "$1" "$3" "$4" "$5" | $K apply -f - >/dev/null
$K -n "$1" patch cacheclusters.cache.example.com "$2" --subresource=status --type=merge -p "{\"status\":{\"phase\":\"$phase\"}}" >/dev/null
}
mk %{ns} catalog redis 2 256
mk %{ns}-shared ratelimit memcached 1 128
mk %{ns} search valkey 3 512
mk %{ns}-shared tokens redis 1 64
cat > "$WS/q59/cache.yaml" <<'EOF'
apiVersion: cache.example.com/v1alpha1
kind: CacheCluster
metadata:
name: %{name}
namespace: %{ns}
spec:
engine: %{engcap}
replicas: 12
memory: %{mem}
EOF
""",
"checks": [
C("shortname.txt holds cc", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q59/shortname.txt" 2>/dev/null)" cc"""),
C("degraded.txt lists exactly the Degraded cache clusters", 2, r"""
want=$($K get cacheclusters.cache.example.com -A -o jsonpath='{range .items[?(@.status.phase=="Degraded")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}' 2>/dev/null | sort)
[ "$(echo "$want" | grep -c .)" = 2 ] || exit 1
eq "$(tr -d ' \t\r' < "$WS/q59/degraded.txt" 2>/dev/null | grep . | sort)" "$want" """),
C("CacheCluster %{name}: engine %{eng}, memoryMb %{mem}", 2, r"""
eq "$(j -n %{ns} cacheclusters.cache.example.com %{name} '{.spec.engine}/{.spec.memoryMb}')" "%{eng}/%{mem}" """),
C("it runs the most replicas the CRD allows", 1, r"""
max=$(j crd cacheclusters.cache.example.com '{.spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.replicas.maximum}')
[ -n "$max" ] && eq "$(j -n %{ns} cacheclusters.cache.example.com %{name} '{.spec.replicas}')" "$max" """),
C("cache.yaml itself now applies", 1, r"""
f="$WS/q59/cache.yaml"; grep -Eq '^[[:space:]]*name:[[:space:]]*%{name}[[:space:]]*$' "$f" 2>/dev/null \
&& $K apply --dry-run=server -f "$f" >/dev/null 2>&1"""),
],
"solution": """
k api-resources --api-group=cache.example.com # cacheclusters cc cache.example.com/v1 true CacheCluster
echo cc > {WS}/q59/shortname.txt
k get cc -A # the PHASE column is status.phase
k get cc -A --no-headers | awk '$NF == "Degraded" {print $1 "/" $2}' > {WS}/q59/degraded.txt
k apply -f {WS}/q59/cache.yaml --dry-run=server # one complaint after another:
# no matches for kind "CacheCluster" in version "cache.example.com/v1alpha1" -> only v1 is served
# unknown field "spec.memory" -> the field is memoryMb
# spec.engine: Unsupported value: "%{engcap}": supported values: "redis", "valkey", "memcached"
# spec.replicas: Invalid value: 12: spec.replicas in body should be less than or equal to %{maxr}
k explain cc.spec # engine (enum: redis, valkey, memcached), memoryMb, replicas - all required
k get crd cacheclusters.cache.example.com -o yaml | grep -n -A1 'maximum'
vi {WS}/q59/cache.yaml
# apiVersion: cache.example.com/v1
# spec: {engine: %{eng}, replicas: %{maxr}, memoryMb: %{mem}}
k apply -f {WS}/q59/cache.yaml
# A CRD's schema is the contract: required fields, enum values and bounds are all in it, and
# --dry-run=server shows every violation without creating anything. .status belongs to the
# operator - you read it, you do not write it.
""",
},
# ---------------------------------------------------------------- DESIGN ----
{
"id": 60, "domain": "DESIGN", "title": "Entrypoint, default argument and a non-root user",
"params": lambda r: {"img": name_pick(r, r.choice(["reporter", "exporter", "notifier"])), "arg": r.choice(["daily", "weekly", "hourly"]),
"arg2": r.choice(["audit", "billing", "usage"]), "team": r.choice(["payments", "search", "identity"])},
"task": """
`{WS}/q60/image` is the build context of `localhost/%{img}:1.0`, which is already built on this
machine (use `podman`). That image runs as root, and running it with an argument fails.
1. Change the **Dockerfile** so that
- `/app/report.sh` is the image's entrypoint and `%{arg}` its default argument,
- the image runs as the non-root user `10001`,
- the image carries the label `team=%{team}`,
and build it as `localhost/%{img}:1.1`.
2. Run `1.1` once with its defaults and write the output to `{WS}/q60/default.txt`. Run it once
more with the argument `%{arg2}` instead and write that output to `{WS}/q60/%{arg2}.txt`.
3. Remove the old image `localhost/%{img}:1.0` from this machine.
""",
"setup": r"""
command -v podman >/dev/null || { echo "podman is not installed on this machine - the task needs it"; exit 1; }
podman rmi -f localhost/%{img}:1.0 localhost/%{img}:1.1 >/dev/null 2>&1 || true
rm -rf "$WS/q60"; mkdir -p "$WS/q60/image"
cat > "$WS/q60/image/Dockerfile" <<'EOF'
FROM docker.io/library/busybox:1.36
COPY report.sh /app/report.sh
RUN chmod 0755 /app/report.sh
CMD ["/app/report.sh", "all"]
EOF
cat > "$WS/q60/image/report.sh" <<'EOF'
#!/bin/sh
echo "%{img} report: ${1:-none} (uid $(id -u))"
EOF
podman build -q -t localhost/%{img}:1.0 "$WS/q60/image" >/dev/null 2>&1 \
|| { echo "could not build localhost/%{img}:1.0 - is docker.io/library/busybox:1.36 available to podman?"; exit 1; }
""",
"checks": [
C("1.1 runs /app/report.sh with %{arg} by default, and with another argument when given one", 2, r"""
eq "$(podman run --rm localhost/%{img}:1.1 2>/dev/null | cut -d'(' -f1)" "%{img} report: %{arg} " || exit 1
eq "$(podman run --rm localhost/%{img}:1.1 probe 2>/dev/null | cut -d'(' -f1)" "%{img} report: probe " """),
C("1.1 runs as user 10001 and carries the label team=%{team}", 2, r"""
podman run --rm localhost/%{img}:1.1 2>/dev/null | grep -q '(uid 10001)$' || exit 1
eq "$(podman image inspect localhost/%{img}:1.1 --format '{{index .Config.Labels "team"}}' 2>/dev/null)" %{team}"""),
C("the Dockerfile itself sets ENTRYPOINT, USER and LABEL", 1, r"""
f="$WS/q60/image/Dockerfile"
grep -Eiq '^ENTRYPOINT' "$f" 2>/dev/null && grep -Eiq '^USER[[:space:]]+10001' "$f" && grep -Eiq '^LABEL.*team' "$f" """),
C("default.txt and %{arg2}.txt hold the two outputs", 1, r"""
eq "$(tr -d '\r' < "$WS/q60/default.txt" 2>/dev/null)" "%{img} report: %{arg} (uid 10001)" \
&& eq "$(tr -d '\r' < "$WS/q60/%{arg2}.txt" 2>/dev/null)" "%{img} report: %{arg2} (uid 10001)" """),
C("localhost/%{img}:1.0 is gone, 1.1 is there", 1, r"""
podman image exists localhost/%{img}:1.1 && ! podman image exists localhost/%{img}:1.0"""),
],
"solution": """
cd {WS}/q60/image
podman run --rm localhost/%{img}:1.0 %{arg2} # fails: with only a CMD, the argument replaces the whole command
vi Dockerfile
# FROM docker.io/library/busybox:1.36
# LABEL team=%{team}
# COPY report.sh /app/report.sh
# RUN chmod 0755 /app/report.sh
# USER 10001
# ENTRYPOINT ["/app/report.sh"]
# CMD ["%{arg}"]
podman build -t localhost/%{img}:1.1 .
podman run --rm localhost/%{img}:1.1 > {WS}/q60/default.txt # %{img} report: %{arg} (uid 10001)
podman run --rm localhost/%{img}:1.1 %{arg2} > {WS}/q60/%{arg2}.txt
podman rmi localhost/%{img}:1.0
podman inspect localhost/%{img}:1.1 --format '{{.Config.User}} {{.Config.Entrypoint}} {{.Config.Cmd}} {{.Config.Labels}}'
# ENTRYPOINT is the program, CMD its default arguments, and what follows the image name on the
# command line replaces CMD only. Both need the exec form (a JSON array): a shell-form ENTRYPOINT
# ignores CMD and arguments. In a Pod the same pair is `command` (ENTRYPOINT) and `args` (CMD).
# USER goes after the RUN steps that need root.
""",
},
# ---------------------------------------------------------------- DEPLOY ----
{
"id": 61, "domain": "DEPLOY", "title": "A kustomization that does not build",
"params": lambda r: {"ns": ns_pick(r, "shop"), "sfx": r.choice(["eu", "us", "apac"]), "tok": name_pick(r, "tok"), "n": r.choice([2, 3])},
"task": """
`{WS}/q61/app` holds a kustomization for the app `storefront`, but
`kubectl kustomize {WS}/q61/app` fails.
1. Make it build. Two things are wrong. The Deployment and the Service are fine as they are - do
not change `deployment.yaml` or `service.yaml`.
2. Extend `kustomization.yaml` so that the result
- goes into Namespace `%{ns}`,
- has every resource name end in `-%{sfx}`,
- runs image `nginx:1.27`,
- includes a generated Secret `storefront-token` whose key `token` is `%{tok}`, **without** a
hash at the end of its name.
3. Apply it. The Deployment must come up with the %{n} replicas its patch asks for.
""",
"setup": NS + r"""
mkns %{ns}
$K -n %{ns} delete deploy/storefront-%{sfx} svc/storefront-%{sfx} secret/storefront-token-%{sfx} --ignore-not-found >/dev/null 2>&1
rm -rf "$WS/q61"; mkdir -p "$WS/q61/app"
cat > "$WS/q61/app/deployment.yaml" <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: {name: storefront, labels: {app: storefront}}
spec:
replicas: 1
selector: {matchLabels: {app: storefront}}
template:
metadata: {labels: {app: storefront}}
spec:
containers:
- name: web
image: nginx:1.25
ports: [{containerPort: 80}]
env:
- name: API_TOKEN
valueFrom: {secretKeyRef: {name: storefront-token, key: token}}
EOF
cat > "$WS/q61/app/service.yaml" <<'EOF'
apiVersion: v1
kind: Service
metadata: {name: storefront, labels: {app: storefront}}
spec: {selector: {app: storefront}, ports: [{port: 80, targetPort: 80}]}
EOF
cat > "$WS/q61/app/replicas-patch.yaml" <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: store-front
spec:
replicas: %{n}
EOF
cat > "$WS/q61/app/kustomization.yaml" <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yaml
- service.yaml
patches:
- path: replicas-patch.yaml
EOF
(cd "$WS/q61/app" && md5sum deployment.yaml service.yaml) > "$WS/.q61-app.md5"
""",
"checks": [
C("builds: Deployment and Service storefront-%{sfx} in %{ns}; %{n} replicas of nginx:1.27", 3, r"""
$K kustomize "$WS/q61/app" 2>/dev/null | $K create --dry-run=client -o json -f - 2>/dev/null | python3 -c '
import sys, json
s = sys.stdin.read(); dec = json.JSONDecoder(); i = 0; items = []
while i < len(s):
while i < len(s) and s[i].isspace(): i += 1
if i >= len(s): break
o, i = dec.raw_decode(s, i); items.extend(o.get("items", [o]))
dep = [x for x in items if x["kind"] == "Deployment"]; svc = [x for x in items if x["kind"] == "Service"]
ok = len(dep) == 1 and len(svc) == 1 and all(x["metadata"].get("namespace") == "%{ns}" for x in items) \
and dep[0]["metadata"]["name"] == "storefront-%{sfx}" and svc[0]["metadata"]["name"] == "storefront-%{sfx}" \
and dep[0]["spec"]["replicas"] == %{n} and dep[0]["spec"]["template"]["spec"]["containers"][0]["image"] == "nginx:1.27"
sys.exit(0 if ok else 1)'"""),
C("generated Secret storefront-token-%{sfx} (no hash) holds the token, and the Deployment refers to it", 2, r"""
$K kustomize "$WS/q61/app" 2>/dev/null | $K create --dry-run=client -o json -f - 2>/dev/null | python3 -c '
import sys, json, base64
s = sys.stdin.read(); dec = json.JSONDecoder(); i = 0; items = []
while i < len(s):
while i < len(s) and s[i].isspace(): i += 1
if i >= len(s): break
o, i = dec.raw_decode(s, i); items.extend(o.get("items", [o]))
sec = [x for x in items if x["kind"] == "Secret"]; dep = [x for x in items if x["kind"] == "Deployment"]
ok = len(sec) == 1 and len(dep) == 1 and sec[0]["metadata"]["name"] == "storefront-token-%{sfx}" \
and base64.b64decode(sec[0].get("data", {}).get("token", "")).decode() == "%{tok}" \
and dep[0]["spec"]["template"]["spec"]["containers"][0]["env"][0]["valueFrom"]["secretKeyRef"]["name"] == "storefront-token-%{sfx}"
sys.exit(0 if ok else 1)'"""),
C("applied: %{n} Pods available, and they see API_TOKEN=%{tok}", 2, r"""
d="$K -n %{ns} get deploy storefront-%{sfx}"; ok=
for i in $(seq 1 30); do [ "$($d -o jsonpath='{.status.availableReplicas}' 2>/dev/null)" = %{n} ] && { ok=1; break; }; sleep 3; done
[ -n "$ok" ] || exit 1
eq "$($K -n %{ns} exec deploy/storefront-%{sfx} -- printenv API_TOKEN 2>/dev/null | tr -d '\r')" %{tok}"""),
C("deployment.yaml and service.yaml left as they were", 1, r"""
[ -n "$($K kustomize "$WS/q61/app" 2>/dev/null)" ] && (cd "$WS/q61/app" && md5sum -c --quiet "$WS/.q61-app.md5" >/dev/null 2>&1)"""),
],
"solution": """
cd {WS}/q61/app
k kustomize . # ... deploy.yaml: no such file or directory -> the file is deployment.yaml
k kustomize . # no matches for Id Deployment...store-front; failed to find unique target for patch
sed -i 's/name: store-front/name: storefront/' replicas-patch.yaml # a patch finds its target by kind and name
# (or leave the file and name the target in kustomization.yaml: target: {kind: Deployment, name: storefront})
cat > kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
patches:
- path: replicas-patch.yaml
namespace: %{ns}
nameSuffix: -%{sfx}
images:
- {name: nginx, newTag: "1.27"}
secretGenerator:
- name: storefront-token
literals: [token=%{tok}]
generatorOptions:
disableNameSuffixHash: true
EOF
k kustomize . # read it: storefront-%{sfx} x3 kinds, and secretKeyRef.name became storefront-token-%{sfx}
k apply -k .
k -n %{ns} get deploy,svc,secret
# Patches, images and replicas use the names in the files; nameSuffix is applied afterwards, to
# the generated Secret as well, and every reference to a renamed object is rewritten for you.
# Without disableNameSuffixHash the Secret would be storefront-token-%{sfx}-<hash>.
""",
},
]
# written grouped by domain, listed by number
QUESTIONS.sort(key=lambda q: q["id"])