diff --git a/CKA_EXAM_GUIDE.md b/CKA_EXAM_GUIDE.md index c74a6f2..0e2d468 100644 --- a/CKA_EXAM_GUIDE.md +++ b/CKA_EXAM_GUIDE.md @@ -51,7 +51,7 @@ EOF | **Troubleshoot Node (Kubelet)** | `systemctl status kubelet`
`journalctl -u kubelet -e --no-pager` | Check `/etc/kubernetes/kubelet.conf` & `/var/lib/kubelet/config.yaml` | | **Control Plane Troubleshooting** | Inspect `/etc/kubernetes/manifests/` | Check `kube-apiserver.yaml`, `kube-scheduler.yaml`, `kube-controller-manager.yaml` | | **Etcd Backup** | `etcdctl snapshot save ` | `--endpoints=https://127.0.0.1:2379 --cacert=... --cert=... --key=...` | -| **Etcd Restore** | `etcdctl snapshot restore ` | `--data-dir=/var/lib/etcd-backup` (update `etcd.yaml` volume hostPath) | +| **Etcd Restore** | `etcdutl snapshot restore ` | `--data-dir=/var/lib/etcd-backup` (update `etcd.yaml` volume hostPath). `etcdctl snapshot restore` is gone from etcd 3.6+ | | **Upgrade Node (Control Plane / Worker)** | `apt-get update && apt-get install -y --allow-change-held-packages kubeadm=1.31.x-1.1` | `kubeadm upgrade plan` -> `kubeadm upgrade apply` -> upgrade `kubelet`/`kubectl` -> `systemctl restart kubelet` | --- @@ -98,12 +98,16 @@ ETCDCTL_API=3 etcdctl \ ### Restore: ```bash -ETCDCTL_API=3 etcdctl \ - --data-dir=/var/lib/etcd-from-backup \ - snapshot restore /opt/snapshot-pre-boot.db +etcdutl snapshot restore /opt/snapshot-pre-boot.db \ + --data-dir=/var/lib/etcd-from-backup ``` Then edit `/etc/kubernetes/manifests/etcd.yaml` to update the hostPath volume for `etcd-data` to `/var/lib/etcd-from-backup`. +Restore is an offline file operation, so it needs no endpoints or certificates - and it lives in +`etcdutl`, not `etcdctl`: `etcdctl snapshot restore` was deprecated in etcd 3.5 and is gone from +3.6+ (this cluster runs 3.7, where it fails with `unknown flag: --data-dir`). Check first with +`etcdctl version`; on a 3.5 node the old `ETCDCTL_API=3 etcdctl snapshot restore` still works. + --- ## 5. Official Kubernetes Documentation URLs Allowed in CKA diff --git a/CKA_SOLUTIONS_EXPLAINED.md b/CKA_SOLUTIONS_EXPLAINED.md index 7f19fe9..fd6194a 100644 --- a/CKA_SOLUTIONS_EXPLAINED.md +++ b/CKA_SOLUTIONS_EXPLAINED.md @@ -830,7 +830,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor - labelSelector: {matchLabels: {app: %{dep}}} topologyKey: kubernetes.io/hostname tolerations: - - {key: %{key}, operator: {Equal, value: "%{val}", effect: NoSchedule}} + - {key: %{key}, operator: Equal, value: "%{val}", effect: NoSchedule} containers: - name: app image: nginx:1.27 @@ -838,6 +838,8 @@ The exam-day skill this table is really pointing at, regardless of which platfor ``` ```bash k create priorityclass %{dep}-prio --value=50000 + # the seeded quota (pods: 2, requests.cpu: 250m) cannot hold 3 x 100m - raise BOTH, keep the quota + k -n %{ns} patch quota %{ns}-quota -p '{"spec":{"hard":{"pods":"3","requests.cpu":"300m"}}}' ``` - **Gotchas**: with exactly 3 worker nodes and `replicas: 3` under required anti-affinity, there is zero slack — if the quota's `pods` cap is lower than 3, or CPU requests don't fit the quota's `requests.cpu` budget, pods stay `Pending` with no scheduling error beyond an event; check `kubectl -n %{ns} get quota` and size requests accordingly rather than raising replicas blindly. @@ -874,12 +876,12 @@ The exam-day skill this table is really pointing at, regardless of which platfor - name: agent image: busybox:1.36 restartPolicy: Always - command: [sh, -c, "while true; do date >> /var/log/agent.log; sleep 1; done"] - volumeMounts: [{name: logs, mountPath: /var/log}] + command: [sh, -c, "while true; do date >> /var/log/agent/agent.log; sleep 1; done"] + volumeMounts: [{name: logs, mountPath: /var/log/agent}] containers: - name: app image: nginx:1.27 - volumeMounts: [{name: logs, mountPath: /var/log}] + volumeMounts: [{name: logs, mountPath: /var/log/agent}] # NOT /var/log - see Gotchas readinessProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5} livenessProbe: {tcpSocket: {port: 80}} volumes: [{name: logs, emptyDir: {}}] @@ -889,7 +891,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor | tr ' ' '\n' | grep . | sort > {WS}/q109/containers.txt k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' | tr -d '\n' > {WS}/q109/ip.txt ``` -- **Gotchas**: `restartPolicy: Always` goes **inside the initContainer entry itself**, not at the pod level — that per-container field is exactly what flips a normal init container into a native sidecar. The report file must list container names sorted, one per line, no header — matching the check's own `sort` byte-for-byte. +- **Gotchas**: never mount an emptyDir over `/var/log` in an nginx container — it hides nginx's own `/var/log/nginx` and nginx dies with `could not open error log file`, crash-looping forever. A volume mount **replaces** the directory's contents; mount a subdirectory instead. `restartPolicy: Always` goes **inside the initContainer entry itself**, not at the pod level — that per-container field is exactly what flips a normal init container into a native sidecar. The report file must list container names sorted, one per line, no header — matching the check's own `sort` byte-for-byte. --- @@ -1093,7 +1095,8 @@ The exam-day skill this table is really pointing at, regardless of which platfor externalName: %{ext} ``` ```bash - k -n %{ns} exec %{sts}-0 -- nslookup %{sts}-0.%{sts} > {WS}/q43/pod0.txt + # full name: busybox's nslookup applies no search domains, so the short form returns NXDOMAIN + k -n %{ns} exec %{sts}-0 -- nslookup %{sts}-0.%{sts}.%{ns}.svc.cluster.local > {WS}/q43/pod0.txt ``` - **Gotchas**: the headless Service's name **must match** `spec.serviceName` on the StatefulSet — that field is what wires the StatefulSet's per-pod DNS into that specific headless Service; a mismatched name means the stable pod hostnames never resolve, even though both objects individually look fine. `type: ExternalName` Services never get a `ClusterIP` or endpoints at all — that's expected, not a misconfiguration. @@ -1225,7 +1228,8 @@ The exam-day skill this table is really pointing at, regardless of which platfor # forward . 10.255.255.254 -> forward . /etc/resolv.conf k -n kube-system rollout restart deploy coredns - k run dns-check --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default + # FULL names: busybox's nslookup applies no search domains - `kubernetes.default` fails even when DNS works + k run dns-check --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default.svc.cluster.local ``` - **Gotchas**: fixing the Corefile alone looks successful (`kubectl edit` returns cleanly) but does nothing while replicas are still at 0 and/or a NetworkPolicy blocks ingress to the pods — always verify the actual symptom (`nslookup` from a real pod) rather than trusting that each individual fix "should" have worked. `rollout restart` is necessary after editing the ConfigMap because CoreDNS doesn't hot-reload `Corefile` changes on its own. @@ -1392,9 +1396,11 @@ The exam-day skill this table is really pointing at, regardless of which platfor # re-point legacy: swap claimName old-data -> new-data, then recreate the pod k -n %{ns} delete pod legacy --now k -n %{ns} apply -f legacy.yaml + # the Job's Completed pod still references old-data and pins it - delete that pod first + k -n %{ns} delete pod -l job-name=migrate k -n %{ns} delete pvc old-data ``` -- **Gotchas**: `cp -a /src/. /dst/` (trailing `/.`) copies the **contents** of `/src` into `/dst`, preserving permissions/timestamps — dropping the trailing dot instead creates a nested `/dst/src/` directory, which silently fails any check expecting files directly under the new mount. Repointing `legacy`'s claim requires deleting and recreating the pod (`volumes[].persistentVolumeClaim.claimName` isn't mutable in place) — but the task requires the *Deployment/PVC* naming to stay the same, not the pod's UID, so this is safe here. +- **Gotchas**: a pod that references a claim holds it in `pvc-protection` even after it has **Completed** — `delete pvc old-data` hangs in Terminating until the Job's pod is gone (`k describe pvc old-data` names it under "Used By"). Deleting the pod leaves the Job object and its `succeeded: 1` intact. `cp -a /src/. /dst/` (trailing `/.`) copies the **contents** of `/src` into `/dst`, preserving permissions/timestamps — dropping the trailing dot instead creates a nested `/dst/src/` directory, which silently fails any check expecting files directly under the new mount. Repointing `legacy`'s claim requires deleting and recreating the pod (`volumes[].persistentVolumeClaim.claimName` isn't mutable in place) — but the task requires the *Deployment/PVC* naming to stay the same, not the pod's UID, so this is safe here. --- @@ -1484,7 +1490,8 @@ The exam-day skill this table is really pointing at, regardless of which platfor # forward . 10.255.255.1 -> forward . /etc/resolv.conf k -n kube-system rollout restart deploy coredns - k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default + # FULL names: busybox's nslookup applies no search domains - `kubernetes.default` fails even when DNS works + k run test-dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default.svc.cluster.local ``` - **Gotchas**: `forward . /etc/resolv.conf` tells CoreDNS to use the **node's own** resolver config for anything outside the cluster zone — that's the correct, portable default; pointing `forward` at a hardcoded IP address is exactly the kind of change that breaks the moment that IP stops being reachable, which is the fault being simulated here. @@ -1579,7 +1586,9 @@ The exam-day skill this table is really pointing at, regardless of which platfor k -n %{ns} describe deploy %{dep} k -n %{ns} describe rs -l app=%{dep} # "exceeded quota: pods, requested: pods=1" - k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"3"}}}' + # 4, not 3: the rolling update needs a surge pod, and with maxUnavailable 0 (25% of 3 rounds + # down) the controller will not remove an old pod first - pods=3 deadlocks the rollout + k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"4"}}}' k -n %{ns} set image deploy %{dep} web=nginx:1.27 k -n %{ns} patch secret %{dep}-env -p '{"stringData":{"DATABASE_URL":"postgres://db"}}' diff --git a/README.md b/README.md index b791f1c..fd74ad0 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,9 @@ Entry point is `./exam` (renamed from `cka-sim` so it can't collide with the dir Hands-on exam tasks graded **per check** against a real cluster, an exam clock, a browser front end modelled on the real PSI exam, attempt history, and drills that rotate through the -domains before you sit a full exam. Tracks: **CKA** (65-task bank: 43 normal + 22 hard), -CKAD and CKS (scaffolded — same format, add questions when you get there). +domains before you sit a full exam. Tracks: **CKA** (136 tasks: a random pool of 43 normal + +24 hard, plus four fixed simulator sets A–D of 17–18 tasks each), CKAD and CKS (scaffolded — same +format, add questions when you get there). ## How a session is built @@ -65,8 +66,10 @@ The same state is available from the terminal if you prefer: ``` ./exam drill # next least-drilled domain, 5 tasks (add --hard) -./exam drill weak | drill 2 17 20 | drill TSHOOT NET +./exam drill weak | drill 2 17 20 | drill TSHOOT NET | drill setd ./exam start [--hard] # full exam (gated until all domains drilled; --force) +./exam sim a|b|c|d # a fixed 17/18-task simulator set, exam rules, 120 min +./exam study [DOMAIN|weak|N..] # no clock, solutions visible, nothing recorded ./exam list | show N | timer | status # during ./exam end # submit ./exam weak | history # afterwards @@ -82,10 +85,12 @@ exam. Every submit appends to `history-.jsonl` and regenerates `weak-spot latest domain scores, a "drill next" section listing the *failing checks* per task, and a ✓ / n-of-m matrix across all attempts. -A task counts as weak if it failed most recently, has failed twice ever, or was never -attempted. Drills seed only those tasks, give 6 minutes per task, and — unlike exam mode — -let you check a task while you work. A task stops being weak only when you pass it again -under the clock. +A task counts as weak if it failed on its latest attempt, or has failed at least twice and has +not yet been passed twice in a row since, or was never attempted (fixed-set tasks excepted — note +that this makes an early `drill weak` very large; name tasks explicitly for a short drill). Drills +give 6 minutes per task and — unlike exam mode — let you check a task while you work. History +entries marked `"synthetic": true` (scripted verification runs, not a person) are kept in the file +but left out of every statistic. ## Using your own cluster instead of kind @@ -111,7 +116,7 @@ fails loudly if it cannot. `./etcd-tools.sh` does the same by hand. ## CKA bank -| Domain | Normal (43) | Hard (22) | +| Domain | Normal (43) | Hard (24) | |---|---|---| | Architecture | RBAC, etcd snapshot, Kustomize, CRD, static pod, certs/CSR, Helm, aggregated ClusterRole, `kubectl explain` on a CRD, kubeadm join token, component versions | exact RBAC with an over-grant to remove, **etcd restore**, static pod on a relocated path, user → self-contained kubeconfig, Kustomize generators + patches, CRD validation/defaults, renew all control-plane certs | | Workloads | rolling update, sidecar, node affinity + resources, ConfigMap/Secret, metrics-server repair + HPA, PriorityClass, taints, DaemonSet, CronJob, init container, immutable ConfigMap rotation, parallel Job, quota + LimitRange | anti-affinity + toleration + priority under a quota, rollback to a specific revision then pause, native sidecar + probes + exact report, cluster audit to exact files, work in the context's namespace | @@ -120,12 +125,24 @@ fails loudly if it cannot. `./etcd-tools.sh` does the same by hand. | Troubleshooting | node NotReady, broken deployment + service, CrashLoopBackOff, scheduler down, CoreDNS misconfig, controller-manager down, kubelet config drift | worker with three faults (+ uncordon check), two control-plane components down, app down for four reasons fixed in place (UID check), cluster facts exactly | Disruptive tasks (anything that breaks a node or control-plane component) are seeded last and -can block other tasks until fixed — read everything first, as in the real exam. kindnet doesn't -enforce NetworkPolicy and no ingress/gateway controller runs, so those are graded on the objects. +can block other tasks until fixed — read everything first, as in the real exam. Tasks that damage +or depend on the same shared state declare it in `breaks` and never share a session, and the etcd +restore (Q102) is `solo`: restoring rolls the whole cluster back, so it only runs on its own. +Before every session a baseline pass repairs anything a previous, unfinished session left broken. + +The kindnet CNI here **enforces** NetworkPolicy (it embeds a policy engine), so a wrong policy +really drops traffic; no ingress or Gateway controller runs, so those are graded on the objects. + +**Simulator sets** (`./exam sim a|b|c|d`, never drawn into random sessions): A and B are +killer.sh-grade investigate-and-write-to-file sets, C is modelled on an 18-question commercial +mock sitting, and D is killer.sh-style — control-plane inventory, event forensics with crictl, +a broken kubelet unit, kubelet certificates, manual scheduling with the scheduler stopped, Helm +housekeeping with a stuck release, and an enforced AND-vs-OR NetworkPolicy. All are original +scenarios, not copies of any commercial simulator. ## Adding tasks or a new track -Banks live in `tracks/cka/{questions,bank_extra,bank_hard}.py`; a task is: +Banks live in `tracks/cka/{questions,bank_extra,bank_hard,bank_sim_*}.py`; a task is: ```python {"id": 44, "domain": "NET", "difficulty": "normal", # or "hard" @@ -133,7 +150,9 @@ Banks live in `tracks/cka/{questions,bank_extra,bank_hard}.py`; a task is: "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...", "setup": "bash that seeds the scenario (may use %{ns})", "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...], - "disruptive": False} + "disruptive": False, # True: seeded last, after everything else is scheduled + "breaks": ["w1-taint"], # shared state it damages or needs intact; never paired with a task sharing any + "solo": False} # True: only ever drilled on its own ``` Scripts get `$K` (pinned to the grading context), `$WS`, `$CP/$W1/$W2/$W3`, `nx NODE 'cmd'`, diff --git a/exam b/exam index d190774..a9c204e 100755 --- a/exam +++ b/exam @@ -212,8 +212,8 @@ def q_by_id(n: int) -> dict[str, Any]: for q in QUESTIONS: if q["id"] == n: return q - # ids are not contiguous (1-43 base, 101-124 hard, 201-217 / 301-317 sim sets), so print the - # real ranges rather than "1-", which named ids that do not exist. + # ids are not contiguous (1-43 base, 101-124 hard, 201-217 / 301-317 / 401-418 / 501-517 sim + # sets), so print the real ranges rather than "1-", which named ids that do not exist. ids = sorted(q["id"] for q in QUESTIONS) ranges, start, prev = [], ids[0], ids[0] for i in ids[1:] + [None]: @@ -286,11 +286,15 @@ def score(st, results=None): return earned, total, (round(100 * earned / total, 1) if total else 0) # ----------------------------------------------------------- history ---- -def load_history(): +def load_history(include_synthetic=False): + """Attempts from history-.jsonl. Entries marked "synthetic" (a scripted verification + run, not a person sitting the tasks) stay in the file for the record but are left out of every + statistic, weak list and "latest exam" unless explicitly asked for.""" if not os.path.exists(HISTORY_FILE): return [] with open(HISTORY_FILE) as f: - return [json.loads(l) for l in f if l.strip()] + entries = [json.loads(l) for l in f if l.strip()] + return entries if include_synthetic else [e for e in entries if not e.get("synthetic")] def record_attempt(st): earned, total, pct = score(st) @@ -313,6 +317,7 @@ class AggEntry: self.last_fail_at: str = "" self.failed: list[str] = [] self.last_earned: str = "" + self.streak: int = 0 # consecutive full passes at the end of the history def __getitem__(self, item: str) -> Any: return getattr(self, item) @@ -328,6 +333,7 @@ class AggEntry: "last_fail_at": self.last_fail_at, "failed": self.failed, "last_earned": self.last_earned, + "streak": self.streak, } @@ -341,6 +347,7 @@ def stats() -> dict[int, AggEntry]: a.attempts += 1 a.last = r.get("pass") a.last_earned = f"{r.get('earned', 0)}/{r.get('total', 0)}" + a.streak = a.streak + 1 if r.get("pass") else 0 if not r.get("pass"): a.fails += 1 a.last_fail_at = str(e.get("at", "")) @@ -366,27 +373,44 @@ def drilled_domains(): out[q["domain"]] += 1 return out +def breaks_of(q): + """A task's "breaks" as a set. It is a string or a list: Q118 damages the scheduler AND the + controller-manager, and declaring only one let it share a session with Q38/Q317.""" + b = q.get("breaks") + return set() if not b else ({b} if isinstance(b, str) else set(b)) + def no_conflicts(cands, n, taken=()): - """Take up to n tasks, never two that break the same thing (or clash with tasks already taken).""" - used = {q.get("breaks") for q in taken if q.get("breaks")} + """Take up to n tasks, never two that break the same thing (or clash with tasks already taken). + A "solo" task (Q102's etcd restore rolls the WHOLE cluster back to its snapshot, wiping every + other task's seed and answers) is only ever taken on its own.""" + used = set().union(*(breaks_of(q) for q in taken)) if taken else set() out = [] for q in cands: - b = q.get("breaks") - if b and b in used: + if q.get("solo") and (taken or out): + continue + if any(x.get("solo") for x in list(taken) + out): + break + b = breaks_of(q) + if b & used: continue out.append(q) - if b: - used.add(b) + used |= b if len(out) == n: break return out +def split_solo(qs): + """Pull solo tasks out of a multi-task selection; they have to be drilled alone.""" + if len(qs) <= 1: + return list(qs), [] + return [q for q in qs if not q.get("solo")], [q for q in qs if q.get("solo")] + def pick(domain, n, hard, rng, exclude=(), taken=(), unseen_only=False): """n tasks from a domain: unseen first, then least recently seen; hard sessions prefer hard tasks.""" seen = last_seen() def build(allow_hard): - pool = [q for q in POOL if q["domain"] == domain and q["id"] not in exclude + pool = [q for q in POOL if q["domain"] == domain and q["id"] not in exclude and not q.get("solo") and (allow_hard or q["difficulty"] == "normal") and not (unseen_only and q["id"] in seen)] rng.shuffle(pool) pool.sort(key=lambda q: (0 if (hard and q["difficulty"] == "hard") else 1, seen.get(q["id"], -1))) @@ -425,7 +449,7 @@ def campaign_batch(hard, rng): """'drill all': DRILL_SIZE tasks you have never attempted, round-robin across domains, hard tasks included when --hard. When everything has been seen once, falls back to weak tasks.""" seen = last_seen() - pool = [q for q in POOL if q["id"] not in seen and (hard or q["difficulty"] == "normal")] + pool = [q for q in POOL if q["id"] not in seen and not q.get("solo") and (hard or q["difficulty"] == "normal")] if not pool: return None, [] qs, domains = [], list(DOMAINS) @@ -475,23 +499,46 @@ def cmd_progress(args): print(dim(f"\n{len(QUESTIONS) - len(left)}/{len(QUESTIONS)} tasks attempted at least once." + (f" Never seen: Q{', Q'.join(map(str, left))}" if left and len(left) <= 25 else ""))) print(dim("passed = latest attempt was a full pass. ./exam drill all [--hard] works through what's left.")) +def is_weak(q, a): + """Weak = failed on the latest attempt, or failed at least twice and not yet passed twice in a + row since - a flaky task has to prove itself - or never attempted (outside the fixed sets). + The old rule ("failed twice, ever") kept a task weak forever: 13 of 29 listed were passing.""" + if a.attempts == 0: + return not q.get("set") + return a.last is False or (a.fails >= 2 and a.streak < 2) + def weak_ids(agg=None): agg = agg or stats() - return [q["id"] for q in QUESTIONS - if (agg[q["id"]]["last"] is False or agg[q["id"]]["fails"] >= 2) - or (agg[q["id"]]["attempts"] == 0 and not q.get("set"))] + return [q["id"] for q in QUESTIONS if is_weak(q, agg[q["id"]])] + +def weak_lines(a): + """What to show under a weak task: the failing checks if it failed last time, otherwise how + close it is to dropping off the list - not the ✗ list of some older, since-fixed attempt.""" + if a.last is False: + return [f"✗ {n}" for n in (a.failed or [])] + return [f"passed last time ({a.last_earned}) - one more clean pass takes it off this list"] + +def exam_domain_scores(e): + """domain -> (earned, total) over the tasks that were actually IN that attempt. Summing the + weight of every task in the bank instead made a 100% run read "ARCH 38/266".""" + out = {} + for k, r in e.get("results", {}).items(): + q = QBYID.get(int(k)) + if q: + de, dt = out.get(q["domain"], (0, 0)) + out[q["domain"]] = (de + r.get("earned", 0), dt + r.get("total", 0)) + return out def write_weak_spots(): agg = stats(); hist = load_history() L = [f"# {TRACK_ID.upper()} weak spots", "", f"_Generated by exam from {len(hist)} attempt(s); regenerated on every `end`._", ""] exams = [e for e in hist if e["mode"] == "exam"] if exams: - e = exams[-1] + e = exams[-1]; ds = exam_domain_scores(e) L += [f"## Domain scores (latest full exam, {e['at']}, {e['score_pct']}%)", ""] for d, name in DOMAINS.items(): - dq = [q for q in QUESTIONS if q["domain"] == d] - dt = sum(q["weight"] for q in dq); de = sum(e["results"].get(str(q["id"]), {}).get("earned", 0) for q in dq) - L.append(f"- **{d}** {de}/{dt} — {name}") + if d in ds: + L.append(f"- **{d}** {ds[d][0]}/{ds[d][1]} — {name}") L.append("") L += ["## Drill next (`./exam drill weak`)", ""] weak = [q for q in QUESTIONS if q["id"] in weak_ids(agg) and agg[q["id"]]["attempts"] > 0] @@ -499,9 +546,8 @@ def write_weak_spots(): L.append("Nothing failing right now.") for q in weak: a = agg[q["id"]] - L.append(f"### Q{q['id']:02d} · {q['domain']} · {q['title']} ({a['fails']}/{a['attempts']} failed, last {a['last_earned']} pts, {a['last_fail_at']})") - failed_list = list(a["failed"]) if isinstance(a["failed"], (list, tuple)) else [] - L += [f"- ✗ {n}" for n in failed_list] + L.append(f"### Q{q['id']:02d} · {q['domain']} · {q['title']} ({a['fails']}/{a['attempts']} failed, last {a['last_earned']} pts, last failed {a['last_fail_at']})") + L += [f"- {n}" for n in weak_lines(a)] L.append("") L += ["## Attempt matrix", "", "| Q | Domain | Title | " + " | ".join(f"{e['mode'][0].upper()}{i + 1}" for i, e in enumerate(hist)) + " |", "|---|---|---|" + "---|" * len(hist)] @@ -517,13 +563,13 @@ def write_weak_spots(): # ------------------------------------------------------------ session ---- def preflight(): - out = subprocess.run(["kubectl", "get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"], + out = subprocess.run(["kubectl", "--context", CONTEXT, "get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"], capture_output=True, text=True).stdout.split() missing = [v for v in NODES.values() if v not in out] if missing: die(f"Nodes not found in cluster: {missing}\n cluster has: {out}\n" " Set CKA_CP, CKA_W1, CKA_W2, CKA_W3 to your node names (needs 1 control-plane + 3 workers).") - rc, _ = run('nx "$CP" "true" node names must be kind container names (kind get nodes --name cka-sim)\n" @@ -598,17 +644,26 @@ if nx "$CP" 'grep -q "^ *- kube-apiserverr$" /etc/kubernetes/manifests/kube-apis for _ in $(seq 1 30); do kubectl --context "${CKA_CONTEXT:-kind-cka-sim}" get --raw=/readyz >/dev/null 2>&1 && break; sleep 2; done fi -# scheduler: undo the "kube-scheduler" -> "kube-schedulerr" command typo (Q20/Q118) +# scheduler: undo the "kube-scheduler" -> "kube-schedulerr" command typo (Q20/Q118), and put the +# manifest back if a task had the candidate stop the scheduler by moving it out (Q512) and they +# never moved it back - that setup keeps a pristine copy outside the static pod directory. nx "$CP" ' f=/etc/kubernetes/manifests/kube-scheduler.yaml +[ -f $f ] || { [ -f /etc/kubernetes/.cka-sim/kube-scheduler.yaml ] && cp /etc/kubernetes/.cka-sim/kube-scheduler.yaml $f; } sed -i "s|^\( *\)- kube-schedulerr$|\1- kube-scheduler|" $f 2>/dev/null ' /dev/null sed -i "s|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)-broken|\1|" $f 2>/dev/null +sed -i "s|--cluster-signing-cert-file=/etc/kubernetes/pki/ca-signing.crt|--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt|" $f 2>/dev/null +sed -i "s|,-certificatesigningrequest-signing-controller||; s|,-csrsigning||" $f 2>/dev/null ' /dev/null 2>&1 $K -n kube-system rollout restart deploy coredns >/dev/null 2>&1 fi +# ...and Q114's other two faults. kindnet ENFORCES NetworkPolicy, so a leftover lockdown policy on +# the CoreDNS pods kills name resolution for the whole cluster in every later session, and so does +# a Deployment left at 0 replicas. +$K -n kube-system delete netpol maintenance-lockdown --ignore-not-found >/dev/null 2>&1 +r=$($K -n kube-system get deploy coredns -o jsonpath='{.spec.replicas}' 2>/dev/null) +if [ -n "$r" ] && [ "$r" -lt 2 ]; then $K -n kube-system scale deploy coredns --replicas=2 >/dev/null 2>&1; fi # w1/w2/w3 kubelet: undo Q17/Q39-style tampering, then make sure it's enabled and running. # staticPodPath is normalised unconditionally: Q103 relocates it to /etc/kubernetes/static, and # the config.yaml.orig copy below only helps if something made that backup first - nothing does, # so a relocated path used to survive every later session and silently break Q05 (whose check # looks in /etc/kubernetes/manifests). Only restart the kubelet if we actually changed the file. +# clientCAFile gets the same treatment: Q39 points it at ca-bundle.crt and Q117 at ca-wrong.crt, and +# only W1 ever gets a config.yaml.orig (Q39 makes it), so an unfinished Q117 left W2's kubelet +# crash-looping in every later session - enable/start below cannot help a kubelet that exits. +# The systemd drop-in is checked too: Q507 points ExecStart at a kubelet binary that does not exist. for n in "$W1" "$W2" "$W3"; do nx "$n" ' - [ -f /var/lib/kubelet/config.yaml.orig ] && cp -f /var/lib/kubelet/config.yaml.orig /var/lib/kubelet/config.yaml - if grep -q "^staticPodPath:" /var/lib/kubelet/config.yaml 2>/dev/null \ - && ! grep -q "^staticPodPath: /etc/kubernetes/manifests$" /var/lib/kubelet/config.yaml; then - sed -i "s|^staticPodPath: .*|staticPodPath: /etc/kubernetes/manifests|" /var/lib/kubelet/config.yaml - systemctl restart kubelet >/dev/null 2>&1 + c=/var/lib/kubelet/config.yaml; d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf + [ -f $c.orig ] && cp -f $c.orig $c + restart=0 + if grep -q "^staticPodPath:" $c 2>/dev/null && ! grep -q "^staticPodPath: /etc/kubernetes/manifests$" $c; then + sed -i "s|^staticPodPath: .*|staticPodPath: /etc/kubernetes/manifests|" $c; restart=1 fi + ca=$(awk "/clientCAFile:/{print \$2}" $c 2>/dev/null) + if [ -n "$ca" ] && [ ! -f "$ca" ]; then + sed -i "s|clientCAFile: .*|clientCAFile: /etc/kubernetes/pki/ca.crt|" $c; restart=1 + fi + bin=$(sed -n "s|^ExecStart=\([^ ]*kubelet\) .*|\1|p" $d 2>/dev/null | tail -1) + if [ -n "$bin" ] && [ ! -x "$bin" ]; then + sed -i "s|^ExecStart=$bin |ExecStart=/usr/bin/kubelet |" $d; systemctl daemon-reload; restart=1 + fi + [ $restart = 1 ] && systemctl restart kubelet >/dev/null 2>&1 systemctl enable kubelet >/dev/null 2>&1 systemctl start kubelet >/dev/null 2>&1 ' /dev/null 2>&1; then + for sc in $($K get sc -o jsonpath='{.items[*].metadata.name}'); do + [ "$sc" = standard ] && continue + [ "$($K get sc "$sc" -o jsonpath='{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" = true ] \ + && $K annotate sc "$sc" storageclass.kubernetes.io/is-default-class- >/dev/null 2>&1 + done + $K annotate sc standard storageclass.kubernetes.io/is-default-class=true --overwrite >/dev/null 2>&1 +fi + # w1-taint: clear every randomized demo taint key the bank can apply. Q25 draws from # dedicated/team/workload, Q107 from pool/lane, and Q307 pins maintenance=inprogress - miss one # and that node silently stops accepting pods for every later session. @@ -794,7 +885,7 @@ def cmd_drill(args): label, qs = campaign_batch(hard, random.Random()) if not qs: print(bold("Every task in the bank has been attempted." ) + " Switching to weak tasks.") - qs = [q for q in QUESTIONS if q["id"] in weak_ids()] + qs, _ = split_solo([q for q in QUESTIONS if q["id"] in weak_ids()]) qs = no_conflicts(qs, DRILL_SIZE) if not qs: die("Nothing weak either - you have passed everything on its latest attempt. Time for ./exam start --hard.") @@ -816,7 +907,10 @@ def cmd_drill(args): else: ids.append(q_by_id(int(a))["id"]) ids = sorted(set(ids)) - qs = [q for q in QUESTIONS if q["id"] in ids] + qs, solo = split_solo([q for q in QUESTIONS if q["id"] in ids]) + if solo: + print(yellow(f"Dropped Q{', Q'.join(str(q['id']) for q in solo)}: restoring it rolls the whole cluster back and would " + f"wipe the other tasks. Drill it on its own: ./exam drill {solo[0]['id']}")) kept = no_conflicts(qs, len(qs)) if len(kept) < len(qs): dropped = [q["id"] for q in qs if q not in kept] @@ -840,10 +934,10 @@ def cmd_study(args): qs = pick(domain, n, hard, rng) print(bold(f"Study: {DOMAINS[domain]}")) elif rest and rest[0] == "weak": - qs = no_conflicts([q for q in QUESTIONS if q["id"] in weak_ids()], n) + qs = no_conflicts(split_solo([q for q in QUESTIONS if q["id"] in weak_ids()])[0], n) print(bold("Study: tasks you have been failing")) elif rest: - qs = no_conflicts([q_by_id(int(a)) for a in rest], len(rest)) + qs = no_conflicts(split_solo([q_by_id(int(a)) for a in rest])[0], len(rest)) print(bold("Study: " + ", ".join(f"Q{q['id']}" for q in qs))) else: domain, qs = next_drill(hard, rng) @@ -851,7 +945,7 @@ def cmd_study(args): begin_session(qs, "drill", EXAM_SECONDS, force, hard=hard, untimed_session=True) def cmd_sim(args): - """Fixed 17-task simulator set, full exam rules and clock.""" + """A fixed simulator set, full exam rules and clock.""" force = "--force" in args names = [a.lower() for a in args if not a.startswith("--")] if not names or names[0] not in SETS: @@ -895,8 +989,13 @@ def cmd_show(args): def cmd_grade(args): st = require_started() + force = "--force" in args; args = [a for a in args if a != "--force"] if not args: die("usage: ./exam grade N | all") + # The web UI already refused this; the CLI handed out per-check feedback mid-exam, which a sim + # meant to rehearse the real thing (no feedback until you submit) must not do by default. + if st.get("mode") == "exam" and not st.get("ended_at") and not force: + die("No feedback during an exam - the real one gives none. Submit with ./exam end (--force to peek anyway).") ids = [q["id"] for q in session_questions(st)] if args[0] == "all" else [int(a) for a in args] for n in ids: q = q_by_id(n); r = grade_one(q) @@ -980,7 +1079,7 @@ def cmd_end(args): print_score(st) if entry is None: print(dim("Study session - not added to history or weak-spots.")); return - print(dim(f"Time used: {fmt(st['ended_at'] - st['started_at'])}" + (" (over time)" if st.get("over_time") else ""))) + print(dim(f"Time used: {fmt(st['ended_at'] - st['started_at'] - paused_total(st))}" + (" (over time)" if st.get("over_time") else ""))) failed = sorted(int(k) for k, v in entry["results"].items() if not v["pass"]) print(dim(f"Attempt #{len(load_history())} logged to {os.path.basename(HISTORY_FILE)}; {os.path.basename(WEAK_FILE)} updated.")) if failed: @@ -1012,31 +1111,34 @@ def cmd_weak(args): print(green(" none - everything attempted has passed on its latest try")) for i in ids: q = q_by_id(i); a = agg[i] - print(f"\n{red('Q%02d' % i)} {q['domain']:<7} {q['title']} " + dim(f"{a['fails']}/{a['attempts']} failed, last {a['last_earned']} pts, {a['last_fail_at']}")) - for n in a["failed"]: - print(dim(" ✗ " + n)) + print(f"\n{red('Q%02d' % i)} {q['domain']:<7} {q['title']} " + dim(f"{a['fails']}/{a['attempts']} failed, last {a['last_earned']} pts, last failed {a['last_fail_at']}")) + for n in weak_lines(a): + print(dim(" " + n)) never = [q["id"] for q in QUESTIONS if agg[q["id"]]["attempts"] == 0] if never: print(dim(f"\nNever attempted: Q{', Q'.join(map(str, never))}")) print(dim(f"\nFull log: {WEAK_FILE} Drill them: ./exam drill weak")) def cmd_history(args): - hist = load_history() + hist = load_history(include_synthetic=True) if not hist: die("No history yet.") print(bold(f"{'#':>3} {'when':<17}{'mode':<7}{'score':>7} {'time':<9} tasks")) for i, e in enumerate(hist, 1): col = green if e["score_pct"] >= PASS_PCT else red tasks = f"all {len(QUESTIONS)}" if len(e["ids"]) == len(QUESTIONS) else "Q" + ",".join(map(str, e["ids"])) - print(f"{i:>3} {e['at']:<17}{(e['mode'] + ('*' if e.get('hard') else '')):<7}{col('%5.1f%%' % e['score_pct']):>16} {fmt(e['time_used']):<9} {tasks}") - exams = [e for e in hist if e["mode"] == "exam"] + tag = dim(" [scripted verification - not counted]") if e.get("synthetic") else "" + print(f"{i:>3} {e['at']:<17}{(e['mode'] + ('*' if e.get('hard') else '')):<7}{col('%5.1f%%' % e['score_pct']):>16} {fmt(e['time_used']):<9} {tasks}{tag}") + exams = [e for e in hist if e["mode"] == "exam" and not e.get("synthetic")] print(dim("* = hard")) if len(exams) >= 2: - print(bold("\nDomain trend (full exams, % of points):")) + print(bold("\nDomain trend (full exams, % of that exam's points in the domain):")) for d in DOMAINS: - dq = [q for q in QUESTIONS if q["domain"] == d]; dt = sum(q["weight"] for q in dq) - pts = [round(100 * sum(e["results"].get(str(q["id"]), {}).get("earned", 0) for q in dq) / dt) for e in exams] - print(f" {d:<7} " + " → ".join(f"{p:>3}%" for p in pts)) + cells = [] + for e in exams: + de, dt = exam_domain_scores(e).get(d, (0, 0)) + cells.append(f"{round(100 * de / dt):>3}%" if dt else " -") + print(f" {d:<7} " + " → ".join(cells)) def cmd_reset(args): if not args: @@ -1114,12 +1216,12 @@ class Api(http.server.BaseHTTPRequestHandler): elif body.get("campaign"): _, qs = campaign_batch(hard, rng) if not qs: - qs = no_conflicts([q for q in QUESTIONS if q["id"] in weak_ids()], DRILL_SIZE) + qs = no_conflicts(split_solo([q for q in QUESTIONS if q["id"] in weak_ids()])[0], DRILL_SIZE) elif body.get("next"): _, qs = next_drill(hard, rng) else: ids = body.get("ids") or weak_ids() - qs = no_conflicts([q for q in QUESTIONS if q["id"] in ids], len(ids)) + qs = no_conflicts(split_solo([q for q in QUESTIONS if q["id"] in ids])[0], len(ids)) per = DRILL_SECONDS_PER_TASK_HARD if hard else DRILL_SECONDS_PER_TASK begin_session(qs, "drill", per * len(qs), True, log=ulog, hard=hard, untimed_session=bool(body.get("study"))) @@ -1269,9 +1371,9 @@ def cmd_help(args): exact-output tasks, compound failures, collateral checks and exam-style contexts. list tasks in this session, with current + historical results show N display task N - grade N|all grade one task (or all) per check - drills only, in the UI + grade N|all grade one task (or all) per check - drills only (--force to peek in an exam) status / timer time remaining (timer = live countdown) - pause / resume stop the clock (recorded on the attempt - the real exam has no pause) + pause / resume stop the clock (not recorded anywhere - but the real exam has no pause) end submit: grade per check, score, log attempt, update weak-spots weak what you keep getting wrong, with the failing checks history every attempt, scores, and per-domain trend diff --git a/history-cka.jsonl b/history-cka.jsonl index 1abda7a..ade7c4d 100644 --- a/history-cka.jsonl +++ b/history-cka.jsonl @@ -8,4 +8,4 @@ {"at": "2026-09-15T20:55", "mode": "drill", "hard": false, "ids": [13, 15, 16, 19, 25, 28, 29, 30, 33, 37, 39, 40, 42, 106, 112, 115, 120, 121], "score_pct": 41.7, "earned": 40, "total": 96, "time_used": 608, "results": {"13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "19": {"pass": true, "earned": 6, "total": 6, "failed": []}, "33": {"pass": true, "earned": 5, "total": 5, "failed": []}, "30": {"pass": true, "earned": 4, "total": 4, "failed": []}, "37": {"pass": true, "earned": 6, "total": 6, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~6h"]}, "42": {"pass": true, "earned": 4, "total": 4, "failed": []}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries team=finance:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "115": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "120": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "121": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-21 in ac-echo (the context's namespace), 2 available, owner=ac", "Service ledger-21 port 9000 in the same namespace", "ns.txt has the namespace"]}}} {"at": "2026-09-16T19:40", "mode": "drill", "hard": false, "ids": [15, 16, 25, 28, 29, 40], "score_pct": 50.0, "earned": 14, "total": 28, "time_used": 1258, "results": {"15": {"pass": true, "earned": 6, "total": 6, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "29": {"pass": true, "earned": 4, "total": 4, "failed": []}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~24h"]}}} {"at": "2026-09-19T12:27", "mode": "drill", "hard": false, "ids": [106, 217, 34, 202, 102, 301, 39, 13, 32, 204, 25, 2, 8, 112, 4, 109, 313, 9], "score_pct": 39.7, "earned": 48, "total": 121, "time_used": 4197, "results": {"8": {"pass": true, "earned": 4, "total": 4, "failed": []}, "34": {"pass": true, "earned": 4, "total": 4, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "301": {"pass": true, "earned": 9, "total": 9, "failed": []}, "4": {"pass": true, "earned": 4, "total": 4, "failed": []}, "9": {"pass": true, "earned": 4, "total": 4, "failed": []}, "106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "217": {"pass": false, "earned": 0, "total": 10, "failed": ["Pod tiger-probe Running with both labels", "container.txt holds a real container ID for that Pod", "container.txt records the runtimeType", "container.log holds the httpd startup output"]}, "202": {"pass": false, "earned": 3, "total": 8, "failed": ["watchAllNamespaces=true reached the pod env", "Ledger main-ledger has currency USD and retentionDays 30"]}, "102": {"pass": false, "earned": 2, "total": 10, "failed": ["ConfigMap restore-proof-67 exists again with its original data", "etcd uses a restored data directory (manifest changed)"]}, "32": {"pass": false, "earned": 0, "total": 4, "failed": ["ResourceQuota limits pods/6, requests.cpu 1, limits.memory 2Gi", "LimitRange defaults set", "worker pods Running with defaulted requests"]}, "204": {"pass": false, "earned": 0, "total": 8, "failed": ["evicted-first.txt lists exactly the BestEffort pods", "api-qos.txt says Burstable", "guaranteed.txt names orion-ledger"]}, "2": {"pass": false, "earned": 0, "total": 7, "failed": ["/opt/cka/etcd-snapshot.db exists on the control-plane node", "snapshot is a real etcd backup (> 100 KB)"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret secure.lab.local-tls is kubernetes.io/tls with CN secure.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "109": {"pass": false, "earned": 2, "total": 8, "failed": ["native sidecar: initContainer agent with restartPolicy Always", "probes on app as specified", "Running, shared log written by agent, grace period 5"]}, "313": {"pass": false, "earned": 0, "total": 11, "failed": ["Secret classic-tls is a TLS secret with CN classic.r700.io", "both host rules route to the right Services on /", "TLS covers both hosts with classic-tls", "defaultBackend points at web-classic:80, class is nginx"]}}} -{"at": "2026-09-20T17:32", "mode": "exam", "hard": true, "ids": [401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418], "score_pct": 100.0, "earned": 163, "total": 163, "time_used": 2909, "results": {"402": {"pass": true, "earned": 9, "total": 9, "failed": []}, "404": {"pass": true, "earned": 9, "total": 9, "failed": []}, "405": {"pass": true, "earned": 11, "total": 11, "failed": []}, "401": {"pass": true, "earned": 8, "total": 8, "failed": []}, "403": {"pass": true, "earned": 9, "total": 9, "failed": []}, "406": {"pass": true, "earned": 7, "total": 7, "failed": []}, "407": {"pass": true, "earned": 10, "total": 10, "failed": []}, "408": {"pass": true, "earned": 9, "total": 9, "failed": []}, "409": {"pass": true, "earned": 9, "total": 9, "failed": []}, "410": {"pass": true, "earned": 8, "total": 8, "failed": []}, "411": {"pass": true, "earned": 7, "total": 7, "failed": []}, "412": {"pass": true, "earned": 11, "total": 11, "failed": []}, "413": {"pass": true, "earned": 10, "total": 10, "failed": []}, "414": {"pass": true, "earned": 9, "total": 9, "failed": []}, "415": {"pass": true, "earned": 9, "total": 9, "failed": []}, "416": {"pass": true, "earned": 10, "total": 10, "failed": []}, "417": {"pass": true, "earned": 10, "total": 10, "failed": []}, "418": {"pass": true, "earned": 8, "total": 8, "failed": []}}} +{"at": "2026-09-20T17:32", "mode": "exam", "hard": true, "ids": [401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418], "score_pct": 100.0, "earned": 163, "total": 163, "time_used": 2909, "results": {"402": {"pass": true, "earned": 9, "total": 9, "failed": []}, "404": {"pass": true, "earned": 9, "total": 9, "failed": []}, "405": {"pass": true, "earned": 11, "total": 11, "failed": []}, "401": {"pass": true, "earned": 8, "total": 8, "failed": []}, "403": {"pass": true, "earned": 9, "total": 9, "failed": []}, "406": {"pass": true, "earned": 7, "total": 7, "failed": []}, "407": {"pass": true, "earned": 10, "total": 10, "failed": []}, "408": {"pass": true, "earned": 9, "total": 9, "failed": []}, "409": {"pass": true, "earned": 9, "total": 9, "failed": []}, "410": {"pass": true, "earned": 8, "total": 8, "failed": []}, "411": {"pass": true, "earned": 7, "total": 7, "failed": []}, "412": {"pass": true, "earned": 11, "total": 11, "failed": []}, "413": {"pass": true, "earned": 10, "total": 10, "failed": []}, "414": {"pass": true, "earned": 9, "total": 9, "failed": []}, "415": {"pass": true, "earned": 9, "total": 9, "failed": []}, "416": {"pass": true, "earned": 10, "total": 10, "failed": []}, "417": {"pass": true, "earned": 10, "total": 10, "failed": []}, "418": {"pass": true, "earned": 8, "total": 8, "failed": []}}, "synthetic": true, "note": "Scripted end-to-end verification of Set C (tasks solved programmatically), not a person sitting it"} diff --git a/setup.sh b/setup.sh index db6c4c4..263d863 100755 --- a/setup.sh +++ b/setup.sh @@ -98,4 +98,4 @@ echo echo "Cluster ready. Nodes:" kubectl get nodes -o wide echo -echo "Next: ./exam start (seeds all 20 tasks and starts the 120-minute clock)" +echo "Next: ./exam web (browser UI) or ./exam drill (5 tasks from your least-drilled domain)" diff --git a/tracks/cka/bank_extra.py b/tracks/cka/bank_extra.py index 890716a..5278a5c 100644 --- a/tracks/cka/bank_extra.py +++ b/tracks/cka/bank_extra.py @@ -169,8 +169,9 @@ for t in $($K -n kube-system get secrets -o jsonpath='{.items[?(@.type=="bootstr C("the token in the file exists in the cluster with TTL ~%{ttl}", 3, r""" tok=$(grep -o -- '--token [a-z0-9]\{6\}\.[a-z0-9]\{16\}' "$WS/q40/join.sh" 2>/dev/null | awk '{print $2}'); [ -n "$tok" ] || exit 1 id=${tok%%.*}; exp=$(j -n kube-system secret bootstrap-token-$id '{.data.expiration}' | base64 -d 2>/dev/null); [ -n "$exp" ] || exit 1 -secs=$(( $(date -d "$exp" +%s) - $(date +%s) )); want=$(( $(echo %{ttl} | tr -d h) * 3600 )) -[ $secs -gt $((want - 900)) ] && [ $secs -le $((want + 60)) ]"""), +ct=$(j -n kube-system secret bootstrap-token-$id '{.metadata.creationTimestamp}'); [ -n "$ct" ] || exit 1 +secs=$(( $(date -d "$exp" +%s) - $(date -d "$ct" +%s) )); want=$(( $(echo %{ttl} | tr -d h) * 3600 )) +[ $secs -ge $((want - 120)) ] && [ $secs -le $((want + 120)) ] && [ "$(date -d "$exp" +%s)" -gt "$(date +%s)" ]"""), ], }, { @@ -197,9 +198,9 @@ kub=$(j node %{node} '{.status.nodeInfo.kubeletVersion}') grep -qx "apiserver=$api" "$f" && grep -qx "kubelet=$kub" "$f" """), C("etcd and runtime versions correct", 2, r""" f="$WS/q41/versions.txt" -etcd=$(j -n kube-system pod -l component=etcd '{.items[0].spec.containers[0].image}' | sed 's/.*://') +etcd=$(j -n kube-system pod -l component=etcd '{.items[0].spec.containers[0].image}' | sed 's/.*://'); ev=${etcd%-*} rt=$(j node %{node} '{.status.nodeInfo.containerRuntimeVersion}') -grep -qx "etcd=$etcd" "$f" && grep -qx "runtime=$rt" "$f" """), +grep -qxE "etcd=v?($etcd|$ev)" "$f" && grep -qx "runtime=$rt" "$f" """), ], }, # ---------------------------------------------------------------- WORK ---- @@ -244,11 +245,17 @@ c["args"]=[a for a in c["args"] if a!="--kubelet-insecure-tls"]; print(json.dump "setup": NS + r""" mkns %{ns} $K delete priorityclass %{pc} --ignore-not-found >/dev/null +$K -n %{ns} delete deploy payments --ignore-not-found --wait=true >/dev/null 2>&1 $K -n %{ns} create deploy payments --image=nginx:1.25 --replicas=2 --dry-run=client -o yaml | $K apply -f - >/dev/null """, "checks": [ C("PriorityClass %{pc} value %{val}, not default", 2, r"""eq "$(j priorityclass %{pc} '{.value}')" %{val} && [ "$(j priorityclass %{pc} '{.globalDefault}')" != true ]"""), - C("payments pods running with priority %{val}", 2, r"""eq "$(j -n %{ns} deploy payments '{.status.availableReplicas}')" 2 && for p in $(j -n %{ns} pod -l app=payments '{.items[*].spec.priority}'); do eq "$p" %{val} || exit 1; done"""), + C("payments pods running with priority %{val}", 2, r""" +for t in $(seq 1 15); do # the old ReplicaSet's pods linger for a moment while they terminate + ok=1; eq "$(j -n %{ns} deploy payments '{.status.availableReplicas}')" 2 || ok=0 + for p in $(j -n %{ns} pod -l app=payments '{.items[*].spec.priority}'); do eq "$p" %{val} || ok=0; done + [ $ok = 1 ] && exit 0; sleep 2 +done; exit 1"""), ], }, { @@ -275,6 +282,8 @@ $K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null }, { "id": 26, "domain": "WORK", "title": "DaemonSet on every node", + # needs every node schedulable: Q25/Q107 taint W1 and must keep the taint, Q307 blocks it + "breaks": ["w1-taint", "w1-block"], "params": lambda r: {"ns": ns_pick(r, "agents"), "ds": name_pick(r, "node-agent")}, "task": """ Create a DaemonSet `%{ds}` in namespace `%{ns}` (image `busybox:1.36`, command `sleep 3600`) @@ -284,7 +293,9 @@ Requests: cpu 10m, memory 16Mi. "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete ds %{ds} --ignore-not-found >/dev/null""", "checks": [ C("DaemonSet exists with requests set", 1, r"""eq "$(j -n %{ns} ds %{ds} '{.spec.template.spec.containers[0].resources.requests.cpu}')" 10m"""), - C("tolerates the control-plane taint", 1, r"""j -n %{ns} ds %{ds} '{.spec.template.spec.tolerations[*].key}' | grep -q control-plane"""), + C("tolerates the control-plane taint", 1, r"""$K -n %{ns} get ds %{ds} -o json 2>/dev/null | python3 -c ' +import sys,json; t=json.load(sys.stdin)["spec"]["template"]["spec"].get("tolerations") or [] +sys.exit(0 if any("control-plane" in (x.get("key") or "") or (x.get("operator")=="Exists" and not x.get("key")) for x in t) else 1)'"""), C("one ready pod per node (including control-plane)", 2, r"""n=$($K get nodes --no-headers 2>/dev/null | wc -l); eq "$(j -n %{ns} ds %{ds} '{.status.numberReady}')" "$n" && eq "$(j -n %{ns} ds %{ds} '{.status.desiredNumberScheduled}')" "$n" """), ], }, @@ -418,7 +429,8 @@ $K -n %{ns} create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o y $K -n %{ns} create deploy api --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null $K -n %{ns} expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null $K -n %{ns} expose deploy api --port=8080 --target-port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null -$K delete gatewayclass %{gc} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete gateway %{gw} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete httproute %{host}-route --ignore-not-found >/dev/null 2>&1 +# these fail harmlessly when the Gateway API CRDs are not installed yet - installing them is step 1 +$K delete gatewayclass %{gc} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete gateway %{gw} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete httproute %{host}-route --ignore-not-found >/dev/null 2>&1 || true """, "checks": [ C("Gateway API CRDs installed (gateways, httproutes)", 1, r"""$K get crd gateways.gateway.networking.k8s.io httproutes.gateway.networking.k8s.io >/dev/null 2>&1"""), @@ -430,7 +442,7 @@ eq "$($h -o jsonpath='{.spec.parentRefs[0].name}')" %{gw} || exit 1 $h -o json | python3 -c ' import sys,json; r=json.load(sys.stdin)["spec"]["rules"] api=[x for x in r if any(m.get("path",{}).get("value")=="/api" for m in x.get("matches",[]))] -web=[x for x in r if x is not api and x.get("backendRefs",[{}])[0].get("name")=="web"] +web=[x for x in r if x not in api and x.get("backendRefs",[{}])[0].get("name")=="web"] ok=api and api[0]["backendRefs"][0]["name"]=="api" and api[0]["backendRefs"][0]["port"]==8080 and web and web[0]["backendRefs"][0]["port"]==80 sys.exit(0 if ok else 1)'"""), ], @@ -492,8 +504,10 @@ sys.exit(0 if ok else 1)'"""), 1. In `%{ns}`, StatefulSet `%{sts}` (2 replicas) exists but has no Service. Create the **headless** Service `%{sts}` it needs (port 5432) so that `%{sts}-0.%{sts}.%{ns}.svc.cluster.local` resolves. 2. Create Service `legacy` of type ExternalName pointing at `%{ext}`. -3. From a pod in the namespace, resolve `%{sts}-0.%{sts}` and write the full nslookup output - to `{WS}/q43/pod0.txt`. +3. From a pod in the namespace, resolve pod 0's stable name + `%{sts}-0.%{sts}.%{ns}.svc.cluster.local` and write the full nslookup output to + `{WS}/q43/pod0.txt`. (busybox's nslookup applies no search domains, so a short name such as + `%{sts}-0.%{sts}` fails even when DNS is healthy.) """, "setup": NS + r""" mkns %{ns}; rm -rf "$WS/q43"; mkdir -p "$WS/q43" @@ -550,11 +564,18 @@ spec: {accessModes: [ReadWriteOnce], storageClassName: manual, resources: {reque EOF for i in $(seq 1 20); do [ "$(j -n %{ns} pvc old-claim '{.status.phase}')" = Bound ] && break; sleep 1; done $K -n %{ns} delete pvc old-claim >/dev/null +# every PV that exists now; the "no extra PV" check compares against this, so Retain leftovers from +# earlier sessions (this task's own, Q15's pv-data, Set C's) can no longer fail it +$K get pv -o jsonpath='{.items[*].metadata.name}' > "$WS/.q35-pvs" """, "checks": [ C("PV %{pv} still exists and was not recreated", 2, r"""[ -n "$(j pv %{pv} '{.metadata.uid}')" ] && eq "$(j pv %{pv} '{.spec.persistentVolumeReclaimPolicy}')" Retain"""), C("PVC %{pvc} Bound to %{pv}", 3, r"""eq "$(j -n %{ns} pvc %{pvc} '{.status.phase}')" Bound && eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}')" %{pv}"""), - C("no extra PVs were created", 1, r"""[ "$($K get pv --no-headers 2>/dev/null | grep -c manual)" -le 2 ]"""), + C("no extra PVs were created", 1, r""" +before=" $(cat "$WS/.q35-pvs" 2>/dev/null) "; [ "$before" != " " ] || exit 1 +for pv in $($K get pv -o jsonpath='{range .items[?(@.spec.storageClassName=="manual")]}{.metadata.name}{" "}{end}' 2>/dev/null); do + case "$before" in *" $pv "*) ;; *) exit 1 ;; esac +done"""), ], }, { diff --git a/tracks/cka/bank_hard.py b/tracks/cka/bank_hard.py index 8e6b5ad..d5a39a0 100644 --- a/tracks/cka/bank_hard.py +++ b/tracks/cka/bank_hard.py @@ -32,14 +32,18 @@ $K delete clusterrole %{sa}-nodes --ignore-not-found >/dev/null; $K delete clust $K create clusterrolebinding %{sa}-legacy-admin --clusterrole=cluster-admin --serviceaccount=%{ns}:%{sa} >/dev/null """, "checks": [ - C("can get/list/watch/patch deployments and replicasets in %{ns}", 2, r"""for res in deployments.apps replicasets.apps; do for v in get list watch patch; do eq "$($K auth can-i $v $res -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes || exit 1; done; done"""), - C("can create pods/exec in %{ns}; can list nodes", 2, r"""eq "$($K auth can-i create pods/exec -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes && eq "$($K auth can-i list nodes --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes"""), + C("can get/list/watch/patch deployments and replicasets in %{ns}", 2, r"""! $K get clusterrolebinding %{sa}-legacy-admin >/dev/null 2>&1 || exit 1 # not via cluster-admin +for res in deployments.apps replicasets.apps; do for v in get list watch patch; do eq "$($K auth can-i $v $res -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes || exit 1; done; done"""), + C("can create pods/exec in %{ns}; can list nodes", 2, r"""! $K get clusterrolebinding %{sa}-legacy-admin >/dev/null 2>&1 || exit 1 +eq "$($K auth can-i create pods --subresource=exec -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes && eq "$($K auth can-i list nodes --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes"""), C("no over-grant: cannot delete deployments, cannot get secrets, cannot get nodes", 3, r"""for t in "delete deployments -n %{ns}" "get secrets -n %{ns}" "get nodes" "create pods -n default"; do eq "$($K auth can-i $t --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null || true)" no || exit 1; done"""), - C("the cluster-admin binding is gone and named objects exist", 2, r"""! $K get clusterrolebinding %{sa}-legacy-admin >/dev/null 2>&1 && $K -n %{ns} get role %{sa}-role rolebinding %{sa}-rb >/dev/null 2>&1 && $K get clusterrole %{sa}-nodes clusterrolebinding %{sa}-crb >/dev/null 2>&1"""), + C("the cluster-admin binding is gone and named objects exist", 2, r"""! $K get clusterrolebinding %{sa}-legacy-admin >/dev/null 2>&1 && $K -n %{ns} get role/%{sa}-role rolebinding/%{sa}-rb >/dev/null 2>&1 && $K get clusterrole/%{sa}-nodes clusterrolebinding/%{sa}-crb >/dev/null 2>&1"""), ], }, { "id": 102, "domain": "ARCH", "breaks": "etcd", "difficulty": "hard", "title": "etcd restore", + # solo: restoring rolls EVERY API object back to the snapshot - other tasks' seeds and answers too + "solo": True, "params": lambda r: {"ctx": "k8s-c1-H", "cm": name_pick(r, "restore-proof")}, "task": CTX + """ An etcd snapshot taken earlier is at `/opt/cka/backup.db` on the control-plane node. @@ -49,6 +53,9 @@ been deleted. **Restore the cluster's etcd from that snapshot** so the ConfigMap exists again. The cluster must be healthy afterwards (API server up, all kube-system pods Running). Do not recreate the ConfigMap by hand — its data must come from the snapshot. + +A restore rolls every API object back to the moment of the snapshot — on a real exam cluster that +would include the work you did on other tasks, so a restore is the last thing you do. """, "setup": r""" nx "$CP" 'mkdir -p /opt/cka; rm -f /opt/cka/backup.db' @@ -58,15 +65,24 @@ $K -n default create cm %{cm} --from-literal=proof=$(date +%s)-%{cm} >/dev/null out=$(nx "$CP" 'etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/cka/backup.db' 2>&1) \ || { echo "snapshot save on $CP failed: $out"; exit 1; } sleep 2; $K -n default delete cm %{cm} >/dev/null +nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' "$WS/.q102-datadir" +date +%s > "$WS/.q102-t0" """, "checks": [ C("ConfigMap %{cm} exists again with its original data", 6, r"""j -n default cm %{cm} '{.data.proof}' | grep -q -- '-%{cm}$'"""), - C("etcd uses a restored data directory (manifest changed)", 2, r"""! nx "$CP" 'grep -q "path: /var/lib/etcd$" /etc/kubernetes/manifests/etcd.yaml' """), - C("control plane healthy: etcd, apiserver, scheduler, controller-manager Running", 2, r"""for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done"""), + C("etcd runs on a data directory restored from the snapshot", 2, r""" +was=$(cat "$WS/.q102-datadir" 2>/dev/null); t0=$(cat "$WS/.q102-t0" 2>/dev/null); [ -n "$was" ] && [ -n "$t0" ] || exit 1 +now=$(nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' /dev/null" /dev/null | grep -q 'O *= *developers'"""), C("kubeconfig is self-contained and works without flags", 3, r"""f="$WS/q104/%{user}.kubeconfig"; [ -f "$f" ] || exit 1; grep -q 'client-certificate-data' "$f" && grep -q 'certificate-authority-data' "$f" && ! grep -q 'client-certificate:' "$f" && kubectl --kubeconfig "$f" get pods >/dev/null 2>&1"""), C("context %{user}@lab defaults to %{ns}", 1, r"""eq "$(kubectl --kubeconfig "$WS/q104/%{user}.kubeconfig" config view -o jsonpath='{.contexts[?(@.name=="%{user}@lab")].context.namespace}')" %{ns}"""), - C("edit rights in %{ns}, none elsewhere", 2, r"""eq "$($K auth can-i create deployments -n %{ns} --as=%{user} 2>/dev/null)" yes && eq "$($K auth can-i delete pods -n %{ns} --as=%{user} 2>/dev/null)" yes && eq "$($K auth can-i get pods -n default --as=%{user} 2>/dev/null || true)" no && eq "$($K auth can-i get secrets -n kube-system --as=%{user} 2>/dev/null || true)" no"""), + C("edit rights in %{ns}, none elsewhere", 2, r"""eq "$($K auth can-i create deployments -n %{ns} --as=%{user} --as-group=developers 2>/dev/null)" yes && eq "$($K auth can-i delete pods -n %{ns} --as=%{user} --as-group=developers 2>/dev/null)" yes && eq "$($K auth can-i get pods -n default --as=%{user} --as-group=developers 2>/dev/null || true)" no && eq "$($K auth can-i get secrets -n kube-system --as=%{user} --as-group=developers 2>/dev/null || true)" no"""), ], }, { @@ -159,18 +175,22 @@ $K delete ns %{ns} --ignore-not-found --wait=false >/dev/null "id": 106, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults", "params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])}, "task": CTX + """ -Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural lowercase+s, shortName `%{kind}`-initials) with schema: +Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the kind in lowercase plus +`s`, shortName = the first two letters of the kind in lowercase) with schema: - `spec.replicas` integer, **required**, minimum 1, maximum 10, - `spec.environment` string, enum `prod|stage|dev`, **default** `dev`, - `spec.image` string, required. Then: 1. Create `%{kind}` `good` in `default` with replicas 3, image `nginx:1.27`, environment `%{env}`. 2. Save the API server's rejection message for a `%{kind}` with `replicas: 0` to `{WS}/q106/rejected.txt`. -3. Create `%{kind}` `defaulted` with only replicas 2 and image set; its environment must be `dev` after creation. +3. Create `%{kind}` `defaulted` in `default` with only replicas 2 and image set; its environment + must be `dev` after creation. + +This context's default namespace is not necessarily `default` — say `-n default` explicitly. """, "setup": r"""rm -rf "$WS/q106"; mkdir -p "$WS/q106"; k=$(echo %{kind} | tr A-Z a-z); $K delete crd ${k}s.deploy.example.com --ignore-not-found >/dev/null""", "checks": [ - C("CRD exists, Namespaced, replicas has min/max and is required", 2, r"""k=$(echo %{kind} | tr A-Z a-z); c="$K get crd ${k}s.deploy.example.com"; eq "$($c -o jsonpath='{.spec.scope}' 2>/dev/null)" Namespaced && s=$($c -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec}'); echo "$s" | grep -q '"minimum":1' && echo "$s" | grep -q '"maximum":10' && echo "$s" | grep -q '"required":\[' """), + C("CRD exists, Namespaced, replicas has min/max and is required", 2, r"""s=""; k=$(echo %{kind} | tr A-Z a-z); c="$K get crd ${k}s.deploy.example.com"; eq "$($c -o jsonpath='{.spec.scope}' 2>/dev/null)" Namespaced && s=$($c -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec}'); echo "$s" | grep -q '"minimum":1' && echo "$s" | grep -q '"maximum":10' && echo "$s" | grep -q '"required":\[' """), C("good created with environment %{env}", 1, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com good '{.spec.environment}')" %{env}"""), C("replicas 0 is rejected and the message was saved", 2, r"""grep -qiE 'minimum|greater than|>= 1|invalid' "$WS/q106/rejected.txt" 2>/dev/null"""), C("defaulting works: defaulted.spec.environment == dev", 2, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com defaulted '{.spec.environment}')" dev"""), @@ -191,7 +211,9 @@ Write the new `apiserver` certificate's expiry (as printed by `kubeadm certs che C("apiserver certificate was renewed (newer notBefore)", 3, r"""b=$(sed 's/notBefore=//' "$WS/.q122-before"); a=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' | sed 's/notBefore=//'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""), C("all components restarted and Running with the new certs (apiserver serving cert matches file)", 3, r""" for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done -srv=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial'); f=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial'); [ -n "$srv" ] && eq "$srv" "$f" """), +srv=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial -startdate'); f=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial -startdate'); [ -n "$srv" ] && eq "$srv" "$f" || exit 1 +# ...and it is the RENEWED certificate: an untouched cluster also serves the file it has on disk +b=$(sed 's/notBefore=//' "$WS/.q122-before"); a=$(echo "$srv" | sed -n 's/notBefore=//p'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""), C("admin.conf uses the renewed client cert", 2, r"""nx "$CP" 'grep client-certificate-data /etc/kubernetes/admin.conf | awk "{print \$2}" | base64 -d | openssl x509 -noout -startdate' | sed 's/notBefore=//' > /tmp/q122a; b=$(sed 's/notBefore=//' "$WS/.q122-before"); [ "$(date -d "$(cat /tmp/q122a)" +%s)" -gt "$(date -d "$b" +%s)" ]"""), C("expiry.txt matches kubeadm's report for apiserver", 1, r"""exp=$(nx "$CP" 'kubeadm certs check-expiration 2>/dev/null' | awk '$1=="apiserver"{print $2,$3,$4,$5,$6}'); [ -n "$exp" ] && grep -qF "$exp" "$WS/q122/expiry.txt" 2>/dev/null"""), ], @@ -291,9 +313,15 @@ $K -n %{ns} set image deploy %{dep} nginx=nginx:1.27 >/dev/null; $K -n %{ns} rol $K -n %{ns} set image deploy %{dep} nginx=nginx:9.99-broken >/dev/null """, "checks": [ - C("running pods use nginx:1.25 and 3 are available", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 && for i in $(j -n %{ns} pod -l app=%{dep} '{.items[*].spec.containers[0].image}'); do eq "$i" nginx:1.25 || exit 1; done"""), + C("running pods use nginx:1.25 and 3 are available", 3, r""" +for t in $(seq 1 15); do # pods of the replaced ReplicaSets linger for a moment while they terminate + ok=1; eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 || ok=0 + for i in $(j -n %{ns} pod -l app=%{dep} '{.items[*].spec.containers[0].image}'); do eq "$i" nginx:1.25 || ok=0; done + [ $ok = 1 ] && exit 0; sleep 2 +done; exit 1"""), C("deployment is paused with template image nginx:1.27 and RELEASE=next", 2, r"""d="$K -n %{ns} get deploy %{dep}"; eq "$($d -o jsonpath='{.spec.paused}' 2>/dev/null)" true && eq "$($d -o jsonpath='{.spec.template.spec.containers[0].image}')" nginx:1.27 && eq "$($d -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="RELEASE")].value}')" next"""), - C("no ReplicaSet exists for the paused template (1.27 + RELEASE)", 2, r"""! $K -n %{ns} get rs -o json 2>/dev/null | python3 -c ' + C("no ReplicaSet exists for the paused template (1.27 + RELEASE)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.containers[0].image}')" nginx:1.27 || exit 1 +! $K -n %{ns} get rs -o json 2>/dev/null | python3 -c ' import sys,json for rs in json.load(sys.stdin)["items"]: c=rs["spec"]["template"]["spec"]["containers"][0] @@ -307,44 +335,99 @@ sys.exit(1)'"""), "task": CTX + """ Create pod `%{pod}` in `%{ns}`: - a **native sidecar** (an init container with `restartPolicy: Always`) named `agent` (busybox:1.36) - that appends `$(date)` to `/var/log/agent.log` every second, + that appends `$(date)` to `/var/log/agent/agent.log` every second, - main container `app` (nginx:1.27) with a readiness probe (HTTP GET / on 80, period 5s) and a liveness probe (TCP 80), -- shared `emptyDir` `logs` mounted at `/var/log` in both, +- shared `emptyDir` `logs` mounted at `/var/log/agent` in both (not at `/var/log`: nginx keeps its + own logs under `/var/log/nginx`, and an emptyDir mounted over `/var/log` hides that directory), - `terminationGracePeriodSeconds: 5`. When it is Running, write the names of **all** containers of the pod (init and regular), sorted, one per line, to `{WS}/q109/containers.txt`, and the pod's `.status.podIP` to `{WS}/q109/ip.txt`. """, - "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q109"; mkdir -p "$WS/q109"; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""", + "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q109"; mkdir -p "$WS/q109"; $K -n %{ns} delete pod %{pod} --ignore-not-found --force --grace-period=0 --wait=true >/dev/null 2>&1""", "checks": [ C("native sidecar: initContainer agent with restartPolicy Always", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.spec.initContainers[?(@.name=="agent")].restartPolicy}')" Always"""), C("probes on app as specified", 2, r"""c='{.spec.containers[?(@.name=="app")]'; eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.httpGet.path}")" / && eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.periodSeconds}")" 5 && eq "$(j -n %{ns} pod %{pod} "$c.livenessProbe.tcpSocket.port}")" 80"""), - C("Running, shared log written by agent, grace period 5", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.terminationGracePeriodSeconds}')" 5 && [ -n "$($K -n %{ns} exec %{pod} -c app -- sh -c 'tail -1 /var/log/agent.log' 2>/dev/null)" ]"""), - C("report files exact", 2, r"""want=$( (j -n %{ns} pod %{pod} '{.spec.initContainers[*].name}'; echo; j -n %{ns} pod %{pod} '{.spec.containers[*].name}') | tr ' ' '\n' | grep . | sort); [ "$(grep . "$WS/q109/containers.txt" 2>/dev/null | sort)" = "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q109/ip.txt" 2>/dev/null)" "$(j -n %{ns} pod %{pod} '{.status.podIP}')" """), + C("Running, shared log written by agent, grace period 5", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.terminationGracePeriodSeconds}')" 5 && [ -n "$($K -n %{ns} exec %{pod} -c app -- sh -c 'tail -1 /var/log/agent/agent.log' 2>/dev/null)" ]"""), + C("report files exact", 2, r"""want=$( (j -n %{ns} pod %{pod} '{.spec.initContainers[*].name}'; echo; j -n %{ns} pod %{pod} '{.spec.containers[*].name}') | tr ' ' '\n' | grep . | sort); [ -n "$want" ] && [ "$(grep . "$WS/q109/containers.txt" 2>/dev/null | sort)" = "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q109/ip.txt" 2>/dev/null)" "$(j -n %{ns} pod %{pod} '{.status.podIP}')" """), ], }, { - "id": 110, "domain": "WORK", "difficulty": "hard", "title": "Cluster audit to exact files", - "params": lambda r: {"ctx": "k8s-c1-H"}, + # Scoped to its own namespace: the cluster-wide version compared a file written mid-exam with the + # whole cluster at submit time, so every pod another task (or a grader probe) created or deleted + # in between failed it. killer.sh-style exact-output tasks work on data the task owns. + "id": 110, "domain": "WORK", "difficulty": "hard", "title": "Namespace audit to exact files", + "params": lambda r: {"ctx": "k8s-c1-H", "ns": ns_pick(r, "audit")}, "task": CTX + """ -Produce three files, contents **exactly** as specified (no headers, sorted with `sort`, one per line): -1. `{WS}/q110/no-readiness.txt` — `/` for every pod in the cluster whose first - container has **no** readiness probe. Exclude namespace `kube-system`. -2. `{WS}/q110/per-node.txt` — ` ` of Running pods per node, all namespaces. -3. `{WS}/q110/images.txt` — every distinct container image running in namespace `kube-system`. +Namespace `%{ns}` runs a mix of workloads. Produce three files with contents **exactly** as +specified — no headers, one entry per line, sorted with `sort`: +1. `{WS}/q110/no-readiness.txt` — the name of every Pod in `%{ns}` whose **first** container has + no readiness probe (whatever the Pod's phase). +2. `{WS}/q110/per-node.txt` — ` `: how many Pods in `%{ns}` are **Running** on each + node, listing only nodes that have at least one. +3. `{WS}/q110/images.txt` — every distinct container image used by Pods in `kube-system`. +""", + "setup": NS + r""" +mkns %{ns}; rm -rf "$WS/q110"; mkdir -p "$WS/q110" +$K -n %{ns} delete pod --all --force --grace-period=0 --wait=true >/dev/null 2>&1 +$K apply -f - >/dev/null </dev/null)" = Succeeded ] && break; sleep 2; done """, - "setup": r"""rm -rf "$WS/q110"; mkdir -p "$WS/q110" """, "checks": [ - C("no-readiness.txt exact", 3, r"""want=$($K get pods -A -o json 2>/dev/null | python3 -c ' + C("no-readiness.txt exact (first container only, every phase)", 3, r"""want=$($K -n %{ns} get pods -o json 2>/dev/null | python3 -c ' import sys,json for p in json.load(sys.stdin)["items"]: - if p["metadata"]["namespace"]=="kube-system": continue - if not p["spec"]["containers"][0].get("readinessProbe"): print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort); [ "$(grep . "$WS/q110/no-readiness.txt" 2>/dev/null | sort)" = "$want" ]"""), - C("per-node.txt exact", 2, r"""want=$($K get pods -A -o json 2>/dev/null | python3 -c ' + if not p["spec"]["containers"][0].get("readinessProbe"): print(p["metadata"]["name"])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q110/no-readiness.txt" 2>/dev/null)" = "$want" ]"""), + C("per-node.txt exact (Running pods only)", 2, r"""want=$($K -n %{ns} get pods -o json 2>/dev/null | python3 -c ' import sys,json,collections c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"] if p["status"]["phase"]=="Running" and p["spec"].get("nodeName")) -for n in sorted(c): print(n, c[n])'); [ "$(grep . "$WS/q110/per-node.txt" 2>/dev/null | sort)" = "$(echo "$want" | sort)" ]"""), - C("images.txt exact", 2, r"""want=$(j -n kube-system pods '{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u); [ "$(grep . "$WS/q110/images.txt" 2>/dev/null | sort -u)" = "$want" ]"""), +for n in c: print(n, c[n])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q110/per-node.txt" 2>/dev/null)" = "$want" ]"""), + C("images.txt exact", 2, r"""want=$(j -n kube-system pods '{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u); [ -n "$want" ] && [ "$(grep . "$WS/q110/images.txt" 2>/dev/null)" = "$want" ]"""), ], }, { @@ -361,7 +444,7 @@ by `kubectl config view --minify`, to `{WS}/q121/ns.txt`. """, "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q121"; mkdir -p "$WS/q121"; $K -n %{ns} delete deploy,svc %{dep} --ignore-not-found >/dev/null; $K -n default delete deploy,svc %{dep} --ignore-not-found >/dev/null""", "checks": [ - C("Deployment %{dep} in %{ns} (the context's namespace), 2 available, owner=ac", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && eq "$(j -n %{ns} deploy %{dep} '{.spec.template.metadata.labels.owner}')" ac"""), + C("Deployment %{dep} in %{ns} (the context's namespace), 2 available, owner=ac", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && { eq "$(j -n %{ns} deploy %{dep} '{.metadata.labels.owner}')" ac || eq "$(j -n %{ns} deploy %{dep} '{.spec.template.metadata.labels.owner}')" ac; }"""), C("Service %{dep} port 9000 in the same namespace", 1, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].port}')" 9000"""), C("nothing was created in namespace default by mistake", 1, r"""! $K -n default get deploy %{dep} >/dev/null 2>&1"""), C("ns.txt has the namespace", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q121/ns.txt" 2>/dev/null)" %{ns}"""), @@ -461,7 +544,7 @@ ok=any(any(any(h.get("name","").lower()=="x-canary" and h.get("value")=="always" sys.exit(0 if ok else 1)'"""), C("default rule weights v1 %{w2} / v2 %{w}", 2, r"""$K -n %{ns} get httproute %{host}-route -o json 2>/dev/null | python3 -c ' import sys,json; r=json.load(sys.stdin)["spec"]["rules"] -ok=any(not x.get("matches") or all(not m.get("headers") for m in x["matches"]) and {b["name"]:b.get("weight") for b in x["backendRefs"]}=={"v1":%{w2},"v2":%{w}} for x in r if len(x.get("backendRefs",[]))==2) +ok=any((not x.get("matches") or all(not m.get("headers") for m in x["matches"])) and {b["name"]:b.get("weight") for b in x["backendRefs"]}=={"v1":%{w2},"v2":%{w}} for x in r if len(x.get("backendRefs",[]))==2) sys.exit(0 if ok else 1)'"""), ], }, @@ -513,7 +596,18 @@ mkns %{ns} for n in "$W1" "$W2" "$W3"; do $K label node $n disk- >/dev/null 2>&1 || true; done $K label node "$W2" disk=nvme --overwrite >/dev/null $K -n %{ns} delete pod consumer --ignore-not-found --wait=false >/dev/null; $K -n %{ns} delete pvc %{pvc} --ignore-not-found >/dev/null -$K delete pv lv-hdd lv-nvme --ignore-not-found >/dev/null +# The PV names are fixed but every session draws a new namespace, so a claim from an earlier run +# (another namespace, often with its pod still running) keeps lv-hdd/lv-nvme Bound. pv-protection +# then holds `kubectl delete pv` forever and the seed timed out. Release them from whoever has them. +for pv in lv-hdd lv-nvme; do + cns=$(j pv $pv '{.spec.claimRef.namespace}'); cn=$(j pv $pv '{.spec.claimRef.name}') + [ -n "$cns" ] && [ -n "$cn" ] || continue + for p in $($K -n "$cns" get pod -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}{end}' 2>/dev/null | awk -v c="$cn" '{for(i=2;i<=NF;i++) if($i==c) print $1}'); do + $K -n "$cns" delete pod "$p" --grace-period=0 --force >/dev/null 2>&1 + done + $K -n "$cns" delete pvc "$cn" --ignore-not-found --wait=false >/dev/null 2>&1 +done +$K delete pv lv-hdd lv-nvme --ignore-not-found --timeout=90s >/dev/null 2>&1 || true for pair in "lv-hdd:$W1:hdd:2Gi" "lv-nvme:$W2:nvme:1Gi"; do IFS=: read name node kind size <<<"$pair" nx "$node" "mkdir -p /mnt/$name" $K apply -f - >/dev/null < "$WS/.q116-pv" }, # ============================================================== TSHOOT ==== { - "id": 117, "domain": "TSHOOT", "breaks": "w2-kubelet", "difficulty": "hard", "title": "Worker node: three things wrong", "disruptive": True, + "id": 117, "domain": "TSHOOT", "breaks": ["w2-kubelet", "w2-drain"], "difficulty": "hard", "title": "Worker node: three things wrong", "disruptive": True, "params": lambda r: {"ctx": "k8s-c1-H"}, "task": CTX + """ Node `{W2}` is unusable. Make it fully usable again: `Ready`, schedulable, kubelet running and @@ -600,7 +694,7 @@ $K -n default delete pod node-probe --wait=false >/dev/null 2>&1; exit $ok"""), ], }, { - "id": 118, "domain": "TSHOOT", "breaks": "scheduler", "difficulty": "hard", "title": "Two control-plane components down", "disruptive": True, + "id": 118, "domain": "TSHOOT", "breaks": ["scheduler", "controller-manager"], "difficulty": "hard", "title": "Two control-plane components down", "disruptive": True, "params": lambda r: {"ctx": "k8s-c1-H"}, "task": CTX + """ New Deployments create no pods, and pods that do get created stay `Pending`. Two different @@ -673,7 +767,8 @@ j -n %{ns} deploy %{dep} '{.metadata.uid}' > "$WS/.q119-uid"; j -n %{ns} svc %{d "checks": [ C("3 replicas Available", 3, r"""for i in $(seq 1 6); do [ "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" = 3 ] && exit 0; sleep 5; done; exit 1"""), C("Service routes: endpoints ready, targetPort 80", 2, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].targetPort}')" 80 && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""), - C("Deployment UID and Service ClusterIP unchanged (fixed in place)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.metadata.uid}')" "$(cat "$WS/.q119-uid")" && eq "$(j -n %{ns} svc %{dep} '{.spec.clusterIP}')" "$(cat "$WS/.q119-ip")" """), + C("Deployment UID and Service ClusterIP unchanged (fixed in place)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 || exit 1 # only once it is actually fixed +eq "$(j -n %{ns} deploy %{dep} '{.metadata.uid}')" "$(cat "$WS/.q119-uid")" && eq "$(j -n %{ns} svc %{dep} '{.spec.clusterIP}')" "$(cat "$WS/.q119-ip")" """), C("quota still present and permits the 3 pods", 1, r"""p=$(j -n %{ns} quota tight '{.spec.hard.pods}'); [ -n "$p" ] && [ "$p" -ge 3 ]"""), ], }, diff --git a/tracks/cka/bank_sim_a.py b/tracks/cka/bank_sim_a.py index 30b56e4..685720b 100644 --- a/tracks/cka/bank_sim_a.py +++ b/tracks/cka/bank_sim_a.py @@ -243,12 +243,15 @@ Namespace `project-orion` is close to filling its nodes. setup=NS + r""" mkns project-orion rm -rf "$WS/q204"; mkdir -p "$WS/q204" -$K -n project-orion delete deploy --all >/dev/null 2>&1; $K -n project-orion delete pod orion-api-0 --ignore-not-found --wait=false >/dev/null -$K apply -f - >/dev/null <<'EOF' +# the pod is re-created below under the same name: a delete that does not wait lets the apply land on +# the terminating old pod, which then vanishes and leaves the task without orion-api-0 +$K -n project-orion delete deploy --all >/dev/null 2>&1; $K -n project-orion delete pod orion-api-0 --ignore-not-found --force --grace-period=0 --wait=true >/dev/null 2>&1 +$K apply -f - >/dev/null </dev/null | python3 -c ' + # Every BestEffort pod this task seeded must be listed and no Burstable/Guaranteed one may be. + # BestEffort pods other tasks add to the namespace later (212's spread-check has no + # resources at all) are tolerated either way - they did not exist when the file was written. + C("evicted-first.txt lists the BestEffort pods and nothing else", 4, r""" +f="$WS/q204/evicted-first.txt"; [ -s "$f" ] || exit 1 +$K -n project-orion get pod -o json 2>/dev/null | python3 -c ' import sys,json -print("\n".join(sorted(p["metadata"]["name"] for p in json.load(sys.stdin)["items"] if p["status"].get("qosClass")=="BestEffort")))') -[ -n "$want" ] && [ "$(grep . "$WS/q204/evicted-first.txt" 2>/dev/null | sort)" = "$want" ]"""), +items=json.load(sys.stdin)["items"]; got={l.strip() for l in open(sys.argv[1]) if l.strip()} +qos={p["metadata"]["name"]:p["status"].get("qosClass") for p in items} +seeded={p["metadata"]["name"] for p in items if p["metadata"].get("labels",{}).get("app")=="orion-batch"} +ok=seeded and seeded<=got and not any(qos.get(n) in ("Burstable","Guaranteed") for n in got) +sys.exit(0 if ok else 1)' "$f" """), C("api-qos.txt says Burstable", 2, r"""grep -qi burstable "$WS/q204/api-qos.txt" 2>/dev/null"""), C("guaranteed.txt names orion-ledger", 2, r"""grep -q 'orion-ledger' "$WS/q204/guaranteed.txt" 2>/dev/null"""), ], @@ -504,6 +516,11 @@ chmod +x {WS}/q207/usage-nodes.sh {WS}/q207/usage-pods.sh ) # ------------------------------------------------------------------ 208 ---- +# Own namespace (project-vega): it used to share project-atlas with 203, whose "running pods" +# count then dropped by one when this drain deleted the bare pod. And the grader checks the pods +# that were on the node when it was seeded, not "nothing on the node at grading time" - after the +# required uncordon the scheduler is free to put new pods there (212's one-per-node spread always +# does), which failed this task no matter what the candidate did. task( id=208, domain="ARCH", title="Drain a node for maintenance", breaks="w2-drain", task=""" @@ -512,23 +529,28 @@ Node `{W2}` needs kernel maintenance. 1. Evict every non-DaemonSet Pod from it. Pods with `emptyDir` volumes must not stop you, and neither must Pods that aren't managed by a controller. 2. Write the exact command you used to `{WS}/q208/drain.sh`. -3. Deployment `atlas-web` in Namespace `project-atlas` must still have all 3 replicas Available afterwards. +3. Deployment `vega-web` in Namespace `project-vega` must still have all 3 replicas Available afterwards. 4. When you're done, make the node schedulable again. + +A Pod without a controller is **deleted** by a forced drain, not rescheduled. Before you drain, +look at what else runs on the node (`k get pods -A -o wide --field-selector spec.nodeName={W2}`) — +anything you created there for another task has to be recreated afterwards. """, setup=NS + r""" -mkns project-atlas +mkns project-vega rm -rf "$WS/q208"; mkdir -p "$WS/q208" -$K -n project-atlas delete deploy atlas-web --ignore-not-found >/dev/null +$K -n project-vega delete deploy vega-web --ignore-not-found >/dev/null +$K -n project-vega delete pod vega-orphan --ignore-not-found --force --grace-period=0 --wait=true >/dev/null 2>&1 $K uncordon "$W2" >/dev/null 2>&1 || true $K apply -f - >/dev/null </dev/null 2>&1 -$K -n project-atlas run atlas-orphan --image=busybox:1.36 --restart=Never --overrides="{\"spec\":{\"nodeName\":\"$W2\"}}" -- sleep 86400 >/dev/null 2>&1 +$K -n project-vega rollout status deploy vega-web --timeout=120s >/dev/null 2>&1 +$K -n project-vega run vega-orphan --image=busybox:1.36 --restart=Never --overrides="{\"spec\":{\"nodeName\":\"$W2\"}}" -- sleep 86400 >/dev/null 2>&1 +$K -n project-vega wait --for=condition=Ready pod/vega-orphan --timeout=90s >/dev/null 2>&1 +# what the drain has to clear: every non-DaemonSet, non-mirror, non-kube-system pod on the node now +$K get pods -A -o json | python3 -c ' +import sys,json +n=sys.argv[1] +for p in json.load(sys.stdin)["items"]: + m=p["metadata"]; o=(m.get("ownerReferences") or [{}])[0].get("kind") + if p["spec"].get("nodeName")==n and m["namespace"]!="kube-system" and o not in ("DaemonSet","Node"): + print(m["uid"])' "$W2" > "$WS/.q208-uids" """, checks=[ - C("no non-DaemonSet workload pods remain on the node", 4, r""" -left=$($K get pods -A -o json 2>/dev/null | python3 -c ' -import sys,json,os -n=os.environ["CKA_W2"] -print(sum(1 for p in json.load(sys.stdin)["items"] - if p["spec"].get("nodeName")==n and p["status"]["phase"]=="Running" - and (p["metadata"].get("ownerReferences") or [{}])[0].get("kind") not in ("DaemonSet",) - and p["metadata"]["namespace"] not in ("kube-system",)))') -eq "$left" 0"""), + C("every pod that was on the node when it was seeded has been evicted", 4, r""" +[ -s "$WS/.q208-uids" ] || exit 1 +now=$($K get pods -A -o jsonpath='{range .items[*]}{.metadata.uid}{" "}{.spec.nodeName}{"\n"}{end}' 2>/dev/null) +while read -r u; do echo "$now" | grep -q "^$u $W2\$" && exit 1; done < "$WS/.q208-uids" +! $K -n project-vega get pod vega-orphan >/dev/null 2>&1"""), C("drain.sh records a drain with --ignore-daemonsets and --delete-emptydir-data", 2, r""" f="$WS/q208/drain.sh"; grep -q 'drain' "$f" 2>/dev/null && grep -q -- '--ignore-daemonsets' "$f" && grep -q -- '--delete-emptydir-data' "$f" """), - C("atlas-web still has 3 replicas Available", 2, r"""eq "$(j -n project-atlas deploy atlas-web '{.status.availableReplicas}')" 3"""), + C("vega-web still has 3 replicas Available", 2, r"""eq "$(j -n project-vega deploy vega-web '{.status.availableReplicas}')" 3"""), C("node is schedulable again", 2, r"""[ "$(j node "$W2" '{.spec.unschedulable}')" != true ]"""), ], solution=""" +k get pods -A -o wide --field-selector spec.nodeName={W2} # what is about to be evicted or deleted k drain {W2} --ignore-daemonsets --delete-emptydir-data --force # --ignore-daemonsets DaemonSet pods can't be evicted, skip them -# --delete-emptydir-data required because atlas-web mounts an emptyDir -# --force required for the bare pod atlas-orphan (no controller) +# --delete-emptydir-data required because vega-web mounts an emptyDir (its data is lost) +# --force required for the bare pod vega-orphan - which is DELETED, not moved echo 'kubectl drain {W2} --ignore-daemonsets --delete-emptydir-data --force' > {WS}/q208/drain.sh -k get pods -A -o wide --field-selector spec.nodeName={W2} -k -n project-atlas rollout status deploy atlas-web +k get pods -A -o wide --field-selector spec.nodeName={W2} # only DaemonSet pods left +k -n project-vega rollout status deploy vega-web k uncordon {W2} """, ) @@ -657,7 +687,7 @@ k -n project-atlas auth can-i get secret --as=system:serviceaccount:project-a # ------------------------------------------------------------------ 211 ---- task( - id=211, domain="WORK", title="DaemonSet on every node", + id=211, domain="WORK", title="DaemonSet on every node", breaks=["w1-taint", "w1-block"], task=""" In Namespace `project-orion` create a DaemonSet `node-collector`: @@ -697,7 +727,7 @@ k -n project-orion get ds node-collector -o wide # ------------------------------------------------------------------ 212 ---- task( - id=212, domain="WORK", title="One Pod per node, on purpose", + id=212, domain="WORK", title="One Pod per node, on purpose", breaks=["w1-taint", "w1-block"], task=""" In Namespace `project-orion` create Deployment `spread-check`: @@ -722,7 +752,7 @@ echo "$t" | grep -q '"topologyKey":"kubernetes.io/hostname"' && { echo "$t" | gr C("exactly 3 Pods Running, one per node, 1 Pending", 3, r""" r=$($K -n project-orion get pod -l id=spread-check --no-headers 2>/dev/null | grep -c Running) p=$($K -n project-orion get pod -l id=spread-check --no-headers 2>/dev/null | grep -c Pending) -eq "$r" 3 && eq "$p" 1 && [ "$(j -n project-orion pod -l id=spread-check '{.items[*].spec.nodeName}' | tr ' ' '\n' | grep . | sort -u | wc -l)" = 3"""), +eq "$r" 3 && eq "$p" 1 && [ "$(j -n project-orion pod -l id=spread-check '{.items[*].spec.nodeName}' | tr ' ' '\n' | grep . | sort -u | wc -l)" = 3 ]"""), C("pending.txt names the Pending pod", 2, r""" want=$($K -n project-orion get pod -l id=spread-check --no-headers 2>/dev/null | awk '$3=="Pending"{print $1}') [ -n "$want" ] && grep -qF "$want" "$WS/q212/pending.txt" 2>/dev/null"""), diff --git a/tracks/cka/bank_sim_b.py b/tracks/cka/bank_sim_b.py index eeb8fe0..fc90112 100644 --- a/tracks/cka/bank_sim_b.py +++ b/tracks/cka/bank_sim_b.py @@ -233,14 +233,16 @@ k -n project-token get secret reporter-legacy -o yaml # the controller fills ) # ------------------------------------------------------------------ 305 ---- +# {W2}, not {W3}: 309 drains {W3} in this same set, and a drain without --force refuses to touch +# bare Pods - while one with --force deletes them, taking this task's answer with it. task( id=305, domain="WORK", title="Schedule without the scheduler", task=""" In Namespace `project-orion`: 1. Create Pod `direct-assign` (image `busybox:1.36`, command `sleep 86400`) that runs on node - `{W3}` **without the scheduler being involved at all**. -2. Create Pod `scheduled` (same image and command) that lands on `{W3}` **through** the scheduler. + `{W2}` **without the scheduler being involved at all**. +2. Create Pod `scheduled` (same image and command) that lands on `{W2}` **through** the scheduler. 3. Write to `{WS}/q305/proof.txt` the value of the field you used for Pod 1, in the form `field=`, and on a second line `scheduler=`. """, @@ -248,16 +250,26 @@ In Namespace `project-orion`: mkns project-orion rm -rf "$WS/q305"; mkdir -p "$WS/q305" $K -n project-orion delete pod direct-assign scheduled --ignore-not-found --wait=false >/dev/null -$K label node "$W3" tier- >/dev/null 2>&1 || true """, + # Who set spec.nodeName is recorded permanently in managedFields: a client that submits it + # (kubectl create/apply/run) owns f:nodeName, while a scheduler binding goes through the + # pods/binding subresource and leaves no managedFields entry for it (verified on 1.37). Events + # were the old signal, but plain `get events` has no SOURCE column and they expire after 1h. checks=[ - C("direct-assign Running on the right node with spec.nodeName set", 3, r""" -eq "$(j -n project-orion pod direct-assign '{.status.phase}')" Running && eq "$(j -n project-orion pod direct-assign '{.spec.nodeName}')" "$W3" """), - C("direct-assign was never scheduled (no Scheduled event from default-scheduler)", 3, r""" -! $K -n project-orion get events --field-selector involvedObject.name=direct-assign 2>/dev/null | grep -q 'default-scheduler'"""), - C("scheduled Running on the same node via the default scheduler", 3, r""" -eq "$(j -n project-orion pod scheduled '{.status.phase}')" Running && eq "$(j -n project-orion pod scheduled '{.spec.nodeName}')" "$W3" \ - && $K -n project-orion get events --field-selector involvedObject.name=scheduled 2>/dev/null | grep -q 'default-scheduler'"""), + C("direct-assign Running on {W2}", 3, r""" +eq "$(j -n project-orion pod direct-assign '{.status.phase}')" Running && eq "$(j -n project-orion pod direct-assign '{.spec.nodeName}')" "$W2" """), + C("direct-assign bypassed the scheduler (spec.nodeName came from your manifest)", 3, r""" +$K -n project-orion get pod direct-assign -o json --show-managed-fields 2>/dev/null | python3 -c ' +import sys,json +mf=json.load(sys.stdin)["metadata"].get("managedFields",[]) +sys.exit(0 if any("f:nodeName" in json.dumps(m.get("fieldsV1",{})) for m in mf if m.get("subresource","")!="status") else 1)'"""), + C("scheduled Running on {W2}, placed there by default-scheduler (no nodeName in its manifest)", 3, r""" +eq "$(j -n project-orion pod scheduled '{.status.phase}')" Running && eq "$(j -n project-orion pod scheduled '{.spec.nodeName}')" "$W2" \ + && eq "$(j -n project-orion pod scheduled '{.spec.schedulerName}')" default-scheduler || exit 1 +$K -n project-orion get pod scheduled -o json --show-managed-fields 2>/dev/null | python3 -c ' +import sys,json +mf=json.load(sys.stdin)["metadata"].get("managedFields",[]) +sys.exit(1 if any("f:nodeName" in json.dumps(m.get("fieldsV1",{})) for m in mf) else 0)'"""), C("proof.txt names nodeName and default-scheduler", 2, r""" grep -q 'field= *nodeName' "$WS/q305/proof.txt" 2>/dev/null && grep -q 'scheduler= *default-scheduler' "$WS/q305/proof.txt" """), ], @@ -267,7 +279,7 @@ apiVersion: v1 kind: Pod metadata: {name: direct-assign, namespace: project-orion} spec: - nodeName: {W3} # bypasses the scheduler entirely - the kubelet just picks it up + nodeName: {W2} # bypasses the scheduler entirely - the kubelet just picks it up containers: [{name: c, image: busybox:1.36, command: [sleep, "86400"]}] EOF cat < {WS}/q305/proof.txt +# `k get events` hides who reported an event; -o wide (or describe) shows the SOURCE column. +# Events expire after an hour, so for lasting proof look at who owns spec.nodeName: +# k -n project-orion get pod direct-assign -o yaml --show-managed-fields | grep -B2 -A2 nodeName """, ) @@ -294,8 +309,13 @@ In Namespace `project-orbit`: `volumeBindingMode: WaitForFirstConsumer`, and allowing volume expansion. 2. Create PVC `growing` (1Gi, RWO) using that class and Pod `grower` (busybox:1.36, `sleep 86400`) mounting it at `/data`, so the claim binds. -3. Then grow the claim to **2Gi** without deleting it, and write the claim's - `.status.capacity.storage` after the resize to `{WS}/q306/capacity.txt`. +3. Then grow the claim to **2Gi** without deleting it, and write both sizes the claim reports + afterwards to `{WS}/q306/capacity.txt`, one per line: + `requested=<.spec.resources.requests.storage>` and `status=<.status.capacity.storage>`. + +(The local-path provisioner has no volume resizer, so on this cluster the status will not catch up +with the request — that gap is what step 3 asks you to observe. With a CSI driver that supports +expansion, the status would follow within seconds.) """, setup=NS + r""" mkns project-orbit @@ -312,10 +332,17 @@ eq "$($s -o jsonpath='{.provisioner}' 2>/dev/null)" rancher.io/local-path && eq C("PVC growing Bound on that class with grower Running", 3, r""" eq "$(j -n project-orbit pvc growing '{.spec.storageClassName}')" expandable && eq "$(j -n project-orbit pvc growing '{.status.phase}')" Bound \ && eq "$(j -n project-orbit pod grower '{.status.phase}')" Running"""), - C("claim requests 2Gi and reports at least 2Gi capacity", 4, r""" + # local-path cannot resize (verified: ExternalExpanding, capacity never moves), so a bound PV + # that is still 1Gi under a claim requesting 2Gi is exactly what "grown in place" looks like + # here. A claim deleted and recreated at 2Gi would have been provisioned a 2Gi volume instead. + C("claim grown in place: requests 2Gi on the original 1Gi volume", 4, r""" eq "$(j -n project-orbit pvc growing '{.spec.resources.requests.storage}')" 2Gi || exit 1 -for i in $(seq 1 12); do c=$(j -n project-orbit pvc growing '{.status.capacity.storage}'); [ "$c" = 2Gi ] && exit 0; sleep 5; done; exit 1"""), - C("capacity.txt records the new capacity", 1, r"""grep -q '2Gi' "$WS/q306/capacity.txt" 2>/dev/null"""), +pv=$(j -n project-orbit pvc growing '{.spec.volumeName}'); [ -n "$pv" ] || exit 1 +eq "$(j pv "$pv" '{.spec.capacity.storage}')" 1Gi"""), + C("capacity.txt records requested and status sizes as the cluster reports them", 1, r""" +f="$WS/q306/capacity.txt" +eq "$(sed -n 's/^ *requested *= *//p' "$f" 2>/dev/null | tr -d '[:space:]')" 2Gi \ + && eq "$(sed -n 's/^ *status *= *//p' "$f" | tr -d '[:space:]')" "$(j -n project-orbit pvc growing '{.status.capacity.storage}')" """), ], solution=""" cat < {WS}/q306/capacity.txt -# Shrinking is never allowed. Without allowVolumeExpansion the patch is rejected outright. +k -n project-orbit describe pvc growing # event ExternalExpanding: waiting for an external controller +k -n project-orbit get pvc growing -o jsonpath='requested={.spec.resources.requests.storage}{"\\n"}status={.status.capacity.storage}{"\\n"}' > {WS}/q306/capacity.txt +cat {WS}/q306/capacity.txt # requested=2Gi status=1Gi +# allowVolumeExpansion is the gate: without it the API rejects the patch outright, and shrinking is +# never allowed. Accepting the request is all the API does - a resizer (the CSI driver's +# external-resizer sidecar) then grows the volume and the kubelet the filesystem, and only then does +# .status.capacity change. local-path has no resizer, so here the status stays at 1Gi forever. """, ) # ------------------------------------------------------------------ 307 ---- task( - id=307, domain="TSHOOT", title="Node accepts nothing new", breaks="w1-block", + id=307, domain="TSHOOT", title="Node accepts nothing new", breaks=["w1-block", "w1-taint"], task=""" Node `{W1}` is `Ready` and its kubelet is healthy, but no new Pod will schedule onto it. @@ -451,6 +482,10 @@ at all times. 2. Drain `{W3}` (DaemonSet Pods aside), then make it schedulable again. 3. Write the value of the PDB's `.status.disruptionsAllowed` after your change to `{WS}/q309/allowed.txt`. + +Look before you drain: `k get pods -A -o wide --field-selector spec.nodeName={W3}`. A Pod without a +controller blocks a plain drain, and `--force` deletes it for good — if it belongs to another +task, recreate it once the node is back. """, setup=NS + r""" mkns project-atlas @@ -498,6 +533,7 @@ k -n project-atlas get pdb atlas-api # a drain would hang forever waiting for the eviction API. k -n project-atlas patch pdb atlas-api -p '{"spec":{"minAvailable":1}}' k -n project-atlas get pdb atlas-api # ALLOWED DISRUPTIONS: 1 +k get pods -A -o wide --field-selector spec.nodeName={W3} # bare pods here? they need --force and are then gone k drain {W3} --ignore-daemonsets --delete-emptydir-data k -n project-atlas rollout status deploy atlas-api k uncordon {W3} @@ -583,24 +619,33 @@ Then write the number of **distinct** images in use across those Deployments to """, setup=r"""rm -rf "$WS/q311"; mkdir -p "$WS/q311" """, checks=[ - C("images.txt matches the cluster exactly", 5, r""" -want=$($K get deploy -A -o json 2>/dev/null | python3 -c ' -import sys,json -out=[] + # Graded against the Deployments that existed when the file was written (its mtime): other + # set-B tasks create Deployments later (303's cidr-demo), and an exact cluster-wide + # comparison at submit time failed a correct answer whenever one of them came after this. + C("images.txt matches the cluster as it was when you wrote it", 5, r""" +f="$WS/q311/images.txt"; [ -s "$f" ] || exit 1 +$K get deploy -A -o json 2>/dev/null | python3 -c ' +import sys,json,os,datetime +f=sys.argv[1]; T=os.path.getmtime(f) +def ts(d): return datetime.datetime.strptime(d["metadata"]["creationTimestamp"],"%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=datetime.timezone.utc).timestamp() +want=set(); later=set() for d in json.load(sys.stdin)["items"]: - s=d["spec"]["template"]["spec"] - imgs=[c["image"] for c in s.get("initContainers",[])]+[c["image"] for c in s["containers"]] - out.append("%s/%s %s"%(d["metadata"]["namespace"],d["metadata"]["name"],",".join(imgs))) -print("\n".join(sorted(out)))') -[ "$(grep . "$WS/q311/images.txt" 2>/dev/null | sort)" = "$want" ]"""), + key="%s/%s"%(d["metadata"]["namespace"],d["metadata"]["name"]); s=d["spec"]["template"]["spec"] + if ts(d) > T: later.add(key); continue + want.add("%s %s"%(key,",".join([c["image"] for c in s.get("initContainers",[])]+[c["image"] for c in s["containers"]]))) +got={l.strip() for l in open(f) if l.strip()} +extra={g for g in got-want if g.split(" ")[0] in {w.split(" ")[0] for w in want}} # a real deployment, wrong images +sys.exit(0 if want<=got and not extra else 1)' "$f" """), C("distinct.txt has the right count", 3, r""" +f="$WS/q311/images.txt"; [ -s "$f" ] || exit 1 want=$($K get deploy -A -o json 2>/dev/null | python3 -c ' -import sys,json -s=set() +import sys,json,os,datetime +T=os.path.getmtime(sys.argv[1]); s=set() for d in json.load(sys.stdin)["items"]: + if datetime.datetime.strptime(d["metadata"]["creationTimestamp"],"%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=datetime.timezone.utc).timestamp() > T: continue sp=d["spec"]["template"]["spec"] for c in sp.get("initContainers",[])+sp["containers"]: s.add(c["image"]) -print(len(s))') +print(len(s))' "$f") eq "$(tr -dc '0-9' < "$WS/q311/distinct.txt" 2>/dev/null)" "$want" """), ], solution=""" @@ -784,8 +829,10 @@ for i in $(seq 1 12); do [ "$(j -n default pod "edge-cache-$W3" '{.status.phase}')" = Running ] && [ "$(j -n default pod "edge-cache-$W3" '{.spec.containers[0].image}')" = nginx:1.27 ] && exit 0 sleep 5 done; exit 1"""), + # -h: with two file operands GNU grep prefixes every line with the file name, which made + # awk's $2 the literal word "path:" and this check unpassable. C("the hostPath the manifest references now exists on the node", 3, r""" -p=$(nx "$W3" 'grep -A2 hostPath /etc/kubernetes/manifests/edge-cache.yaml /etc/kubernetes/static/edge-cache.yaml 2>/dev/null | grep "path:" | head -1' | awk '{print $2}') +p=$(nx "$W3" 'grep -h -A2 hostPath /etc/kubernetes/manifests/edge-cache.yaml /etc/kubernetes/static/edge-cache.yaml 2>/dev/null | grep "path:" | head -1' | awk '{print $2}') [ -n "$p" ] && nx "$W3" "test -d $p" """), C("reason.txt mentions the missing host directory", 2, r"""grep -Eqi 'director|hostpath|/opt/edge-cache|not exist|missing' "$WS/q314/reason.txt" 2>/dev/null"""), ], @@ -916,11 +963,16 @@ EOF ) # ------------------------------------------------------------------ 317 ---- +# The sabotage switches off ONE controller. It used to point --cluster-signing-cert-file at a +# missing file instead, but a controller that fails to start is fatal to kube-controller-manager +# (verified on 1.37: "Error building controllers" -> exit), so the whole KCM crash-looped and every +# other set-B task that needs a controller (Deployments, Jobs, EndpointSlices, PV phases, PDB +# status, token Secrets) silently stopped working until this task - the last one - was fixed. task( id=317, domain="TSHOOT", title="Certificates stop being issued", breaks="controller-manager", task=""" Certificate signing requests are approved but never get a certificate: `kubectl get csr` shows -them `Approved` with an empty `CONDITION`-issued certificate forever. +them `Approved` and never `Approved,Issued`. Everything else in the cluster works. Find the responsible control-plane component, repair it on the control-plane node, and prove it works: a CertificateSigningRequest named `issue-probe` (CN `probe`, signer @@ -931,14 +983,14 @@ receive a certificate. $K delete csr issue-probe --ignore-not-found >/dev/null 2>&1 nx "$CP" ' f=/etc/kubernetes/manifests/kube-controller-manager.yaml -grep -q "cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt" $f && sed -i "s|--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt|--cluster-signing-cert-file=/etc/kubernetes/pki/ca-signing.crt|" $f +grep -q -- "-certificatesigningrequest-signing-controller" $f || sed -i "s|^\( *- --controllers=.*\)$|\1,-certificatesigningrequest-signing-controller|" $f ' """, checks=[ - C("kube-controller-manager points at a signing certificate that exists", 4, r""" -p=$(nx "$CP" 'grep -o "\-\-cluster-signing-cert-file=[^ ]*" /etc/kubernetes/manifests/kube-controller-manager.yaml' | cut -d= -f2) -[ -n "$p" ] && nx "$CP" "test -f $p" """), - C("kube-controller-manager pod is Running", 3, r""" + C("kube-controller-manager runs the CSR signing controller again", 4, r""" +c=$(nx "$CP" 'grep -o -- "--controllers=[^ ]*" /etc/kubernetes/manifests/kube-controller-manager.yaml') +[ -n "$c" ] && ! echo "$c" | grep -Eq -- '-(certificatesigningrequest-signing-controller|csrsigning)(,|$)'"""), + C("kube-controller-manager still Running after your edit", 3, r""" for i in $(seq 1 12); do [ "$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].status.phase}')" = Running ] && exit 0; sleep 5; done; exit 1"""), C("an approved CSR now receives a certificate", 5, r""" $K get csr issue-probe >/dev/null 2>&1 || exit 1 @@ -946,15 +998,18 @@ $K get csr issue-probe -o jsonpath='{.status.conditions[*].type}' | grep -q Appr for i in $(seq 1 15); do [ -n "$(j csr issue-probe '{.status.certificate}')" ] && exit 0; sleep 4; done; exit 1"""), ], solution=""" -k get csr # Approved but never Issued -k -n kube-system logs kube-controller-manager-{CP} | tail -20 -# ... unable to load client CA file .../ca-signing.crt: no such file or directory +k get csr # Approved - but never Approved,Issued +# Approving is the API's job; ISSUING is a controller's job, and controllers live in +# kube-controller-manager. So: is it running, and is the right controller in it running? +k -n kube-system get pod -l component=kube-controller-manager # Running - so not a crash +k -n kube-system logs kube-controller-manager-{CP} | grep -i disabled +# "Warning: controller is disabled" controller="certificatesigningrequest-signing-controller" {SHELL_CP} -grep signing /etc/kubernetes/manifests/kube-controller-manager.yaml -ls /etc/kubernetes/pki/ca.crt # the real file -vi /etc/kubernetes/manifests/kube-controller-manager.yaml -# --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt +grep -- --controllers /etc/kubernetes/manifests/kube-controller-manager.yaml +# --controllers=*,bootstrapsigner,tokencleaner,-certificatesigningrequest-signing-controller +vi /etc/kubernetes/manifests/kube-controller-manager.yaml # remove ,-certificatesigningrequest-signing-controller exit +k -n kube-system get pod -l component=kube-controller-manager -w # the kubelet restarts the static pod openssl genrsa -out probe.key 2048 openssl req -new -key probe.key -subj "/CN=probe" -out probe.csr cat </dev/null 2> want=$($K -n kube-system get ds -o name 2>/dev/null | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1) [ -n "$want" ] || want=kindnet grep -qi "$want" "$WS/q410/cni.txt" 2>/dev/null"""), - C("enforced.txt correctly says this CNI does not enforce NetworkPolicy", 2, r""" -grep -qix 'no' "$WS/q410/enforced.txt" 2>/dev/null"""), + # The right answer depends on the build, so derive it: recent kindnet embeds a policy engine + # (nftables table `kindnet-network-policies` on every node - verified enforcing on this + # cluster), calico/cilium/weave enforce, flannel does not. + C("enforced.txt gives the right answer for this cluster's CNI", 2, r""" +ds=$($K -n kube-system get ds -o name 2>/dev/null | sed 's|.*/||' | tr '\n' ' ') +case "$ds" in + *calico*|*cilium*|*weave*|*antrea*) want=yes ;; + *kindnet*) nx "$W1" 'nft list table inet kindnet-network-policies >/dev/null 2>&1' /dev/null"""), C("default-deny-all selects every pod in both directions", 4, r""" n="$K -n zero-trust get netpol default-deny-all" eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \ @@ -702,7 +714,10 @@ eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \ ], solution=""" k -n kube-system get ds # kindnet -> {WS}/q410/cni.txt -echo no > {WS}/q410/enforced.txt # kindnet has no policy engine +# Don't answer from memory - prove it. Two pods, one deny policy, one wget: +k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv; k -n zero-trust expose pod srv --port 80 +k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600 +k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null && echo reachable cat </dev/null || echo BLOCKED # -> BLOCKED here +echo yes > {WS}/q410/enforced.txt +k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv # leave only the policy # podSelector {} means every pod in the namespace; naming both policyTypes with no ingress/egress -# blocks listed means "deny everything". kindnet (like flannel) has no policy engine, so the API -# server stores these objects and nothing enforces them - a real trap in a hardening review. -# Calico and Cilium do enforce; the exam expects you to know which is which. +# blocks listed means "deny everything". Recent kindnet builds (this cluster's included) embed a +# policy engine - look for `nft list table inet kindnet-network-policies` on a node. Flannel stores +# NetworkPolicies and enforces nothing; Calico and Cilium enforce. A policy that "works" on a +# non-enforcing CNI is the classic hardening-review trap. """, ) @@ -1066,7 +1085,8 @@ In Namespace `safety-monitor`, create NetworkPolicy `safety-api-policy` for the - **egress**: DNS on port 53, both UDP and TCP, to any destination, - everything else denied in both directions. -(kindnet does not enforce policies; you are graded on the object.) +(This cluster's CNI enforces policies, so a wrong policy really breaks traffic — but you are graded on +the object.) """, setup=NS + r""" mkns safety-monitor @@ -1195,8 +1215,13 @@ $K -n booking-tls rollout status deploy booking-nginx --timeout=120s >/dev/null C("the ConfigMap allows TLSv1.3 only", 3, r""" c=$(j -n booking-tls cm booking-nginx-config '{.data.default\.conf}') echo "$c" | grep -q 'ssl_protocols.*TLSv1\.3' && ! echo "$c" | grep -q 'TLSv1\.2'"""), + # rollout restart (restartedAt annotation) or deleting the pods - either way every running + # pod must be younger than the last write to the ConfigMap C("the pods were restarted to pick the change up", 2, r""" -j -n booking-tls deploy booking-nginx '{.spec.template.metadata.annotations}' | grep -q 'restartedAt'"""), +j -n booking-tls deploy booking-nginx '{.spec.template.metadata.annotations}' | grep -q 'restartedAt' && exit 0 +cm_t=$($K -n booking-tls get cm booking-nginx-config -o json --show-managed-fields 2>/dev/null | python3 -c 'import sys,json; print(max(m.get("time","") for m in json.load(sys.stdin)["metadata"].get("managedFields",[])))') +ts=$(j -n booking-tls pod -l app=booking-nginx '{.items[*].metadata.creationTimestamp}'); [ -n "$cm_t" ] && [ -n "$ts" ] || exit 1 +for t in $ts; do [[ "$t" > "$cm_t" ]] || exit 1; done"""), C("ssl-line.txt records the directive", 2, r""" f="$WS/q417/ssl-line.txt" grep -q 'ssl_protocols' "$f" 2>/dev/null && grep -q 'TLSv1\.3' "$f" && ! grep -q 'TLSv1\.2' "$f" """), diff --git a/tracks/cka/bank_sim_d.py b/tracks/cka/bank_sim_d.py new file mode 100644 index 0000000..153f3da --- /dev/null +++ b/tracks/cka/bank_sim_d.py @@ -0,0 +1,988 @@ +"""Simulator Set D - 17 tasks, fixed set, killer.sh-style. + +Original scenarios, not copies of any commercial simulator. They exercise the techniques the +killer.sh CKA environment is known for that sets A-C do not already cover: saying how every +control-plane piece is run, cluster facts to an exact file, readiness gated on a Service, +`--sort-by` scripts, event forensics around kube-proxy with crictl, API-resource and RBAC census, +a kubelet whose systemd unit points at the wrong binary, Secrets mounted read-only and renamed into +env vars, a static Pod on the control plane behind a Service, kubelet client/serving certificate +forensics, control-plane-only scheduling, stopping the scheduler and scheduling by hand, Helm +release management (including a release stuck in pending-install), StatefulSet storage, the +namespaceSelector-AND-podSelector NetworkPolicy trap (enforced here - kindnet enforces), a +dnsPolicy that silently breaks name resolution, and a claim Pending on an access mode. + +Every value a grader compares against is derived from the live cluster, never hardcoded, and every +task was seeded, solved and graded on an isolated kind cluster before it went in. + +Run with: ./exam sim d +""" +from tracks.cka.common import C, NS + +Q = [] +CTX = "Use context: `kubectl config use-context %{ctx}`\n\n" + + +def task(**kw): + kw.setdefault("difficulty", "hard") + kw.setdefault("params", lambda r: {"ctx": "k8s-c1-H"}) + kw["task"] = CTX + kw["task"].lstrip("\n") + kw["set"] = "d" + Q.append(kw) + + +# ------------------------------------------------------------------ 501 ---- +task( + id=501, domain="ARCH", title="How the control plane is run", + task=""" +The operations handbook has to say how each part of this cluster's control plane is run. Write +`{WS}/q501/controlplane.txt` with exactly these six lines, each value one of `process`, +`static-pod`, `pod`, `deployment` or `daemonset`: + +``` +kubelet: +kube-apiserver: +kube-scheduler: +kube-controller-manager: +etcd: +dns: +``` + +`dns` is whatever serves cluster DNS, for example `dns: daemonset some-name`. +""", + setup=r"""rm -rf "$WS/q501"; mkdir -p "$WS/q501" """, + checks=[ + C("kubelet is a process", 1, r""" +f="$WS/q501/controlplane.txt"; [ -s "$f" ] || exit 1 +want=process; [ "$(nx "$CP" 'systemctl is-active kubelet' /dev/null 2>&1 || exit 1 +for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do + eq "$(sed -n "s/^ *$c *: *//p" "$f" | head -1 | tr -d '[:space:]' | tr A-Z a-z)" static-pod || exit 1 +done"""), + C("dns names the right kind of object and its name", 2, r""" +f="$WS/q501/controlplane.txt" +sel=$(j -n kube-system svc kube-dns '{.spec.selector.k8s-app}') +own=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].kind}') +name=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].name}') +if [ "$own" = ReplicaSet ]; then want="deployment $(j -n kube-system rs "$name" '{.metadata.ownerReferences[0].name}')" +else want="$(echo "$own" | tr A-Z a-z) $name"; fi +eq "$(sed -n 's/^ *dns *: *//p' "$f" | head -1 | tr -s ' ' | tr A-Z a-z | sed 's/ *$//')" "$want" """), + ], + solution=""" +{SHELL_CP} +systemctl status kubelet | head -3 # a systemd service: a plain process on the node +ls /etc/kubernetes/manifests # etcd, kube-apiserver, kube-controller-manager, kube-scheduler +exit +k -n kube-system get pod -o wide # static pods carry the node name as a suffix... +k -n kube-system get pod kube-apiserver-{CP} -o jsonpath='{.metadata.ownerReferences[0].kind}{"\\n"}' # ...and are owned by the Node +k -n kube-system get deploy,ds # coredns is a Deployment (kube-proxy, kindnet are DaemonSets) +cat > {WS}/q501/controlplane.txt <<'EOF' +kubelet: process +kube-apiserver: static-pod +kube-scheduler: static-pod +kube-controller-manager: static-pod +etcd: static-pod +dns: deployment coredns +EOF +# The Service is called kube-dns for historical reasons; what actually answers is the coredns +# Deployment. Follow the chain svc selector -> pod -> ownerReferences when unsure. +""", +) + +# ------------------------------------------------------------------ 502 ---- +task( + id=502, domain="TSHOOT", title="Cluster facts to an exact file", + task=""" +Write `{WS}/q502/cluster-info.txt` — answers only, exactly this format: + +``` +1: +2: +3: +4: , +5: +``` +""", + setup=r"""rm -rf "$WS/q502"; mkdir -p "$WS/q502" """, + checks=[ + C("node counts (lines 1 and 2)", 2, r""" +f="$WS/q502/cluster-info.txt"; [ -s "$f" ] || exit 1 +cp=$($K get nodes -l node-role.kubernetes.io/control-plane --no-headers 2>/dev/null | wc -l); all=$($K get nodes --no-headers | wc -l) +eq "$(sed -n 's/^ *1 *: *//p' "$f" | tr -d '[:space:]')" "$cp" && eq "$(sed -n 's/^ *2 *: *//p' "$f" | tr -d '[:space:]')" "$((all - cp))" """), + C("Service CIDR (line 3)", 2, r""" +sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2) +[ -n "$sc" ] && eq "$(sed -n 's/^ *3 *: *//p' "$WS/q502/cluster-info.txt" | tr -d '[:space:]')" "$sc" """), + C("CNI plugin and its config file (line 4)", 2, r""" +l=$(sed -n 's/^ *4 *: *//p' "$WS/q502/cluster-info.txt" | tr -d '[:space:]') +cni=$($K -n kube-system get ds -o name | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1) +cfg=$(nx "$W1" 'ls /etc/cni/net.d/*.conflist /etc/cni/net.d/*.conf 2>/dev/null | head -1' {WS}/q502/cluster-info.txt <<'EOF' +1: 1 +2: 3 +3: 10.96.0.0/16 +4: kindnet, /etc/cni/net.d/10-kindnet.conflist +5: -{W3} +EOF +# A static Pod's mirror is named -, e.g. kube-apiserver-{CP}. Values come from THIS +# cluster - check each one instead of trusting the ones shown here. +""", +) + +# ------------------------------------------------------------------ 503 ---- +task( + id=503, domain="NET", title="Ready only when its dependency answers", + task=""" +In Namespace `oslo-web`: + +1. Create Pod `gatekeeper` (image `busybox:1.36`, command `sleep 86400`) with an **exec readiness + probe** that runs `wget -T2 -qO- http://ledger-api:8080`, initial delay 5 seconds, period 5 + seconds. For now it must stay **not ready** — confirm that it does. +2. Deployment `ledger` serves HTTP on port 80, but nothing exposes it. Create the Service + `ledger-api` (ClusterIP, port **8080**) for it, so that `gatekeeper` turns Ready without you + touching the Pod again. +""", + setup=NS + r""" +mkns oslo-web +$K -n oslo-web delete pod gatekeeper --ignore-not-found --force --grace-period=0 >/dev/null 2>&1 +$K -n oslo-web delete svc ledger-api --ignore-not-found >/dev/null +$K -n oslo-web create deploy ledger --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n oslo-web rollout status deploy ledger --timeout=120s >/dev/null 2>&1 +""", + checks=[ + C("gatekeeper has the exec readiness probe (delay 5, period 5)", 2, r""" +p='{.spec.containers[0].readinessProbe' +j -n oslo-web pod gatekeeper "$p.exec.command}" | grep -q 'ledger-api:8080' \ + && eq "$(j -n oslo-web pod gatekeeper "$p.initialDelaySeconds}")" 5 && eq "$(j -n oslo-web pod gatekeeper "$p.periodSeconds}")" 5"""), + C("Service ledger-api: 8080 -> 80 with ready endpoints", 2, r""" +eq "$(j -n oslo-web svc ledger-api '{.spec.ports[0].port}')" 8080 && eq "$(j -n oslo-web svc ledger-api '{.spec.ports[0].targetPort}')" 80 \ + && [ -n "$(j -n oslo-web endpointslices -l kubernetes.io/service-name=ledger-api '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""), + C("gatekeeper is Ready", 3, r""" +for i in $(seq 1 20); do [ "$(j -n oslo-web pod gatekeeper '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 3; done; exit 1"""), + ], + solution=""" +k -n oslo-web run gatekeeper --image=busybox:1.36 $do -- sleep 86400 > gk.yaml +# add to the container: +# readinessProbe: +# exec: +# command: ["sh", "-c", "wget -T2 -qO- http://ledger-api:8080"] +# initialDelaySeconds: 5 +# periodSeconds: 5 +k apply -f gk.yaml +k -n oslo-web get pod gatekeeper # 0/1 Running - the probe cannot resolve ledger-api +k -n oslo-web describe pod gatekeeper | tail -3 # Readiness probe failed: wget: bad address 'ledger-api:8080' +k -n oslo-web expose deploy ledger --name=ledger-api --port=8080 --target-port=80 +k -n oslo-web get pod gatekeeper -w # 1/1 after the next probe +# Readiness removes a Pod from Service endpoints and blocks rollouts; liveness restarts it. +# A dependency check belongs in readiness - failing liveness here would just restart-loop the Pod. +""", +) + +# ------------------------------------------------------------------ 504 ---- +task( + id=504, domain="WORK", title="Sorting done by kubectl", + task=""" +Write two executable scripts that call `kubectl` (spelled out — aliases do not exist in a fresh +shell) and let **kubectl itself** do the sorting with `--sort-by`: + +1. `{WS}/q504/pods-by-age.sh` — every Pod in every Namespace, oldest first (by creation time). +2. `{WS}/q504/pods-by-uid.sh` — the Pods in Namespace `kube-system`, sorted by `metadata.uid`. +""", + setup=r"""rm -rf "$WS/q504"; mkdir -p "$WS/q504" """, + checks=[ + C("pods-by-age.sh: all namespaces, --sort-by creationTimestamp, runs", 2, r""" +f="$WS/q504/pods-by-age.sh"; [ -x "$f" ] || exit 1 +grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \ + && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.creationTimestamp" "$f" || exit 1 +out=$(timeout 60 bash "$f" 2>/dev/null); [ "$(echo "$out" | wc -l)" -ge 5 ] && echo "$out" | grep -q kube-system"""), + C("pods-by-uid.sh: kube-system, --sort-by metadata.uid, runs", 2, r""" +f="$WS/q504/pods-by-uid.sh"; [ -x "$f" ] || exit 1 +grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-n|--namespace)[= ]*kube-system' "$f" \ + && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.uid" "$f" || exit 1 +timeout 60 bash "$f" 2>/dev/null | grep -q etcd"""), + C("no aliases in either script", 1, r"""[ -s "$WS/q504/pods-by-age.sh" ] && [ -s "$WS/q504/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q504/pods-by-age.sh" "$WS/q504/pods-by-uid.sh" """), + ], + solution=""" +cat > {WS}/q504/pods-by-age.sh <<'EOF' +kubectl get pods -A --sort-by=.metadata.creationTimestamp +EOF +cat > {WS}/q504/pods-by-uid.sh <<'EOF' +kubectl -n kube-system get pods --sort-by=.metadata.uid +EOF +chmod +x {WS}/q504/*.sh +bash {WS}/q504/pods-by-age.sh | head -3 # the control-plane pods come first - they are oldest +# --sort-by takes a JSONPath into each item; ascending only, so "newest first" is `| tac` or +# sorting by a field that grows the other way. +""", +) + +# ------------------------------------------------------------------ 505 ---- +task( + id=505, domain="TSHOOT", title="Event forensics around kube-proxy", breaks="w2-kube-proxy", + task=""" +1. Write `{WS}/q505/cluster-events.sh`: a script that prints the latest events of the **whole** + cluster (all Namespaces) sorted by creation time, using `kubectl`. Make it executable. +2. Delete the kube-proxy Pod that runs on node `{W2}`. Write the events this caused to + `{WS}/q505/pod-deleted.log`. +3. On node `{W2}`, stop **and remove** the kube-proxy **container** with `crictl` — leave the Pod + object alone. Write the events this caused to `{WS}/q505/container-killed.log`. +4. Which of the two actions made the Pod's `RESTARTS` counter go up? Write `pod` or `container` to + `{WS}/q505/restarts.txt`. + +Do step 2 before step 3. +""", + setup=r""" +rm -rf "$WS/q505"; mkdir -p "$WS/q505" +$K -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 -o jsonpath='{.items[0].metadata.uid}' > "$WS/.q505-uid" +""", + checks=[ + C("cluster-events.sh: all namespaces, sorted by time, runs", 2, r""" +f="$WS/q505/cluster-events.sh"; [ -x "$f" ] || exit 1 +grep -Eq 'kubectl\b.*\bget\b.*\b(ev|events?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \ + && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?(metadata\.creationTimestamp|lastTimestamp|eventTime|firstTimestamp)" "$f" || exit 1 +timeout 60 bash "$f" >/dev/null 2>&1"""), + C("the kube-proxy Pod on {W2} was deleted and replaced", 2, r""" +old=$(cat "$WS/.q505-uid" 2>/dev/null); [ -n "$old" ] || exit 1 +now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}') +[ -n "$now" ] && [ "$now" != "$old" ] && eq "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.phase}')" Running"""), + # counted on the REPLACEMENT pod, which starts at 0 - after a node reboot the original pod + # often already shows a restart, which would otherwise hand these points out for free + C("then its container was killed (restart counter >= 1)", 2, r""" +old=$(cat "$WS/.q505-uid" 2>/dev/null); now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}') +[ -n "$old" ] && [ -n "$now" ] && [ "$now" != "$old" ] || exit 1 +[ "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null"""), + C("pod-deleted.log and container-killed.log hold the events", 2, r""" +for f in pod-deleted.log container-killed.log; do + grep -q 'kube-proxy' "$WS/q505/$f" 2>/dev/null && grep -Eq 'Killing|SuccessfulCreate|SuccessfulDelete|Scheduled|Pulled|Created|Started|BackOff' "$WS/q505/$f" || exit 1 +done"""), + C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q505/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""), + ], + solution=""" +cat > {WS}/q505/cluster-events.sh <<'EOF' +kubectl get events -A --sort-by=.metadata.creationTimestamp +EOF +chmod +x {WS}/q505/cluster-events.sh + +k -n kube-system get pod -l k8s-app=kube-proxy -o wide # the one on {W2} +k -n kube-system delete pod +k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q505/pod-deleted.log +# Killing (old pod), SuccessfulCreate (DaemonSet), Scheduled, Pulled, Created, Started (new pod) + +{SHELL_W2} +crictl ps --name kube-proxy # container id +crictl stop && crictl rm +exit +k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q505/container-killed.log +# Pulled, Created, Started - same Pod, new container +k -n kube-system get pod -l k8s-app=kube-proxy -o wide # RESTARTS 1 on {W2} +echo container > {WS}/q505/restarts.txt +# Deleting the Pod makes the DaemonSet create a brand-new Pod (RESTARTS 0). Killing only the +# container makes the kubelet start a new container inside the SAME Pod - that is a restart. +""", +) + +# ------------------------------------------------------------------ 506 ---- +def _census(r): + n = r.sample(range(2, 10), 4) + return {"ctx": "k8s-c1-H", "north": n[0], "south": n[1], "east": n[2], "west": n[3]} + + +task( + id=506, domain="ARCH", title="API resources and a Role census", params=_census, + task=""" +1. Write the names of every **namespaced** resource type this API server offers to + `{WS}/q506/namespaced.txt`, one per line — plural names, as `kubectl api-resources` prints them. +2. Of the Namespaces `census-north`, `census-south`, `census-east` and `census-west`, which one holds + the most Roles? Write ` ` to `{WS}/q506/most-roles.txt`. +""", + setup=NS + r""" +rm -rf "$WS/q506"; mkdir -p "$WS/q506" +for pair in north:%{north} south:%{south} east:%{east} west:%{west}; do + ns=census-${pair%%:*}; n=${pair##*:} + mkns $ns + $K -n $ns delete role --all >/dev/null 2>&1 + for i in $(seq 1 $n); do $K -n $ns create role reader-$i --verb=get,list --resource=configmaps >/dev/null; done +done +""", + checks=[ + C("namespaced.txt lists exactly the namespaced resource types", 3, r""" +f="$WS/q506/namespaced.txt"; [ -s "$f" ] || exit 1 +want=$($K api-resources --namespaced=true -o name 2>/dev/null | sed 's/\..*//' | sort -u) +got=$(awk '{print $1}' "$f" | sed 's/\..*//' | grep . | sort -u) +[ -n "$want" ] && [ "$got" = "$want" ]"""), + C("most-roles.txt names the namespace and its Role count", 3, r""" +best=$(for ns in census-north census-south census-east census-west; do echo "$ns $($K -n $ns get role --no-headers 2>/dev/null | wc -l)"; done | sort -k2 -n | tail -1) +eq "$(tr -s ' ' < "$WS/q506/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """), + ], + solution=""" +k api-resources --namespaced=true -o name > /tmp/r.txt # deployments.apps, ... (group suffix) +k api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q506/namespaced.txt +for ns in census-north census-south census-east census-west; do echo "$ns $(k -n $ns get role --no-headers | wc -l)"; done +# pick the largest, e.g. echo "census-east 8" > {WS}/q506/most-roles.txt +# --namespaced=false gives the cluster-scoped half (nodes, pv, storageclasses, clusterroles...). +# `k get role -A --no-headers | awk '{print $1}' | sort | uniq -c | sort -n` counts every +# namespace at once. +""", +) + +# ------------------------------------------------------------------ 507 ---- +task( + id=507, domain="TSHOOT", title="Kubelet unit points at nothing", breaks="w1-kubelet", disruptive=True, + task=""" +Node `{W1}` went `NotReady` after someone "tidied up" its kubelet installation. Bring it back to +`Ready`; the kubelet has to run as the systemd service it always was. Then write the full path of the +file you had to fix to `{WS}/q507/fixed-file.txt`. + +Reach the node with `{SHELL_W1}`. Pods on a node that stays NotReady are evicted after five minutes. +""", + setup=r""" +rm -rf "$WS/q507"; mkdir -p "$WS/q507" +nx "$W1" ' +d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf +grep -q "^ExecStart=/usr/bin/kubelet " $d && sed -i "s|^ExecStart=/usr/bin/kubelet |ExecStart=/usr/local/bin/kubelet |" $d +systemctl daemon-reload; systemctl restart kubelet' /dev/null 2>&1 +exit 0 +""", + checks=[ + C("{W1} is Ready", 4, r"""for i in $(seq 1 15); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 4; done; exit 1"""), + C("kubelet active, and its unit runs a kubelet binary that exists", 2, r""" +nx "$W1" 'systemctl is-active --quiet kubelet && b=$(sed -n "s|^ExecStart=\([^ ]*kubelet\) .*|\1|p" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | tail -1) && [ -x "$b" ]' /dev/null"""), + ], + solution=""" +k get nodes # {W1} NotReady +{SHELL_W1} +systemctl status kubelet +# Process: ... ExecStart=/usr/local/bin/kubelet ... (code=exited, status=203/EXEC) +journalctl -u kubelet -n 20 --no-pager # Unable to locate executable '/usr/local/bin/kubelet' +which kubelet # /usr/bin/kubelet - the binary was never moved +systemctl cat kubelet # shows the unit AND every drop-in, with file names +vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf # ExecStart=/usr/bin/kubelet ... +systemctl daemon-reload # systemd caches unit files - without this nothing changes +systemctl restart kubelet +systemctl status kubelet # active (running) +exit +echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q507/fixed-file.txt +k get nodes -w +# status=203/EXEC means systemd could not exec the binary at all - a path problem, not a kubelet +# configuration problem. `ExecStart=` (empty) in a drop-in clears the unit's own ExecStart first. +""", +) + +# ------------------------------------------------------------------ 508 ---- +task( + id=508, domain="WORK", title="Secrets as a read-only volume and renamed env vars", + params=lambda r: {"ctx": "k8s-c1-H", "pw": "ledger-" + "".join(r.choice("abcdefghjkmnpqrstuvwxyz23456789") for _ in range(10))}, + task=""" +In Namespace `oslo-secure`: + +1. Create Secret `bundle` from the file `{WS}/q508/tls-bundle.pem`; the key must be the file's name. +2. Create Secret `db-login` with `user=svc-ledger` and `pass=%{pw}`. +3. Create Pod `vault-reader` (image `busybox:1.36`, command `sleep 86400`) that + - mounts Secret `bundle` **read-only** at `/etc/bundle`, + - receives `db-login`'s keys as environment variables named **`APP_USER`** and **`APP_PASS`**. +""", + setup=NS + r""" +mkns oslo-secure +rm -rf "$WS/q508"; mkdir -p "$WS/q508" +openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=ledger.oslo.internal" -keyout /dev/null -out "$WS/q508/tls-bundle.pem" >/dev/null 2>&1 +$K -n oslo-secure delete pod vault-reader --ignore-not-found --force --grace-period=0 >/dev/null 2>&1 +$K -n oslo-secure delete secret bundle db-login --ignore-not-found >/dev/null +""", + checks=[ + C("Secrets bundle (key tls-bundle.pem, the file's content) and db-login", 2, r""" +[ "$(j -n oslo-secure secret bundle '{.data.tls-bundle\.pem}' | base64 -d 2>/dev/null)" = "$(cat "$WS/q508/tls-bundle.pem")" ] \ + && eq "$(j -n oslo-secure secret db-login '{.data.user}' | base64 -d)" svc-ledger && eq "$(j -n oslo-secure secret db-login '{.data.pass}' | base64 -d)" '%{pw}'"""), + C("vault-reader Running with bundle mounted read-only at /etc/bundle", 3, r""" +p="$K -n oslo-secure get pod vault-reader" +eq "$($p -o jsonpath='{.status.phase}' 2>/dev/null)" Running || exit 1 +v=$($p -o jsonpath='{.spec.volumes[?(@.secret.secretName=="bundle")].name}'); [ -n "$v" ] || exit 1 +eq "$($p -o jsonpath="{.spec.containers[0].volumeMounts[?(@.name==\"$v\")].mountPath}")" /etc/bundle \ + && eq "$($p -o jsonpath="{.spec.containers[0].volumeMounts[?(@.name==\"$v\")].readOnly}")" true \ + && $K -n oslo-secure exec vault-reader -- cat /etc/bundle/tls-bundle.pem 2>/dev/null | grep -q 'BEGIN CERTIFICATE'"""), + C("APP_USER / APP_PASS come from db-login", 3, r""" +e=$(j -n oslo-secure pod vault-reader '{.spec.containers[0].env}') +echo "$e" | grep -q '"name":"APP_USER"' && echo "$e" | grep -q '"name":"APP_PASS"' && echo "$e" | grep -q 'db-login' || exit 1 +env=$($K -n oslo-secure exec vault-reader -- env 2>/dev/null) +echo "$env" | grep -qx 'APP_USER=svc-ledger' && echo "$env" | grep -qx 'APP_PASS=%{pw}'"""), + ], + solution=""" +k -n oslo-secure create secret generic bundle --from-file={WS}/q508/tls-bundle.pem # key = file name +k -n oslo-secure create secret generic db-login --from-literal=user=svc-ledger --from-literal=pass='%{pw}' +k -n oslo-secure run vault-reader --image=busybox:1.36 $do -- sleep 86400 > vr.yaml +# in vr.yaml, pod level: +# volumes: [{name: bundle, secret: {secretName: bundle}}] +# in the container: +# volumeMounts: [{name: bundle, mountPath: /etc/bundle, readOnly: true}] +# env: +# - name: APP_USER +# valueFrom: {secretKeyRef: {name: db-login, key: user}} +# - name: APP_PASS +# valueFrom: {secretKeyRef: {name: db-login, key: pass}} +k apply -f vr.yaml +k -n oslo-secure exec vault-reader -- env | grep APP_ +k -n oslo-secure exec vault-reader -- ls /etc/bundle +# envFrom would inject the keys under their own names (user, pass); renaming needs env + secretKeyRef. +# A Secret volume updates in place when the Secret changes; env vars never do. +""", +) + +# ------------------------------------------------------------------ 509 ---- +task( + id=509, domain="ARCH", title="Static Pod on the control plane behind a Service", + task=""" +1. On the control-plane node (`{SHELL_CP}`) create a **static Pod** `ops-beacon` in Namespace + `default`: image `nginx:1.27`, requests `cpu: 10m` and `memory: 20Mi`, label `app=ops-beacon`. +2. Expose it with a Service `ops-beacon-svc` of type **NodePort** (port 80 → 80) whose endpoints + contain that static Pod. +3. Write the node port the Service was given to `{WS}/q509/nodeport.txt`. +""", + setup=r""" +rm -rf "$WS/q509"; mkdir -p "$WS/q509" +nx "$CP" 'rm -f /etc/kubernetes/manifests/ops-beacon*.yaml' /dev/null +# the kubelet drops the mirror pod on its next sync of the manifest directory +for i in $(seq 1 30); do $K -n default get pod "ops-beacon-$CP" >/dev/null 2>&1 || break; sleep 2; done +""", + checks=[ + C("manifest in the control-plane node's staticPodPath", 1, r""" +nx "$CP" 'd=$(awk "/^staticPodPath:/{print \$2}" /var/lib/kubelet/config.yaml); grep -ls "ops-beacon" "$d"/*.yaml' /dev/null 2>&1"""), + C("mirror Pod ops-beacon-{CP} Running with the requests and label", 3, r""" +p="$K -n default get pod ops-beacon-$CP" +eq "$($p -o jsonpath='{.status.phase}' 2>/dev/null)" Running && eq "$($p -o jsonpath='{.metadata.labels.app}')" ops-beacon \ + && eq "$($p -o jsonpath='{.spec.containers[0].resources.requests.cpu}')" 10m && eq "$($p -o jsonpath='{.spec.containers[0].resources.requests.memory}')" 20Mi"""), + C("NodePort Service ops-beacon-svc 80 -> 80 with the static Pod as endpoint", 3, r""" +s="$K -n default get svc ops-beacon-svc" +eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" NodePort && eq "$($s -o jsonpath='{.spec.ports[0].port}')" 80 && eq "$($s -o jsonpath='{.spec.ports[0].targetPort}')" 80 || exit 1 +ip=$(j -n default pod "ops-beacon-$CP" '{.status.podIP}'); [ -n "$ip" ] || exit 1 +j -n default endpointslices -l kubernetes.io/service-name=ops-beacon-svc '{.items[*].endpoints[*].addresses[*]}' | tr ' ' '\n' | grep -qx "$ip" """), + C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q509/nodeport.txt" 2>/dev/null)" "$np" """), + ], + solution=""" +k run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon $do > beacon.yaml +# add to the container: resources: {requests: {cpu: 10m, memory: 20Mi}} +{SHELL_CP} +grep staticPodPath /var/lib/kubelet/config.yaml # /etc/kubernetes/manifests +vi /etc/kubernetes/manifests/ops-beacon.yaml # paste beacon.yaml +exit +k -n default get pod ops-beacon-{CP} -o wide # mirror pod, named - +k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80 +k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc # the mirror pod's IP +k -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q509/nodeport.txt +# The control-plane taint only keeps the SCHEDULER from placing pods there; the kubelet runs its +# static pods regardless. `expose pod` copies the pod's labels into the selector. +""", +) + +# ------------------------------------------------------------------ 510 ---- +task( + id=510, domain="ARCH", title="The kubelet's two certificates", + task=""" +On node `{W2}` find the certificate the kubelet uses as a **client** (to talk to the API server) +and the one it **serves** its own API with (port 10250). Write exactly this to +`{WS}/q510/kubelet-certs.txt`: + +``` +client-issuer: +client-eku: +server-issuer: +server-eku: +``` + +Reach the node with `{SHELL_W2}`. +""", + setup=r"""rm -rf "$WS/q510"; mkdir -p "$WS/q510" """, + checks=[ + C("client certificate: issuer and extended key usage", 3, r""" +f="$WS/q510/kubelet-certs.txt"; [ -s "$f" ] || exit 1 +iss=$(nx "$W2" 'openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -issuer' Extended Key Usage: TLS Web Server Authentication +exit +# write the four lines with the values YOUR openssl printed: +# client-issuer: CN=kubernetes +# client-eku: TLS Web Client Authentication +# server-issuer: CN={W2}-ca@ +# server-eku: TLS Web Server Authentication +# The client cert is signed by the cluster CA and rotated by the kubelet through CSRs. The serving +# cert here is self-signed by the kubelet at startup (no serverTLSBootstrap) - which is exactly why +# metrics-server needs --kubelet-insecure-tls on this cluster. +""", +) + +# ------------------------------------------------------------------ 511 ---- +task( + id=511, domain="WORK", title="A Pod for the control plane only", + task=""" +Create Pod `cp-watch` in Namespace `default`: image `httpd:2-alpine`, container name `watch`. + +It must be able to run **only** on control-plane nodes. Select them by the node label kubeadm puts +on control-plane nodes — not by host name, and not with `nodeName` — and tolerate whatever keeps +ordinary Pods off them. It must be Running. +""", + setup=r"""$K -n default delete pod cp-watch --ignore-not-found --force --grace-period=0 >/dev/null 2>&1""", + checks=[ + C("cp-watch Running on the control-plane node, container named watch", 2, r""" +eq "$(j -n default pod cp-watch '{.status.phase}')" Running && eq "$(j -n default pod cp-watch '{.spec.nodeName}')" "$CP" && eq "$(j -n default pod cp-watch '{.spec.containers[0].name}')" watch"""), + C("selected by the node-role.kubernetes.io/control-plane label", 2, r""" +s=$(j -n default pod cp-watch '{.spec.nodeSelector}{.spec.affinity.nodeAffinity}') +echo "$s" | grep -q 'node-role.kubernetes.io/control-plane' && ! echo "$s" | grep -q 'kubernetes.io/hostname'"""), + C("tolerates the control-plane taint, no nodeName in the manifest", 2, r""" +$K -n default get pod cp-watch -o json 2>/dev/null | python3 -c ' +import sys,json; t=json.load(sys.stdin)["spec"].get("tolerations") or [] +sys.exit(0 if any("control-plane" in (x.get("key") or "") or (x.get("operator")=="Exists" and not x.get("key")) for x in t) else 1)' || exit 1 +$K -n default get pod cp-watch -o json --show-managed-fields | python3 -c ' +import sys,json +mf=json.load(sys.stdin)["metadata"].get("managedFields",[]) +sys.exit(1 if any("f:nodeName" in json.dumps(m.get("fieldsV1",{})) for m in mf) else 0)'"""), + ], + solution=""" +k get node {CP} --show-labels | tr ',' '\\n' | grep node-role # node-role.kubernetes.io/control-plane= +k describe node {CP} | grep Taints # node-role.kubernetes.io/control-plane:NoSchedule +k -n default run cp-watch --image=httpd:2-alpine $do > cp.yaml +# rename the container to watch, then add at pod level: +# nodeSelector: +# node-role.kubernetes.io/control-plane: "" +# tolerations: +# - {key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule} +k apply -f cp.yaml +k -n default get pod cp-watch -o wide +# The toleration only ALLOWS the control plane; the nodeSelector is what REQUIRES it. You need both. +""", +) + +# ------------------------------------------------------------------ 512 ---- +task( + id=512, domain="TSHOOT", title="Stop the scheduler, schedule by hand", breaks="scheduler", + task=""" +1. Stop the kube-scheduler **temporarily** — it runs as a static Pod on the control-plane node — and + confirm it is gone. +2. Create Pod `manual-1` (image `busybox:1.36`, command `sleep 86400`) in Namespace `default` and + confirm it stays `Pending`. +3. Schedule `manual-1` **by hand** onto the control-plane node and get it Running. +4. Start the kube-scheduler again, then create Pod `manual-2` (same image and command) in `default` + and confirm that the scheduler places it. +""", + setup=r""" +# a pristine copy outside the static pod directory, so the pre-session baseline can put the +# scheduler back if this task is abandoned half-done +nx "$CP" 'mkdir -p /etc/kubernetes/.cka-sim && [ -f /etc/kubernetes/manifests/kube-scheduler.yaml ] && cp /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/.cka-sim/kube-scheduler.yaml' /dev/null 2>&1 +exit 0 +""", + checks=[ + C("kube-scheduler is back and Running", 2, r""" +$K -n default get pod manual-1 >/dev/null 2>&1 || exit 1 # only once the task was actually done +nx "$CP" 'test -f /etc/kubernetes/manifests/kube-scheduler.yaml' m1.yaml +# add under spec: nodeName: {CP} +k replace --force -f m1.yaml +k -n default get pod manual-1 -o wide # Running on {CP} +{SHELL_CP} +mv /etc/kubernetes/kube-scheduler.yaml /etc/kubernetes/manifests/ +exit +k -n kube-system get pod -l component=kube-scheduler -w # Running again +k -n default run manual-2 --image=busybox:1.36 -- sleep 86400 +k -n default get pod manual-2 -o wide # a worker, chosen by the scheduler +# nodeName skips the scheduler entirely - including its taint checks, which is why manual-1 can +# sit on the tainted control plane. (The Binding API does the same without recreating the pod.) +""", +) + +# ------------------------------------------------------------------ 513 ---- +_CHART = r''' +mkchart() { # dir name version image replicas + mkdir -p "$1/templates" + printf 'apiVersion: v2\nname: %s\nversion: %s\nappVersion: "%s"\n' "$2" "$3" "${4##*:}" > "$1/Chart.yaml" + printf 'replicaCount: %s\nimage: %s\n' "$5" "$4" > "$1/values.yaml" + cat > "$1/templates/deployment.yaml" <<'YAML' +apiVersion: apps/v1 +kind: Deployment +metadata: {name: {{ .Release.Name }}, labels: {app: {{ .Release.Name }}}} +spec: + replicas: {{ .Values.replicaCount }} + selector: {matchLabels: {app: {{ .Release.Name }}}} + template: + metadata: {labels: {app: {{ .Release.Name }}}} + spec: + containers: [{name: main, image: "{{ .Values.image }}"}] +YAML +} +''' + +task( + id=513, domain="ARCH", title="Helm release housekeeping", + task=""" +Namespace `oslo-helm` holds several Helm releases; local charts are under `{WS}/q513/charts`. + +1. Delete the release `web-legacy`. +2. Upgrade the release `web-main` to chart version **1.1.0** (`{WS}/q513/charts/nginx-lite-1.1.0`). +3. Install a new release `cache` from `{WS}/q513/charts/redis-lite` with **2** replicas (the chart's + `replicaCount` value). +4. One release in the Namespace never finished installing. Find it and remove it. +""", + setup=NS + _CHART + r""" +mkns oslo-helm +C="$WS/q513/charts"; rm -rf "$WS/q513"; mkdir -p "$C" +mkchart "$C/nginx-lite-1.0.0" nginx-lite 1.0.0 nginx:1.25 1 +mkchart "$C/nginx-lite-1.1.0" nginx-lite 1.1.0 nginx:1.27 1 +mkchart "$C/redis-lite" redis-lite 0.3.0 redis:7-alpine 1 +mkchart /tmp/q513-stuck cache-edge 0.1.0 busybox:0.0-does-not-exist 1 +for r in $(helm -n oslo-helm list -q 2>/dev/null); do helm -n oslo-helm uninstall "$r" >/dev/null 2>&1; done +helm -n oslo-helm install web-legacy "$C/nginx-lite-1.0.0" >/dev/null +helm -n oslo-helm install web-main "$C/nginx-lite-1.0.0" >/dev/null +# helm records the release as pending-install BEFORE it starts waiting; kill it mid-wait and it +# stays that way - the classic result of an interrupted install or a CI job that timed out +timeout -s KILL 6 helm -n oslo-helm install cache-broken /tmp/q513-stuck --wait --timeout 120s >/dev/null 2>&1 +rm -rf /tmp/q513-stuck +helm -n oslo-helm status cache-broken -o json 2>/dev/null | grep -q '"status":"pending-install"' || echo "warn: cache-broken did not end up pending-install" +""", + checks=[ + C("web-legacy is gone", 1, r"""! helm -n oslo-helm status web-legacy >/dev/null 2>&1 && helm -n oslo-helm status web-main >/dev/null 2>&1"""), + C("web-main deployed at chart version 1.1.0", 2, r""" +helm -n oslo-helm list -o json 2>/dev/null | python3 -c ' +import sys,json +r={x["name"]:x for x in json.load(sys.stdin)}.get("web-main") or {} +sys.exit(0 if r.get("chart")=="nginx-lite-1.1.0" and r.get("status")=="deployed" and int(r.get("revision",0))>=2 else 1)'"""), + C("cache deployed from redis-lite with 2 available replicas", 2, r""" +helm -n oslo-helm list -o json 2>/dev/null | python3 -c ' +import sys,json +r={x["name"]:x for x in json.load(sys.stdin)}.get("cache") or {} +sys.exit(0 if r.get("chart","").startswith("redis-lite-") and r.get("status")=="deployed" else 1)' || exit 1 +for i in $(seq 1 20); do [ "$(j -n oslo-helm deploy cache '{.status.availableReplicas}')" = 2 ] && exit 0; sleep 3; done; exit 1"""), + C("the stuck release is removed; nothing pending remains", 2, r""" +! helm -n oslo-helm status cache-broken >/dev/null 2>&1 && [ -z "$(helm -n oslo-helm list --pending -q 2>/dev/null)" ] && ! $K -n oslo-helm get deploy cache-broken >/dev/null 2>&1"""), + ], + solution=""" +helm -n oslo-helm list # Helm 4 shows every status; Helm 3 hides pending ones +helm -n oslo-helm list --pending # works in both: cache-broken pending-install +helm -n oslo-helm uninstall web-legacy +helm -n oslo-helm upgrade web-main {WS}/q513/charts/nginx-lite-1.1.0 +helm -n oslo-helm install cache {WS}/q513/charts/redis-lite --set replicaCount=2 +helm -n oslo-helm uninstall cache-broken # also removes the Deployment it half-created +helm -n oslo-helm list # web-main (1.1.0, rev 2) and cache, both deployed +# A pending-install release blocks a plain re-install under the same name ("cannot re-use a name +# that is still in use"). `helm history ` shows when it got stuck; uninstall clears it. +""", +) + +# ------------------------------------------------------------------ 514 ---- +task( + id=514, domain="STOR", title="Per-replica storage for a StatefulSet", + task=""" +In Namespace `lima-data`: + +1. Create a headless Service `ledger-db` (port 5432) and a StatefulSet `ledger-db` with **3** + replicas: image `busybox:1.36`, command `sh -c 'hostname > /data/id.txt && sleep 86400'`, and a + `volumeClaimTemplates` entry `data` — 100Mi, ReadWriteOnce, StorageClass `standard` — mounted at + `/data`. +2. Once all 3 are Ready, scale the StatefulSet down to **2**. +3. Write the names of the PersistentVolumeClaims that belong to the StatefulSet and still exist + after the scale-down to `{WS}/q514/pvcs.txt`, one per line, sorted. +""", + setup=NS + r""" +mkns lima-data +rm -rf "$WS/q514"; mkdir -p "$WS/q514" +$K -n lima-data delete sts ledger-db --ignore-not-found --wait=true >/dev/null 2>&1 +$K -n lima-data delete svc ledger-db --ignore-not-found >/dev/null +$K -n lima-data delete pvc -l app=ledger-db --ignore-not-found >/dev/null 2>&1 +for i in 0 1 2 3 4; do $K -n lima-data delete pvc data-ledger-db-$i --ignore-not-found --wait=false >/dev/null 2>&1; done +""", + checks=[ + C("headless Service and a StatefulSet with the right claim template", 2, r""" +eq "$(j -n lima-data svc ledger-db '{.spec.clusterIP}')" None && eq "$(j -n lima-data sts ledger-db '{.spec.serviceName}')" ledger-db || exit 1 +t='{.spec.volumeClaimTemplates[?(@.metadata.name=="data")]' +eq "$(j -n lima-data sts ledger-db "$t.spec.resources.requests.storage}")" 100Mi && eq "$(j -n lima-data sts ledger-db "$t.spec.storageClassName}")" standard \ + && j -n lima-data sts ledger-db "$t.spec.accessModes}" | grep -q ReadWriteOnce"""), + C("scaled to 2, both Ready", 2, r"""eq "$(j -n lima-data sts ledger-db '{.spec.replicas}')" 2 && eq "$(j -n lima-data sts ledger-db '{.status.readyReplicas}')" 2"""), + C("each replica wrote its own identity to its own volume", 2, r""" +for i in 0 1; do eq "$($K -n lima-data exec ledger-db-$i -- cat /data/id.txt 2>/dev/null | tr -d '[:space:]')" "ledger-db-$i" || exit 1; done +[ "$(j -n lima-data pod ledger-db-0 '{.spec.volumes[?(@.name=="data")].persistentVolumeClaim.claimName}')" = data-ledger-db-0 ]"""), + C("pvcs.txt lists every claim that survived the scale-down", 2, r""" +want=$($K -n lima-data get pvc --no-headers -o custom-columns=N:.metadata.name 2>/dev/null | grep '^data-ledger-db-' | sort) +[ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q514/pvcs.txt" 2>/dev/null)" = "$want" ]"""), + ], + solution=""" +cat < /data/id.txt && sleep 86400"] + volumeMounts: [{name: data, mountPath: /data}] + volumeClaimTemplates: + - metadata: {name: data} + spec: {accessModes: [ReadWriteOnce], storageClassName: standard, resources: {requests: {storage: 100Mi}}} +EOF +k -n lima-data rollout status sts ledger-db +k -n lima-data scale sts ledger-db --replicas=2 +k -n lima-data get pvc -o custom-columns=N:.metadata.name --no-headers | grep '^data-ledger-db-' | sort > {WS}/q514/pvcs.txt +cat {WS}/q514/pvcs.txt # all THREE: data-ledger-db-0, -1, -2 +# Claims are named