diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..3f69c2c --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +.state/ +*.pyc +__pycache__/ +.DS_Store diff --git a/README.md b/README.md index 4351f76..d9c3bba 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,137 @@ -# cka-sim +# cka-sim — Kubernetes certification exam simulator +Entry point is `./exam` (renamed from `cka-sim` so it can't collide with the directory name). + +Hands-on exam tasks graded **per check** against a real cluster, an exam clock, a browser +front end modelled on the real PSI exam, attempt history, and drills that rotate through the +domains before you sit a full exam. Tracks: **CKA** (65-task bank: 43 normal + 22 hard), +CKAD and CKS (scaffolded — same format, add questions when you get there). + +## How a session is built + +Every session draws **fresh** tasks: `start` samples per domain in the exam's proportions +(ARCH 5, WORK 4, NET 4, STOR 2, TSHOOT 5 = 20), preferring tasks you have never seen, then the +least-recently seen. Most tasks are parameterized — namespaces, names, replica counts, ports, +which node is broken — so even a repeat isn't the same task. `drill` with no arguments picks the +domain you've drilled least and gives you 5 tasks from it; a full exam is gated until every +domain has been drilled at least once (`--force` skips that). + +**Hard mode** (`--hard`, or the toggle in the UI lobby) is modelled on what people report about +killer.sh, not on its content: 25 tasks in the same 120 minutes, hard tasks preferred, no hints +about paths or nodes, traps (an over-granting binding you must find, an allow-all NetworkPolicy +left behind, a relocated `staticPodPath`, two PVs where only one is right), exact-output tasks +graded byte-for-byte, compound failures (three faults on one node, two control-plane components +down for different reasons, DNS broken three ways), collateral-damage checks (still cordoned, +recreated instead of fixed in place, extra PVs), and exam-style contexts `k8s-c1-H` / +`k8s-c2-AC` / `k8s-c3-CCC` with different default namespaces — each task says which to use. +Grading always runs against the real cluster context regardless of which one you're on. + +## Why kind and not k3s / hosted labs + +kind is kubeadm-based: real etcd, static-pod control plane, systemd kubelet on every node, +`kubeadm certs` on the control plane. That is what lets the troubleshooting tasks break real +things (kubelet, scheduler manifest) the way the exam does. Hosted labs and CK-X give you no +node access, which rules out exactly those tasks. `podman exec -it bash` plays the role +of `ssh ` here; use the Vagrant/ssh mode below if you want a real ssh session. + +## Requirements + +podman, kind, kubectl, python3, **ttyd** (for the terminal pane in the browser UI). +~4 GB RAM for the 4-node cluster. Rootless podman needs cgroup v2; otherwise `sudo -E ./setup.sh`. +Docker still works: `CONTAINER_RUNTIME=docker ./setup.sh`. + +## Run + +``` +./setup.sh # create the kind cluster, preload images (idempotent) +./exam web # open http://127.0.0.1:8080 — start an exam or drill from there +``` + +Both the built-in terminal and your own shell can have the exam aliases: + +``` +source ./examrc # bash: k=kubectl, $do, $now, kubectl completion (also for k), vim defaults +source ./examrc.fish # fish equivalent +``` + +`./exam web` sources the right one automatically for its terminal pane. + +That's the whole loop. The UI seeds the scenarios, runs the clock, lets you flag questions and +keep notes, submits (or auto-submits at 00:00), shows the score with every check, and offers +"Drill what I missed". The terminal pane is a real `bash` via ttyd with kubectl already pointed +at the cluster (`--shell fish` if you'd rather practise in fish; the real exam is bash). + +The same state is available from the terminal if you prefer: + +``` +./exam drill # next least-drilled domain, 5 tasks (add --hard) +./exam drill weak | drill 2 17 20 | drill TSHOOT NET +./exam start [--hard] # full exam (gated until all domains drilled; --force) +./exam list | show N | timer | status # during +./exam end # submit +./exam weak | history # afterwards +./exam --track ckad web # other tracks +./teardown.sh # delete cluster + state (history kept) +``` + +## Scoring, history, drills + +Each task is a list of named, weighted checks (`tracks/cka/questions.py`). You get the points +for every check that passes, so a task can score 4/6 — the same partial-credit model as the +exam. Every submit appends to `history-.jsonl` and regenerates `weak-spots-.md`: +latest domain scores, a "drill next" section listing the *failing checks* per task, and a +✓ / n-of-m matrix across all attempts. + +A task counts as weak if it failed most recently, has failed twice ever, or was never +attempted. Drills seed only those tasks, give 6 minutes per task, and — unlike exam mode — +let you check a task while you work. A task stops being weak only when you pass it again +under the clock. + +## Using your own cluster instead of kind + +Any kubeadm-style cluster with 1 control-plane + 3 workers works. Skip `setup.sh`, set (Fish): + +``` +set -x CKA_CONTEXT kubernetes-admin@kubernetes +set -x CKA_NODE_ACCESS ssh # podman | docker | ssh | vagrant +set -x CKA_CP cp1; set -x CKA_W1 node1; set -x CKA_W2 node2; set -x CKA_W3 node3 +./exam web +``` + +`ssh` needs passwordless `ssh ` (for Vagrant: `vagrant ssh-config >> ~/.ssh/config`); +`vagrant` runs `vagrant ssh` from the Vagrantfile directory. Tasks 2 and 20 write to +`/opt/cka` and `/etc/kubernetes/manifests`, so the login must be able to. Managed clusters +(EKS/AKS/GKE) won't work — no node or control-plane access. Startup preflights all of this. + +## CKA bank + +| Domain | Normal (43) | Hard (22) | +|---|---|---| +| Architecture | RBAC, etcd snapshot, Kustomize, CRD, static pod, certs/CSR, Helm, aggregated ClusterRole, `kubectl explain` on a CRD, kubeadm join token, component versions | exact RBAC with an over-grant to remove, **etcd restore**, static pod on a relocated path, user → self-contained kubeconfig, Kustomize generators + patches, CRD validation/defaults, renew all control-plane certs | +| Workloads | rolling update, sidecar, node affinity + resources, ConfigMap/Secret, metrics-server repair + HPA, PriorityClass, taints, DaemonSet, CronJob, init container, immutable ConfigMap rotation, parallel Job, quota + LimitRange | anti-affinity + toleration + priority under a quota, rollback to a specific revision then pause, native sidecar + probes + exact report, cluster audit to exact files, work in the context's namespace | +| Networking | NetworkPolicy, NodePort, Ingress, cluster DNS, Gateway API, named ports, egress-only policy, headless + ExternalName | zero-trust namespace (4 policies + trap), TLS Ingress with generated cert + fingerprint, Gateway canary with header match and weights, DNS down three ways | +| Storage | PV/PVC/pod, default StorageClass, re-bind a Released PV, data survives the pod | bind the right local volume by selector, migrate data between claims via a Job | +| Troubleshooting | node NotReady, broken deployment + service, CrashLoopBackOff, scheduler down, CoreDNS misconfig, controller-manager down, kubelet config drift | worker with three faults (+ uncordon check), two control-plane components down, app down for four reasons fixed in place (UID check), cluster facts exactly | + +Disruptive tasks (anything that breaks a node or control-plane component) are seeded last and +can block other tasks until fixed — read everything first, as in the real exam. kindnet doesn't +enforce NetworkPolicy and no ingress/gateway controller runs, so those are graded on the objects. + +## Adding tasks or a new track + +Banks live in `tracks/cka/{questions,bank_extra,bank_hard}.py`; a task is: + +```python +{"id": 44, "domain": "NET", "difficulty": "normal", # or "hard" + "params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])}, + "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...", + "setup": "bash that seeds the scenario (may use %{ns})", + "checks": [C("what this check proves", 2, "bash; exit 0 = pass"), ...], + "disruptive": False} +``` + +Scripts get `$K` (pinned to the grading context), `$WS`, `$CP/$W1/$W2/$W3`, `nx NODE 'cmd'`, +and helpers `j '{jsonpath}'` and `eq a b`. Task text may use `{WS}`, `{CP}`, `{W2}`, +`{W3}`, `{SHELL_CP}` etc. `TRACK["quota"]` / `["quota_hard"]` set how many tasks per domain a +session draws. For CKAD/CKS, fill in `QUESTIONS` in the +existing placeholder track; `TRACK` already carries the domain table and pass mark. diff --git a/exam b/exam new file mode 100755 index 0000000..97d19af --- /dev/null +++ b/exam @@ -0,0 +1,1105 @@ +#!/usr/bin/env python3 +"""exam - Kubernetes certification exam simulator (CKA / CKAD / CKS tracks). +Hands-on tasks graded per-check against the live cluster, exam clock, attempt history, +weak-spot log, drill mode, and a browser front end modelled on the real exam.""" +import http.server +import importlib +import json +import os +import shutil +import signal +import subprocess +import sys +import threading +import time +import urllib.parse +from collections.abc import Callable +from datetime import datetime +from typing import Any + +VERSION = "2026.09.06.14" +ROOT = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, ROOT) + +# ------------------------------------------------------------ track ---- +ARGV = sys.argv[1:] +TRACKS = ["cka", "ckad", "cks"] +TRACK_ID = "" +TRACK = {} +QUESTIONS = [] +DOMAINS = {} +QUOTA = {} +QUOTA_HARD = {} +QBYID = {} +SETS = {} +POOL = [] +EXAM_SECONDS = 0 +PASS_PCT = 0 +STATE_FILE = "" +HISTORY_FILE = "" +WEAK_FILE = "" + + +def load_track(track_id): + """Bind every track-dependent global. Called at startup and when the UI switches track.""" + try: + mod = importlib.import_module(f"tracks.{track_id}.questions") + except ModuleNotFoundError: + print(f"unknown track '{track_id}' (no tracks/{track_id}/questions.py)"); sys.exit(1) + importlib.reload(mod) + track, questions = mod.TRACK, list(mod.QUESTIONS) + for q in questions: + q["weight"] = sum(c["pts"] for c in q["checks"]) + q.setdefault("difficulty", "normal") + q.setdefault("set", None) + g = globals() + g.update( + TRACK_ID=track_id, TRACK=track, QUESTIONS=questions, DOMAINS=track["domains"], + QUOTA=track.get("quota", {}), QUOTA_HARD=track.get("quota_hard", track.get("quota", {})), + QBYID={q["id"]: q for q in questions}, + SETS=track.get("sets", {}), + POOL=[q for q in questions if not q.get("set")], + EXAM_SECONDS=track["minutes"] * 60, PASS_PCT=track["pass_pct"], + STATE_FILE=os.path.join(ROOT, ".state", f"state-{track_id}.json"), + HISTORY_FILE=os.path.join(ROOT, f"history-{track_id}.jsonl"), + WEAK_FILE=os.path.join(ROOT, f"weak-spots-{track_id}.md"), + ) + + +def set_questions(name): + return [q for q in QUESTIONS if q.get("set") == name] + + +STATE_DIR = os.path.join(ROOT, ".state") +WORKSPACE = os.path.join(ROOT, "workspace") +# How {WS} is shown in task text and solutions. Relative to the repo root by default, since that +# is where you are working; graders always use the absolute $WS internally. +WS_DISPLAY = os.environ.get("CKA_WS_DISPLAY") or os.path.relpath(WORKSPACE, ROOT) +DRILL_SECONDS_PER_TASK = 6 * 60 +DRILL_SECONDS_PER_TASK_HARD = 5 * 60 +DRILL_SIZE = 5 +import random +NAMESPACES_TO_SETTLE = ["apps", "netpol", "svc", "ingress-app", "dns-check", "storage"] + +# --- cluster targeting (override with env vars to use a non-kind cluster) --- +CONTEXT = os.environ.get("CKA_CONTEXT", "kind-cka-sim") +def _default_access(): + try: + return open(os.path.join(STATE_DIR, "runtime")).read().strip() # written by setup.sh + except OSError: + return "podman" +NODE_ACCESS = os.environ.get("CKA_NODE_ACCESS", _default_access()) # podman | docker | ssh | vagrant +NODES = { + "CP": os.environ.get("CKA_CP", "cka-sim-control-plane"), + "W1": os.environ.get("CKA_W1", "cka-sim-worker"), + "W2": os.environ.get("CKA_W2", "cka-sim-worker2"), + "W3": os.environ.get("CKA_W3", "cka-sim-worker3"), +} +def shell_hint(node): + return {"podman": f"podman exec -it {node} bash", + "docker": f"docker exec -it {node} bash", + "ssh": f"ssh {node}", + "vagrant": f"vagrant ssh {node}"}.get(NODE_ACCESS, f"ssh {node}") + +PRELUDE = r""" +set -uo pipefail +K="kubectl --context ${CKA_CONTEXT:-kind-cka-sim}" # grading never depends on the candidate's current context +WS="$WORKSPACE" +# j [kubectl get args...] '{jsonpath}' -> prints the jsonpath result ("" on error) +j() { local n=$#; local path="${!n}"; $K get "${@:1:n-1}" -o "jsonpath=$path" 2>/dev/null; } +eq() { [ "$1" = "$2" ]; } +CP="${CKA_CP:-cka-sim-control-plane}"; W1="${CKA_W1:-cka-sim-worker}" +W2="${CKA_W2:-cka-sim-worker2}"; W3="${CKA_W3:-cka-sim-worker3}" +# nx NODE 'command' -> run a shell command on a node (stdin is forwarded) +nx() { + local n="$1"; shift + case "${CKA_NODE_ACCESS:-podman}" in + ssh) ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new "$n" "$@" ;; + vagrant) vagrant ssh "$n" -c "$*" -- -o LogLevel=QUIET ;; + docker) docker exec -i "$n" sh -c "$*" ;; + *) podman exec -i "$n" sh -c "$*" ;; + esac +} +""" + +def _pick_track_arg(argv): + t = os.environ.get("CKA_TRACK", "cka") + if "--track" in argv: + i = argv.index("--track"); t = argv[i + 1]; del argv[i:i + 2] + return t + + +load_track(_pick_track_arg(ARGV)) + +TTY = sys.stdout.isatty() +def c(code, s): return f"\033[{code}m{s}\033[0m" if TTY else s +def green(s): return c("32", s) +def red(s): return c("31", s) +def yellow(s): return c("33", s) +def bold(s): return c("1", s) +def dim(s): return c("2", s) +def die(msg, code=1): + print(red(msg)); sys.exit(code) + +# ------------------------------------------------------------- state ---- +def load_state() -> dict[str, Any] | None: + if not os.path.exists(STATE_FILE): + return None + with open(STATE_FILE) as f: + return json.load(f) + +def save_state(st: dict[str, Any]) -> None: + os.makedirs(STATE_DIR, exist_ok=True) + with open(STATE_FILE, "w") as f: + json.dump(st, f, indent=2) + +def paused_total(st: dict[str, Any] | None) -> float: + """Seconds spent paused so far, including the current pause if any.""" + if not st: + return 0.0 + t = float(st.get("paused_total", 0)) + if st.get("paused_at"): + t += time.time() - float(st["paused_at"]) + return t + +def untimed(st: dict[str, Any] | None) -> bool: + return bool(st and st.get("untimed")) + +def remaining(st: dict[str, Any] | None) -> float: + if not st: + return 0.0 + if untimed(st): + return float("inf") + return float(st.get("started_at", 0)) + float(st.get("duration", EXAM_SECONDS)) + paused_total(st) - time.time() + +def set_paused(st: dict[str, Any], on: bool) -> None: + if on and not st.get("paused_at"): + st["paused_at"] = time.time(); st["pauses"] = int(st.get("pauses", 0)) + 1 + elif not on and st.get("paused_at"): + st["paused_total"] = float(st.get("paused_total", 0)) + (time.time() - float(st["paused_at"])); st["paused_at"] = None + save_state(st) + +def fmt(secs: float | int) -> str: + secs_int = max(0, int(secs)) + return f"{secs_int // 3600:02d}:{(secs_int % 3600) // 60:02d}:{secs_int % 60:02d}" + +def require_started() -> dict[str, Any]: + st = load_state() + if not st: + die("No session in progress. Run: ./exam start (or ./exam drill weak)") + assert st is not None + return st + +def session_questions(st: dict[str, Any] | None) -> list[dict[str, Any]]: + if not st: + return list(QUESTIONS) + ids = st.get("ids") or [q["id"] for q in QUESTIONS] + return [q for q in QUESTIONS if q["id"] in ids] + +def q_by_id(n: int) -> dict[str, Any]: + for q in QUESTIONS: + if q["id"] == n: + return q + die(f"No question {n}. Valid: 1-{len(QUESTIONS)}") + return {} + +def render(text, params): + """Substitute %{param}, {WS}, {CP}.. and {SHELL_CP}.. placeholders.""" + t = text.replace("{WS}", WS_DISPLAY) + for k, v in (params or {}).items(): + t = t.replace("%{" + k + "}", str(v)) + for k, v in NODES.items(): + t = t.replace("{" + k + "}", v).replace("{SHELL_" + k + "}", shell_hint(v)) + return t + +def session_params(st, q): + """Params drawn for this session; outside a session, a fixed sample so text still reads naturally.""" + p = (st or {}).get("params", {}).get(str(q["id"])) + return p if p is not None else make_params(q, random.Random(q["id"])) + +def task_text(q, params=None): + return render(q["task"].strip(), params if params is not None else session_params(load_state(), q)) + +def make_params(q, rng): + fn = q.get("params") + return fn(rng) if fn else {} + +# -------------------------------------------------------------- shell ---- +def run(script, timeout=180, params=None): + env = dict(os.environ, WORKSPACE=WORKSPACE, CKA_NODE_ACCESS=NODE_ACCESS, CKA_CONTEXT=CONTEXT, + **{f"CKA_{k}": v for k, v in NODES.items()}) + p = subprocess.run(["bash", "-c", PRELUDE + render(script, params)], env=env, capture_output=True, text=True, + timeout=timeout, stdin=subprocess.DEVNULL) + return p.returncode, (p.stdout + p.stderr).strip() + +def kubectl_ok(): + try: + return subprocess.run(["kubectl", "--context", CONTEXT, "get", "--raw=/version"], + capture_output=True, text=True, timeout=15).returncode == 0 + except Exception: + return False + +def grade_one(q, params=None): + """Run every check independently. Returns a result dict with partial credit.""" + if params is None: + params = session_params(load_state(), q) + checks = [] + for ch in q["checks"]: + try: + rc, out = run(ch["script"], timeout=240, params=params) + except subprocess.TimeoutExpired: + rc, out = 1, "timed out" + checks.append({"name": render(ch["name"], params), "pts": ch["pts"], "pass": rc == 0, "out": out[-400:]}) + earned = sum(ch["pts"] for ch in checks if ch["pass"]) + return {"pass": earned == q["weight"], "earned": earned, "total": q["weight"], + "checks": checks, "at": time.time(), "difficulty": q["difficulty"]} + +def result_detail(r): + return "\n".join((" ✓ " if ch["pass"] else " ✗ ") + ch["name"] + f" ({ch['pts']} pt)" for ch in r["checks"]) + +def score(st, results=None): + results = results if results is not None else st["results"] + qs = session_questions(st) + total = sum(q["weight"] for q in qs) + earned = sum(results.get(str(q["id"]), {}).get("earned", 0) for q in qs) + return earned, total, (round(100 * earned / total, 1) if total else 0) + +# ----------------------------------------------------------- history ---- +def load_history(): + if not os.path.exists(HISTORY_FILE): + return [] + with open(HISTORY_FILE) as f: + return [json.loads(l) for l in f if l.strip()] + +def record_attempt(st): + earned, total, pct = score(st) + entry = {"at": datetime.now().isoformat(timespec="minutes"), "mode": st.get("mode", "exam"), "hard": st.get("hard", False), + "ids": [q["id"] for q in session_questions(st)], "score_pct": pct, "earned": earned, "total": total, + "time_used": int(st["ended_at"] - st["started_at"] - paused_total(st)), + "pauses": st.get("pauses", 0), "paused_seconds": int(paused_total(st)), + "results": {k: {"pass": v["pass"], "earned": v["earned"], "total": v["total"], + "failed": [ch["name"] for ch in v["checks"] if not ch["pass"]]} + for k, v in st["results"].items()}} + with open(HISTORY_FILE, "a") as f: + f.write(json.dumps(entry) + "\n") + write_weak_spots() + return entry + +class AggEntry: + def __init__(self) -> None: + self.attempts: int = 0 + self.fails: int = 0 + self.last: Any = None + self.last_fail_at: str = "" + self.failed: list[str] = [] + self.last_earned: str = "" + + def __getitem__(self, item: str) -> Any: + return getattr(self, item) + + def __setitem__(self, item: str, value: Any) -> None: + setattr(self, item, value) + + def to_dict(self) -> dict[str, Any]: + return { + "attempts": self.attempts, + "fails": self.fails, + "last": self.last, + "last_fail_at": self.last_fail_at, + "failed": self.failed, + "last_earned": self.last_earned, + } + + +def stats() -> dict[int, AggEntry]: + agg: dict[int, AggEntry] = {int(q["id"]): AggEntry() for q in QUESTIONS} + for e in load_history(): + for k, r in e.get("results", {}).items(): + a = agg.get(int(k)) + if a is None: + continue + a.attempts += 1 + a.last = r.get("pass") + a.last_earned = f"{r.get('earned', 0)}/{r.get('total', 0)}" + if not r.get("pass"): + a.fails += 1 + a.last_fail_at = str(e.get("at", "")) + a.failed = list(r.get("failed", [])) + return agg + +def last_seen(): + """question id -> index of the most recent session it appeared in (higher = more recent).""" + seen = {} + for i, e in enumerate(load_history()): + for k in e["results"]: + seen[int(k)] = i + return seen + +def drilled_domains(): + out = {d: 0 for d in DOMAINS} + for e in load_history(): + if e["mode"] != "drill": + continue + for k in e["results"]: + q = QBYID.get(int(k)) + if q: + out[q["domain"]] += 1 + return out + +def no_conflicts(cands, n, taken=()): + """Take up to n tasks, never two that break the same thing (or clash with tasks already taken).""" + used = {q.get("breaks") for q in taken if q.get("breaks")} + out = [] + for q in cands: + b = q.get("breaks") + if b and b in used: + continue + out.append(q) + if b: + used.add(b) + if len(out) == n: + break + return out + +def pick(domain, n, hard, rng, exclude=(), taken=(), unseen_only=False): + """n tasks from a domain: unseen first, then least recently seen; hard sessions prefer hard tasks.""" + seen = last_seen() + pool = [q for q in POOL if q["domain"] == domain and q["id"] not in exclude + and (hard or q["difficulty"] == "normal") and not (unseen_only and q["id"] in seen)] + rng.shuffle(pool) + pool.sort(key=lambda q: (0 if (hard and q["difficulty"] == "hard") else 1, seen.get(q["id"], -1))) + return no_conflicts(pool, n, taken) + +def compose_exam(hard, rng): + qs = [] + for d, n in (QUOTA_HARD if hard else QUOTA).items(): + qs += pick(d, n, hard, rng, taken=qs) + rng.shuffle(qs) + return qs + +def campaign_batch(hard, rng): + """'drill all': DRILL_SIZE tasks you have never attempted, round-robin across domains, + hard tasks included when --hard. When everything has been seen once, falls back to weak tasks.""" + seen = last_seen() + pool = [q for q in POOL if q["id"] not in seen and (hard or q["difficulty"] == "normal")] + if not pool: + return None, [] + qs, domains = [], list(DOMAINS) + rng.shuffle(pool) + pool.sort(key=lambda q: 0 if (hard and q["difficulty"] == "hard") else 1) + while len(qs) < DRILL_SIZE and pool: + progressed = False + for d in domains: + cand = [q for q in pool if q["domain"] == d and q not in qs] + got = no_conflicts(cand, 1, qs) + if got: + qs += got; progressed = True + if len(qs) == DRILL_SIZE: + break + if not progressed: + break + total = len([q for q in POOL if hard or q["difficulty"] == "normal"]) + return f"{total - len(pool)}/{total} attempted", qs + +def solution_allowed(st: dict[str, Any] | None, qid: int | str) -> bool: + """Study: always. Drill: once the task has been graded. Exam: only after submit.""" + if not st or st.get("ended_at") or untimed(st): + return True + return bool(st.get("mode") == "drill" and str(qid) in st.get("results", {})) + +def cmd_solution(args): + if not args: + die("usage: ./exam solution N (drills: after grading the task; exams: after submit; --force to peek anyway)") + force = "--force" in args; args = [a for a in args if a != "--force"] + q = q_by_id(int(args[0])); st = load_state() + if not solution_allowed(st, int(q["id"])) and not force: + die("Not yet: in a drill, grade the task first (./exam grade N); in an exam, submit first. --force overrides.") + print(bold(f"Solution - Q{q['id']} {q['title']}")); print("-" * 64) + sol = str(q.get("solution", "")) + print(render(sol, session_params(st, q))); print("-" * 64) + print(dim(f"One correct way, not the only one. $do = --dry-run=client -o yaml, k = kubectl. Paths are relative to {ROOT}")) + +def cmd_progress(args): + seen = last_seen(); agg = stats() + print(bold(f"{'Domain':<8}{'attempted':>11}{'passed':>9} {'hard attempted':>15}{'hard passed':>13}")) + for d in DOMAINS: + dq = [q for q in QUESTIONS if q["domain"] == d] + n = [q for q in dq if q["difficulty"] == "normal"]; h = [q for q in dq if q["difficulty"] == "hard"] + att = lambda L: sum(1 for q in L if q["id"] in seen); ok = lambda L: sum(1 for q in L if agg.get(int(q["id"])) and agg[int(q["id"])].last is True) + print(f"{d:<8}{att(n):>6}/{len(n):<4}{ok(n):>6}/{len(n):<4} {att(h):>9}/{len(h):<4}{ok(h):>8}/{len(h):<4}") + left = [q["id"] for q in QUESTIONS if q["id"] not in seen] + print(dim(f"\n{len(QUESTIONS) - len(left)}/{len(QUESTIONS)} tasks attempted at least once." + (f" Never seen: Q{', Q'.join(map(str, left))}" if left and len(left) <= 25 else ""))) + print(dim("passed = latest attempt was a full pass. ./exam drill all [--hard] works through what's left.")) + +def weak_ids(agg=None): + agg = agg or stats() + return [q["id"] for q in QUESTIONS + if (agg[q["id"]]["last"] is False or agg[q["id"]]["fails"] >= 2) + or (agg[q["id"]]["attempts"] == 0 and not q.get("set"))] + +def write_weak_spots(): + agg = stats(); hist = load_history() + L = [f"# {TRACK_ID.upper()} weak spots", "", f"_Generated by exam from {len(hist)} attempt(s); regenerated on every `end`._", ""] + exams = [e for e in hist if e["mode"] == "exam"] + if exams: + e = exams[-1] + L += [f"## Domain scores (latest full exam, {e['at']}, {e['score_pct']}%)", ""] + for d, name in DOMAINS.items(): + dq = [q for q in QUESTIONS if q["domain"] == d] + dt = sum(q["weight"] for q in dq); de = sum(e["results"].get(str(q["id"]), {}).get("earned", 0) for q in dq) + L.append(f"- **{d}** {de}/{dt} — {name}") + L.append("") + L += ["## Drill next (`./exam drill weak`)", ""] + weak = [q for q in QUESTIONS if q["id"] in weak_ids(agg) and agg[q["id"]]["attempts"] > 0] + if not weak: + L.append("Nothing failing right now.") + for q in weak: + a = agg[q["id"]] + L.append(f"### Q{q['id']:02d} · {q['domain']} · {q['title']} ({a['fails']}/{a['attempts']} failed, last {a['last_earned']} pts, {a['last_fail_at']})") + failed_list = list(a["failed"]) if isinstance(a["failed"], (list, tuple)) else [] + L += [f"- ✗ {n}" for n in failed_list] + L.append("") + L += ["## Attempt matrix", "", "| Q | Domain | Title | " + " | ".join(f"{e['mode'][0].upper()}{i + 1}" for i, e in enumerate(hist)) + " |", + "|---|---|---|" + "---|" * len(hist)] + for q in QUESTIONS: + cells = [] + for e in hist: + r = e["results"].get(str(q["id"])) + cells.append("·" if r is None else ("✓" if r["pass"] else f"{r['earned']}/{r['total']}")) + L.append(f"| {q['id']} | {q['domain']} | {q['title']} | " + " | ".join(cells) + " |") + L += ["", "_E = full exam, D = drill, · = not in that session, n/m = partial credit._", ""] + with open(WEAK_FILE, "w") as f: + f.write("\n".join(L)) + +# ------------------------------------------------------------ session ---- +def preflight(): + out = subprocess.run(["kubectl", "get", "nodes", "-o", "jsonpath={.items[*].metadata.name}"], + capture_output=True, text=True).stdout.split() + missing = [v for v in NODES.values() if v not in out] + if missing: + die(f"Nodes not found in cluster: {missing}\n cluster has: {out}\n" + " Set CKA_CP, CKA_W1, CKA_W2, CKA_W3 to your node names (needs 1 control-plane + 3 workers).") + rc, _ = run('nx "$CP" "true" node names must be kind container names (kind get nodes --name cka-sim)\n" + " ssh -> passwordless ssh to each node name must work\n" + " vagrant -> run from the Vagrantfile directory; names are vagrant machine names") + print(dim(f" track: {TRACK_ID} cluster: {CONTEXT} node access: {NODE_ACCESS} nodes: {', '.join(NODES.values())}")) + for tool, why in (("helm", "Helm tasks (the real exam has helm on the terminal)"), ("openssl", "certificate tasks"), ("curl", "tasks that fetch manifests")): + if not shutil.which(str(tool)): + print(yellow(f" warning: '{tool}' not found on this machine - needed for {why}")) + +HARD_CONTEXTS = {"k8s-c1-H": "default", "k8s-c2-AC": "%{ns}", "k8s-c3-CCC": "kube-system"} + +def setup_contexts(hard, params_by_id): + """Hard mode: several contexts on the same cluster with different default namespaces, like the real exam.""" + cluster = subprocess.run(["kubectl", "config", "view", "-o", f"jsonpath={{.contexts[?(@.name=='{CONTEXT}')].context.cluster}}"], + capture_output=True, text=True).stdout.strip() + user = subprocess.run(["kubectl", "config", "view", "-o", f"jsonpath={{.contexts[?(@.name=='{CONTEXT}')].context.user}}"], + capture_output=True, text=True).stdout.strip() + for name in HARD_CONTEXTS: + subprocess.run(["kubectl", "config", "delete-context", name], capture_output=True) + if not hard: + subprocess.run(["kubectl", "config", "use-context", CONTEXT], capture_output=True) + return + ns_ac = next((p.get("ns") for p in params_by_id.values() if p.get("ns")), "default") + for name, ns in HARD_CONTEXTS.items(): + subprocess.run(["kubectl", "config", "set-context", name, f"--cluster={cluster}", f"--user={user}", + f"--namespace={ns.replace('%{ns}', ns_ac)}"], capture_output=True) + subprocess.run(["kubectl", "config", "use-context", "k8s-c1-H"], capture_output=True) + +def _log_msg(*args: Any, **kwargs: Any) -> None: + print(*args, **kwargs) + + +def seed(qs, log: Callable[..., Any] = _log_msg, params_by_id=None): + params_by_id = params_by_id or {} + normal = [q for q in qs if not q.get("disruptive")] + disruptive = [q for q in qs if q.get("disruptive")] + def one(q): + log(f" … Q{q['id']:02d} {q['title']}") + try: + rc, out = run(q["setup"], params=params_by_id.get(str(q["id"])), timeout=240) + except subprocess.TimeoutExpired: + rc, out = 1, "setup timed out after 240s" + log(f" {'✓' if rc == 0 else '!'} Q{q['id']:02d} {q['title']}" + ("" if rc == 0 else f"\n{dim(out[-300:])}")) + for q in normal: + one(q) + if disruptive: + log(dim(" waiting up to 45s for seeded pods to be scheduled...")) + for _ in range(15): + pending = subprocess.run(["kubectl", "--context", CONTEXT, "get", "pods", "-A", "--field-selector=spec.nodeName=", + "-o", "name"], capture_output=True, text=True).stdout.strip() + if not pending: + break + time.sleep(3) + for q in disruptive: + one(q) + +def begin_session(qs, mode, duration, force, log: Callable[..., Any] = _log_msg, hard=False, untimed_session=False): + st = load_state() + if st and not st.get("ended_at") and not force: + die("A session is already in progress. './exam end' it first, or add --force.") + if not kubectl_ok(): + die(f"Cannot reach the cluster through kubectl context '{CONTEXT}'. Run ./setup.sh (kind) or set CKA_CONTEXT for another cluster.") + if not qs: + die(f"Track '{TRACK_ID}' has no questions yet.") + preflight() + os.makedirs(WORKSPACE, exist_ok=True) + rng = random.Random() + params_by_id = {str(q["id"]): make_params(q, rng) for q in qs} + setup_contexts(hard, params_by_id) + log(bold(f"Seeding {len(qs)} scenario(s){' [HARD]' if hard else ''}...")) + seed(qs, log, params_by_id) + st = {"mode": mode, "track": TRACK_ID, "hard": hard, "untimed": untimed_session, "ids": [q["id"] for q in qs], + "params": params_by_id, "duration": duration, "started_at": time.time(), "ended_at": None, + "results": {}, "flags": [], "notes": ""} + save_state(st) + log("") + log(bold("=" * 64)) + log(bold(f" {TRACK_ID.upper()} {'EXAM' if mode == 'exam' else 'DRILL'}{' · HARD' if hard else ''} - clock started")) + if hard: + log(yellow(" Contexts: k8s-c1-H (default) · k8s-c2-AC · k8s-c3-CCC - each task tells you which to use; you are on k8s-c1-H now")) + log(f" Ends : {datetime.fromtimestamp(st['started_at'] + duration).strftime('%H:%M:%S')} ({duration // 60} min)") + log(f" Tasks : {', '.join('Q%02d' % q['id'] for q in qs) if mode == 'drill' else len(qs)} Pass mark: {PASS_PCT}%") + log(bold("=" * 64)) + log(" ./exam web exam-style browser UI (task pane + terminal + clock)\n" + " ./exam list / show N / grade N / timer / end terminal workflow\n" + + (" Tip: read ALL tasks first. Some are cluster-wide failures that block others.\n" if mode == "exam" else "") + + f" Files the tasks ask for go under {WS_DISPLAY}/ (relative to this directory)\n" + + " Your own terminal: source ./examrc (fish: source ./examrc.fish) -> k, $do, $now, completion") + return st + +def finish_session(st, log: Callable[..., Any] = _log_msg): + qs = session_questions(st) + over = remaining(st) <= 0 + set_paused(st, False) + log(bold(f"Submitting - grading {len(qs)} task(s)...")) + for q in qs: + r = grade_one(q) + st["results"][str(q["id"])] = r + tag = green("PASS") if r["pass"] else (yellow(f"{r['earned']}/{r['total']}") if r["earned"] else red("FAIL")) + log(f"Q{q['id']:02d} {tag} {q['title']}" + ("" if r["pass"] else f"\n{dim(result_detail(r))}")) + st["ended_at"] = time.time() + st["over_time"] = over + save_state(st) + setup_contexts(False, {}) + if untimed(st): + return None + return record_attempt(st) + +# ----------------------------------------------------------- commands ---- +def cmd_start(args): + hard = "--hard" in args; force = "--force" in args + undrilled = [d for d, n in drilled_domains().items() if n == 0] + if undrilled and not force: + die("Drill every domain at least once before a full exam (that is the point of the drills).\n" + f" Not yet drilled: {', '.join(undrilled)}\n Run: ./exam drill (picks the next domain)\n" + " or: ./exam start --force to skip the gate.") + begin_session(compose_exam(hard, random.Random()), "exam", EXAM_SECONDS, force, hard=hard) + +def next_drill(hard, rng): + """No-arg drill: the least-drilled domain, DRILL_SIZE tasks, least-recently-seen first.""" + cov = drilled_domains() + domain = min(DOMAINS, key=lambda d: (cov[d], list(DOMAINS).index(d))) + return domain, pick(domain, DRILL_SIZE, hard, rng) + +def cmd_drill(args): + force = "--force" in args; hard = "--hard" in args + args = [a for a in args if a not in ("--force", "--hard")] + minutes = None + if "--time" in args: + i = args.index("--time"); minutes = int(args[i + 1]); del args[i:i + 2] + per_task = DRILL_SECONDS_PER_TASK_HARD if hard else DRILL_SECONDS_PER_TASK + if not args: + domain, qs = next_drill(hard, random.Random()) + cov = drilled_domains() + print(bold(f"Drill: {DOMAINS[domain]}") + dim(f" ({sum(1 for v in cov.values() if v)}/{len(DOMAINS)} domains drilled so far)")) + begin_session(qs, "drill", (minutes * 60) if minutes else per_task * len(qs), force, hard=hard) + return + if args[0] == "all": + label, qs = campaign_batch(hard, random.Random()) + if not qs: + print(bold("Every task in the bank has been attempted." ) + " Switching to weak tasks.") + qs = [q for q in QUESTIONS if q["id"] in weak_ids()] + qs = no_conflicts(qs, DRILL_SIZE) + if not qs: + die("Nothing weak either - you have passed everything on its latest attempt. Time for ./exam start --hard.") + else: + print(bold(f"Campaign: {label}") + dim(f" this batch: Q{', Q'.join(str(q['id']) for q in qs)}")) + begin_session(qs, "drill", (minutes * 60) if minutes else per_task * len(qs), force, hard=hard) + return + if args[0] == "weak": + ids = weak_ids() + if not ids: + die("Nothing marked weak - finish a session with './exam end' first, or name tasks: ./exam drill 2 17") + else: + ids = [] + for a in args: + if a.lower().replace("set", "") in SETS and a.lower().startswith("set"): + ids += [q["id"] for q in set_questions(a.lower().replace("set", ""))] + elif a.upper() in DOMAINS: + ids += [q["id"] for q in QUESTIONS if q["domain"] == a.upper()] + else: + ids.append(q_by_id(int(a))["id"]) + ids = sorted(set(ids)) + qs = [q for q in QUESTIONS if q["id"] in ids] + kept = no_conflicts(qs, len(qs)) + if len(kept) < len(qs): + dropped = [q["id"] for q in qs if q not in kept] + print(yellow(f"Dropped Q{', Q'.join(map(str, dropped))}: they break the same thing as another task in this set. Drill them separately.")) + begin_session(kept, "drill", (minutes * 60) if minutes else per_task * len(kept), force, hard=hard) + +def cmd_study(args): + """No clock, solutions unlocked from the start, nothing written to history.""" + hard = "--hard" in args; force = "--force" in args + rest = [a for a in args if not a.startswith("--")] + n = DRILL_SIZE + if "--count" in args: + n = int(args[args.index("--count") + 1]); rest = [a for a in rest if a != str(n)] + rng = random.Random() + if rest and rest[0].upper() in DOMAINS: + domain = rest[0].upper() + qs = pick(domain, n, hard, rng) + print(bold(f"Study: {DOMAINS[domain]}")) + elif rest and rest[0] == "weak": + qs = no_conflicts([q for q in QUESTIONS if q["id"] in weak_ids()], n) + print(bold("Study: tasks you have been failing")) + elif rest: + qs = no_conflicts([q_by_id(int(a)) for a in rest], len(rest)) + print(bold("Study: " + ", ".join(f"Q{q['id']}" for q in qs))) + else: + domain, qs = next_drill(hard, rng) + print(bold(f"Study: {DOMAINS[domain]}")) + begin_session(qs, "drill", EXAM_SECONDS, force, hard=hard, untimed_session=True) + +def cmd_sim(args): + """Fixed 17-task simulator set, full exam rules and clock.""" + force = "--force" in args + names = [a.lower() for a in args if not a.startswith("--")] + if not names or names[0] not in SETS: + die("usage: ./exam sim a|b [--force]\n " + "\n ".join(f"{k}: {v} ({len(set_questions(k))} tasks)" for k, v in SETS.items())) + name = names[0] + qs = set_questions(name) + print(bold(f"{SETS[name]} - {len(qs)} tasks, {TRACK['minutes']} minutes, exam rules (no feedback until you submit)")) + begin_session(qs, "exam", EXAM_SECONDS, force, hard=True) + +def cmd_list(args): + st = load_state() + results = st["results"] if st and "results" in st else {} + qs = session_questions(st) if st and not st.get("ended_at") else list(QUESTIONS) + agg = stats() + print(bold(f"{'#':>3} {'Domain':<8}{'Pts':>4} {'Now':<8} {'History':<9} Title")) + for q in qs: + r = results.get(str(q["id"])) + now = "-" if r is None else (green("PASS") if r.get("pass") else yellow(f"{r.get('earned')}/{r.get('total')}")) + a = agg.get(int(q["id"])) or AggEntry() + hist = "" if a.attempts == 0 else f"{a.attempts - a.fails}/{a.attempts}" + if a.attempts and a.last is False: + hist = red(hist + " ✗") + print(f"{q['id']:>3} {q['domain']:<8}{q['weight']:>4} {now:<17} {hist:<18} {q['title']}{' [hard]' if q['difficulty'] == 'hard' else ''}") + print(dim("\nHistory = full passes/attempts across sessions, ✗ = failed most recently.")) + if st and not st.get("ended_at"): + print(status_line(st)) + +def cmd_show(args): + if not args: + die("usage: ./exam show N") + q = q_by_id(int(args[0])) + st0 = load_state() + sq = session_questions(st0) if st0 and not st0.get("ended_at") else [] + pos = f"{sq.index(q) + 1} of {len(sq)} | " if q in sq else "" + print(bold(f"{pos}Q{q['id']} | {DOMAINS[q['domain']]} | {q['weight']} pts{' | HARD' if q['difficulty'] == 'hard' else ''}")) + print(bold(q["title"])); print("-" * 64); print(task_text(q)); print("-" * 64) + st = load_state() + if st and not st.get("ended_at"): + print(status_line(st)) + +def cmd_grade(args): + st = require_started() + if not args: + die("usage: ./exam grade N | all") + ids = [q["id"] for q in session_questions(st)] if args[0] == "all" else [int(a) for a in args] + for n in ids: + q = q_by_id(n); r = grade_one(q) + st["results"][str(n)] = r + tag = green("PASS") if r.get("pass") else yellow(f"{r.get('earned')}/{r.get('total')}") + print(f"Q{n:02d} {tag} {q['title']}"); print(dim(result_detail(r))) + save_state(st) + if args[0] == "all": + print_score(st) + +def print_score(st): + qs = session_questions(st) + earned, total, pct = score(st) + print(); print(bold("Per-domain:")) + for d, name in DOMAINS.items(): + dq = [q for q in qs if q["domain"] == d] + if not dq: + continue + dt = sum(q["weight"] for q in dq); de = sum(st["results"].get(str(q["id"]), {}).get("earned", 0) for q in dq) + n = int(20 * de / dt) + print(f" {d:<7} {'█' * n}{'░' * (20 - n)} {de:>3}/{dt:<3} {dim(name)}") + verdict = green("PASS") if pct >= PASS_PCT else red("FAIL") + print(f"\n{bold('Score:')} {earned}/{total} = {pct}% -> {verdict} (pass mark {PASS_PCT}%)") + +def status_line(st): + if untimed(st): + return green("study mode - no clock") + dim(f" [{len(session_questions(st))} tasks, nothing recorded]") + r = remaining(st) + if st.get("paused_at"): + return yellow(f"⏸ PAUSED - {fmt(r)} remaining") + dim(" ./exam resume") + if r <= 0: + return red("⏰ TIME IS UP - run ./exam end") + col = green if r > 30 * 60 else (yellow if r > 10 * 60 else red) + return col(f"⏱ {fmt(r)} remaining") + dim(f" [{st.get('mode', 'exam')}]") + +def cmd_status(args): + st = require_started() + if st.get("ended_at"): + print("Session ended."); print_score(st); return + print(status_line(st)) + print(dim(f"{sum(1 for v in st['results'].values() if v['pass'])} passed / {len(st['results'])} graded / {len(session_questions(st))} in session")) + +def cmd_timer(args): + st = require_started() + if st.get("ended_at"): + die("Session already ended.") + if untimed(st): + die("This is a study session - there is no clock.") + try: + while True: + st = load_state(); r = remaining(st) + if st.get("paused_at"): + sys.stdout.write("\r" + yellow(bold(f"⏸ {fmt(r)} PAUSED ")) + dim(" ./exam resume ")); sys.stdout.flush(); time.sleep(1); continue + if r <= 0: + print("\r" + red(bold("⏰ 00:00:00 TIME IS UP - run ./exam end "))); break + col = green if r > 30 * 60 else (yellow if r > 10 * 60 else red) + sys.stdout.write("\r" + col(bold(f"⏱ {fmt(r)} remaining")) + dim(" (Ctrl-C to hide)")); sys.stdout.flush() + if int(r) in (30 * 60, 15 * 60, 5 * 60) and TTY: + sys.stdout.write("\a") + time.sleep(1) + except KeyboardInterrupt: + print() + +def cmd_pause(args): + st = require_started() + if st.get("ended_at"): + die("Session already ended.") + set_paused(st, True); print(status_line(st)) + +def cmd_resume(args): + st = require_started() + set_paused(st, False); print(status_line(st)) + +def cmd_end(args): + st = require_started() + if st.get("ended_at"): + die("Session already ended and recorded.") + if remaining(st) <= 0: + print(yellow("Note: time expired before submission.")) + entry = finish_session(st) + print_score(st) + if entry is None: + print(dim("Study session - not added to history or weak-spots.")); return + print(dim(f"Time used: {fmt(st['ended_at'] - st['started_at'])}" + (" (over time)" if st.get("over_time") else ""))) + failed = sorted(int(k) for k, v in entry["results"].items() if not v["pass"]) + print(dim(f"Attempt #{len(load_history())} logged to {os.path.basename(HISTORY_FILE)}; {os.path.basename(WEAK_FILE)} updated.")) + if failed: + print(bold("\nTo drill what you missed: ./exam drill weak") + dim(f" (Q{', Q'.join(map(str, failed))})")) + +def cmd_abort(args): + """End a session without grading it and restore the kubectl context.""" + st = load_state() + if not st or st.get("ended_at"): + die("No session in progress.") + assert st is not None + if "--yes" not in args: + die(f"This discards the {st.get('mode', 'exam')} in progress without grading or recording it.\n" + " Confirm: ./exam abort --yes (to grade and keep it instead: ./exam end)") + st["ended_at"] = time.time(); st["aborted"] = True + save_state(st) + setup_contexts(False, {}) + print(yellow("Session discarded. Nothing was graded or written to history.")) + print(dim(f" kubectl context restored to {CONTEXT}. Seeded objects are still on the cluster;")) + print(dim(" './exam drill ...' or './exam start' re-seeds, './teardown.sh' removes the cluster.")) + +def cmd_weak(args): + agg = stats(); hist = load_history() + if not hist: + die("No history yet - finish a session with './exam end'.") + ids = [i for i in weak_ids(agg) if agg[i]["attempts"] > 0] + print(bold(f"Weak spots after {len(hist)} attempt(s):")) + if not ids: + print(green(" none - everything attempted has passed on its latest try")) + for i in ids: + q = q_by_id(i); a = agg[i] + print(f"\n{red('Q%02d' % i)} {q['domain']:<7} {q['title']} " + dim(f"{a['fails']}/{a['attempts']} failed, last {a['last_earned']} pts, {a['last_fail_at']}")) + for n in a["failed"]: + print(dim(" ✗ " + n)) + never = [q["id"] for q in QUESTIONS if agg[q["id"]]["attempts"] == 0] + if never: + print(dim(f"\nNever attempted: Q{', Q'.join(map(str, never))}")) + print(dim(f"\nFull log: {WEAK_FILE} Drill them: ./exam drill weak")) + +def cmd_history(args): + hist = load_history() + if not hist: + die("No history yet.") + print(bold(f"{'#':>3} {'when':<17}{'mode':<7}{'score':>7} {'time':<9} tasks")) + for i, e in enumerate(hist, 1): + col = green if e["score_pct"] >= PASS_PCT else red + tasks = f"all {len(QUESTIONS)}" if len(e["ids"]) == len(QUESTIONS) else "Q" + ",".join(map(str, e["ids"])) + print(f"{i:>3} {e['at']:<17}{(e['mode'] + ('*' if e.get('hard') else '')):<7}{col('%5.1f%%' % e['score_pct']):>16} {fmt(e['time_used']):<9} {tasks}") + exams = [e for e in hist if e["mode"] == "exam"] + print(dim("* = hard")) + if len(exams) >= 2: + print(bold("\nDomain trend (full exams, % of points):")) + for d in DOMAINS: + dq = [q for q in QUESTIONS if q["domain"] == d]; dt = sum(q["weight"] for q in dq) + pts = [round(100 * sum(e["results"].get(str(q["id"]), {}).get("earned", 0) for q in dq) / dt) for e in exams] + print(f" {d:<7} " + " → ".join(f"{p:>3}%" for p in pts)) + +def cmd_reset(args): + if not args: + die("usage: ./exam reset N (re-seed one task's scenario)") + q = q_by_id(int(args[0])); rc, out = run(str(q.get("setup", "")), params=session_params(load_state(), q)) + print(("re-seeded " if rc == 0 else "problem re-seeding ") + f"Q{q['id']:02d}: {q['title']}") + if out: + print(dim(out)) + +# --------------------------------------------------------------- web ---- +class Api(http.server.BaseHTTPRequestHandler): + """Serves web/index.html plus a small JSON API. State lives in the same files the CLI uses.""" + lock = threading.Lock() + log_lines = [] + seeding = False + seed_error = None + ttyd_port = 7681 + shell = "bash" + + def log_message(self, format: str, *args: Any) -> None: # quiet + pass + + def _json(self, obj, code=200): + body = json.dumps(obj).encode() + self.send_response(code); self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body) + + def _read_json(self): + n = int(self.headers.get("Content-Length") or 0) + return json.loads(self.rfile.read(n) or b"{}") + + def do_GET(self): + p = urllib.parse.urlparse(self.path) + if p.path in ("/", "/index.html"): + with open(os.path.join(ROOT, "web", "index.html"), "rb") as f: + body = f.read() + self.send_response(200); self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body); return + if p.path == "/api/session": + with Api.lock: + return self._json(self.session_payload()) + if p.path == "/api/history": + return self._json({"history": load_history(), "weak": weak_ids(), "stats": {str(k): v.to_dict() for k, v in stats().items()}}) + self._json({"error": "not found"}, 404) + + def do_POST(self): + p = urllib.parse.urlparse(self.path); body = self._read_json() + with Api.lock: + st = load_state() + if p.path == "/api/track": + t = body.get("track") + if t not in TRACKS: + return self._json({"error": f"unknown track {t}"}, 400) + if Api.seeding: + return self._json({"error": "a session is being seeded"}, 409) + load_track(t) + Api.log_lines = [] + return self._json(self.session_payload()) + if p.path == "/api/start": + if Api.seeding: + return self._json({"error": "already seeding"}, 409) + mode = body.get("mode", "exam"); hard = bool(body.get("hard")); rng = random.Random() + Api.log_lines = []; Api.seed_error = None; Api.seeding = True + ansi = __import__("re").compile(r"\x1b\[[0-9;]*m") + ulog = lambda line: Api.log_lines.append(ansi.sub("", line)) + def work(): + try: + if body.get("sim") in SETS: + begin_session(set_questions(body["sim"]), "exam", EXAM_SECONDS, True, log=ulog, hard=True) + elif mode == "exam": + begin_session(compose_exam(hard, rng), "exam", EXAM_SECONDS, True, log=ulog, hard=hard) + else: + if body.get("domain") in DOMAINS: + qs = pick(body["domain"], DRILL_SIZE, hard, rng) + elif body.get("campaign"): + _, qs = campaign_batch(hard, rng) + if not qs: + qs = no_conflicts([q for q in QUESTIONS if q["id"] in weak_ids()], DRILL_SIZE) + elif body.get("next"): + _, qs = next_drill(hard, rng) + else: + ids = body.get("ids") or weak_ids() + qs = no_conflicts([q for q in QUESTIONS if q["id"] in ids], len(ids)) + per = DRILL_SECONDS_PER_TASK_HARD if hard else DRILL_SECONDS_PER_TASK + begin_session(qs, "drill", per * len(qs), True, log=ulog, hard=hard, + untimed_session=bool(body.get("study"))) + except SystemExit: + Api.seed_error = "\n".join(l for l in Api.log_lines if l.strip()) or "start failed - see the terminal running ./exam web" + except Exception as e: # noqa: BLE001 + Api.seed_error = f"{type(e).__name__}: {e}" + finally: + Api.seeding = False + threading.Thread(target=work, daemon=True).start() + return self._json({"seeding": True}) + if not st: + return self._json({"error": "no session"}, 400) + if p.path == "/api/flag": + qid = int(body["id"]); flags = set(st.get("flags", [])) + flags.symmetric_difference_update({qid}); st["flags"] = sorted(flags); save_state(st) + elif p.path == "/api/pause": + set_paused(st, bool(body.get("paused", True))) + elif p.path == "/api/notes": + st["notes"] = body.get("notes", ""); save_state(st) + elif p.path == "/api/grade": + if st.get("mode") != "drill" and not st.get("ended_at"): + return self._json({"error": "No feedback during an exam. Submit to see results."}, 403) + q = q_by_id(int(body["id"])) + if "results" not in st or not isinstance(st["results"], dict): + st["results"] = {} + st["results"][str(q["id"])] = grade_one(q, session_params(st, q)); save_state(st) + elif p.path == "/api/end": + if not st.get("ended_at"): + finish_session(st, log=lambda *_: None) + elif p.path == "/api/reset": + q = q_by_id(int(body["id"])); run(str(q.get("setup", "")), params=session_params(st, q)) + else: + return self._json({"error": "not found"}, 404) + return self._json(self.session_payload()) + + def session_payload(self) -> dict[str, Any]: + st = load_state() + qs = session_questions(st) if st else QUESTIONS + payload: dict[str, Any] = { + "track": TRACK, "track_id": TRACK_ID, "workspace": WS_DISPLAY, "version": VERSION, "ttyd_port": Api.ttyd_port, + "ttyd": shutil.which("ttyd") is not None, "shell": Api.shell, "setup_log": Api.log_lines, + "questions": [{"id": q["id"], "domain": q["domain"], "title": q["title"], "weight": q["weight"], "difficulty": q["difficulty"], + "task": task_text(q, session_params(st, q)), "checks": [{"name": c["name"], "pts": c["pts"]} for c in q["checks"]], + "solution": render(str(q.get("solution", "")), session_params(st, q)) if solution_allowed(st, int(q["id"])) else None} + for q in qs], + "bank": [{"id": q["id"], "domain": q["domain"], "title": q["title"], "difficulty": q["difficulty"]} for q in QUESTIONS], + "tracks": TRACKS, "drilled": drilled_domains(), "sets": {k: {"name": v, "count": len(set_questions(k))} for k, v in SETS.items()}, + "seen": sorted(last_seen()), + "seeding": Api.seeding, "seed_error": Api.seed_error, + "session": None, + } + if st: + earned, total, pct = score(st) + payload["session"] = {"mode": st.get("mode"), "hard": st.get("hard", False), "started_at": st["started_at"], "duration": st.get("duration", EXAM_SECONDS), + "ends_at": st["started_at"] + st.get("duration", EXAM_SECONDS), "ended_at": st.get("ended_at"), + "over_time": st.get("over_time", False), "flags": st.get("flags", []), + "paused": bool(st.get("paused_at")), "paused_total": paused_total(st), "pauses": st.get("pauses", 0), + "untimed": untimed(st), "notes": st.get("notes", ""), + "results": st["results"], "score": {"earned": earned, "total": total, "pct": pct, "pass_pct": PASS_PCT}, + "domains": self.domain_scores(st), "server_time": time.time()} + return payload + + def domain_scores(self, st: dict[str, Any]) -> dict[str, Any]: + qs = session_questions(st); out = {} + st_results: dict[str, Any] = st.get("results", {}) + for d in DOMAINS: + dq = [q for q in qs if q["domain"] == d] + if dq: + out[d] = {"earned": sum(st_results.get(str(q["id"]), {}).get("earned", 0) for q in dq), + "total": sum(q["weight"] for q in dq)} + return out + +def cmd_web(args): + port = 8080 + if "--port" in args: + port = int(args[args.index("--port") + 1]) + Api.shell = os.environ.get("CKA_SHELL", "bash") + if "--shell" in args: + Api.shell = args[args.index("--shell") + 1] + os.makedirs(WORKSPACE, exist_ok=True) + import socket + def port_free(pt): + with socket.socket() as sk: + sk.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + try: + sk.bind(("127.0.0.1", pt)); return True + except OSError: + return False + if not port_free(port): + die(f"Port {port} is already in use - probably another './exam web' still running.\n" + f" Find it: ss -ltnp | grep :{port} or start on another port: ./exam web --port {port + 10}") + if shutil.which("ttyd") and not port_free(Api.ttyd_port): + print(yellow(f" ttyd port {Api.ttyd_port} already in use - reusing the existing terminal server.")) + ttyd = None + if shutil.which("ttyd") and port_free(Api.ttyd_port): + env = dict(os.environ, WORKSPACE=WORKSPACE, KUBECONFIG=os.environ.get("KUBECONFIG", os.path.expanduser("~/.kube/config"))) + rc = os.path.join(ROOT, "examrc.fish" if "fish" in Api.shell else "examrc") + if "fish" in Api.shell: + shell_cmd = [Api.shell, "-C", f"source {rc}"] + else: + shell_cmd = [Api.shell, "--rcfile", rc, "-i"] + ttyd = subprocess.Popen(["ttyd", "-p", str(Api.ttyd_port), "-W", "-t", "fontSize=14", "-t", "theme={\"background\":\"#1a1b1e\"}", + "--cwd", ROOT] + shell_cmd, env=env, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + print(dim(f" terminal: ttyd on :{Api.ttyd_port} running {Api.shell} in {ROOT} with k=kubectl, $do and $now preset")) + else: + print(yellow(" ttyd not found - the terminal pane will show install instructions.\n" + " (dnf/apt/pacman/brew install ttyd, or https://github.com/tsl0922/ttyd/releases)")) + http.server.ThreadingHTTPServer.allow_reuse_address = True + srv = http.server.ThreadingHTTPServer(("127.0.0.1", port), Api) # type: ignore[arg-type] + print(bold(f"\n {TRACK_ID.upper()} exam UI: http://127.0.0.1:{port} (exam {VERSION})\n") + dim(" Ctrl-C stops the UI (not the exam clock).")) + def stop(*_): + if ttyd: + ttyd.terminate() + srv.shutdown() + signal.signal(signal.SIGINT, lambda *_: threading.Thread(target=stop).start()) + signal.signal(signal.SIGTERM, lambda *_: threading.Thread(target=stop).start()) + try: + srv.serve_forever() + finally: + if ttyd: + ttyd.terminate() + +def cmd_version(args): + print(f"exam {VERSION} track={TRACK_ID} bank={len(QUESTIONS)} tasks python={sys.version.split()[0]}") + +def cmd_help(args): + print(__doc__ + f"\nversion {VERSION}") + print(f"""usage: ./exam [--track cka|ckad|cks] (current track: {TRACK_ID}, {len(QUESTIONS)} tasks) + + web [--port 8080] [--shell bash] exam-style browser UI: tasks, terminal, clock, submit + study [DOMAIN|weak|N..] no clock, solutions unlocked, nothing recorded (--count N, default {DRILL_SIZE}) + drill next domain you have drilled least: {DRILL_SIZE} tasks, 6 min each (--hard: 5 min, hard tasks) + drill all [--hard] campaign: {DRILL_SIZE} never-attempted tasks per session until the bank is done, then weak ones + drill weak|N..|DOMAIN.. chosen tasks (--time MIN, --force, --hard) + progress attempted/passed per domain, normal and hard + solution N reference solution (drills: after grading N; exams: after submit) + sim a|b fixed 17-task simulator set, exam rules, {TRACK['minutes']} min + start [--hard] full exam: {sum(QUOTA.values()) or '?'} tasks sampled by domain ({sum(QUOTA_HARD.values()) or '?'} in hard mode), {TRACK['minutes']} min. + Requires every domain drilled once first (--force skips). Hard mode adds traps, + exact-output tasks, compound failures, collateral checks and exam-style contexts. + list tasks in this session, with current + historical results + show N display task N + grade N|all grade one task (or all) per check - drills only, in the UI + status / timer time remaining (timer = live countdown) + pause / resume stop the clock (recorded on the attempt - the real exam has no pause) + end submit: grade per check, score, log attempt, update weak-spots + weak what you keep getting wrong, with the failing checks + history every attempt, scores, and per-domain trend + reset N re-seed the scenario for task N +""") + +COMMANDS = {"start": cmd_start, "drill": cmd_drill, "list": cmd_list, "show": cmd_show, "grade": cmd_grade, + "status": cmd_status, "timer": cmd_timer, "pause": cmd_pause, "resume": cmd_resume, "end": cmd_end, "weak": cmd_weak, "history": cmd_history, + "reset": cmd_reset, "web": cmd_web, "abort": cmd_abort, "progress": cmd_progress, "solution": cmd_solution, "sim": cmd_sim, "study": cmd_study, "version": cmd_version, "help": cmd_help} + +if __name__ == "__main__": + cmd = ARGV[0] if ARGV else "help" + COMMANDS.get(cmd, cmd_help)(ARGV[1:]) diff --git a/examrc b/examrc new file mode 100644 index 0000000..7700af8 --- /dev/null +++ b/examrc @@ -0,0 +1,17 @@ +# cka-sim shell setup. Sourced automatically by the terminal in ./exam web, +# and safe to source yourself: source ./examrc +[ -f ~/.bashrc ] && . ~/.bashrc + +alias k=kubectl +export do='--dry-run=client -o yaml' # k create deploy x --image=nginx $do > x.yaml +export now='--force --grace-period 0' # k delete pod x $now +export KUBE_EDITOR="${KUBE_EDITOR:-vi}" + +# completion, including for the alias +source <(kubectl completion bash) 2>/dev/null +complete -o default -F __start_kubectl k 2>/dev/null + +# vim defaults that make pasted YAML behave +if [ ! -f ~/.vimrc ]; then + printf 'set tabstop=2\nset expandtab\nset shiftwidth=2\n' > ~/.vimrc +fi diff --git a/examrc.fish b/examrc.fish new file mode 100644 index 0000000..b8ec94f --- /dev/null +++ b/examrc.fish @@ -0,0 +1,7 @@ +# cka-sim shell setup for fish: source ./examrc.fish +alias k kubectl +set -gx KUBE_EDITOR (test -n "$KUBE_EDITOR"; and echo $KUBE_EDITOR; or echo vi) +# lists, so they expand to separate arguments +set -g do --dry-run=client -o yaml +set -g now --force --grace-period 0 +kubectl completion fish 2>/dev/null | source diff --git a/kind-config.yaml b/kind-config.yaml new file mode 100644 index 0000000..2e8b488 --- /dev/null +++ b/kind-config.yaml @@ -0,0 +1,8 @@ +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +name: cka-sim +nodes: + - role: control-plane + - role: worker + - role: worker + - role: worker diff --git a/node-tools.sh b/node-tools.sh new file mode 100755 index 0000000..1ad5286 --- /dev/null +++ b/node-tools.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# Install vim/nano/less/jq on every kind node (the real exam has an editor on each node). +# Re-run this after ./setup.sh if the install was skipped or the network was down. +set -euo pipefail +cd "$(dirname "$0")" +RT="${CONTAINER_RUNTIME:-$(cat .state/runtime 2>/dev/null || echo podman)}" +[ "$RT" = podman ] && export KIND_EXPERIMENTAL_PROVIDER=podman +for n in $(kind get nodes --name cka-sim); do + printf '%s: ' "$n" + if "$RT" exec "$n" sh -c 'command -v vim >/dev/null'; then echo "already has vim"; continue; fi + if "$RT" exec "$n" sh -c 'DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq vim nano less jq >/dev/null 2>&1'; then + "$RT" exec "$n" sh -c 'printf "set tabstop=2\nset expandtab\nset shiftwidth=2\n" > /root/.vimrc' + echo ok + else + echo "FAILED (no outbound network?)" + fi +done diff --git a/setup.sh b/setup.sh new file mode 100755 index 0000000..4c15649 --- /dev/null +++ b/setup.sh @@ -0,0 +1,86 @@ +#!/usr/bin/env bash +# Bootstraps the cka-sim kind cluster (1 control-plane + 3 workers) on podman and preloads images. +# Override with CONTAINER_RUNTIME=docker if you ever want docker instead. +set -euo pipefail +cd "$(dirname "$0")" + +RT="${CONTAINER_RUNTIME:-podman}" + +need() { + command -v "$1" >/dev/null 2>&1 || { echo "ERROR: '$1' is not installed. $2"; exit 1; } +} +need "$RT" "https://podman.io/docs/installation" +need kind "https://kind.sigs.k8s.io/docs/user/quick-start/#installation" +need kubectl "https://kubernetes.io/docs/tasks/tools/" +need python3 "" + +if [ "$RT" = podman ]; then + export KIND_EXPERIMENTAL_PROVIDER=podman + podman info >/dev/null 2>&1 || { echo "ERROR: podman is not working (try: podman info)."; exit 1; } + # kind needs cgroup v2 with controller delegation when podman is rootless. + if [ "$(id -u)" != 0 ] && [ "$(podman info --format '{{.Host.CgroupsVersion}}' 2>/dev/null)" != "v2" ]; then + echo "ERROR: rootless podman needs cgroup v2 for kind. Either enable cgroup v2 or run setup as root (sudo -E ./setup.sh)." + exit 1 + fi + # inotify limits are the #1 cause of pods stuck in ContainerCreating on multi-node kind + for k in fs.inotify.max_user_watches:524288 fs.inotify.max_user_instances:512; do + key=${k%%:*}; want=${k##*:}; have=$(sysctl -n "$key" 2>/dev/null || echo 0) + [ "$have" -ge "$want" ] || echo "warn: $key=$have (recommend >= $want: sudo sysctl $key=$want)" + done +else + docker info >/dev/null 2>&1 || { echo "ERROR: docker daemon is not running/reachable."; exit 1; } +fi + +if kind get clusters 2>/dev/null | grep -qx cka-sim; then + echo "==> kind cluster 'cka-sim' already exists, reusing it." +else + echo "==> Creating kind cluster 'cka-sim' on $RT (1 control-plane + 3 workers)..." + kind create cluster --config kind-config.yaml ${KIND_NODE_IMAGE:+--image "$KIND_NODE_IMAGE"} --wait 180s +fi + +kubectl config use-context kind-cka-sim >/dev/null +echo "==> Waiting for nodes..." +kubectl wait --for=condition=Ready nodes --all --timeout=180s + +echo "==> Preloading images into the cluster (non-fatal if a pull fails)..." +for img in nginx:1.25 nginx:1.27 busybox:1.36 redis:7-alpine; do + if "$RT" pull -q "docker.io/library/$img" >/dev/null 2>&1; then + # podman stores the fully-qualified name; kind load accepts either + kind load docker-image --name cka-sim "docker.io/library/$img" >/dev/null 2>&1 \ + && echo " loaded $img" || echo " warn: could not load $img" + else + echo " warn: could not pull $img (tasks will pull at runtime)" + fi +done + +echo "==> Making sure the exam workdir exists on the control-plane node..." +"$RT" exec cka-sim-control-plane mkdir -p /opt/cka + +# The real exam gives you vim (and less/jq) on every node; the kind node image is minimal, so +# install them once here. Skip with CKA_SKIP_NODE_TOOLS=1. +if [ -z "${CKA_SKIP_NODE_TOOLS:-}" ]; then + echo "==> Installing editors and tools on the nodes (vim, nano, less, jq)..." + for n in $(kind get nodes --name cka-sim); do + if "$RT" exec "$n" sh -c 'command -v vim >/dev/null'; then + echo " $n already has vim" + continue + fi + if "$RT" exec "$n" sh -c 'DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null 2>&1 && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq vim nano less jq >/dev/null 2>&1'; then + "$RT" exec "$n" sh -c 'printf "set tabstop=2\nset expandtab\nset shiftwidth=2\n" > /root/.vimrc' + echo " $n ok" + else + echo " warn: could not install tools on $n (no outbound network?). Fall back to:" + echo " $RT cp $n:/path/file /tmp/f && \$EDITOR /tmp/f && $RT cp /tmp/f $n:/path/file" + fi + done +fi + +# remember the runtime so the CLI uses the matching exec command +mkdir -p .state && echo "$RT" > .state/runtime + +chmod +x exam +echo +echo "Cluster ready. Nodes:" +kubectl get nodes -o wide +echo +echo "Next: ./exam start (seeds all 20 tasks and starts the 120-minute clock)" diff --git a/teardown.sh b/teardown.sh new file mode 100755 index 0000000..e9c4f1f --- /dev/null +++ b/teardown.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# Deletes the kind cluster and all local exam state. +set -euo pipefail +cd "$(dirname "$0")" +RT="${CONTAINER_RUNTIME:-$(cat .state/runtime 2>/dev/null || echo podman)}" +[ "$RT" = podman ] && export KIND_EXPERIMENTAL_PROVIDER=podman +kind delete cluster --name cka-sim || true +rm -rf .state workspace +echo "cka-sim cluster and local state removed." diff --git a/tracks/__init__.py b/tracks/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tracks/cka/bank_extra.py b/tracks/cka/bank_extra.py new file mode 100644 index 0000000..ff8a1f1 --- /dev/null +++ b/tracks/cka/bank_extra.py @@ -0,0 +1,646 @@ +"""Extra normal-difficulty CKA tasks (ids 21-43). Parameters (%{...}) are drawn per session.""" +from tracks.cka.common import C, NS, ns_pick, name_pick + +QUESTIONS = [ +# ---------------------------------------------------------------- ARCH ---- +{ + "id": 21, "domain": "ARCH", "title": "Helm release install and upgrade", + "params": lambda r: {"ns": ns_pick(r, "helm"), "rel": name_pick(r, "web"), "n": r.choice([3, 4, 5])}, + "task": """ +A Helm chart is at `{WS}/q21/chart` (values: `replicaCount`, `image.*`, `service.type`, `service.port`). + +1. Install it as release `%{rel}` into namespace `%{ns}` (create the namespace) with + `replicaCount=%{n}` overridden. +2. Then **upgrade** the same release so the Service type becomes `NodePort` + (value `service.type`). The release must be at revision 2 or higher. +""", + "setup": NS + r""" +mkns %{ns} +rm -rf "$WS/q21"; mkdir -p "$WS/q21/chart/templates" +cat > "$WS/q21/chart/Chart.yaml" <<'EOF' +apiVersion: v2 +name: chart +description: A small web chart for the exam simulator +type: application +version: 0.1.0 +appVersion: "1.27" +EOF +cat > "$WS/q21/chart/values.yaml" <<'EOF' +replicaCount: 1 +image: + repository: nginx + tag: "1.27" +service: + type: ClusterIP + port: 80 +EOF +cat > "$WS/q21/chart/templates/_helpers.tpl" <<'EOF' +{{- define "chart.fullname" -}} +{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +EOF +cat > "$WS/q21/chart/templates/deployment.yaml" <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "chart.fullname" . }} + labels: {app: {{ include "chart.fullname" . }}} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: {app: {{ include "chart.fullname" . }}} + template: + metadata: + labels: {app: {{ include "chart.fullname" . }}} + spec: + containers: + - name: web + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + ports: [{containerPort: 80}] +EOF +cat > "$WS/q21/chart/templates/service.yaml" <<'EOF' +apiVersion: v1 +kind: Service +metadata: + name: {{ include "chart.fullname" . }} +spec: + type: {{ .Values.service.type }} + selector: {app: {{ include "chart.fullname" . }}} + ports: [{port: {{ .Values.service.port }}, targetPort: 80}] +EOF +command -v helm >/dev/null || echo "warn: helm is not installed on this machine - Q21 cannot be completed or graded without it" +helm -n %{ns} uninstall %{rel} >/dev/null 2>&1 || true +""", + "checks": [ + C("release %{rel} deployed in %{ns}", 1, r"""helm -n %{ns} status %{rel} -o json 2>/dev/null | python3 -c 'import sys,json;sys.exit(0 if json.load(sys.stdin)["info"]["status"]=="deployed" else 1)'"""), + C("replicaCount=%{n} applied", 1, r"""eq "$(j -n %{ns} deploy %{rel}-chart '{.spec.replicas}')" %{n}"""), + C("upgraded: revision >= 2 and Service is NodePort", 2, r""" +rev=$(helm -n %{ns} history %{rel} -o json 2>/dev/null | python3 -c 'import sys,json;print(max(r["revision"] for r in json.load(sys.stdin)))' 2>/dev/null) +[ -n "$rev" ] && [ "$rev" -ge 2 ] && eq "$(j -n %{ns} svc %{rel}-chart '{.spec.type}')" NodePort"""), + ], +}, +{ + "id": 33, "domain": "ARCH", "title": "Aggregated ClusterRole", + "params": lambda r: {"cr": name_pick(r, "ops"), "user": r.choice(["dana", "lee", "sam"])}, + "task": """ +1. Create a ClusterRole `%{cr}` that has **no rules of its own** but aggregates every + ClusterRole labelled `rbac.example.com/aggregate-to-%{cr}=true`. +2. Create two ClusterRoles carrying that label: `%{cr}-pods` (get/list/watch pods) and + `%{cr}-services` (get/list/watch services). +3. Bind `%{cr}` cluster-wide to user `%{user}` with ClusterRoleBinding `%{cr}-binding`. + +`kubectl auth can-i list pods --as=%{user} -A` must say yes; deleting pods must say no. +""", + "setup": r""" +for x in %{cr} %{cr}-pods %{cr}-services; do $K delete clusterrole $x --ignore-not-found >/dev/null; done +$K delete clusterrolebinding %{cr}-binding --ignore-not-found >/dev/null +""", + "checks": [ + C("%{cr} has an aggregationRule with the right label", 1, r"""j clusterrole %{cr} '{.spec.aggregationRule.clusterRoleSelectors[*].matchLabels}' | grep -q 'aggregate-to-%{cr}":"true'"""), + C("aggregation populated rules from the two labelled roles", 2, r"""j clusterrole %{cr} '{.rules[*].resources}' | grep -q pods && j clusterrole %{cr} '{.rules[*].resources}' | grep -q services"""), + C("%{user} can list pods and services cluster-wide", 1, r"""eq "$($K auth can-i list pods -A --as=%{user} 2>/dev/null)" yes && eq "$($K auth can-i list services -A --as=%{user} 2>/dev/null)" yes"""), + C("%{user} cannot delete pods", 1, r"""eq "$($K auth can-i delete pods -A --as=%{user} 2>/dev/null || true)" no"""), + ], +}, +{ + "id": 34, "domain": "ARCH", "title": "Discover a CRD field with kubectl explain", + "params": lambda r: {"kind": r.choice(["Widget", "Gadget", "Gizmo"]), "field": r.choice(["retries", "attempts", "tries"]), "n": r.randint(2, 7)}, + "task": """ +A CRD for kind `%{kind}` (group `demo.io`) is installed. Somewhere in its schema is an +integer field whose description is exactly **"Number of retry attempts"**. + +1. Using `kubectl explain`, find the field's full path (e.g. `spec.foo.bar`) and write it + on one line to `{WS}/q34/path.txt`. +2. Create a `%{kind}` named `sample` in namespace `default` with that field set to `%{n}`. +""", + "setup": r""" +rm -rf "$WS/q34"; mkdir -p "$WS/q34" +K_LOWER=$(echo %{kind} | tr A-Z a-z) +$K delete crd ${K_LOWER}s.demo.io --ignore-not-found >/dev/null +$K apply -f - >/dev/null </dev/null)" = "spec.tuning.%{field}" ]"""), + C("%{kind}/sample created with the field = %{n}", 2, r"""K_LOWER=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${K_LOWER}s.demo.io sample '{.spec.tuning.%{field}}')" %{n}"""), + ], +}, +{ + "id": 40, "domain": "ARCH", "title": "kubeadm join token", + "params": lambda r: {"ttl": r.choice(["2h", "6h", "24h"])}, + "task": """ +A new worker is going to be added to the cluster. + +On the control-plane node (`{SHELL_CP}`), create a **new** bootstrap token with TTL `%{ttl}` +and save the complete `kubeadm join` command for it (with `--discovery-token-ca-cert-hash`) +to `{WS}/q40/join.sh`. +""", + "setup": r""" +rm -rf "$WS/q40"; mkdir -p "$WS/q40" +for t in $($K -n kube-system get secrets -o jsonpath='{.items[?(@.type=="bootstrap.kubernetes.io/token")].metadata.name}'); do $K -n kube-system delete secret $t >/dev/null; done +""", + "checks": [ + C("join.sh contains a kubeadm join with token and CA hash", 2, r"""grep -q 'kubeadm join' "$WS/q40/join.sh" 2>/dev/null && grep -q -- '--token' "$WS/q40/join.sh" && grep -q -- '--discovery-token-ca-cert-hash sha256:' "$WS/q40/join.sh" """), + C("the token in the file exists in the cluster with TTL ~%{ttl}", 3, r""" +tok=$(grep -o -- '--token [a-z0-9]\{6\}\.[a-z0-9]\{16\}' "$WS/q40/join.sh" 2>/dev/null | awk '{print $2}'); [ -n "$tok" ] || exit 1 +id=${tok%%.*}; exp=$(j -n kube-system secret bootstrap-token-$id '{.data.expiration}' | base64 -d 2>/dev/null); [ -n "$exp" ] || exit 1 +secs=$(( $(date -d "$exp" +%s) - $(date +%s) )); want=$(( $(echo %{ttl} | tr -d h) * 3600 )) +[ $secs -gt $((want - 900)) ] && [ $secs -le $((want + 60)) ]"""), + ], +}, +{ + "id": 41, "domain": "ARCH", "title": "Component versions", + "params": lambda r: {"node": r.choice(["{W1}", "{W2}", "{W3}"])}, + "task": """ +Write the following to `{WS}/q41/versions.txt`, one per line, in this exact format: + +``` +apiserver= +etcd= +kubelet= +runtime= +``` + +Use the values the cluster itself reports (`kubectl`, not what you remember). +""", + "setup": r"""rm -rf "$WS/q41"; mkdir -p "$WS/q41" """, + "checks": [ + C("apiserver and kubelet versions correct", 2, r""" +f="$WS/q41/versions.txt"; [ -f "$f" ] || exit 1 +api=$($K version -o json 2>/dev/null | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])') +kub=$(j node %{node} '{.status.nodeInfo.kubeletVersion}') +grep -qx "apiserver=$api" "$f" && grep -qx "kubelet=$kub" "$f" """), + C("etcd and runtime versions correct", 2, r""" +f="$WS/q41/versions.txt" +etcd=$(j -n kube-system pod -l component=etcd '{.items[0].spec.containers[0].image}' | sed 's/.*://') +rt=$(j node %{node} '{.status.nodeInfo.containerRuntimeVersion}') +grep -qx "etcd=$etcd" "$f" && grep -qx "runtime=$rt" "$f" """), + ], +}, +# ---------------------------------------------------------------- WORK ---- +{ + "id": 23, "domain": "WORK", "title": "metrics-server repair and HPA", + "params": lambda r: {"ns": ns_pick(r, "scale"), "dep": name_pick(r, "api"), "max": r.choice([5, 6, 8]), "cpu": r.choice([60, 70, 75])}, + "task": """ +`metrics-server` was installed in `kube-system` but `kubectl top nodes` does not work. + +1. Fix metrics-server so `kubectl top nodes` returns data (hint: in this cluster the kubelet + serving certificates are not signed by the cluster CA). +2. Create an HPA `%{dep}` for Deployment `%{dep}` in namespace `%{ns}`: + min 2, max %{max}, target average CPU utilisation %{cpu}%. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} create deploy %{dep} --image=nginx:1.25 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} set resources deploy %{dep} --requests=cpu=50m >/dev/null +$K -n %{ns} delete hpa %{dep} --ignore-not-found >/dev/null +if ! $K -n kube-system get deploy metrics-server >/dev/null 2>&1; then + curl -sfL --max-time 60 https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -o /tmp/ms.yaml && $K apply -f /tmp/ms.yaml >/dev/null || echo "warn: could not download metrics-server manifest" +fi +# make it broken (kind kubelets use self-signed serving certs) +$K -n kube-system get deploy metrics-server -o json 2>/dev/null | python3 -c ' +import sys,json; d=json.load(sys.stdin); c=d["spec"]["template"]["spec"]["containers"][0] +c["args"]=[a for a in c["args"] if a!="--kubelet-insecure-tls"]; print(json.dumps(d))' | $K apply -f - >/dev/null 2>&1 || true +""", + "checks": [ + C("kubectl top nodes works", 2, r"""for i in $(seq 1 12); do $K top nodes >/dev/null 2>&1 && exit 0; sleep 5; done; exit 1"""), + C("HPA %{dep}: min 2, max %{max}", 1, r"""eq "$(j -n %{ns} hpa %{dep} '{.spec.minReplicas}')" 2 && eq "$(j -n %{ns} hpa %{dep} '{.spec.maxReplicas}')" %{max}"""), + C("HPA targets CPU %{cpu}% on the deployment", 1, r"""eq "$(j -n %{ns} hpa %{dep} '{.spec.scaleTargetRef.name}')" %{dep} && j -n %{ns} hpa %{dep} '{.spec.metrics[*].resource}' | grep -q '"averageUtilization":%{cpu}'"""), + ], +}, +{ + "id": 24, "domain": "WORK", "title": "PriorityClass", + "params": lambda r: {"ns": ns_pick(r, "prio"), "pc": name_pick(r, "critical"), "val": r.choice([100000, 500000, 900000])}, + "task": """ +1. Create a PriorityClass `%{pc}` with value %{val}, not the global default, description "%{pc} workloads". +2. In namespace `%{ns}`, Deployment `payments` already exists. Make its pods use `%{pc}` and + ensure a rollout completes (pods must show priority %{val}). +""", + "setup": NS + r""" +mkns %{ns} +$K delete priorityclass %{pc} --ignore-not-found >/dev/null +$K -n %{ns} create deploy payments --image=nginx:1.25 --replicas=2 --dry-run=client -o yaml | $K apply -f - >/dev/null +""", + "checks": [ + C("PriorityClass %{pc} value %{val}, not default", 2, r"""eq "$(j priorityclass %{pc} '{.value}')" %{val} && [ "$(j priorityclass %{pc} '{.globalDefault}')" != true ]"""), + C("payments pods running with priority %{val}", 2, r"""eq "$(j -n %{ns} deploy payments '{.status.availableReplicas}')" 2 && for p in $(j -n %{ns} pod -l app=payments '{.items[*].spec.priority}'); do eq "$p" %{val} || exit 1; done"""), + ], +}, +{ + "id": 25, "domain": "WORK", "breaks": "w1-taint", "title": "Taints and tolerations", + "params": lambda r: {"ns": ns_pick(r, "sched"), "key": r.choice(["dedicated", "workload", "team"]), "val": r.choice(["gpu", "batch", "finance"]), "dep": name_pick(r, "worker")}, + "task": """ +1. Taint node `{W1}` with `%{key}=%{val}:NoSchedule`. +2. Create Deployment `%{dep}` in namespace `%{ns}` (image `busybox:1.36`, command `sleep 3600`, + 2 replicas) that tolerates that taint **and** is pinned to `{W1}` with a nodeSelector on + `kubernetes.io/hostname`. + +Both pods must be Running on `{W1}`. Do not remove the taint. +""", + "setup": NS + r""" +mkns %{ns} +$K taint node "$W1" %{key}- >/dev/null 2>&1 || true +$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null +""", + "checks": [ + C("{W1} carries %{key}=%{val}:NoSchedule", 1, r"""j node "$W1" '{.spec.taints}' | grep -q '"key":"%{key}"' && j node "$W1" '{.spec.taints}' | grep -q '"value":"%{val}"' """), + C("deployment tolerates the taint", 1, r"""j -n %{ns} deploy %{dep} '{.spec.template.spec.tolerations}' | grep -q '"key":"%{key}"' """), + C("both pods Running on {W1}", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && for n in $(j -n %{ns} pod -l app=%{dep} '{.items[*].spec.nodeName}'); do eq "$n" "$W1" || exit 1; done"""), + ], +}, +{ + "id": 26, "domain": "WORK", "title": "DaemonSet on every node", + "params": lambda r: {"ns": ns_pick(r, "agents"), "ds": name_pick(r, "node-agent")}, + "task": """ +Create a DaemonSet `%{ds}` in namespace `%{ns}` (image `busybox:1.36`, command `sleep 3600`) +that runs on **every** node, including the control-plane. +Requests: cpu 10m, memory 16Mi. +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete ds %{ds} --ignore-not-found >/dev/null""", + "checks": [ + C("DaemonSet exists with requests set", 1, r"""eq "$(j -n %{ns} ds %{ds} '{.spec.template.spec.containers[0].resources.requests.cpu}')" 10m"""), + C("tolerates the control-plane taint", 1, r"""j -n %{ns} ds %{ds} '{.spec.template.spec.tolerations[*].key}' | grep -q control-plane"""), + C("one ready pod per node (including control-plane)", 2, r"""n=$($K get nodes --no-headers 2>/dev/null | wc -l); eq "$(j -n %{ns} ds %{ds} '{.status.numberReady}')" "$n" && eq "$(j -n %{ns} ds %{ds} '{.status.desiredNumberScheduled}')" "$n" """), + ], +}, +{ + "id": 27, "domain": "WORK", "title": "CronJob with a manual run", + "params": lambda r: {"ns": ns_pick(r, "jobs"), "cj": name_pick(r, "report"), "every": r.choice([5, 10, 15])}, + "task": """ +1. Create CronJob `%{cj}` in `%{ns}`: runs every %{every} minutes, image `busybox:1.36`, + command `date`, `concurrencyPolicy: Forbid`, keep 3 successful and 1 failed job, + each run limited to 30 seconds (`activeDeadlineSeconds`), restartPolicy Never. +2. Trigger it manually once as a Job named `%{cj}-manual` and let it complete. +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete cj %{cj} --ignore-not-found >/dev/null; $K -n %{ns} delete job %{cj}-manual --ignore-not-found >/dev/null""", + "checks": [ + C("schedule every %{every} minutes, Forbid, history limits", 2, r"""c="$K -n %{ns} get cj %{cj}"; eq "$($c -o jsonpath='{.spec.schedule}' 2>/dev/null)" "*/%{every} * * * *" && eq "$($c -o jsonpath='{.spec.concurrencyPolicy}')" Forbid && eq "$($c -o jsonpath='{.spec.successfulJobsHistoryLimit}')" 3 && eq "$($c -o jsonpath='{.spec.failedJobsHistoryLimit}')" 1"""), + C("activeDeadlineSeconds 30, restartPolicy Never", 1, r"""eq "$(j -n %{ns} cj %{cj} '{.spec.jobTemplate.spec.activeDeadlineSeconds}')" 30 && eq "$(j -n %{ns} cj %{cj} '{.spec.jobTemplate.spec.template.spec.restartPolicy}')" Never"""), + C("manual job %{cj}-manual completed", 1, r"""for i in $(seq 1 10); do [ "$(j -n %{ns} job %{cj}-manual '{.status.succeeded}')" = 1 ] && exit 0; sleep 3; done; exit 1"""), + ], +}, +{ + "id": 28, "domain": "WORK", "title": "Init container prepares content", + "params": lambda r: {"ns": ns_pick(r, "web"), "pod": name_pick(r, "site"), "text": r.choice(["hello from init", "prepared by init", "init was here"])}, + "task": """ +Create pod `%{pod}` in namespace `%{ns}`: +- init container `setup` (busybox:1.36) writes the text `%{text}` into `/work/index.html`, +- main container `web` (nginx:1.25) serves that file from `/usr/share/nginx/html`, +- both share an `emptyDir` volume named `content`. + +`curl` from inside the cluster to the pod on port 80 must return the text. +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""", + "checks": [ + C("pod Running, init container named setup completed", 1, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.initContainers[0].name}')" setup"""), + C("emptyDir 'content' shared by both containers", 1, r"""[ -n "$(j -n %{ns} pod %{pod} '{.spec.volumes[?(@.name=="content")].emptyDir}')" ] && j -n %{ns} pod %{pod} '{.spec.containers[0].volumeMounts[*].name}' | grep -qw content"""), + C("nginx serves the init-written content", 2, r"""$K -n %{ns} exec %{pod} -c web -- cat /usr/share/nginx/html/index.html 2>/dev/null | grep -q '%{text}'"""), + ], +}, +{ + "id": 29, "domain": "WORK", "title": "Immutable ConfigMap rotation", + "params": lambda r: {"ns": ns_pick(r, "cfg"), "dep": name_pick(r, "app"), "cm": name_pick(r, "settings")}, + "task": """ +Deployment `%{dep}` in `%{ns}` mounts ConfigMap `%{cm}` at `/etc/cfg`. The ConfigMap is +**immutable** and its key `mode` is `old`. + +The application must now read `mode=new`. Without deleting `%{cm}` and without editing it +(it cannot be edited), make the running pods see `new` at `/etc/cfg/mode`. +Hint: version the ConfigMap. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null +$K -n %{ns} delete cm -l q=29 --ignore-not-found >/dev/null; $K -n %{ns} delete cm %{cm} --ignore-not-found >/dev/null +$K apply -f - >/dev/null </dev/null)" new || exit 1; done"""), + ], +}, +{ + "id": 31, "domain": "WORK", "title": "Parallel Job", + "params": lambda r: {"ns": ns_pick(r, "batch"), "job": name_pick(r, "crunch"), "c": r.choice([6, 8, 9]), "p": r.choice([2, 3])}, + "task": """ +Create Job `%{job}` in `%{ns}`: image `busybox:1.36`, command `sh -c "echo done; sleep 2"`, +%{c} completions, %{p} in parallel, `backoffLimit: 2`, restartPolicy Never. Let it finish. +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete job %{job} --ignore-not-found >/dev/null""", + "checks": [ + C("completions %{c}, parallelism %{p}, backoffLimit 2", 2, r"""eq "$(j -n %{ns} job %{job} '{.spec.completions}')" %{c} && eq "$(j -n %{ns} job %{job} '{.spec.parallelism}')" %{p} && eq "$(j -n %{ns} job %{job} '{.spec.backoffLimit}')" 2"""), + C("job succeeded %{c} times", 2, r"""for i in $(seq 1 20); do [ "$(j -n %{ns} job %{job} '{.status.succeeded}')" = %{c} ] && exit 0; sleep 3; done; exit 1"""), + ], +}, +{ + "id": 32, "domain": "WORK", "title": "ResourceQuota and LimitRange", + "params": lambda r: {"ns": ns_pick(r, "team"), "pods": r.choice([4, 5, 6]), "dcpu": r.choice(["100m", "150m", "200m"])}, + "task": """ +In namespace `%{ns}`: +1. ResourceQuota `%{ns}-quota`: at most %{pods} pods, total CPU requests 1, total memory limits 2Gi. +2. LimitRange `%{ns}-defaults`: for containers, default request cpu %{dcpu} / memory 64Mi and + default limit cpu 500m / memory 256Mi. +3. Create Deployment `worker` (nginx:1.25, 2 replicas) **without** specifying resources. + Its pods must be Running with the defaults applied. +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete quota,limitrange --all >/dev/null 2>&1; $K -n %{ns} delete deploy worker --ignore-not-found >/dev/null""", + "checks": [ + C("ResourceQuota limits pods/%{pods}, requests.cpu 1, limits.memory 2Gi", 2, r"""q="$K -n %{ns} get quota %{ns}-quota"; eq "$($q -o jsonpath='{.spec.hard.pods}' 2>/dev/null)" %{pods} && eq "$($q -o jsonpath='{.spec.hard.requests\.cpu}')" 1 && eq "$($q -o jsonpath='{.spec.hard.limits\.memory}')" 2Gi"""), + C("LimitRange defaults set", 1, r"""l="$K -n %{ns} get limitrange %{ns}-defaults"; eq "$($l -o jsonpath='{.spec.limits[0].defaultRequest.cpu}' 2>/dev/null)" %{dcpu} && eq "$($l -o jsonpath='{.spec.limits[0].default.memory}')" 256Mi"""), + C("worker pods Running with defaulted requests", 1, r"""eq "$(j -n %{ns} deploy worker '{.status.availableReplicas}')" 2 && eq "$(j -n %{ns} pod -l app=worker '{.items[0].spec.containers[0].resources.requests.cpu}')" %{dcpu}"""), + ], +}, +# ----------------------------------------------------------------- NET ---- +{ + "id": 22, "domain": "NET", "title": "Gateway API HTTPRoute", + "params": lambda r: {"ns": ns_pick(r, "gw"), "gc": name_pick(r, "lab-gc"), "gw": name_pick(r, "edge"), "host": r.choice(["shop", "api", "app"])}, + "task": """ +1. Install the Gateway API **standard** channel CRDs (v1.2.1 or newer) into the cluster. +2. Create a GatewayClass `%{gc}` (controllerName `example.com/gateway`). +3. In namespace `%{ns}` create Gateway `%{gw}` using that class, one HTTP listener on port 80 + for hostname `%{host}.example.com`. +4. Create HTTPRoute `%{host}-route` attached to `%{gw}` that sends `/api` (PathPrefix) to + Service `api` port 8080 and everything else to Service `web` port 80. + +(No controller runs here; you are graded on the objects.) +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} create deploy api --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} expose deploy api --port=8080 --target-port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K delete gatewayclass %{gc} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete gateway %{gw} --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete httproute %{host}-route --ignore-not-found >/dev/null 2>&1 +""", + "checks": [ + C("Gateway API CRDs installed (gateways, httproutes)", 1, r"""$K get crd gateways.gateway.networking.k8s.io httproutes.gateway.networking.k8s.io >/dev/null 2>&1"""), + C("GatewayClass %{gc} with the controllerName", 1, r"""eq "$(j gatewayclass %{gc} '{.spec.controllerName}')" example.com/gateway"""), + C("Gateway %{gw}: class, HTTP listener :80 for %{host}.example.com", 2, r"""g="$K -n %{ns} get gateway %{gw}"; eq "$($g -o jsonpath='{.spec.gatewayClassName}' 2>/dev/null)" %{gc} && eq "$($g -o jsonpath='{.spec.listeners[0].port}')" 80 && eq "$($g -o jsonpath='{.spec.listeners[0].hostname}')" %{host}.example.com"""), + C("HTTPRoute: parentRef %{gw}, /api -> api:8080, default -> web:80", 2, r""" +h="$K -n %{ns} get httproute %{host}-route"; $h >/dev/null 2>&1 || exit 1 +eq "$($h -o jsonpath='{.spec.parentRefs[0].name}')" %{gw} || exit 1 +$h -o json | python3 -c ' +import sys,json; r=json.load(sys.stdin)["spec"]["rules"] +api=[x for x in r if any(m.get("path",{}).get("value")=="/api" for m in x.get("matches",[]))] +web=[x for x in r if x is not api and x.get("backendRefs",[{}])[0].get("name")=="web"] +ok=api and api[0]["backendRefs"][0]["name"]=="api" and api[0]["backendRefs"][0]["port"]==8080 and web and web[0]["backendRefs"][0]["port"]==80 +sys.exit(0 if ok else 1)'"""), + ], +}, +{ + "id": 30, "domain": "NET", "title": "Named ports", + "params": lambda r: {"ns": ns_pick(r, "ports"), "dep": name_pick(r, "catalog"), "pname": r.choice(["http-web", "app-port", "svc-http"])}, + "task": """ +1. In `%{ns}`, Deployment `%{dep}` (nginx:1.25, 1 replica) must declare its container port 80 + with the **name** `%{pname}`. +2. Create Service `%{dep}` (ClusterIP, port 8080) whose `targetPort` references the port + **by name**, not by number. +3. Add a readiness probe on the container that also references the port by name (HTTP GET /). +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete deploy,svc %{dep} --ignore-not-found >/dev/null""", + "checks": [ + C("container port 80 named %{pname}", 1, r"""eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.containers[0].ports[?(@.containerPort==80)].name}')" %{pname}"""), + C("Service port 8080 -> targetPort %{pname} (by name)", 2, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].port}')" 8080 && eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].targetPort}')" %{pname}"""), + C("readiness probe uses the named port; endpoints ready", 1, r"""eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.containers[0].readinessProbe.httpGet.port}')" %{pname} && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""), + ], +}, +{ + "id": 42, "domain": "NET", "title": "Egress-only NetworkPolicy", + "params": lambda r: {"ns": ns_pick(r, "egress"), "app": name_pick(r, "client"), "cidr": r.choice(["10.96.0.0/12", "172.16.0.0/16", "192.168.100.0/24"])}, + "task": """ +Namespace `%{ns}` has a Deployment `%{app}` (label `app=%{app}`). + +Create NetworkPolicy `%{app}-egress` that applies only to those pods and allows **egress** to: +- DNS (UDP and TCP 53) to pods in `kube-system` labelled `k8s-app=kube-dns`, +- TCP 443 to the CIDR `%{cidr}`, +and denies all other egress. Do not restrict ingress. +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} create deploy %{app} --image=busybox:1.36 --dry-run=client -o yaml -- sleep 3600 | $K apply -f - >/dev/null; $K -n %{ns} delete netpol %{app}-egress --ignore-not-found >/dev/null""", + "checks": [ + C("applies to app=%{app}, Egress only", 1, r"""n="$K -n %{ns} get netpol %{app}-egress"; eq "$($n -o jsonpath='{.spec.podSelector.matchLabels.app}' 2>/dev/null)" %{app} && eq "$($n -o jsonpath='{.spec.policyTypes}')" '["Egress"]'"""), + C("DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP", 2, r""" +$K -n %{ns} get netpol %{app}-egress -o json 2>/dev/null | python3 -c ' +import sys,json; e=json.load(sys.stdin)["spec"].get("egress",[]) +dns=[r for r in e if any((p.get("port")==53 or p.get("port")=="53") for p in r.get("ports",[]))] +ok=False +for r in dns: + protos={p.get("protocol","TCP") for p in r["ports"] if str(p.get("port"))=="53"} + to=r.get("to",[]) + nsok=any("kube-system" in json.dumps(t.get("namespaceSelector",{})) for t in to) + podok=any(t.get("podSelector",{}).get("matchLabels",{}).get("k8s-app")=="kube-dns" for t in to) + if protos>={"UDP","TCP"} and nsok and podok: ok=True +sys.exit(0 if ok else 1)'"""), + C("TCP 443 to %{cidr} only", 1, r""" +$K -n %{ns} get netpol %{app}-egress -o json 2>/dev/null | python3 -c ' +import sys,json; e=json.load(sys.stdin)["spec"].get("egress",[]) +ok=any(any(t.get("ipBlock",{}).get("cidr")=="%{cidr}" for t in r.get("to",[])) and any(str(p.get("port"))=="443" for p in r.get("ports",[])) for r in e) +sys.exit(0 if ok else 1)'"""), + ], +}, +{ + "id": 43, "domain": "NET", "title": "Headless and ExternalName services", + "params": lambda r: {"ns": ns_pick(r, "svc2"), "sts": name_pick(r, "db"), "ext": r.choice(["api.example.com", "db.corp.example.org", "cache.internal.example.net"])}, + "task": """ +1. In `%{ns}`, StatefulSet `%{sts}` (2 replicas) exists but has no Service. Create the **headless** + Service `%{sts}` it needs (port 5432) so that `%{sts}-0.%{sts}.%{ns}.svc.cluster.local` resolves. +2. Create Service `legacy` of type ExternalName pointing at `%{ext}`. +3. From a pod in the namespace, resolve `%{sts}-0.%{sts}` and write the full nslookup output + to `{WS}/q43/pod0.txt`. +""", + "setup": NS + r""" +mkns %{ns}; rm -rf "$WS/q43"; mkdir -p "$WS/q43" +$K -n %{ns} delete svc %{sts} legacy --ignore-not-found >/dev/null +$K apply -f - >/dev/null < %{ext}", 1, r"""eq "$(j -n %{ns} svc legacy '{.spec.type}')" ExternalName && eq "$(j -n %{ns} svc legacy '{.spec.externalName}')" %{ext}"""), + C("pod0.txt shows %{sts}-0 resolving to its pod IP", 1, r"""ip=$(j -n %{ns} pod %{sts}-0 '{.status.podIP}'); [ -n "$ip" ] && grep -q "$ip" "$WS/q43/pod0.txt" 2>/dev/null"""), + ], +}, +# ---------------------------------------------------------------- STOR ---- +{ + "id": 35, "domain": "STOR", "title": "Re-bind a Released PersistentVolume", + "params": lambda r: {"ns": ns_pick(r, "store"), "pv": name_pick(r, "pv-archive"), "pvc": name_pick(r, "archive-claim")}, + "task": """ +PersistentVolume `%{pv}` (1Gi, hostPath, class `manual`) is in status **Released**: the claim that used +it was deleted, but its data must be kept and reused. + +Make `%{pv}` bindable again without deleting it, then create PVC `%{pvc}` in `%{ns}` +(class `manual`, 500Mi, RWO) so it binds to **exactly** that PV. Do not create any other PV. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} delete pvc %{pvc} --ignore-not-found >/dev/null; $K delete pv %{pv} --ignore-not-found >/dev/null +$K apply -f - >/dev/null </dev/null +""", + "checks": [ + C("PV %{pv} still exists and was not recreated", 2, r"""[ -n "$(j pv %{pv} '{.metadata.uid}')" ] && eq "$(j pv %{pv} '{.spec.persistentVolumeReclaimPolicy}')" Retain"""), + C("PVC %{pvc} Bound to %{pv}", 3, r"""eq "$(j -n %{ns} pvc %{pvc} '{.status.phase}')" Bound && eq "$(j -n %{ns} pvc %{pvc} '{.spec.volumeName}')" %{pv}"""), + C("no extra PVs were created", 1, r"""[ "$($K get pv --no-headers 2>/dev/null | grep -c manual)" -le 2 ]"""), + ], +}, +{ + "id": 36, "domain": "STOR", "title": "Data survives the pod", + "params": lambda r: {"ns": ns_pick(r, "persist"), "pvc": name_pick(r, "notes"), "text": r.choice(["order-9981", "ticket-4471", "batch-2026"])}, + "task": """ +1. In `%{ns}` create PVC `%{pvc}` (200Mi, RWO) using the cluster's default StorageClass. +2. Run pod `writer` (busybox:1.36) that mounts it at `/data` and writes the text `%{text}` + into `/data/marker.txt`, then exits (restartPolicy Never). +3. After `writer` has completed, run pod `reader` (busybox:1.36, `sleep 3600`) mounting the + same claim at `/data`. `cat /data/marker.txt` inside `reader` must show the text. +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod writer reader --ignore-not-found --wait=false >/dev/null; $K -n %{ns} delete pvc %{pvc} --ignore-not-found >/dev/null""", + "checks": [ + C("PVC %{pvc} Bound with the default StorageClass", 1, r"""eq "$(j -n %{ns} pvc %{pvc} '{.status.phase}')" Bound && [ -n "$(j -n %{ns} pvc %{pvc} '{.spec.storageClassName}')" ]"""), + C("writer completed (Succeeded)", 1, r"""eq "$(j -n %{ns} pod writer '{.status.phase}')" Succeeded"""), + C("reader sees the marker written by writer", 2, r"""eq "$($K -n %{ns} exec reader -- cat /data/marker.txt 2>/dev/null | tr -d '[:space:]')" %{text}"""), + ], +}, +# -------------------------------------------------------------- TSHOOT ---- +{ + "id": 37, "domain": "TSHOOT", "breaks": "coredns", "title": "Cluster DNS is broken", "disruptive": True, + "params": lambda r: {"bad": r.choice(["forward . 10.255.255.1", "forward . 192.0.2.53"])}, + "task": """ +Pods across the cluster can no longer resolve names (in-cluster or external). +Find the cause in CoreDNS and fix it. `nslookup kubernetes.default` from any pod must work +again, and CoreDNS must be healthy. +""", + "setup": r""" +$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > /tmp/Corefile.orig +sed -i 's|forward \. /etc/resolv.conf|%{bad}|' /tmp/Corefile.orig +$K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.orig --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n kube-system rollout restart deploy coredns >/dev/null +""", + "checks": [ + C("CoreDNS Corefile forwards to the node resolver again", 2, r"""j -n kube-system cm coredns '{.data.Corefile}' | grep -q 'forward \. /etc/resolv.conf'"""), + C("CoreDNS pods available", 1, r"""[ "$(j -n kube-system deploy coredns '{.status.availableReplicas}')" -ge 1 ]"""), + C("in-cluster and external names resolve from a pod", 3, r""" +$K -n default delete pod dns-probe --ignore-not-found --wait=true >/dev/null 2>&1 +$K -n default run dns-probe --image=busybox:1.36 --restart=Never -- sh -c 'for i in 1 2 3 4 5 6; do nslookup kubernetes.default >/dev/null 2>&1 && nslookup kubernetes.io >/dev/null 2>&1 && echo OK && exit 0; sleep 5; done; exit 1' >/dev/null 2>&1 +for i in $(seq 1 20); do ph=$(j -n default pod dns-probe '{.status.phase}'); [ "$ph" = Succeeded ] && { $K -n default delete pod dns-probe --wait=false >/dev/null; exit 0; }; [ "$ph" = Failed ] && break; sleep 3; done +$K -n default delete pod dns-probe --wait=false >/dev/null; exit 1"""), + ], +}, +{ + "id": 38, "domain": "TSHOOT", "breaks": "controller-manager", "title": "Deployments never create pods", "disruptive": True, + "task": """ +Creating a Deployment results in **no** ReplicaSet or pods, and nothing is reported in +`kubectl describe deploy`. Existing pods still run. + +Find the failing control-plane component, repair it on the control-plane node +(`{SHELL_CP}`) and make sure new Deployments get their pods. +""", + "setup": r""" +nx "$CP" ' +f=/etc/kubernetes/manifests/kube-controller-manager.yaml +grep -q "^ *- kube-controller-manager$" $f && sed -i "s|^\( *\)- kube-controller-manager$|\1- kube-controller-manager\n\1- --leader-elect-lease-duration=1s|" $f +' +""", + "checks": [ + C("kube-controller-manager manifest repaired and pod Running", 3, r"""! nx "$CP" 'grep -q "leader-elect-lease-duration=1s" /etc/kubernetes/manifests/kube-controller-manager.yaml' && [ "$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].status.phase}')" = Running ]"""), + C("a new Deployment gets a ReplicaSet and running pods", 4, r""" +$K -n default delete deploy cm-probe --ignore-not-found --wait=true >/dev/null 2>&1 +$K -n default create deploy cm-probe --image=busybox:1.36 -- sleep 300 >/dev/null 2>&1 +ok=1; for i in $(seq 1 24); do [ "$(j -n default deploy cm-probe '{.status.availableReplicas}')" = 1 ] && { ok=0; break; }; sleep 5; done +$K -n default delete deploy cm-probe --wait=false >/dev/null 2>&1; exit $ok"""), + ], +}, +{ + "id": 39, "domain": "TSHOOT", "breaks": "w1-kubelet", "title": "kubelet misconfiguration", "disruptive": True, + "task": """ +Node `{W1}` is `NotReady`. Someone "tuned" the kubelet on it last night. +Bring the node back to `Ready`; the kubelet must be running and enabled. +Reach the node with `{SHELL_W1}`. `journalctl -u kubelet` is your friend. +""", + "setup": r""" +nx "$W1" ' +cp -n /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.orig 2>/dev/null +sed -i "s|clientCAFile: /etc/kubernetes/pki/ca.crt|clientCAFile: /etc/kubernetes/pki/ca-bundle.crt|" /var/lib/kubelet/config.yaml +systemctl restart kubelet +' +""", + "checks": [ + C("kubelet config points at a CA file that exists", 2, r"""f=$(nx "$W1" 'grep clientCAFile /var/lib/kubelet/config.yaml' | awk '{print $2}'); [ -n "$f" ] && nx "$W1" "test -f $f" """), + C("node Ready", 4, r"""for i in $(seq 1 12); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 5; done; exit 1"""), + C("kubelet active and enabled", 1, r"""eq "$(nx "$W1" 'systemctl is-active kubelet')" active && eq "$(nx "$W1" 'systemctl is-enabled kubelet')" enabled"""), + ], +}, +] diff --git a/tracks/cka/bank_hard.py b/tracks/cka/bank_hard.py new file mode 100644 index 0000000..2324682 --- /dev/null +++ b/tracks/cka/bank_hard.py @@ -0,0 +1,660 @@ +"""Hard-difficulty CKA tasks (ids 101-122): multi-part, no hints, traps, exact output, +compound failures, collateral-damage checks, exam-style contexts. Modelled on the kind of +difficulty people report from killer.sh, not on its content.""" +from tracks.cka.common import C, NS, ns_pick, name_pick + +CTX = "Use context: `kubectl config use-context %{ctx}`\n\n" + +def ctx_pick(r): + return r.choice(["k8s-c1-H", "k8s-c2-AC", "k8s-c3-CCC"]) + +QUESTIONS = [ +# ================================================================ ARCH ==== +{ + "id": 101, "domain": "ARCH", "difficulty": "hard", "title": "Exact RBAC with an existing over-grant", + "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "ci"), "sa": name_pick(r, "ci-bot")}, + "task": CTX + """ +ServiceAccount `%{sa}` in namespace `%{ns}` is used by a CI system. Its permissions must be +**exactly**: +- in `%{ns}`: `get`, `list`, `watch`, `patch` on `deployments` and `replicasets` (apps), + and `create` on `pods/exec`, +- cluster-wide: `list` on `nodes`. + +Nothing more. Audit what it can do now and correct it. Use Role `%{sa}-role`, RoleBinding +`%{sa}-rb`, ClusterRole `%{sa}-nodes`, ClusterRoleBinding `%{sa}-crb`. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} create sa %{sa} --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} delete role,rolebinding --all >/dev/null 2>&1 +$K delete clusterrole %{sa}-nodes --ignore-not-found >/dev/null; $K delete clusterrolebinding %{sa}-crb %{sa}-legacy-admin --ignore-not-found >/dev/null +# trap: someone bound cluster-admin to this SA "temporarily" +$K create clusterrolebinding %{sa}-legacy-admin --clusterrole=cluster-admin --serviceaccount=%{ns}:%{sa} >/dev/null +""", + "checks": [ + C("can get/list/watch/patch deployments and replicasets in %{ns}", 2, r"""for res in deployments.apps replicasets.apps; do for v in get list watch patch; do eq "$($K auth can-i $v $res -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes || exit 1; done; done"""), + C("can create pods/exec in %{ns}; can list nodes", 2, r"""eq "$($K auth can-i create pods/exec -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes && eq "$($K auth can-i list nodes --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes"""), + C("no over-grant: cannot delete deployments, cannot get secrets, cannot get nodes", 3, r"""for t in "delete deployments -n %{ns}" "get secrets -n %{ns}" "get nodes" "create pods -n default"; do eq "$($K auth can-i $t --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null || true)" no || exit 1; done"""), + C("the cluster-admin binding is gone and named objects exist", 2, r"""! $K get clusterrolebinding %{sa}-legacy-admin >/dev/null 2>&1 && $K -n %{ns} get role %{sa}-role rolebinding %{sa}-rb >/dev/null 2>&1 && $K get clusterrole %{sa}-nodes clusterrolebinding %{sa}-crb >/dev/null 2>&1"""), + ], +}, +{ + "id": 102, "domain": "ARCH", "breaks": "etcd", "difficulty": "hard", "title": "etcd restore", + "params": lambda r: {"ctx": "k8s-c1-H", "cm": name_pick(r, "restore-proof")}, + "task": CTX + """ +An etcd snapshot taken earlier is at `/opt/cka/backup.db` on the control-plane node. +A ConfigMap `%{cm}` in namespace `default` existed when the snapshot was taken and has since +been deleted. + +**Restore the cluster's etcd from that snapshot** so the ConfigMap exists again. The cluster +must be healthy afterwards (API server up, all kube-system pods Running). +Do not recreate the ConfigMap by hand — its data must come from the snapshot. +""", + "setup": r""" +nx "$CP" 'mkdir -p /opt/cka; rm -f /opt/cka/backup.db' +$K -n default delete cm %{cm} --ignore-not-found >/dev/null +$K -n default create cm %{cm} --from-literal=proof=$(date +%s)-%{cm} >/dev/null +POD=$($K -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].metadata.name}') +$K -n kube-system exec "$POD" -- sh -c 'ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/etcd/q102-backup.db' >/dev/null 2>&1 +nx "$CP" 'mv /var/lib/etcd/q102-backup.db /opt/cka/backup.db' +for bin in etcdctl etcdutl; do $K -n kube-system exec "$POD" -- cat /usr/local/bin/$bin > /tmp/cka-$bin 2>/dev/null && [ -s /tmp/cka-$bin ] && nx "$CP" "cat > /usr/local/bin/$bin && chmod +x /usr/local/bin/$bin" < /tmp/cka-$bin; rm -f /tmp/cka-$bin; done +sleep 2; $K -n default delete cm %{cm} >/dev/null +""", + "checks": [ + C("ConfigMap %{cm} exists again with its original data", 6, r"""j -n default cm %{cm} '{.data.proof}' | grep -q -- '-%{cm}$'"""), + C("etcd uses a restored data directory (manifest changed)", 2, r"""! nx "$CP" 'grep -q "path: /var/lib/etcd$" /etc/kubernetes/manifests/etcd.yaml' """), + C("control plane healthy: etcd, apiserver, scheduler, controller-manager Running", 2, r"""for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done"""), + ], +}, +{ + "id": 103, "domain": "ARCH", "breaks": "w3-kubelet", "difficulty": "hard", "title": "Static pod on a relocated manifest path", + "params": lambda r: {"ctx": "k8s-c1-H", "ns": ns_pick(r, "edge"), "pod": name_pick(r, "edge-proxy")}, + "task": CTX + """ +Exactly one worker node is labelled `tier=edge`. On that node create a **static pod**: +- name `%{pod}`, namespace `%{ns}` (create the namespace), +- image `nginx:1.27`, requests cpu 50m / memory 32Mi, label `role=edge`. + +The mirror pod must be Running. Note that the kubelet on that node may not be configured the +way you expect. +""", + "setup": NS + r""" +mkns %{ns} +for n in "$W1" "$W2" "$W3"; do $K label node $n tier- >/dev/null 2>&1 || true; done +$K label node "$W3" tier=edge --overwrite >/dev/null +nx "$W3" ' +rm -f /etc/kubernetes/manifests/%{pod}*.yaml /etc/kubernetes/static/%{pod}*.yaml 2>/dev/null; mkdir -p /etc/kubernetes/static +sed -i "s|^staticPodPath: .*|staticPodPath: /etc/kubernetes/static|" /var/lib/kubelet/config.yaml +systemctl restart kubelet' +""", + "checks": [ + C("mirror pod %{pod}- Running in %{ns}", 4, r"""eq "$(j -n %{ns} pod "%{pod}-$W3" '{.status.phase}')" Running"""), + C("image, requests and label correct", 2, r"""p="$K -n %{ns} get pod %{pod}-$W3"; eq "$($p -o jsonpath='{.spec.containers[0].image}' 2>/dev/null)" nginx:1.27 && eq "$($p -o jsonpath='{.spec.containers[0].resources.requests.cpu}')" 50m && eq "$($p -o jsonpath='{.metadata.labels.role}')" edge"""), + C("manifest lives in the kubelet's actual staticPodPath", 2, r"""d=$(nx "$W3" 'grep staticPodPath /var/lib/kubelet/config.yaml' | awk '{print $2}'); nx "$W3" "grep -ls %{pod} $d/*.yaml" >/dev/null 2>&1"""), + ], +}, +{ + "id": 104, "domain": "ARCH", "difficulty": "hard", "title": "User onboarding to a working kubeconfig", + "params": lambda r: {"ctx": "k8s-c1-H", "user": r.choice(["maya", "theo", "iris"]), "ns": ns_pick(r, "dev")}, + "task": CTX + """ +Onboard user `%{user}` (group `developers`) end to end: +1. Key + CSR with CN=%{user}, O=developers; Kubernetes CSR object `%{user}`, approved, certificate issued. +2. A **self-contained** kubeconfig at `{WS}/q104/%{user}.kubeconfig` (certs embedded, not file paths) + with a context `%{user}@lab` whose default namespace is `%{ns}`. +3. `%{user}` may do everything to pods, deployments and services in `%{ns}` (edit-level), nothing in other namespaces. + +`kubectl --kubeconfig {WS}/q104/%{user}.kubeconfig get pods` must work with no extra flags. +""", + "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q104"; mkdir -p "$WS/q104"; $K delete csr %{user} --ignore-not-found >/dev/null; $K -n %{ns} delete rolebinding --all >/dev/null 2>&1""", + "checks": [ + C("CSR %{user} approved with certificate, O=developers", 2, r"""[ -n "$(j csr %{user} '{.status.certificate}')" ] && j csr %{user} '{.status.certificate}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'O *= *developers'"""), + C("kubeconfig is self-contained and works without flags", 3, r"""f="$WS/q104/%{user}.kubeconfig"; [ -f "$f" ] || exit 1; grep -q 'client-certificate-data' "$f" && grep -q 'certificate-authority-data' "$f" && ! grep -q 'client-certificate:' "$f" && kubectl --kubeconfig "$f" get pods >/dev/null 2>&1"""), + C("context %{user}@lab defaults to %{ns}", 1, r"""eq "$(kubectl --kubeconfig "$WS/q104/%{user}.kubeconfig" config view -o jsonpath='{.contexts[?(@.name=="%{user}@lab")].context.namespace}')" %{ns}"""), + C("edit rights in %{ns}, none elsewhere", 2, r"""eq "$($K auth can-i create deployments -n %{ns} --as=%{user} 2>/dev/null)" yes && eq "$($K auth can-i delete pods -n %{ns} --as=%{user} 2>/dev/null)" yes && eq "$($K auth can-i get pods -n default --as=%{user} 2>/dev/null || true)" no && eq "$($K auth can-i get secrets -n kube-system --as=%{user} 2>/dev/null || true)" no"""), + ], +}, +{ + "id": 105, "domain": "ARCH", "difficulty": "hard", "title": "Kustomize with generators and patches", + "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "kz"), "rep": r.choice([3, 4]), "lvl": r.choice(["debug", "trace", "warn"])}, + "task": CTX + """ +Base at `{WS}/q105/base` (Deployment `web`, Service `web`). Build overlay `{WS}/q105/overlays/prod` that: +- targets namespace `%{ns}` and adds label `tier=prod` to all resources (labels only, not selectors), +- sets replicas to %{rep} via a **strategic merge patch** file `replicas.yaml`, +- adds a `configMapGenerator` named `web-config` with literal `LOG_LEVEL=%{lvl}` (hash suffix enabled), +- injects that ConfigMap into the container via `envFrom` (JSON 6902 patch file `envfrom.yaml`). + +Apply with `kubectl apply -k`. The Deployment must reference the hashed ConfigMap name. +""", + "setup": r""" +rm -rf "$WS/q105"; mkdir -p "$WS/q105/base" +cat > "$WS/q105/base/kustomization.yaml" <<'EOF' +resources: [deployment.yaml, service.yaml] +EOF +cat > "$WS/q105/base/deployment.yaml" <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: {name: web} +spec: + replicas: 1 + selector: {matchLabels: {app: web}} + template: + metadata: {labels: {app: web}} + spec: + containers: [{name: web, image: nginx:1.25, ports: [{containerPort: 80}]}] +EOF +cat > "$WS/q105/base/service.yaml" <<'EOF' +apiVersion: v1 +kind: Service +metadata: {name: web} +spec: {selector: {app: web}, ports: [{port: 80}]} +EOF +$K delete ns %{ns} --ignore-not-found --wait=false >/dev/null +""", + "checks": [ + C("overlay has replicas.yaml (SMP) and envfrom.yaml (JSON6902)", 1, r"""[ -f "$WS/q105/overlays/prod/replicas.yaml" ] && [ -f "$WS/q105/overlays/prod/envfrom.yaml" ] && grep -q 'op:' "$WS/q105/overlays/prod/envfrom.yaml" """), + C("deployed to %{ns}: %{rep} replicas, tier=prod label, selector untouched", 2, r"""d="$K -n %{ns} get deploy web"; eq "$($d -o jsonpath='{.spec.replicas}' 2>/dev/null)" %{rep} && eq "$($d -o jsonpath='{.metadata.labels.tier}')" prod && [ -z "$($d -o jsonpath='{.spec.selector.matchLabels.tier}')" ]"""), + C("hashed ConfigMap with LOG_LEVEL=%{lvl}, referenced via envFrom", 3, r"""cm=$(j -n %{ns} deploy web '{.spec.template.spec.containers[0].envFrom[0].configMapRef.name}'); [ -n "$cm" ] && echo "$cm" | grep -Eq '^web-config-[a-z0-9]{6,}$' && eq "$(j -n %{ns} cm "$cm" '{.data.LOG_LEVEL}')" %{lvl}"""), + C("pods available and env visible", 1, r"""eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" %{rep} && $K -n %{ns} exec deploy/web -- env 2>/dev/null | grep -q '^LOG_LEVEL=%{lvl}$'"""), + ], +}, +{ + "id": 106, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults", + "params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])}, + "task": CTX + """ +Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural lowercase+s, shortName `%{kind}`-initials) with schema: +- `spec.replicas` integer, **required**, minimum 1, maximum 10, +- `spec.environment` string, enum `prod|stage|dev`, **default** `dev`, +- `spec.image` string, required. +Then: +1. Create `%{kind}` `good` in `default` with replicas 3, image `nginx:1.27`, environment `%{env}`. +2. Save the API server's rejection message for a `%{kind}` with `replicas: 0` to `{WS}/q106/rejected.txt`. +3. Create `%{kind}` `defaulted` with only replicas 2 and image set; its environment must be `dev` after creation. +""", + "setup": r"""rm -rf "$WS/q106"; mkdir -p "$WS/q106"; k=$(echo %{kind} | tr A-Z a-z); $K delete crd ${k}s.deploy.example.com --ignore-not-found >/dev/null""", + "checks": [ + C("CRD exists, Namespaced, replicas has min/max and is required", 2, r"""k=$(echo %{kind} | tr A-Z a-z); c="$K get crd ${k}s.deploy.example.com"; eq "$($c -o jsonpath='{.spec.scope}' 2>/dev/null)" Namespaced && s=$($c -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec}'); echo "$s" | grep -q '"minimum":1' && echo "$s" | grep -q '"maximum":10' && echo "$s" | grep -q '"required":\[' """), + C("good created with environment %{env}", 1, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com good '{.spec.environment}')" %{env}"""), + C("replicas 0 is rejected and the message was saved", 2, r"""grep -qiE 'minimum|greater than|>= 1|invalid' "$WS/q106/rejected.txt" 2>/dev/null"""), + C("defaulting works: defaulted.spec.environment == dev", 2, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com defaulted '{.spec.environment}')" dev"""), + ], +}, +{ + "id": 122, "domain": "ARCH", "breaks": "certs", "difficulty": "hard", "title": "Renew control-plane certificates", + "params": lambda r: {"ctx": "k8s-c1-H"}, + "task": CTX + """ +Security policy requires the control-plane certificates to be renewed now. +On the control-plane node, renew **all** kubeadm-managed certificates, make every control-plane +component pick up the new certificates, and update the admin kubeconfig on the node +(`/etc/kubernetes/admin.conf`) so it uses the renewed client certificate. +Write the new `apiserver` certificate's expiry (as printed by `kubeadm certs check-expiration`) to `{WS}/q122/expiry.txt`. +""", + "setup": r"""rm -rf "$WS/q122"; mkdir -p "$WS/q122"; nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' > "$WS/.q122-before" """, + "checks": [ + C("apiserver certificate was renewed (newer notBefore)", 3, r"""b=$(sed 's/notBefore=//' "$WS/.q122-before"); a=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' | sed 's/notBefore=//'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""), + C("all components restarted and Running with the new certs (apiserver serving cert matches file)", 3, r""" +for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done +srv=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial'); f=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial'); [ -n "$srv" ] && eq "$srv" "$f" """), + C("admin.conf uses the renewed client cert", 2, r"""nx "$CP" 'grep client-certificate-data /etc/kubernetes/admin.conf | awk "{print \$2}" | base64 -d | openssl x509 -noout -startdate' | sed 's/notBefore=//' > /tmp/q122a; b=$(sed 's/notBefore=//' "$WS/.q122-before"); [ "$(date -d "$(cat /tmp/q122a)" +%s)" -gt "$(date -d "$b" +%s)" ]"""), + C("expiry.txt matches kubeadm's report for apiserver", 1, r"""exp=$(nx "$CP" 'kubeadm certs check-expiration 2>/dev/null' | awk '$1=="apiserver"{print $2,$3,$4,$5,$6}'); [ -n "$exp" ] && grep -qF "$exp" "$WS/q122/expiry.txt" 2>/dev/null"""), + ], +}, +# ================================================================ WORK ==== +{ + "id": 107, "domain": "WORK", "breaks": "w1-taint", "difficulty": "hard", "title": "Spread, tolerate, prioritise — under a quota", + "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "spread"), "dep": name_pick(r, "gateway"), "key": r.choice(["pool", "lane"]), "val": r.choice(["ingress", "edge"])}, + "task": CTX + """ +Node `{W1}` is tainted `%{key}=%{val}:NoSchedule`. In namespace `%{ns}` create Deployment `%{dep}` +(nginx:1.27, **3 replicas**) such that: +- exactly one pod runs on each worker node (required pod anti-affinity on `kubernetes.io/hostname`), +- it tolerates the taint on `{W1}`, +- it uses PriorityClass `%{dep}-prio` (value 50000, create it), +- requests cpu 100m / memory 64Mi. + +All 3 pods must be Running. The namespace has a ResourceQuota — respect it or adjust it, but the +quota must still cap pods in the namespace. +""", + "setup": NS + r""" +mkns %{ns} +$K taint node "$W1" %{key}- >/dev/null 2>&1 || true; $K taint node "$W1" %{key}=%{val}:NoSchedule >/dev/null +$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null; $K delete priorityclass %{dep}-prio --ignore-not-found >/dev/null +$K apply -f - >/dev/null <=3)", 2, r"""p=$(j -n %{ns} quota %{ns}-quota '{.spec.hard.pods}'); [ -n "$p" ] && [ "$p" -ge 3 ]"""), + ], +}, +{ + "id": 108, "domain": "WORK", "difficulty": "hard", "title": "Rollback to a specific revision, then pause", + "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "rel"), "dep": name_pick(r, "portal")}, + "task": CTX + """ +Deployment `%{dep}` in `%{ns}` has gone through several releases; the current one is broken. +1. Roll it back to the revision that used image `nginx:1.25` (not simply the previous revision). +2. Scale it to 3 replicas and wait until available. +3. **Pause** the rollout, then change the image to `nginx:1.27` and add env `RELEASE=next`. + Because it is paused, no new ReplicaSet may be created and the running pods must stay on `nginx:1.25`. +Leave it paused. +""", + "setup": NS + r""" +mkns %{ns}; $K -n %{ns} delete deploy %{dep} --ignore-not-found --wait=true >/dev/null +$K -n %{ns} create deploy %{dep} --image=nginx:1.25 >/dev/null +$K -n %{ns} rollout status deploy %{dep} --timeout=90s >/dev/null 2>&1 +$K -n %{ns} set image deploy %{dep} nginx=nginx:1.26 >/dev/null; $K -n %{ns} rollout status deploy %{dep} --timeout=90s >/dev/null 2>&1 +$K -n %{ns} set image deploy %{dep} nginx=nginx:1.27 >/dev/null; $K -n %{ns} rollout status deploy %{dep} --timeout=90s >/dev/null 2>&1 +$K -n %{ns} set image deploy %{dep} nginx=nginx:9.99-broken >/dev/null +""", + "checks": [ + C("running pods use nginx:1.25 and 3 are available", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 && for i in $(j -n %{ns} pod -l app=%{dep} '{.items[*].spec.containers[0].image}'); do eq "$i" nginx:1.25 || exit 1; done"""), + C("deployment is paused with template image nginx:1.27 and RELEASE=next", 2, r"""d="$K -n %{ns} get deploy %{dep}"; eq "$($d -o jsonpath='{.spec.paused}' 2>/dev/null)" true && eq "$($d -o jsonpath='{.spec.template.spec.containers[0].image}')" nginx:1.27 && eq "$($d -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="RELEASE")].value}')" next"""), + C("no ReplicaSet exists for the paused template (1.27 + RELEASE)", 2, r"""! $K -n %{ns} get rs -o json 2>/dev/null | python3 -c ' +import sys,json +for rs in json.load(sys.stdin)["items"]: + c=rs["spec"]["template"]["spec"]["containers"][0] + if c["image"]=="nginx:1.27" and any(e.get("name")=="RELEASE" for e in c.get("env",[])): sys.exit(0) +sys.exit(1)'"""), + ], +}, +{ + "id": 109, "domain": "WORK", "difficulty": "hard", "title": "Native sidecar with probes and an exact report", + "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "side"), "pod": name_pick(r, "collector")}, + "task": CTX + """ +Create pod `%{pod}` in `%{ns}`: +- a **native sidecar** (an init container with `restartPolicy: Always`) named `agent` (busybox:1.36) + that appends `$(date)` to `/var/log/agent.log` every second, +- main container `app` (nginx:1.27) with a readiness probe (HTTP GET / on 80, period 5s) + and a liveness probe (TCP 80), +- shared `emptyDir` `logs` mounted at `/var/log` in both, +- `terminationGracePeriodSeconds: 5`. +When it is Running, write the names of **all** containers of the pod (init and regular), sorted, +one per line, to `{WS}/q109/containers.txt`, and the pod's `.status.podIP` to `{WS}/q109/ip.txt`. +""", + "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q109"; mkdir -p "$WS/q109"; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""", + "checks": [ + C("native sidecar: initContainer agent with restartPolicy Always", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.spec.initContainers[?(@.name=="agent")].restartPolicy}')" Always"""), + C("probes on app as specified", 2, r"""c='{.spec.containers[?(@.name=="app")]'; eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.httpGet.path}")" / && eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.periodSeconds}")" 5 && eq "$(j -n %{ns} pod %{pod} "$c.livenessProbe.tcpSocket.port}")" 80"""), + C("Running, shared log written by agent, grace period 5", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.terminationGracePeriodSeconds}')" 5 && [ -n "$($K -n %{ns} exec %{pod} -c app -- sh -c 'tail -1 /var/log/agent.log' 2>/dev/null)" ]"""), + C("report files exact", 2, r"""want=$( (j -n %{ns} pod %{pod} '{.spec.initContainers[*].name}'; echo; j -n %{ns} pod %{pod} '{.spec.containers[*].name}') | tr ' ' '\n' | grep . | sort); [ "$(grep . "$WS/q109/containers.txt" 2>/dev/null | sort)" = "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q109/ip.txt" 2>/dev/null)" "$(j -n %{ns} pod %{pod} '{.status.podIP}')" """), + ], +}, +{ + "id": 110, "domain": "WORK", "difficulty": "hard", "title": "Cluster audit to exact files", + "params": lambda r: {"ctx": "k8s-c1-H"}, + "task": CTX + """ +Produce three files, contents **exactly** as specified (no headers, sorted with `sort`, one per line): +1. `{WS}/q110/no-readiness.txt` — `/` for every pod in the cluster whose first + container has **no** readiness probe. Exclude namespace `kube-system`. +2. `{WS}/q110/per-node.txt` — ` ` of Running pods per node, all namespaces. +3. `{WS}/q110/images.txt` — every distinct container image running in namespace `kube-system`. +""", + "setup": r"""rm -rf "$WS/q110"; mkdir -p "$WS/q110" """, + "checks": [ + C("no-readiness.txt exact", 3, r"""want=$($K get pods -A -o json 2>/dev/null | python3 -c ' +import sys,json +for p in json.load(sys.stdin)["items"]: + if p["metadata"]["namespace"]=="kube-system": continue + if not p["spec"]["containers"][0].get("readinessProbe"): print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort); [ "$(grep . "$WS/q110/no-readiness.txt" 2>/dev/null | sort)" = "$want" ]"""), + C("per-node.txt exact", 2, r"""want=$($K get pods -A -o json 2>/dev/null | python3 -c ' +import sys,json,collections +c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"] if p["status"]["phase"]=="Running" and p["spec"].get("nodeName")) +for n in sorted(c): print(n, c[n])'); [ "$(grep . "$WS/q110/per-node.txt" 2>/dev/null | sort)" = "$(echo "$want" | sort)" ]"""), + C("images.txt exact", 2, r"""want=$(j -n kube-system pods '{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u); [ "$(grep . "$WS/q110/images.txt" 2>/dev/null | sort -u)" = "$want" ]"""), + ], +}, +{ + "id": 121, "domain": "WORK", "difficulty": "hard", "title": "The context's namespace", + "params": lambda r: {"ctx": "k8s-c2-AC", "ns": ns_pick(r, "ac"), "dep": name_pick(r, "ledger")}, + "task": CTX + """ +In the **current namespace of that context** (do not pass `-n`), create Deployment `%{dep}` +(busybox:1.36, `sleep 3600`, 2 replicas, label `owner=ac`) and a ClusterIP Service `%{dep}` +on port 9000 targeting port 9000. Then write the namespace name you deployed into, as reported +by `kubectl config view --minify`, to `{WS}/q121/ns.txt`. +""", + "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q121"; mkdir -p "$WS/q121"; $K -n %{ns} delete deploy,svc %{dep} --ignore-not-found >/dev/null; $K -n default delete deploy,svc %{dep} --ignore-not-found >/dev/null""", + "checks": [ + C("Deployment %{dep} in %{ns} (the context's namespace), 2 available, owner=ac", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && eq "$(j -n %{ns} deploy %{dep} '{.spec.template.metadata.labels.owner}')" ac"""), + C("Service %{dep} port 9000 in the same namespace", 1, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].port}')" 9000"""), + C("nothing was created in namespace default by mistake", 1, r"""! $K -n default get deploy %{dep} >/dev/null 2>&1"""), + C("ns.txt has the namespace", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q121/ns.txt" 2>/dev/null)" %{ns}"""), + ], +}, +# ================================================================= NET ==== +{ + "id": 111, "domain": "NET", "difficulty": "hard", "title": "Zero-trust namespace", + "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "zt"), "from": ns_pick(r, "front")}, + "task": CTX + """ +Namespace `%{ns}` runs `api` (label `app=api`, port 8080) and `db` (label `app=db`, port 5432). +Implement zero trust with NetworkPolicies: +1. `default-deny`: deny all ingress **and** egress for every pod in `%{ns}`. +2. `allow-dns`: every pod may egress to DNS (UDP/TCP 53) in `kube-system` only. +3. `api-ingress`: `api` accepts ingress on 8080 only from pods in namespaces labelled `team=%{from}`. +4. `api-to-db`: `api` may egress to `db` on 5432; `db` accepts ingress on 5432 from `api` only. +Remove anything that contradicts this. Label namespace `%{from}` appropriately. +""", + "setup": NS + r""" +mkns %{ns}; mkns %{from} +$K -n %{ns} create deploy api --image=nginx:1.25 --port=8080 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} create deploy db --image=nginx:1.25 --port=5432 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n %{ns} delete netpol --all >/dev/null 2>&1 +# trap: an allow-all left behind +$K apply -f - >/dev/null </dev/null 2>&1 || true +""", + "checks": [ + C("default-deny covers Ingress and Egress for all pods; allow-all removed", 2, r"""n="$K -n %{ns} get netpol default-deny"; eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' && $n -o jsonpath='{.spec.policyTypes}' | grep -q Ingress && $n -o jsonpath='{.spec.policyTypes}' | grep -q Egress && [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ] && ! $K -n %{ns} get netpol temp-allow-all >/dev/null 2>&1"""), + C("allow-dns: egress 53 UDP+TCP to kube-system only", 2, r"""$K -n %{ns} get netpol allow-dns -o json 2>/dev/null | python3 -c ' +import sys,json; s=json.load(sys.stdin)["spec"]; e=s.get("egress",[]) +ok=s.get("podSelector")=={} and any({p.get("protocol") for p in r.get("ports",[]) if str(p.get("port"))=="53"}>={"UDP","TCP"} and all("kube-system" in json.dumps(t.get("namespaceSelector",{})) for t in r.get("to",[])) and r.get("to") for r in e) +sys.exit(0 if ok else 1)'"""), + C("api-ingress: from namespaces team=%{from} on 8080 only; namespace labelled", 2, r"""eq "$(j ns %{from} '{.metadata.labels.team}')" %{from} && $K -n %{ns} get netpol api-ingress -o json 2>/dev/null | python3 -c ' +import sys,json; s=json.load(sys.stdin)["spec"] +ok=s["podSelector"].get("matchLabels",{}).get("app")=="api" and len(s.get("ingress",[]))==1 and all(str(p.get("port"))=="8080" for p in s["ingress"][0].get("ports",[])) and s["ingress"][0].get("ports") and all(t.get("namespaceSelector",{}).get("matchLabels",{}).get("team")=="%{from}" for t in s["ingress"][0].get("from",[])) and s["ingress"][0].get("from") +sys.exit(0 if ok else 1)'"""), + C("api-to-db: api egress to db:5432 and db ingress from api:5432", 2, r"""$K -n %{ns} get netpol -o json 2>/dev/null | python3 -c ' +import sys,json; items=json.load(sys.stdin)["items"]; eg=ing=False +for n in items: + s=n["spec"]; app=s["podSelector"].get("matchLabels",{}).get("app") + for r in s.get("egress",[]): + if app=="api" and any(t.get("podSelector",{}).get("matchLabels",{}).get("app")=="db" for t in r.get("to",[])) and any(str(p.get("port"))=="5432" for p in r.get("ports",[])): eg=True + for r in s.get("ingress",[]): + if app=="db" and any(t.get("podSelector",{}).get("matchLabels",{}).get("app")=="api" for t in r.get("from",[])) and any(str(p.get("port"))=="5432" for p in r.get("ports",[])): ing=True +sys.exit(0 if eg and ing else 1)'"""), + ], +}, +{ + "id": 112, "domain": "NET", "difficulty": "hard", "title": "TLS Ingress with a generated certificate", + "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "tls"), "host": r.choice(["secure.lab.local", "portal.lab.local", "admin.lab.local"])}, + "task": CTX + """ +Namespace `%{ns}` has Deployment+Service `web` (port 80). +1. Generate a self-signed certificate (RSA 2048, 365 days) with CN `%{host}` and store it as a + Secret `%{host}-tls` of type `kubernetes.io/tls`. +2. Create Ingress `web` (class `nginx`) serving host `%{host}` with that TLS secret, path `/` Prefix → `web:80`, + and the annotation `nginx.ingress.kubernetes.io/ssl-redirect: "true"`. +3. Write the certificate's SHA-256 fingerprint (`openssl x509 -fingerprint -sha256`, the value only) to `{WS}/q112/fp.txt`. +""", + "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q112"; mkdir -p "$WS/q112"; $K -n %{ns} create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} delete secret %{host}-tls --ignore-not-found >/dev/null; $K -n %{ns} delete ingress web --ignore-not-found >/dev/null""", + "checks": [ + C("Secret %{host}-tls is kubernetes.io/tls with CN %{host}", 2, r"""eq "$(j -n %{ns} secret %{host}-tls '{.type}')" kubernetes.io/tls && j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'CN *= *%{host}'"""), + C("Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", 3, r"""i="$K -n %{ns} get ingress web"; eq "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" nginx && eq "$($i -o jsonpath='{.spec.tls[0].secretName}')" %{host}-tls && eq "$($i -o jsonpath='{.spec.tls[0].hosts[0]}')" %{host} && eq "$($i -o jsonpath='{.spec.rules[0].host}')" %{host} && eq "$($i -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}')" web && eq "$($i -o jsonpath='{.metadata.annotations.nginx\.ingress\.kubernetes\.io/ssl-redirect}')" true"""), + C("fp.txt matches the secret's certificate fingerprint", 2, r"""fp=$(j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//'); [ -n "$fp" ] && grep -qiF "$fp" "$WS/q112/fp.txt" 2>/dev/null"""), + ], +}, +{ + "id": 113, "domain": "NET", "difficulty": "hard", "title": "Gateway API canary with header match", + "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "canary"), "host": r.choice(["shop", "pay", "cart"]), "w": r.choice([10, 20, 25])}, + "task": CTX + """ +Gateway API CRDs are installed. Namespace `%{ns}` has Services `v1` and `v2` (port 80). +Create in `%{ns}`: +- Gateway `%{host}-gw` (class `%{host}-gc`, create the GatewayClass, controllerName `example.com/gw`) + with two HTTP listeners: `web` for `%{host}.example.com` and `api` for `api.%{host}.example.com`, both port 80. +- HTTPRoute `%{host}-route` attached to listener `web` only (`sectionName`), with: + - a rule matching header `X-Canary: always` → `v2` (100%), + - a default rule splitting `v1` %{w2}% / `v2` %{w}% by weight. +""", + "params_post": None, + "setup": NS + r""" +mkns %{ns} +$K get crd gateways.gateway.networking.k8s.io >/dev/null 2>&1 || { curl -sfL --max-time 60 https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml -o /tmp/gwapi.yaml && $K apply -f /tmp/gwapi.yaml >/dev/null 2>&1; } || echo "warn: could not install Gateway API CRDs" +for v in v1 v2; do $K -n %{ns} create deploy $v --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} expose deploy $v --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; done +$K delete gatewayclass %{host}-gc --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete gateway %{host}-gw --ignore-not-found >/dev/null 2>&1; $K -n %{ns} delete httproute %{host}-route --ignore-not-found >/dev/null 2>&1 +""", + "checks": [ + C("Gateway with listeners web and api on the right hostnames", 2, r"""g="$K -n %{ns} get gateway %{host}-gw"; eq "$($g -o jsonpath='{.spec.gatewayClassName}' 2>/dev/null)" %{host}-gc && eq "$($g -o jsonpath='{.spec.listeners[?(@.name=="web")].hostname}')" %{host}.example.com && eq "$($g -o jsonpath='{.spec.listeners[?(@.name=="api")].hostname}')" api.%{host}.example.com"""), + C("route attaches to listener web only (sectionName)", 1, r"""eq "$(j -n %{ns} httproute %{host}-route '{.spec.parentRefs[0].sectionName}')" web && [ -z "$(j -n %{ns} httproute %{host}-route '{.spec.parentRefs[1]}')" ]"""), + C("header rule X-Canary: always -> v2", 2, r"""$K -n %{ns} get httproute %{host}-route -o json 2>/dev/null | python3 -c ' +import sys,json; r=json.load(sys.stdin)["spec"]["rules"] +ok=any(any(any(h.get("name","").lower()=="x-canary" and h.get("value")=="always" for h in m.get("headers",[])) for m in x.get("matches",[])) and x["backendRefs"][0]["name"]=="v2" and len(x["backendRefs"])==1 for x in r) +sys.exit(0 if ok else 1)'"""), + C("default rule weights v1 %{w2} / v2 %{w}", 2, r"""$K -n %{ns} get httproute %{host}-route -o json 2>/dev/null | python3 -c ' +import sys,json; r=json.load(sys.stdin)["spec"]["rules"] +ok=any(not x.get("matches") or all(not m.get("headers") for m in x["matches"]) and {b["name"]:b.get("weight") for b in x["backendRefs"]}=={"v1":%{w2},"v2":%{w}} for x in r if len(x.get("backendRefs",[]))==2) +sys.exit(0 if ok else 1)'"""), + ], +}, +{ + "id": 114, "domain": "NET", "breaks": "coredns", "difficulty": "hard", "title": "DNS down: three faults", "disruptive": True, + "params": lambda r: {"ctx": "k8s-c1-H"}, + "task": CTX + """ +Nothing in the cluster resolves DNS. Restore it. There is more than one cause. +When done, `nslookup kubernetes.default` and `nslookup kubernetes.io` must succeed from a pod +in `default`, CoreDNS must have 2 available replicas, and no NetworkPolicy may remain in `kube-system`. +""", + "setup": r""" +$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > /tmp/Corefile.q114 +sed -i 's|forward \. /etc/resolv.conf|forward . 10.255.255.254|' /tmp/Corefile.q114 +$K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.q114 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n kube-system scale deploy coredns --replicas=0 >/dev/null +$K apply -f - >/dev/null </dev/null | wc -l)" = 0 ]"""), + C("resolution works from a pod", 3, r""" +$K -n default delete pod dns-probe --ignore-not-found --wait=true >/dev/null 2>&1 +$K -n default run dns-probe --image=busybox:1.36 --restart=Never -- sh -c 'for i in 1 2 3 4 5 6; do nslookup kubernetes.default >/dev/null 2>&1 && nslookup kubernetes.io >/dev/null 2>&1 && exit 0; sleep 5; done; exit 1' >/dev/null 2>&1 +for i in $(seq 1 20); do ph=$(j -n default pod dns-probe '{.status.phase}'); [ "$ph" = Succeeded ] && { $K -n default delete pod dns-probe --wait=false >/dev/null; exit 0; }; [ "$ph" = Failed ] && break; sleep 3; done +$K -n default delete pod dns-probe --wait=false >/dev/null; exit 1"""), + ], +}, +# ================================================================ STOR ==== +{ + "id": 115, "domain": "STOR", "difficulty": "hard", "title": "Bind the right local volume", + "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "lv"), "pvc": name_pick(r, "fast-data")}, + "task": CTX + """ +Two PersistentVolumes of class `local-fast` exist. Only one of them lives on the node labelled +`disk=nvme`. Create PVC `%{pvc}` in `%{ns}` (RWO, 1Gi, class `local-fast`) that binds to **that** +PV — use a label selector, do not edit the PVs — and pod `consumer` (busybox:1.36, `sleep 3600`) +mounting it at `/data`. The pod must run on the nvme node. Do not delete either PV. +""", + "setup": NS + r""" +mkns %{ns} +for n in "$W1" "$W2" "$W3"; do $K label node $n disk- >/dev/null 2>&1 || true; done +$K label node "$W2" disk=nvme --overwrite >/dev/null +$K -n %{ns} delete pod consumer --ignore-not-found --wait=false >/dev/null; $K -n %{ns} delete pvc %{pvc} --ignore-not-found >/dev/null +$K delete pv lv-hdd lv-nvme --ignore-not-found >/dev/null +for pair in "lv-hdd:$W1:hdd:2Gi" "lv-nvme:$W2:nvme:1Gi"; do IFS=: read name node kind size <<<"$pair" +nx "$node" "mkdir -p /mnt/$name" +$K apply -f - >/dev/null </dev/null; $K -n %{ns} delete job migrate --ignore-not-found >/dev/null; $K -n %{ns} delete pvc old-data new-data --ignore-not-found >/dev/null +$K apply -f - >/dev/null < /data/file-\$i.txt; done; sleep 3600"], volumeMounts: [{name: d, mountPath: /data}]}] + volumes: [{name: d, persistentVolumeClaim: {claimName: old-data}}] +EOF +for i in $(seq 1 30); do [ "$(j -n %{ns} pod legacy '{.status.phase}')" = Running ] && break; sleep 2; done +pv=$(j -n %{ns} pvc old-data '{.spec.volumeName}'); [ -n "$pv" ] && $K patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' >/dev/null +echo "$pv" > "$WS/.q116-pv" +""", + "checks": [ + C("Job migrate completed", 2, r"""eq "$(j -n %{ns} job migrate '{.status.succeeded}')" 1"""), + C("legacy Running on new-data with all %{n} files intact", 3, r"""eq "$(j -n %{ns} pod legacy '{.status.phase}')" Running && eq "$(j -n %{ns} pod legacy '{.spec.volumes[?(@.name!="kube-api-access")].persistentVolumeClaim.claimName}' | tr ' ' '\n' | grep -v '^$' | head -1)" new-data && eq "$($K -n %{ns} exec legacy -- sh -c 'ls /data/file-*.txt | wc -l' 2>/dev/null | tr -d ' ')" %{n} && eq "$($K -n %{ns} exec legacy -- cat /data/file-%{n}.txt 2>/dev/null)" record-%{n}"""), + C("old-data deleted; its PV Released (not deleted)", 2, r"""! $K -n %{ns} get pvc old-data >/dev/null 2>&1 && pv=$(cat "$WS/.q116-pv" 2>/dev/null) && eq "$(j pv $pv '{.status.phase}')" Released"""), + ], +}, +# ============================================================== TSHOOT ==== +{ + "id": 117, "domain": "TSHOOT", "breaks": "w2-kubelet", "difficulty": "hard", "title": "Worker node: three things wrong", "disruptive": True, + "params": lambda r: {"ctx": "k8s-c1-H"}, + "task": CTX + """ +Node `{W2}` is unusable. Make it fully usable again: `Ready`, schedulable, kubelet running and +enabled, and a new pod pinned to it must reach Running. Nothing else on the cluster may be changed. +""", + "setup": r""" +$K cordon "$W2" >/dev/null 2>&1 || true +nx "$W2" ' +sed -i "s|clientCAFile: /etc/kubernetes/pki/ca.crt|clientCAFile: /etc/kubernetes/pki/ca-wrong.crt|" /var/lib/kubelet/config.yaml +systemctl disable kubelet >/dev/null 2>&1; systemctl stop kubelet' +""", + "checks": [ + C("node Ready", 3, r"""for i in $(seq 1 12); do [ "$(j node "$W2" '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 5; done; exit 1"""), + C("node uncordoned (collateral check)", 2, r"""[ "$(j node "$W2" '{.spec.unschedulable}')" != true ]"""), + C("kubelet active and enabled, config points at a real CA", 2, r"""eq "$(nx "$W2" 'systemctl is-active kubelet')" active && eq "$(nx "$W2" 'systemctl is-enabled kubelet')" enabled && f=$(nx "$W2" 'grep clientCAFile /var/lib/kubelet/config.yaml' | awk '{print $2}') && nx "$W2" "test -f $f" """), + C("a pod pinned to the node runs", 2, r""" +$K -n default delete pod node-probe --ignore-not-found --wait=true >/dev/null 2>&1 +$K -n default run node-probe --image=busybox:1.36 --restart=Never --overrides="{\"spec\":{\"nodeName\":\"$W2\"}}" -- sleep 120 >/dev/null 2>&1 +ok=1; for i in $(seq 1 20); do [ "$(j -n default pod node-probe '{.status.phase}')" = Running ] && { ok=0; break; }; sleep 3; done +$K -n default delete pod node-probe --wait=false >/dev/null 2>&1; exit $ok"""), + ], +}, +{ + "id": 118, "domain": "TSHOOT", "breaks": "scheduler", "difficulty": "hard", "title": "Two control-plane components down", "disruptive": True, + "params": lambda r: {"ctx": "k8s-c1-H"}, + "task": CTX + """ +New Deployments create no pods, and pods that do get created stay `Pending`. Two different +control-plane components are failing for two different reasons. Repair both on the control-plane node. +Afterwards a new Deployment must reach Available within a minute. +""", + "setup": r""" +nx "$CP" ' +f=/etc/kubernetes/manifests/kube-scheduler.yaml; grep -q "^ *- kube-scheduler$" $f && sed -i "s|^\( *\)- kube-scheduler$|\1- kube-schedulerr|" $f +g=/etc/kubernetes/manifests/kube-controller-manager.yaml; grep -q "image: registry.k8s.io/kube-controller-manager:v" $g && sed -i "s|\(image: registry.k8s.io/kube-controller-manager:v[0-9.]*\)|\1-broken|" $g +' +""", + "checks": [ + C("kube-scheduler manifest repaired and Running", 3, r"""nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml' && eq "$(j -n kube-system pod -l component=kube-scheduler '{.items[0].status.phase}')" Running"""), + C("kube-controller-manager image repaired and Running", 3, r"""! nx "$CP" 'grep -q "\-broken" /etc/kubernetes/manifests/kube-controller-manager.yaml' && eq "$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].status.phase}')" Running"""), + C("a new Deployment becomes Available", 3, r""" +$K -n default delete deploy cp-probe --ignore-not-found --wait=true >/dev/null 2>&1 +$K -n default create deploy cp-probe --image=busybox:1.36 -- sleep 300 >/dev/null 2>&1 +ok=1; for i in $(seq 1 20); do [ "$(j -n default deploy cp-probe '{.status.availableReplicas}')" = 1 ] && { ok=0; break; }; sleep 3; done +$K -n default delete deploy cp-probe --wait=false >/dev/null 2>&1; exit $ok"""), + ], +}, +{ + "id": 119, "domain": "TSHOOT", "difficulty": "hard", "title": "Application down for four reasons", + "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "checkout"), "dep": name_pick(r, "checkout")}, + "task": CTX + """ +Deployment `%{dep}` in `%{ns}` should have 3 Available replicas and Service `%{dep}` should route +to them on port 80. Right now nothing works. Fix it **in place** — the Deployment must keep its +UID (do not delete and recreate it) and the Service must keep its ClusterIP. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} delete deploy,svc,quota,secret --all >/dev/null 2>&1 +$K apply -f - >/dev/null < "$WS/.q119-uid"; j -n %{ns} svc %{dep} '{.spec.clusterIP}' > "$WS/.q119-ip" +""", + "checks": [ + C("3 replicas Available", 3, r"""for i in $(seq 1 6); do [ "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" = 3 ] && exit 0; sleep 5; done; exit 1"""), + C("Service routes: endpoints ready, targetPort 80", 2, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].targetPort}')" 80 && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""), + C("Deployment UID and Service ClusterIP unchanged (fixed in place)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.metadata.uid}')" "$(cat "$WS/.q119-uid")" && eq "$(j -n %{ns} svc %{dep} '{.spec.clusterIP}')" "$(cat "$WS/.q119-ip")" """), + C("quota still present and permits the 3 pods", 1, r"""p=$(j -n %{ns} quota tight '{.spec.hard.pods}'); [ -n "$p" ] && [ "$p" -ge 3 ]"""), + ], +}, +{ + "id": 120, "domain": "TSHOOT", "difficulty": "hard", "title": "Cluster facts, exactly", + "params": lambda r: {"ctx": "k8s-c1-H", "node": r.choice(["{W1}", "{W2}", "{W3}"])}, + "task": CTX + """ +Write `{WS}/q120/facts.txt` with these lines, values taken from the running cluster: +``` +pod-cidr= +service-cidr= +dns-ip= +proxy-mode= +%{node}-pod-cidr=<.spec.podCIDR of that node> +``` +""", + "setup": r"""rm -rf "$WS/q120"; mkdir -p "$WS/q120" """, + "checks": [ + C("pod-cidr and service-cidr", 2, r"""f="$WS/q120/facts.txt"; pc=$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'cluster-cidr=[^"]*' | cut -d= -f2); sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2); grep -qx "pod-cidr=$pc" "$f" 2>/dev/null && grep -qx "service-cidr=$sc" "$f" """), + C("dns-ip and proxy-mode", 2, r"""f="$WS/q120/facts.txt"; d=$(j -n kube-system svc kube-dns '{.spec.clusterIP}'); m=$(j -n kube-system cm kube-proxy '{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"'); [ -n "$m" ] || m=iptables; grep -qx "dns-ip=$d" "$f" 2>/dev/null && grep -qx "proxy-mode=$m" "$f" """), + C("%{node} pod CIDR", 1, r"""grep -qx "%{node}-pod-cidr=$(j node %{node} '{.spec.podCIDR}')" "$WS/q120/facts.txt" 2>/dev/null"""), + ], +}, +] + +# derived params: w2 = 100 - w for the canary split +for _q in QUESTIONS: + if _q["id"] == 113: + _orig = _q["params"] + _q["params"] = lambda r, _o=_orig: (lambda p: {**p, "w2": 100 - p["w"]})(_o(r)) diff --git a/tracks/cka/bank_sim_a.py b/tracks/cka/bank_sim_a.py new file mode 100644 index 0000000..30b56e4 --- /dev/null +++ b/tracks/cka/bank_sim_a.py @@ -0,0 +1,1048 @@ +"""Simulator Set A - 17 tasks, fixed set, killer.sh-grade difficulty. + +Original scenarios (not copies of any commercial simulator) exercising the same techniques: +investigate-and-write-to-file, node-level work via crictl/openssl/systemd, Kustomize overlays, +Helm with CRDs, QoS reasoning, Gateway API conversion, NetworkPolicy match semantics, CoreDNS. + +Run with: ./exam sim a +""" +from tracks.cka.common import C, NS + +Q = [] + + +def task(**kw): + kw.setdefault("difficulty", "hard") + kw["set"] = "a" + Q.append(kw) + + +# ------------------------------------------------------------------ 201 ---- +task( + id=201, domain="ARCH", title="kubeconfig forensics", + task=""" +The file `{WS}/q201/kubeconfig` was handed over by another team. + +1. Write every context name, one per line, to `{WS}/q201/contexts`. +2. Write the name of the current context to `{WS}/q201/current-context`. +3. The user `svc-reporting` has an embedded client certificate. Write it **base64-decoded** + (the PEM block) to `{WS}/q201/cert`. +4. Write the Common Name (CN) of that certificate to `{WS}/q201/cn`. + +Do not modify the kubeconfig. +""", + setup=r""" +rm -rf "$WS/q201"; mkdir -p "$WS/q201"; cd "$WS/q201" +openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=svc-reporting/O=reporting" -keyout r.key -out r.crt >/dev/null 2>&1 +openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=svc-billing/O=billing" -keyout b.key -out b.crt >/dev/null 2>&1 +CA=$(base64 -w0 r.crt); RC=$(base64 -w0 r.crt); RK=$(base64 -w0 r.key); BC=$(base64 -w0 b.crt); BK=$(base64 -w0 b.key) +cat > kubeconfig </dev/null | sort | tr '\n' ' ')" = "prod-billing prod-reporting staging-reporting " ]"""), + C("current-context file is correct", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q201/current-context" 2>/dev/null)" prod-billing"""), + C("cert file is the decoded PEM of svc-reporting", 3, r""" +grep -q 'BEGIN CERTIFICATE' "$WS/q201/cert" 2>/dev/null || exit 1 +a=$(openssl x509 -in "$WS/q201/cert" -noout -fingerprint -sha256 2>/dev/null); b=$(openssl x509 -in "$WS/q201/.expected.crt" -noout -fingerprint -sha256) +eq "$a" "$b" """), + C("cn file contains svc-reporting", 1, r"""grep -q 'svc-reporting' "$WS/q201/cn" 2>/dev/null"""), + ], + solution=""" +export KC={WS}/q201/kubeconfig +k --kubeconfig $KC config get-contexts -o name > {WS}/q201/contexts +k --kubeconfig $KC config current-context > {WS}/q201/current-context +k --kubeconfig $KC config view --raw \\ + -o jsonpath='{.users[?(@.name=="svc-reporting")].user.client-certificate-data}' | base64 -d > {WS}/q201/cert +openssl x509 -in {WS}/q201/cert -noout -subject | sed 's/.*CN *= *//' > {WS}/q201/cn +# --raw is what reveals the data fields; without it kubectl prints REDACTED +""", +) + +# ------------------------------------------------------------------ 202 ---- +task( + id=202, domain="ARCH", title="Helm chart with CRDs, then a custom resource", + task=""" +A local Helm chart is at `{WS}/q202/chart`. It ships a CRD (`Ledger`, group `finance.example.com`). + +1. Create Namespace `finance`. +2. Install the chart into `finance` as release `ledger-operator`, with value `watchAllNamespaces=true`. +3. The file `{WS}/q202/ledger.yaml` creates a `Ledger`. Add `spec.retentionDays: 30` to it + (the CRD's schema accepts it — check with `kubectl explain`), then apply it. + +The release must report `deployed` and `kubectl get ledger -A` must list it. +""", + setup=r""" +rm -rf "$WS/q202"; mkdir -p "$WS/q202/chart/crds" "$WS/q202/chart/templates" +cat > "$WS/q202/chart/Chart.yaml" <<'EOF' +apiVersion: v2 +name: ledger-operator +version: 0.2.0 +appVersion: "0.2.0" +description: Toy operator shipping a Ledger CRD +EOF +cat > "$WS/q202/chart/values.yaml" <<'EOF' +watchAllNamespaces: false +replicaCount: 1 +EOF +cat > "$WS/q202/chart/crds/ledger-crd.yaml" <<'EOF' +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: {name: ledgers.finance.example.com} +spec: + group: finance.example.com + scope: Namespaced + names: {kind: Ledger, plural: ledgers, singular: ledger, shortNames: [ldg]} + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + currency: {type: string, description: "ISO currency code"} + retentionDays: {type: integer, description: "How many days entries are kept"} +EOF +cat > "$WS/q202/chart/templates/deployment.yaml" <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: {name: {{ .Release.Name }}} +spec: + replicas: {{ .Values.replicaCount }} + selector: {matchLabels: {app: {{ .Release.Name }}}} + template: + metadata: {labels: {app: {{ .Release.Name }}}} + spec: + containers: + - name: operator + image: busybox:1.36 + command: [sleep, "86400"] + env: + - {name: WATCH_ALL_NAMESPACES, value: "{{ .Values.watchAllNamespaces }}"} +EOF +cat > "$WS/q202/ledger.yaml" <<'EOF' +apiVersion: finance.example.com/v1 +kind: Ledger +metadata: + name: main-ledger + namespace: finance +spec: + currency: USD +EOF +helm -n finance uninstall ledger-operator >/dev/null 2>&1 || true +$K delete crd ledgers.finance.example.com --ignore-not-found >/dev/null +$K delete ns finance --ignore-not-found --wait=false >/dev/null +command -v helm >/dev/null || echo "warn: helm not installed - this task needs it" +""", + checks=[ + C("release ledger-operator deployed in finance", 2, r"""helm -n finance status ledger-operator -o json 2>/dev/null | python3 -c 'import sys,json;sys.exit(0 if json.load(sys.stdin)["info"]["status"]=="deployed" else 1)'"""), + C("watchAllNamespaces=true reached the pod env", 2, r"""eq "$(j -n finance deploy ledger-operator '{.spec.template.spec.containers[0].env[?(@.name=="WATCH_ALL_NAMESPACES")].value}')" true"""), + C("CRD installed and shortname works", 1, r"""$K get crd ledgers.finance.example.com >/dev/null 2>&1 && $K -n finance get ldg >/dev/null 2>&1"""), + C("Ledger main-ledger has currency USD and retentionDays 30", 3, r"""eq "$(j -n finance ledgers main-ledger '{.spec.currency}')" USD && eq "$(j -n finance ledgers main-ledger '{.spec.retentionDays}')" 30"""), + ], + solution=""" +k create ns finance +helm -n finance install ledger-operator {WS}/q202/chart --set watchAllNamespaces=true +helm -n finance ls; k get crd | grep ledgers +k explain ledger.spec # confirms retentionDays exists +vi {WS}/q202/ledger.yaml # add: retentionDays: 30 under spec +k apply -f {WS}/q202/ledger.yaml +k -n finance get ldg +# CRDs in a chart's crds/ directory are installed by helm before the templates, +# which is why the Ledger can be created straight after install. +""", +) + +# ------------------------------------------------------------------ 203 ---- +task( + id=203, domain="WORK", title="Find the controller, then scale it", + task=""" +Namespace `project-atlas` runs Pods named `atlas-cache-*`. Management wants them down to **one** replica. + +1. Find out which resource manages them and write its kind and name, in the form `kind/name` + (lowercase kind, as `kubectl get` prints it), to `{WS}/q203/owner.txt`. +2. Scale it to 1 replica without deleting anything. +3. Write the number of Pods still running in the Namespace after scaling to `{WS}/q203/count.txt`. +""", + setup=NS + r""" +mkns project-atlas +rm -rf "$WS/q203"; mkdir -p "$WS/q203" +$K -n project-atlas delete sts atlas-cache --ignore-not-found >/dev/null +$K apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: Service +metadata: {name: atlas-cache, namespace: project-atlas} +spec: {clusterIP: None, selector: {app: atlas-cache}, ports: [{port: 6379}]} +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: {name: atlas-cache, namespace: project-atlas} +spec: + serviceName: atlas-cache + replicas: 3 + selector: {matchLabels: {app: atlas-cache}} + template: + metadata: {labels: {app: atlas-cache}} + spec: + containers: [{name: cache, image: busybox:1.36, command: [sleep, "86400"]}] +EOF +""", + checks=[ + C("owner.txt identifies statefulset/atlas-cache", 2, r"""tr -d '[:space:]' < "$WS/q203/owner.txt" 2>/dev/null | grep -Eq '^(statefulset|statefulset.apps)/atlas-cache$'"""), + C("StatefulSet scaled to 1 and still exists", 3, r"""eq "$(j -n project-atlas sts atlas-cache '{.spec.replicas}')" 1 && eq "$(j -n project-atlas sts atlas-cache '{.status.readyReplicas}')" 1"""), + C("count.txt matches the running pod count", 1, r"""n=$($K -n project-atlas get pod --no-headers 2>/dev/null | grep -c Running); eq "$(tr -dc '0-9' < "$WS/q203/count.txt" 2>/dev/null)" "$n" """), + ], + solution=""" +k -n project-atlas get pod --show-labels +k -n project-atlas get deploy,sts,ds,rs # -> statefulset.apps/atlas-cache +# or from the pod itself: +k -n project-atlas get pod atlas-cache-0 -o jsonpath='{.metadata.ownerReferences[0].kind}{"/"}{.metadata.ownerReferences[0].name}' +echo statefulset/atlas-cache > {WS}/q203/owner.txt +k -n project-atlas scale sts atlas-cache --replicas 1 +k -n project-atlas get pod | grep -c Running > {WS}/q203/count.txt +""", +) + +# ------------------------------------------------------------------ 204 ---- +task( + id=204, domain="WORK", title="Which Pods get evicted first", + task=""" +Namespace `project-orion` is close to filling its nodes. + +1. Write the names of all Pods in `project-orion` that the kubelet would evict **first** under + node memory pressure — one name per line, sorted — to `{WS}/q204/evicted-first.txt`. +2. Write the Quality of Service class of Pod `orion-api-0` to `{WS}/q204/api-qos.txt`. +3. One Deployment's Pods are `Guaranteed`. Write its name to `{WS}/q204/guaranteed.txt`. +""", + setup=NS + r""" +mkns project-orion +rm -rf "$WS/q204"; mkdir -p "$WS/q204" +$K -n project-orion delete deploy --all >/dev/null 2>&1; $K -n project-orion delete pod orion-api-0 --ignore-not-found --wait=false >/dev/null +$K apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: Pod +metadata: {name: orion-api-0, namespace: project-orion, labels: {app: orion-api}} +spec: + containers: + - name: api + image: busybox:1.36 + command: [sleep, "86400"] + resources: {requests: {cpu: 20m, memory: 24Mi}, limits: {cpu: 100m, memory: 64Mi}} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: orion-batch, namespace: project-orion} +spec: + replicas: 2 + selector: {matchLabels: {app: orion-batch}} + template: + metadata: {labels: {app: orion-batch}} + spec: + containers: [{name: batch, image: busybox:1.36, command: [sleep, "86400"]}] +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: orion-ledger, namespace: project-orion} +spec: + replicas: 2 + selector: {matchLabels: {app: orion-ledger}} + template: + metadata: {labels: {app: orion-ledger}} + spec: + containers: + - name: ledger + image: busybox:1.36 + command: [sleep, "86400"] + resources: {requests: {cpu: 25m, memory: 32Mi}, limits: {cpu: 25m, memory: 32Mi}} +EOF +""", + checks=[ + C("evicted-first.txt lists exactly the BestEffort pods", 4, r""" +want=$($K -n project-orion get pod -o json 2>/dev/null | python3 -c ' +import sys,json +print("\n".join(sorted(p["metadata"]["name"] for p in json.load(sys.stdin)["items"] if p["status"].get("qosClass")=="BestEffort")))') +[ -n "$want" ] && [ "$(grep . "$WS/q204/evicted-first.txt" 2>/dev/null | sort)" = "$want" ]"""), + C("api-qos.txt says Burstable", 2, r"""grep -qi burstable "$WS/q204/api-qos.txt" 2>/dev/null"""), + C("guaranteed.txt names orion-ledger", 2, r"""grep -q 'orion-ledger' "$WS/q204/guaranteed.txt" 2>/dev/null"""), + ], + solution=""" +k -n project-orion get pod -o jsonpath='{range .items[*]}{.metadata.name} {.status.qosClass}{"\\n"}{end}' +# BestEffort (no requests and no limits at all) is evicted first, then Burstable +# exceeding its requests, Guaranteed (requests == limits on every container) last. +k -n project-orion get pod -o jsonpath='{range .items[*]}{.metadata.name} {.status.qosClass}{"\\n"}{end}' \\ + | awk '$2=="BestEffort"{print $1}' | sort > {WS}/q204/evicted-first.txt +k -n project-orion get pod orion-api-0 -o jsonpath='{.status.qosClass}' > {WS}/q204/api-qos.txt +echo orion-ledger > {WS}/q204/guaranteed.txt +""", +) + +# ------------------------------------------------------------------ 205 ---- +task( + id=205, domain="ARCH", title="Kustomize: drop a ConfigMap, add an HPA, override in prod", + task=""" +The app `checkout` is deployed from `{WS}/q205/checkout` into Namespaces `checkout-staging` +and `checkout-prod`: + +``` +kubectl kustomize {WS}/q205/checkout/staging | kubectl apply -f - +kubectl kustomize {WS}/q205/checkout/prod | kubectl apply -f - +``` + +Change the Kustomize config so that: +1. the ConfigMap `legacy-scaler-config` is gone from the built output of **both** overlays, +2. a HorizontalPodAutoscaler named `checkout` targets the `checkout` Deployment with + min 2 / max 5 replicas at 60% average CPU utilisation, +3. in **prod** the same HPA has max 8 replicas, +4. both overlays are applied, and the ConfigMap no longer exists in either Namespace. +""", + setup=r""" +rm -rf "$WS/q205"; mkdir -p "$WS/q205/checkout/base" "$WS/q205/checkout/staging" "$WS/q205/checkout/prod" +B="$WS/q205/checkout" +cat > "$B/base/kustomization.yaml" <<'EOF' +resources: [checkout.yaml] +EOF +cat > "$B/base/checkout.yaml" <<'EOF' +apiVersion: v1 +kind: ConfigMap +metadata: {name: legacy-scaler-config} +data: {threshold: "70"} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: checkout} +spec: + replicas: 1 + selector: {matchLabels: {id: checkout}} + template: + metadata: {labels: {id: checkout}} + spec: + containers: + - name: web + image: httpd:2-alpine + resources: {requests: {cpu: 10m, memory: 20Mi}} +EOF +for env in staging prod; do +cat > "$B/$env/kustomization.yaml" < "$B/$env/checkout.yaml" </dev/null; done +$K kustomize "$B/staging" 2>/dev/null | $K apply -f - >/dev/null 2>&1 +$K kustomize "$B/prod" 2>/dev/null | $K apply -f - >/dev/null 2>&1 +""", + checks=[ + C("built output of both overlays contains no ConfigMap", 3, r""" +for e in staging prod; do $K kustomize "$WS/q205/checkout/$e" 2>/dev/null | grep -q 'kind: ConfigMap' && exit 1; done +$K kustomize "$WS/q205/checkout/staging" >/dev/null 2>&1"""), + C("HPA in staging: min 2, max 5, 60% CPU, targets the Deployment", 3, r""" +h="$K -n checkout-staging get hpa checkout" +eq "$($h -o jsonpath='{.spec.minReplicas}' 2>/dev/null)" 2 && eq "$($h -o jsonpath='{.spec.maxReplicas}')" 5 \ + && eq "$($h -o jsonpath='{.spec.scaleTargetRef.name}')" checkout \ + && $h -o jsonpath='{.spec.metrics[*].resource}' | grep -q '"averageUtilization":60'"""), + C("HPA in prod has max 8 (min and target unchanged)", 2, r""" +h="$K -n checkout-prod get hpa checkout" +eq "$($h -o jsonpath='{.spec.maxReplicas}' 2>/dev/null)" 8 && eq "$($h -o jsonpath='{.spec.minReplicas}')" 2"""), + C("legacy-scaler-config removed from both Namespaces", 2, r""" +! $K -n checkout-staging get cm legacy-scaler-config >/dev/null 2>&1 && ! $K -n checkout-prod get cm legacy-scaler-config >/dev/null 2>&1"""), + ], + solution=""" +cd {WS}/q205/checkout +# 1. the ConfigMap is patched by BOTH overlays, so removing it only from base breaks the build. +# Delete it from base/checkout.yaml AND from staging/checkout.yaml AND prod/checkout.yaml. +# 2. add the HPA to base/checkout.yaml (no namespace - the overlays set it): +# apiVersion: autoscaling/v2 +# kind: HorizontalPodAutoscaler +# metadata: {name: checkout} +# spec: +# scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: checkout} +# minReplicas: 2 +# maxReplicas: 5 +# metrics: [{type: Resource, resource: {name: cpu, target: {type: Utilization, averageUtilization: 60}}}] +# 3. in prod/checkout.yaml add a patch with only the changed field: +# apiVersion: autoscaling/v2 +# kind: HorizontalPodAutoscaler +# metadata: {name: checkout} +# spec: {maxReplicas: 8} +k kustomize staging | k apply -f - +k kustomize prod | k apply -f - +# 4. Kustomize keeps no state: it never deletes resources that only exist in the cluster. +k -n checkout-staging delete cm legacy-scaler-config +k -n checkout-prod delete cm legacy-scaler-config +""", +) + +# ------------------------------------------------------------------ 206 ---- +task( + id=206, domain="STOR", title="PV, PVC and a Deployment that mounts it", + task=""" +1. Create a PersistentVolume `orbit-pv`: 3Gi, accessMode `ReadWriteOnce`, `hostPath` `/Volumes/orbit`, + and **no** storageClassName. +2. In Namespace `project-orbit` create PVC `orbit-pvc` requesting 3Gi, `ReadWriteOnce`, also with + **no** storageClassName, bound to that PV. +3. Create Deployment `orbit` in the same Namespace, image `httpd:2-alpine`, that mounts the claim + at `/usr/local/apache2/htdocs/data`. + +The claim must be `Bound` and the Pod `Running`. +""", + setup=NS + r""" +mkns project-orbit +$K -n project-orbit delete deploy orbit --ignore-not-found >/dev/null +$K -n project-orbit delete pvc orbit-pvc --ignore-not-found >/dev/null +$K delete pv orbit-pv --ignore-not-found >/dev/null +""", + checks=[ + C("PV orbit-pv: 3Gi, RWO, hostPath /Volumes/orbit, no storageClassName", 3, r""" +p="$K get pv orbit-pv" +eq "$($p -o jsonpath='{.spec.capacity.storage}' 2>/dev/null)" 3Gi && eq "$($p -o jsonpath='{.spec.hostPath.path}')" /Volumes/orbit \ + && $p -o jsonpath='{.spec.accessModes}' | grep -q ReadWriteOnce && [ -z "$($p -o jsonpath='{.spec.storageClassName}')" ]"""), + C("PVC orbit-pvc Bound to orbit-pv with no storageClassName", 3, r""" +eq "$(j -n project-orbit pvc orbit-pvc '{.status.phase}')" Bound && eq "$(j -n project-orbit pvc orbit-pvc '{.spec.volumeName}')" orbit-pv \ + && [ -z "$(j -n project-orbit pvc orbit-pvc '{.spec.storageClassName}')" ]"""), + C("Deployment orbit available and mounts the claim at the right path", 3, r""" +eq "$(j -n project-orbit deploy orbit '{.status.availableReplicas}')" 1 \ + && [ -n "$(j -n project-orbit deploy orbit '{.spec.template.spec.volumes[?(@.persistentVolumeClaim.claimName=="orbit-pvc")].name}')" ] \ + && j -n project-orbit deploy orbit '{.spec.template.spec.containers[0].volumeMounts[*].mountPath}' | grep -q '/usr/local/apache2/htdocs/data'"""), + ], + solution=""" +cat < orbit.yaml +# add to the pod spec: +# volumes: [{name: data, persistentVolumeClaim: {claimName: orbit-pvc}}] +# and to the container: +# volumeMounts: [{name: data, mountPath: /usr/local/apache2/htdocs/data}] +k apply -f orbit.yaml +k -n project-orbit get pv,pvc +""", +) + +# ------------------------------------------------------------------ 207 ---- +task( + id=207, domain="TSHOOT", title="Resource usage scripts", + task=""" +Write two executable bash scripts that use `kubectl`: + +1. `{WS}/q207/usage-nodes.sh` — prints CPU and memory usage of all **nodes**, sorted by CPU + (highest first). +2. `{WS}/q207/usage-pods.sh` — prints CPU and memory usage of all **Pods in all Namespaces**, + broken down **per container**. + +Use full command names, no aliases. The scripts must be executable. +""", + setup=r"""rm -rf "$WS/q207"; mkdir -p "$WS/q207" """, + checks=[ + C("usage-nodes.sh is executable and uses kubectl top node with cpu sorting", 3, r""" +f="$WS/q207/usage-nodes.sh"; [ -x "$f" ] || exit 1 +grep -q 'kubectl top node' "$f" && grep -Eq -- '--sort-by[= ]?.?cpu' "$f" && ! grep -qE '(^| )k( |$)' "$f" """), + C("usage-pods.sh is executable, all namespaces, per container", 3, r""" +f="$WS/q207/usage-pods.sh"; [ -x "$f" ] || exit 1 +grep -q 'kubectl top pod' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" && grep -q -- '--containers' "$f" """), + ], + solution=""" +cat > {WS}/q207/usage-nodes.sh <<'EOF' +kubectl top node --sort-by=cpu +EOF +cat > {WS}/q207/usage-pods.sh <<'EOF' +kubectl top pod --all-namespaces --containers=true +EOF +chmod +x {WS}/q207/usage-nodes.sh {WS}/q207/usage-pods.sh +# `kubectl top -h` shows both flags. Aliases (k) don't exist in a fresh shell, so write kubectl. +""", +) + +# ------------------------------------------------------------------ 208 ---- +task( + id=208, domain="ARCH", title="Drain a node for maintenance", breaks="w2-drain", + task=""" +Node `{W2}` needs kernel maintenance. + +1. Evict every non-DaemonSet Pod from it. Pods with `emptyDir` volumes must not stop you, + and neither must Pods that aren't managed by a controller. +2. Write the exact command you used to `{WS}/q208/drain.sh`. +3. Deployment `atlas-web` in Namespace `project-atlas` must still have all 3 replicas Available afterwards. +4. When you're done, make the node schedulable again. +""", + setup=NS + r""" +mkns project-atlas +rm -rf "$WS/q208"; mkdir -p "$WS/q208" +$K -n project-atlas delete deploy atlas-web --ignore-not-found >/dev/null +$K uncordon "$W2" >/dev/null 2>&1 || true +$K apply -f - >/dev/null </dev/null 2>&1 +$K -n project-atlas run atlas-orphan --image=busybox:1.36 --restart=Never --overrides="{\"spec\":{\"nodeName\":\"$W2\"}}" -- sleep 86400 >/dev/null 2>&1 +""", + checks=[ + C("no non-DaemonSet workload pods remain on the node", 4, r""" +left=$($K get pods -A -o json 2>/dev/null | python3 -c ' +import sys,json,os +n=os.environ["CKA_W2"] +print(sum(1 for p in json.load(sys.stdin)["items"] + if p["spec"].get("nodeName")==n and p["status"]["phase"]=="Running" + and (p["metadata"].get("ownerReferences") or [{}])[0].get("kind") not in ("DaemonSet",) + and p["metadata"]["namespace"] not in ("kube-system",)))') +eq "$left" 0"""), + C("drain.sh records a drain with --ignore-daemonsets and --delete-emptydir-data", 2, r""" +f="$WS/q208/drain.sh"; grep -q 'drain' "$f" 2>/dev/null && grep -q -- '--ignore-daemonsets' "$f" && grep -q -- '--delete-emptydir-data' "$f" """), + C("atlas-web still has 3 replicas Available", 2, r"""eq "$(j -n project-atlas deploy atlas-web '{.status.availableReplicas}')" 3"""), + C("node is schedulable again", 2, r"""[ "$(j node "$W2" '{.spec.unschedulable}')" != true ]"""), + ], + solution=""" +k drain {W2} --ignore-daemonsets --delete-emptydir-data --force +# --ignore-daemonsets DaemonSet pods can't be evicted, skip them +# --delete-emptydir-data required because atlas-web mounts an emptyDir +# --force required for the bare pod atlas-orphan (no controller) +echo 'kubectl drain {W2} --ignore-daemonsets --delete-emptydir-data --force' > {WS}/q208/drain.sh +k get pods -A -o wide --field-selector spec.nodeName={W2} +k -n project-atlas rollout status deploy atlas-web +k uncordon {W2} +""", +) + +# ------------------------------------------------------------------ 209 ---- +task( + id=209, domain="ARCH", title="Query the API from inside a Pod", + task=""" +Namespace `project-orbit` has ServiceAccount `cm-reader`, which may read ConfigMaps cluster-wide. + +1. Create Pod `api-probe` in `project-orbit`, image `nginx:1-alpine`, using that ServiceAccount. +2. From **inside** the Pod, query the Kubernetes API for all ConfigMaps in Namespace `project-orbit` + using `curl` and the Pod's ServiceAccount token, verifying the server certificate with the + mounted CA (do not use `-k`). +3. Save the JSON response to `{WS}/q209/configmaps.json`. +4. Write the API server URL you used, as seen from inside the Pod, to `{WS}/q209/endpoint.txt`. +""", + setup=NS + r""" +mkns project-orbit +rm -rf "$WS/q209"; mkdir -p "$WS/q209" +$K -n project-orbit delete pod api-probe --ignore-not-found --wait=false >/dev/null +$K -n project-orbit create sa cm-reader --dry-run=client -o yaml | $K apply -f - >/dev/null +$K delete clusterrole cm-reader --ignore-not-found >/dev/null; $K delete clusterrolebinding cm-reader --ignore-not-found >/dev/null +$K create clusterrole cm-reader --verb=get,list --resource=configmaps >/dev/null +$K create clusterrolebinding cm-reader --clusterrole=cm-reader --serviceaccount=project-orbit:cm-reader >/dev/null +$K -n project-orbit create cm orbit-settings --from-literal=region=us-east --dry-run=client -o yaml | $K apply -f - >/dev/null +""", + checks=[ + C("Pod api-probe Running with ServiceAccount cm-reader", 2, r""" +eq "$(j -n project-orbit pod api-probe '{.status.phase}')" Running && eq "$(j -n project-orbit pod api-probe '{.spec.serviceAccountName}')" cm-reader"""), + C("configmaps.json is a real ConfigMapList containing orbit-settings", 4, r""" +python3 -c ' +import json,sys +d=json.load(open("'"$WS"'/q209/configmaps.json")) +sys.exit(0 if d.get("kind")=="ConfigMapList" and any(i["metadata"]["name"]=="orbit-settings" for i in d.get("items",[])) else 1)' 2>/dev/null"""), + C("the response was authorised (not a 403 Status object)", 2, r"""! grep -q '"kind": *"Status"' "$WS/q209/configmaps.json" 2>/dev/null"""), + C("endpoint.txt names the in-cluster API service", 1, r"""grep -Eq 'kubernetes(\.default)?' "$WS/q209/endpoint.txt" 2>/dev/null"""), + ], + solution=""" +k -n project-orbit run api-probe --image=nginx:1-alpine $do > p.yaml +# add: spec.serviceAccountName: cm-reader +k apply -f p.yaml +k -n project-orbit exec -it api-probe -- sh + TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) + CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt + curl --cacert $CACERT -H "Authorization: Bearer $TOKEN" \\ + https://kubernetes.default/api/v1/namespaces/project-orbit/configmaps > /tmp/cm.json + exit +k -n project-orbit exec api-probe -- cat /tmp/cm.json > {WS}/q209/configmaps.json +echo https://kubernetes.default > {WS}/q209/endpoint.txt +# Without the token you are system:anonymous and get 403; with -k you skip verification, +# which the task forbids - the CA is mounted next to the token. +""", +) + +# ------------------------------------------------------------------ 210 ---- +task( + id=210, domain="ARCH", title="ServiceAccount limited to two resources", + task=""" +In Namespace `project-atlas` create ServiceAccount `pipeline`, plus a Role and a RoleBinding both +named `pipeline`, so that the ServiceAccount may **only**: + +- `create` and `update` **secrets** and **configmaps** in that Namespace. + +It must not be able to read them back, delete them, or touch anything in another Namespace. +""", + setup=NS + r""" +mkns project-atlas +$K -n project-atlas delete sa pipeline --ignore-not-found >/dev/null +$K -n project-atlas delete role pipeline --ignore-not-found >/dev/null +$K -n project-atlas delete rolebinding pipeline --ignore-not-found >/dev/null +""", + checks=[ + C("ServiceAccount, Role and RoleBinding named pipeline exist", 2, r"""$K -n project-atlas get sa pipeline >/dev/null 2>&1 && $K -n project-atlas get role pipeline >/dev/null 2>&1 && $K -n project-atlas get rolebinding pipeline >/dev/null 2>&1"""), + C("can create and update secrets and configmaps in project-atlas", 3, r""" +for r in secrets configmaps; do for v in create update; do + eq "$($K auth can-i $v $r -n project-atlas --as=system:serviceaccount:project-atlas:pipeline 2>/dev/null)" yes || exit 1 +done; done"""), + C("cannot get/list/delete them, and has nothing elsewhere", 3, r""" +for t in "get secrets -n project-atlas" "list configmaps -n project-atlas" "delete secrets -n project-atlas" "create secrets -n default"; do + eq "$($K auth can-i $t --as=system:serviceaccount:project-atlas:pipeline 2>/dev/null || true)" no || exit 1 +done"""), + ], + solution=""" +k -n project-atlas create sa pipeline +k -n project-atlas create role pipeline --verb=create,update --resource=secrets,configmaps +k -n project-atlas create rolebinding pipeline --role=pipeline --serviceaccount=project-atlas:pipeline +k -n project-atlas auth can-i create secret --as=system:serviceaccount:project-atlas:pipeline # yes +k -n project-atlas auth can-i get secret --as=system:serviceaccount:project-atlas:pipeline # no +""", +) + +# ------------------------------------------------------------------ 211 ---- +task( + id=211, domain="WORK", title="DaemonSet on every node", + task=""" +In Namespace `project-orion` create a DaemonSet `node-collector`: + +- image `httpd:2-alpine` +- labels on the DaemonSet **and** its Pods: `id=node-collector` and `tier=infra` +- each Pod requests 15 millicores of CPU and 15 mebibytes of memory +- a Pod must run on **every** node in the cluster, control-plane included + +Every desired Pod must be Ready. +""", + setup=NS + r"""mkns project-orion; $K -n project-orion delete ds node-collector --ignore-not-found >/dev/null""", + checks=[ + C("DaemonSet exists with both labels on the object and the pod template", 2, r""" +d="$K -n project-orion get ds node-collector" +eq "$($d -o jsonpath='{.metadata.labels.id}' 2>/dev/null)" node-collector && eq "$($d -o jsonpath='{.metadata.labels.tier}')" infra \ + && eq "$($d -o jsonpath='{.spec.template.metadata.labels.id}')" node-collector && eq "$($d -o jsonpath='{.spec.template.metadata.labels.tier}')" infra"""), + C("requests are 15m CPU / 15Mi memory", 2, r""" +eq "$(j -n project-orion ds node-collector '{.spec.template.spec.containers[0].resources.requests.cpu}')" 15m \ + && eq "$(j -n project-orion ds node-collector '{.spec.template.spec.containers[0].resources.requests.memory}')" 15Mi"""), + C("one Ready pod on every node including the control-plane", 4, r""" +n=$($K get nodes --no-headers 2>/dev/null | wc -l) +eq "$(j -n project-orion ds node-collector '{.status.desiredNumberScheduled}')" "$n" && eq "$(j -n project-orion ds node-collector '{.status.numberReady}')" "$n" \ + && j -n project-orion pod -l id=node-collector '{.items[*].spec.nodeName}' | tr ' ' '\n' | grep -q control-plane"""), + ], + solution=""" +k -n project-orion create deploy node-collector --image=httpd:2-alpine $do > ds.yaml +# change kind Deployment -> DaemonSet, drop replicas/strategy/status, then add: +# metadata.labels: {id: node-collector, tier: infra} +# spec.selector.matchLabels + spec.template.metadata.labels: same two labels +# container resources.requests: {cpu: 15m, memory: 15Mi} +# spec.template.spec.tolerations: +# - {key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule} +k apply -f ds.yaml +k -n project-orion get ds node-collector -o wide +""", +) + +# ------------------------------------------------------------------ 212 ---- +task( + id=212, domain="WORK", title="One Pod per node, on purpose", + task=""" +In Namespace `project-orion` create Deployment `spread-check`: + +- 4 replicas, Pods labelled `id=spread-check` +- container `main` with image `nginx:1-alpine`, and a second container `pause` with image + `registry.k8s.io/pause:3.10` +- at most **one** of its Pods on any worker node (use `topologyKey: kubernetes.io/hostname`) + +Because there are only three worker nodes, one Pod will stay `Pending` — that is expected. +Write the name of the Pending Pod to `{WS}/q212/pending.txt`. +""", + setup=NS + r"""mkns project-orion; rm -rf "$WS/q212"; mkdir -p "$WS/q212"; $K -n project-orion delete deploy spread-check --ignore-not-found >/dev/null""", + checks=[ + C("Deployment has 4 replicas and both containers", 2, r""" +d="$K -n project-orion get deploy spread-check" +eq "$($d -o jsonpath='{.spec.replicas}' 2>/dev/null)" 4 \ + && $d -o jsonpath='{.spec.template.spec.containers[*].name}' | grep -qw main \ + && $d -o jsonpath='{.spec.template.spec.containers[*].image}' | grep -q 'pause:3.10'"""), + C("anti-affinity or spread constraint on kubernetes.io/hostname", 3, r""" +t=$(j -n project-orion deploy spread-check '{.spec.template.spec}') +echo "$t" | grep -q '"topologyKey":"kubernetes.io/hostname"' && { echo "$t" | grep -q 'podAntiAffinity' || echo "$t" | grep -q 'topologySpreadConstraints'; }"""), + C("exactly 3 Pods Running, one per node, 1 Pending", 3, r""" +r=$($K -n project-orion get pod -l id=spread-check --no-headers 2>/dev/null | grep -c Running) +p=$($K -n project-orion get pod -l id=spread-check --no-headers 2>/dev/null | grep -c Pending) +eq "$r" 3 && eq "$p" 1 && [ "$(j -n project-orion pod -l id=spread-check '{.items[*].spec.nodeName}' | tr ' ' '\n' | grep . | sort -u | wc -l)" = 3"""), + C("pending.txt names the Pending pod", 2, r""" +want=$($K -n project-orion get pod -l id=spread-check --no-headers 2>/dev/null | awk '$3=="Pending"{print $1}') +[ -n "$want" ] && grep -qF "$want" "$WS/q212/pending.txt" 2>/dev/null"""), + ], + solution=""" +k -n project-orion create deploy spread-check --image=nginx:1-alpine --replicas=4 $do > s.yaml +# set the pod label id=spread-check (selector + template), rename the container to main, +# add the second container, and add either: +# affinity: +# podAntiAffinity: +# requiredDuringSchedulingIgnoredDuringExecution: +# - labelSelector: {matchLabels: {id: spread-check}} +# topologyKey: kubernetes.io/hostname +# or: +# topologySpreadConstraints: +# - {maxSkew: 1, topologyKey: kubernetes.io/hostname, whenUnsatisfiable: DoNotSchedule, +# labelSelector: {matchLabels: {id: spread-check}}} +k apply -f s.yaml +k -n project-orion get pod -l id=spread-check -o wide +k -n project-orion get pod -l id=spread-check --field-selector status.phase=Pending -o name > {WS}/q212/pending.txt +k -n project-orion describe pod # "didn't match pod anti-affinity rules" +""", +) + +# ------------------------------------------------------------------ 213 ---- +task( + id=213, domain="NET", title="Convert an Ingress to Gateway API, then split on a header", + task=""" +Namespace `project-r700` has Services `web-classic` and `web-next` (both port 80) and an existing +Gateway named `edge`. The legacy Ingress is at `{WS}/q213/ingress.yaml`. + +Create an HTTPRoute named `router` in that Namespace, attached to the Gateway `edge`, which: + +1. reproduces the routes of the old Ingress (host and paths included), +2. adds a path `/beta` that goes to `web-next` **only when** the header `X-Channel` is exactly + `beta`, and to `web-classic` in every other case. + +Path matching is `PathPrefix` throughout. Order matters. +""", + setup=NS + r""" +mkns project-r700 +rm -rf "$WS/q213"; mkdir -p "$WS/q213" +for s in web-classic web-next; do + $K -n project-r700 create deploy $s --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null + $K -n project-r700 expose deploy $s --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +done +cat > "$WS/q213/ingress.yaml" <<'EOF' +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: {name: router, namespace: project-r700} +spec: + ingressClassName: nginx + rules: + - host: r700.gateway + http: + paths: + - {path: /classic, pathType: Prefix, backend: {service: {name: web-classic, port: {number: 80}}}} + - {path: /next, pathType: Prefix, backend: {service: {name: web-next, port: {number: 80}}}} +EOF +$K get crd gateways.gateway.networking.k8s.io >/dev/null 2>&1 || { + curl -sfL --max-time 60 https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml -o /tmp/gwapi.yaml && $K apply -f /tmp/gwapi.yaml >/dev/null 2>&1; } || echo "warn: could not install Gateway API CRDs" +$K -n project-r700 delete httproute router --ignore-not-found >/dev/null 2>&1 +$K apply -f - >/dev/null 2>&1 <<'EOF' +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass +metadata: {name: edge-class} +spec: {controllerName: example.com/edge} +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: {name: edge, namespace: project-r700} +spec: + gatewayClassName: edge-class + listeners: [{name: http, protocol: HTTP, port: 80, hostname: r700.gateway}] +EOF +""", + checks=[ + C("HTTPRoute router attached to Gateway edge, hostname r700.gateway", 2, r""" +h="$K -n project-r700 get httproute router" +eq "$($h -o jsonpath='{.spec.parentRefs[0].name}' 2>/dev/null)" edge && $h -o jsonpath='{.spec.hostnames[*]}' | grep -q r700.gateway"""), + C("/classic -> web-classic:80 and /next -> web-next:80", 3, r""" +$K -n project-r700 get httproute router -o json 2>/dev/null | python3 -c ' +import sys,json +rs=json.load(sys.stdin)["spec"]["rules"] +def be(path): + for r in rs: + for m in r.get("matches",[]): + if m.get("path",{}).get("value")==path and m.get("path",{}).get("type")=="PathPrefix" and not m.get("headers"): + b=r["backendRefs"][0]; return b["name"], b.get("port") + return None +sys.exit(0 if be("/classic")==("web-classic",80) and be("/next")==("web-next",80) else 1)'"""), + C("/beta with X-Channel=beta goes to web-next (path AND header in one match)", 3, r""" +$K -n project-r700 get httproute router -o json 2>/dev/null | python3 -c ' +import sys,json +rs=json.load(sys.stdin)["spec"]["rules"] +ok=False +for r in rs: + for m in r.get("matches",[]): + hs=m.get("headers") or [] + if m.get("path",{}).get("value")=="/beta" and any(h.get("name","").lower()=="x-channel" and h.get("value")=="beta" for h in hs): + if len(r["backendRefs"])==1 and r["backendRefs"][0]["name"]=="web-next": ok=True +sys.exit(0 if ok else 1)'"""), + C("/beta catch-all goes to web-classic and comes after the header rule", 3, r""" +$K -n project-r700 get httproute router -o json 2>/dev/null | python3 -c ' +import sys,json +rs=json.load(sys.stdin)["spec"]["rules"] +hi=ci=None +for i,r in enumerate(rs): + for m in r.get("matches",[]): + if m.get("path",{}).get("value")=="/beta": + if m.get("headers"): + if hi is None: hi=i + else: + if ci is None and r["backendRefs"][0]["name"]=="web-classic": ci=i +sys.exit(0 if hi is not None and ci is not None and hi/dev/null && grep -qi "$mon" "$f" && grep -q "$day" "$f" """), + C("kubeadm-line.txt is the apiserver-etcd-client row from kubeadm", 3, r"""grep -q 'apiserver-etcd-client' "$WS/q214/kubeadm-line.txt" 2>/dev/null && grep -Eq '[0-9]{4}' "$WS/q214/kubeadm-line.txt" """), + C("renew.sh renews only that certificate", 2, r"""grep -Eq 'kubeadm +certs +renew +apiserver-etcd-client' "$WS/q214/renew.sh" 2>/dev/null"""), + ], + solution=""" +{SHELL_CP} +openssl x509 -in /etc/kubernetes/pki/apiserver-etcd-client.crt -noout -enddate +# notAfter=... -> copy into {WS}/q214/etcd-client-expiry.txt +kubeadm certs check-expiration | grep apiserver-etcd-client +# -> copy that line into {WS}/q214/kubeadm-line.txt +echo 'kubeadm certs renew apiserver-etcd-client' > {WS}/q214/renew.sh +# openssl -noout -text | grep -A2 Validity shows both notBefore and notAfter if you prefer. +""", +) + +# ------------------------------------------------------------------ 215 ---- +task( + id=215, domain="NET", title="Egress policy with paired destinations", + task=""" +Namespace `project-snake2` runs Pods labelled `app=api`, `app=cache`, `app=metrics` and `app=audit`. + +After a breach, the `api` Pods must be restricted. Create a NetworkPolicy `np-api` in that +Namespace so that Pods with `app=api` may open connections **only**: + +- to `app=cache` Pods on TCP port 6379, +- to `app=metrics` Pods on TCP port 9090. + +Any other egress — including to `cache` on 9090 or to `metrics` on 6379 — must be denied. +Ingress must not be restricted by this policy. +""", + setup=NS + r""" +mkns project-snake2 +for a in api cache metrics audit; do + $K -n project-snake2 create deploy $a --image=nginx:1.27 --dry-run=client -o yaml | $K apply -f - >/dev/null + $K -n project-snake2 label deploy $a app=$a --overwrite >/dev/null + $K -n project-snake2 patch deploy $a -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app\":\"$a\"}}}}}" >/dev/null +done +$K -n project-snake2 delete netpol np-api --ignore-not-found >/dev/null +""", + checks=[ + C("policy selects app=api and is Egress-only", 2, r""" +n="$K -n project-snake2 get netpol np-api" +eq "$($n -o jsonpath='{.spec.podSelector.matchLabels.app}' 2>/dev/null)" api && eq "$($n -o jsonpath='{.spec.policyTypes}')" '["Egress"]'"""), + C("two separate rules pair each destination with its own port", 5, r""" +$K -n project-snake2 get netpol np-api -o json 2>/dev/null | python3 -c ' +import sys,json +e=json.load(sys.stdin)["spec"].get("egress",[]) +def pair(app,port): + for r in e: + tos=r.get("to",[]); ports=r.get("ports",[]) + if len(tos)==1 and tos[0].get("podSelector",{}).get("matchLabels",{}).get("app")==app \ + and len(ports)==1 and str(ports[0].get("port"))==str(port) and ports[0].get("protocol","TCP")=="TCP": + return True + return False +sys.exit(0 if len(e)==2 and pair("cache",6379) and pair("metrics",9090) else 1)'"""), + C("no rule allows a destination on the wrong port (single combined rule fails here)", 3, r""" +$K -n project-snake2 get netpol np-api -o json 2>/dev/null | python3 -c ' +import sys,json +e=json.load(sys.stdin)["spec"].get("egress",[]) +for r in e: + apps={t.get("podSelector",{}).get("matchLabels",{}).get("app") for t in r.get("to",[])} + ports={str(p.get("port")) for p in r.get("ports",[])} + if len(apps)>1 or len(ports)>1: sys.exit(1) # combined rule = cross product = too permissive +sys.exit(0)'"""), + ], + solution=""" +cat < cache:9090. That is the trap. +""", +) + +# ------------------------------------------------------------------ 216 ---- +task( + id=216, domain="NET", title="CoreDNS: back up, extend, verify", breaks="coredns", + task=""" +1. Back up the current CoreDNS configuration to `{WS}/q216/coredns-backup.yaml` in a form you + could restore from directly with `kubectl apply -f`. +2. Change CoreDNS so that `SERVICE.NAMESPACE.svc.internal.zone` resolves to the same address as + `SERVICE.NAMESPACE.svc.cluster.local`, **in addition** to the existing name. +3. Roll CoreDNS so the change is live. +4. Prove it: write the output of `nslookup kubernetes.default.svc.internal.zone` from a Pod to + `{WS}/q216/lookup.txt`. +""", + setup=r""" +rm -rf "$WS/q216"; mkdir -p "$WS/q216" +cur=$(j -n kube-system cm coredns '{.data.Corefile}') +echo "$cur" | grep -q internal.zone && { echo "$cur" | sed 's/ internal.zone//' > /tmp/cf216; $K -n kube-system create cm coredns --from-file=Corefile=/tmp/cf216 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n kube-system rollout restart deploy coredns >/dev/null; } +$K -n default delete pod dns-client --ignore-not-found --wait=false >/dev/null +""", + checks=[ + C("backup file is a restorable ConfigMap manifest with the original Corefile", 3, r""" +f="$WS/q216/coredns-backup.yaml"; grep -q 'kind: ConfigMap' "$f" 2>/dev/null && grep -q 'name: coredns' "$f" && grep -q 'Corefile' "$f" && grep -q 'kubernetes cluster.local' "$f" """), + C("Corefile serves internal.zone alongside cluster.local", 3, r"""j -n kube-system cm coredns '{.data.Corefile}' | grep -E '^ *kubernetes ' | grep -q 'internal.zone' """), + C("CoreDNS rolled out and available", 2, r"""for i in $(seq 1 12); do [ "$(j -n kube-system deploy coredns '{.status.availableReplicas}')" -ge 1 ] 2>/dev/null && exit 0; sleep 5; done; exit 1"""), + C("lookup.txt resolves the new name to the kubernetes Service IP", 3, r""" +ip=$(j -n default svc kubernetes '{.spec.clusterIP}'); grep -q "$ip" "$WS/q216/lookup.txt" 2>/dev/null && grep -q 'internal.zone' "$WS/q216/lookup.txt" """), + ], + solution=""" +k -n kube-system get cm coredns -o yaml > {WS}/q216/coredns-backup.yaml +k -n kube-system edit cm coredns +# kubernetes cluster.local in-addr.arpa ip6.arpa { +# becomes +# kubernetes cluster.local internal.zone in-addr.arpa ip6.arpa { +k -n kube-system rollout restart deploy coredns +k -n kube-system rollout status deploy coredns +k run dns-client --image=busybox:1.36 --restart=Never -- sleep 3600 +k exec dns-client -- nslookup kubernetes.default.svc.internal.zone | tee {WS}/q216/lookup.txt +# Restore if needed: k delete -f {WS}/q216/coredns-backup.yaml && k apply -f {WS}/q216/coredns-backup.yaml +""", +) + +# ------------------------------------------------------------------ 217 ---- +task( + id=217, domain="TSHOOT", title="Inspect a container with crictl", + task=""" +1. In Namespace `project-tiger2` create Pod `tiger-probe`, image `httpd:2-alpine`, with labels + `pod=probe` and `probe=pod`. +2. Find which node it runs on, connect to that node, and locate its container with `crictl`. +3. Write the container ID and its `runtimeType` (from `crictl inspect`), separated by a space, + to `{WS}/q217/container.txt`. +4. Write the container's logs to `{WS}/q217/container.log`. +""", + setup=NS + r""" +mkns project-tiger2 +rm -rf "$WS/q217"; mkdir -p "$WS/q217" +$K -n project-tiger2 delete pod tiger-probe --ignore-not-found --wait=false >/dev/null +""", + checks=[ + C("Pod tiger-probe Running with both labels", 2, r""" +eq "$(j -n project-tiger2 pod tiger-probe '{.status.phase}')" Running && eq "$(j -n project-tiger2 pod tiger-probe '{.metadata.labels.pod}')" probe && eq "$(j -n project-tiger2 pod tiger-probe '{.metadata.labels.probe}')" pod"""), + C("container.txt holds a real container ID for that Pod", 4, r""" +id=$(awk '{print $1}' "$WS/q217/container.txt" 2>/dev/null); [ -n "$id" ] || exit 1 +node=$(j -n project-tiger2 pod tiger-probe '{.spec.nodeName}') +cid=$(j -n project-tiger2 pod tiger-probe '{.status.containerStatuses[0].containerID}' | sed 's|.*//||') +case "$cid" in "$id"*) exit 0;; esac; exit 1"""), + C("container.txt records the runtimeType", 2, r"""grep -q 'io.containerd.runc' "$WS/q217/container.txt" 2>/dev/null"""), + C("container.log holds the httpd startup output", 2, r"""grep -qi 'httpd' "$WS/q217/container.log" 2>/dev/null"""), + ], + solution=""" +k -n project-tiger2 run tiger-probe --image=httpd:2-alpine --labels "pod=probe,probe=pod" +k -n project-tiger2 get pod tiger-probe -o wide # note the NODE +{SHELL_W1} # ...whichever node it landed on +crictl ps | grep tiger-probe # -> container id +crictl inspect | grep runtimeType # -> io.containerd.runc.v2 +crictl logs +# write " io.containerd.runc.v2" into {WS}/q217/container.txt and the logs into container.log +# crictl ps -a also shows exited containers; crictl pods lists sandboxes. +""", +) + +QUESTIONS = Q diff --git a/tracks/cka/bank_sim_b.py b/tracks/cka/bank_sim_b.py new file mode 100644 index 0000000..eeb8fe0 --- /dev/null +++ b/tracks/cka/bank_sim_b.py @@ -0,0 +1,973 @@ +"""Simulator Set B - 17 tasks, fixed set, killer.sh-grade difficulty. + +Original scenarios. Weighted toward node-level and control-plane work: etcd flags, kube-proxy +rules, ServiceCIDR, bound tokens, PDB-blocked drains, PV reclaim, upgrade planning, CSR signing. + +Run with: ./exam sim b +""" +from tracks.cka.common import C, NS + +Q = [] + + +def task(**kw): + kw.setdefault("difficulty", "hard") + kw["set"] = "b" + Q.append(kw) + + +# ------------------------------------------------------------------ 301 ---- +task( + id=301, domain="ARCH", title="etcd configuration facts", + task=""" +The security team wants the following about the etcd running on the control-plane node. +Write `{WS}/q301/etcd-info.txt` with exactly these five lines (values after the colon): + +``` +data-dir: +server-key: +trusted-ca: +client-cert-auth: +server-cert-expiry: +``` + +Take every value from how etcd is actually started, not from memory. +""", + setup=r"""rm -rf "$WS/q301"; mkdir -p "$WS/q301" """, + checks=[ + C("data-dir and server-key are correct", 3, r""" +f="$WS/q301/etcd-info.txt"; [ -f "$f" ] || exit 1 +cmd=$(j -n kube-system pod -l component=etcd '{.items[0].spec.containers[0].command}') +dd=$(echo "$cmd" | tr ',' '\n' | grep -o 'data-dir=[^"]*' | cut -d= -f2) +sk=$(echo "$cmd" | tr ',' '\n' | grep -o '\-\-key-file=[^"]*' | cut -d= -f2) +grep -q "data-dir: *$dd" "$f" && grep -q "server-key: *$sk" "$f" """), + C("trusted-ca and client-cert-auth are correct", 3, r""" +f="$WS/q301/etcd-info.txt" +cmd=$(j -n kube-system pod -l component=etcd '{.items[0].spec.containers[0].command}') +ca=$(echo "$cmd" | tr ',' '\n' | grep -o '\-\-trusted-ca-file=[^"]*' | cut -d= -f2) +grep -q "trusted-ca: *$ca" "$f" && grep -Eqi 'client-cert-auth: *(yes|true)' "$f" """), + C("server-cert-expiry matches the certificate on disk", 3, r""" +exp=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -enddate' | sed 's/notAfter=//') +yr=$(echo "$exp" | awk '{print $4}'); mon=$(echo "$exp" | awk '{print $1}'); day=$(echo "$exp" | awk '{print $2}') +f="$WS/q301/etcd-info.txt"; grep -q "$yr" "$f" 2>/dev/null && grep -qi "$mon" "$f" && grep -q "$day" "$f" """), + ], + solution=""" +{SHELL_CP} +cat /etc/kubernetes/manifests/etcd.yaml # every flag etcd runs with +# --data-dir=/var/lib/etcd +# --key-file=/etc/kubernetes/pki/etcd/server.key +# --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt +# --client-cert-auth=true +openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -enddate +# etcd runs as a static pod, so the manifest in /etc/kubernetes/manifests is the source of truth. +# k -n kube-system get pod -l component=etcd -o yaml would show the same command. +""", +) + +# ------------------------------------------------------------------ 302 ---- +task( + id=302, domain="NET", title="Service rules on the node", breaks="proxy-rules", + task=""" +Confirm that kube-proxy is programming this cluster. + +1. In Namespace `project-proxy` create Pod `proxy-pod` (image `nginx:1-alpine`) and expose it + internally as Service `proxy-svc` on port **4000** targeting container port 80. +2. From the node where the Pod runs, capture the proxy rules that mention `proxy-svc` into + `{WS}/q302/rules.txt` (the node uses iptables; `iptables-save` is available as root). +3. Write the Service's ClusterIP to `{WS}/q302/clusterip.txt`. +4. Write the proxy mode kube-proxy is running in to `{WS}/q302/mode.txt` + (an empty `mode` in its ConfigMap means `iptables`). +""", + setup=NS + r""" +mkns project-proxy +rm -rf "$WS/q302"; mkdir -p "$WS/q302" +$K -n project-proxy delete pod proxy-pod --ignore-not-found --wait=false >/dev/null +$K -n project-proxy delete svc proxy-svc --ignore-not-found >/dev/null +""", + checks=[ + C("Pod and Service exist, 4000 -> 80, with endpoints", 3, r""" +eq "$(j -n project-proxy pod proxy-pod '{.status.phase}')" Running \ + && eq "$(j -n project-proxy svc proxy-svc '{.spec.ports[0].port}')" 4000 && eq "$(j -n project-proxy svc proxy-svc '{.spec.ports[0].targetPort}')" 80 \ + && [ -n "$(j -n project-proxy endpointslices -l kubernetes.io/service-name=proxy-svc '{.items[*].endpoints[*].addresses[*]}')" ]"""), + C("clusterip.txt matches the Service", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q302/clusterip.txt" 2>/dev/null)" "$(j -n project-proxy svc proxy-svc '{.spec.clusterIP}')" """), + C("rules.txt contains real proxy rules naming the Service and its ClusterIP", 4, r""" +f="$WS/q302/rules.txt"; grep -q 'proxy-svc' "$f" 2>/dev/null && grep -q "$(j -n project-proxy svc proxy-svc '{.spec.clusterIP}')" "$f" && grep -Eq 'KUBE-(SVC|SEP|SERVICES)' "$f" """), + C("mode.txt says iptables", 2, r"""grep -qi 'iptables' "$WS/q302/mode.txt" 2>/dev/null"""), + ], + solution=""" +k -n project-proxy run proxy-pod --image=nginx:1-alpine +k -n project-proxy expose pod proxy-pod --name proxy-svc --port 4000 --target-port 80 +k -n project-proxy get svc proxy-svc -o jsonpath='{.spec.clusterIP}' > {WS}/q302/clusterip.txt +k -n project-proxy get pod proxy-pod -o wide # note the node +{SHELL_W1} # the node it landed on +iptables-save | grep proxy-svc # -> copy into {WS}/q302/rules.txt +exit +k -n kube-system get cm kube-proxy -o yaml | grep -E '^ *mode:' # empty -> iptables +echo iptables > {WS}/q302/mode.txt +# Deleting the Service makes the rules disappear within a second - kube-proxy watches the API +# and rewrites the rules on every Service/EndpointSlice change. +""", +) + +# ------------------------------------------------------------------ 303 ---- +task( + id=303, domain="NET", title="A second Service CIDR", + task=""" +The cluster is running out of Service IPs. + +1. Write the CIDR the cluster currently allocates Service IPs from to `{WS}/q303/current-cidr.txt`. +2. Add a **second** range `10.180.0.0/16` to the cluster by creating a `ServiceCIDR` object named + `extra-range` (do not remove the existing one). +3. In Namespace `default` create Deployment `cidr-demo` (image `nginx:1-alpine`) and a ClusterIP + Service `cidr-demo` on port 80 whose ClusterIP is taken **from the new range** + (request the address explicitly if you need to). +4. Write that Service's ClusterIP to `{WS}/q303/new-ip.txt`. +""", + setup=r""" +rm -rf "$WS/q303"; mkdir -p "$WS/q303" +$K delete svc cidr-demo --ignore-not-found >/dev/null 2>&1 +$K delete deploy cidr-demo --ignore-not-found >/dev/null 2>&1 +$K delete servicecidr extra-range --ignore-not-found >/dev/null 2>&1 +$K get servicecidr >/dev/null 2>&1 || echo "warn: this cluster has no ServiceCIDR API (needs MultiCIDRServiceAllocator)" +""", + checks=[ + C("current-cidr.txt matches the apiserver's service range", 2, r""" +sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2) +grep -qF "$sc" "$WS/q303/current-cidr.txt" 2>/dev/null"""), + C("ServiceCIDR extra-range covers 10.180.0.0/16 and is Ready", 4, r""" +j servicecidr extra-range '{.spec.cidrs[*]}' | grep -q '10.180.0.0/16' || exit 1 +for i in $(seq 1 10); do [ "$(j servicecidr extra-range '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 2; done; exit 1"""), + C("Service cidr-demo has an IP from the new range and ready endpoints", 4, r""" +ip=$(j -n default svc cidr-demo '{.spec.clusterIP}'); case "$ip" in 10.180.*) ;; *) exit 1;; esac +[ -n "$(j -n default endpointslices -l kubernetes.io/service-name=cidr-demo '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""), + C("new-ip.txt records that ClusterIP", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q303/new-ip.txt" 2>/dev/null)" "$(j -n default svc cidr-demo '{.spec.clusterIP}')" """), + ], + solution=""" +k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range +# -> e.g. 10.96.0.0/16 ; write it into {WS}/q303/current-cidr.txt +k get servicecidr # the built-in "kubernetes" ServiceCIDR +cat < {WS}/q303/new-ip.txt +# New ranges are additive: existing Services keep their IPs, and allocation from a new range +# happens once the ServiceCIDR is Ready. Requesting --cluster-ip makes it deterministic. +""", +) + +# ------------------------------------------------------------------ 304 ---- +task( + id=304, domain="ARCH", title="Bound tokens versus legacy tokens", + task=""" +In Namespace `project-token` there is ServiceAccount `reporter`. + +1. Issue a **bound token** for it that is valid for exactly 2 hours and save it to + `{WS}/q304/token.jwt`. +2. Decode the token's payload and write its `exp` and `iat` claims, as + `exp=` and `iat=` on two lines, to `{WS}/q304/claims.txt`. +3. Also create a **long-lived** token for the same ServiceAccount, as a Secret named + `reporter-legacy` of the correct type, and confirm the controller populated it. +""", + setup=NS + r""" +mkns project-token +rm -rf "$WS/q304"; mkdir -p "$WS/q304" +$K -n project-token create sa reporter --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n project-token delete secret reporter-legacy --ignore-not-found >/dev/null +""", + checks=[ + C("token.jwt is a JWT issued for the reporter ServiceAccount", 3, r""" +f="$WS/q304/token.jwt"; [ -s "$f" ] || exit 1 +python3 - "$f" <<'PY' +import base64,json,sys +t=open(sys.argv[1]).read().strip().split(".") +if len(t)!=3: sys.exit(1) +p=json.loads(base64.urlsafe_b64decode(t[1]+"=="*3)) +sub=p.get("sub",""); ko=p.get("kubernetes.io",{}).get("serviceaccount",{}).get("name","") +sys.exit(0 if "reporter" in sub or ko=="reporter" else 1) +PY"""), + C("the token's lifetime is 2 hours", 3, r""" +python3 - "$WS/q304/token.jwt" <<'PY' +import base64,json,sys +p=json.loads(base64.urlsafe_b64decode(open(sys.argv[1]).read().strip().split(".")[1]+"=="*3)) +d=p["exp"]-p["iat"] +sys.exit(0 if 7000 <= d <= 7400 else 1) +PY"""), + C("claims.txt records exp and iat matching the token", 2, r""" +python3 - "$WS/q304/token.jwt" "$WS/q304/claims.txt" <<'PY' +import base64,json,sys,re +p=json.loads(base64.urlsafe_b64decode(open(sys.argv[1]).read().strip().split(".")[1]+"=="*3)) +try: t=open(sys.argv[2]).read() +except OSError: sys.exit(1) +ok=re.search(r"exp\s*=\s*%d"%p["exp"],t) and re.search(r"iat\s*=\s*%d"%p["iat"],t) +sys.exit(0 if ok else 1) +PY"""), + C("Secret reporter-legacy is a populated service-account-token for reporter", 3, r""" +eq "$(j -n project-token secret reporter-legacy '{.type}')" kubernetes.io/service-account-token \ + && eq "$(j -n project-token secret reporter-legacy '{.metadata.annotations.kubernetes\.io/service-account\.name}')" reporter \ + && [ -n "$(j -n project-token secret reporter-legacy '{.data.token}')" ]"""), + ], + solution=""" +k -n project-token create token reporter --duration=2h > {WS}/q304/token.jwt +# decode the middle segment (JWTs are base64url, pad it out): +cut -d. -f2 {WS}/q304/token.jwt | base64 -d 2>/dev/null | python3 -m json.tool +# -> copy exp= and iat= into {WS}/q304/claims.txt +cat <`, and on a second line `scheduler=`. +""", + setup=NS + r""" +mkns project-orion +rm -rf "$WS/q305"; mkdir -p "$WS/q305" +$K -n project-orion delete pod direct-assign scheduled --ignore-not-found --wait=false >/dev/null +$K label node "$W3" tier- >/dev/null 2>&1 || true +""", + checks=[ + C("direct-assign Running on the right node with spec.nodeName set", 3, r""" +eq "$(j -n project-orion pod direct-assign '{.status.phase}')" Running && eq "$(j -n project-orion pod direct-assign '{.spec.nodeName}')" "$W3" """), + C("direct-assign was never scheduled (no Scheduled event from default-scheduler)", 3, r""" +! $K -n project-orion get events --field-selector involvedObject.name=direct-assign 2>/dev/null | grep -q 'default-scheduler'"""), + C("scheduled Running on the same node via the default scheduler", 3, r""" +eq "$(j -n project-orion pod scheduled '{.status.phase}')" Running && eq "$(j -n project-orion pod scheduled '{.spec.nodeName}')" "$W3" \ + && $K -n project-orion get events --field-selector involvedObject.name=scheduled 2>/dev/null | grep -q 'default-scheduler'"""), + C("proof.txt names nodeName and default-scheduler", 2, r""" +grep -q 'field= *nodeName' "$WS/q305/proof.txt" 2>/dev/null && grep -q 'scheduler= *default-scheduler' "$WS/q305/proof.txt" """), + ], + solution=""" +cat < {WS}/q305/proof.txt +""", +) + +# ------------------------------------------------------------------ 306 ---- +task( + id=306, domain="STOR", title="Grow a volume in place", + task=""" +In Namespace `project-orbit`: + +1. Create StorageClass `expandable` using provisioner `rancher.io/local-path`, + `volumeBindingMode: WaitForFirstConsumer`, and allowing volume expansion. +2. Create PVC `growing` (1Gi, RWO) using that class and Pod `grower` (busybox:1.36, `sleep 86400`) + mounting it at `/data`, so the claim binds. +3. Then grow the claim to **2Gi** without deleting it, and write the claim's + `.status.capacity.storage` after the resize to `{WS}/q306/capacity.txt`. +""", + setup=NS + r""" +mkns project-orbit +rm -rf "$WS/q306"; mkdir -p "$WS/q306" +$K -n project-orbit delete pod grower --ignore-not-found --wait=false >/dev/null +$K -n project-orbit delete pvc growing --ignore-not-found >/dev/null +$K delete sc expandable --ignore-not-found >/dev/null +""", + checks=[ + C("StorageClass expandable: local-path, WaitForFirstConsumer, expansion allowed", 3, r""" +s="$K get sc expandable" +eq "$($s -o jsonpath='{.provisioner}' 2>/dev/null)" rancher.io/local-path && eq "$($s -o jsonpath='{.volumeBindingMode}')" WaitForFirstConsumer \ + && eq "$($s -o jsonpath='{.allowVolumeExpansion}')" true"""), + C("PVC growing Bound on that class with grower Running", 3, r""" +eq "$(j -n project-orbit pvc growing '{.spec.storageClassName}')" expandable && eq "$(j -n project-orbit pvc growing '{.status.phase}')" Bound \ + && eq "$(j -n project-orbit pod grower '{.status.phase}')" Running"""), + C("claim requests 2Gi and reports at least 2Gi capacity", 4, r""" +eq "$(j -n project-orbit pvc growing '{.spec.resources.requests.storage}')" 2Gi || exit 1 +for i in $(seq 1 12); do c=$(j -n project-orbit pvc growing '{.status.capacity.storage}'); [ "$c" = 2Gi ] && exit 0; sleep 5; done; exit 1"""), + C("capacity.txt records the new capacity", 1, r"""grep -q '2Gi' "$WS/q306/capacity.txt" 2>/dev/null"""), + ], + solution=""" +cat < {WS}/q306/capacity.txt +# Shrinking is never allowed. Without allowVolumeExpansion the patch is rejected outright. +""", +) + +# ------------------------------------------------------------------ 307 ---- +task( + id=307, domain="TSHOOT", title="Node accepts nothing new", breaks="w1-block", + task=""" +Node `{W1}` is `Ready` and its kubelet is healthy, but no new Pod will schedule onto it. + +Find every reason and remove them, so that a Pod with a `nodeSelector` pinning it to `{W1}` +starts successfully. Do not delete or recreate the node, and do not use `nodeName` to work around it. +""", + setup=r""" +$K cordon "$W1" >/dev/null 2>&1 || true +$K taint node "$W1" maintenance=inprogress:NoSchedule --overwrite >/dev/null 2>&1 || true +$K label node "$W1" kubernetes.io/os- >/dev/null 2>&1 || true +$K label node "$W1" kubernetes.io/os=linux --overwrite >/dev/null 2>&1 || true +""", + checks=[ + C("node is schedulable (uncordoned)", 3, r"""[ "$(j node "$W1" '{.spec.unschedulable}')" != true ]"""), + C("the maintenance taint is gone", 3, r"""! j node "$W1" '{.spec.taints}' | grep -q 'maintenance'"""), + C("a pinned Pod actually runs there", 4, r""" +$K -n default delete pod pin-probe --ignore-not-found --wait=true >/dev/null 2>&1 +cat </dev/null 2>&1 +apiVersion: v1 +kind: Pod +metadata: {name: pin-probe, namespace: default} +spec: + nodeSelector: {kubernetes.io/hostname: $W1} + containers: [{name: c, image: busybox:1.36, command: [sleep, "120"]}] +EOF +ok=1; for i in $(seq 1 20); do [ "$(j -n default pod pin-probe '{.status.phase}')" = Running ] && { ok=0; break; }; sleep 3; done +$K -n default delete pod pin-probe --wait=false >/dev/null 2>&1; exit $ok"""), + ], + solution=""" +k describe node {W1} | head -20 +# Taints: maintenance=inprogress:NoSchedule +# Unschedulable: true +k uncordon {W1} +k taint node {W1} maintenance- # trailing dash removes the taint +k get node {W1} -o jsonpath='{.spec.taints}{"\\n"}{.spec.unschedulable}{"\\n"}' +# Cordon and taints are two independent mechanisms - `kubectl drain` sets the cordon, +# and a taint may have been left by an operator or added by hand. Both must go. +""", +) + +# ------------------------------------------------------------------ 308 ---- +task( + id=308, domain="ARCH", title="API resource reference card", + task=""" +Write `{WS}/q308/resources.txt` with one line per resource below, in this exact format: + +``` +||| +``` + +for these four: `networkpolicies`, `persistentvolumes`, `storageclasses`, `certificatesigningrequests`. + +Take the values from the cluster (`kubectl api-resources`), not from memory. Use `-` when a +resource has no shortname. Lines in the order given above. +""", + setup=r"""rm -rf "$WS/q308"; mkdir -p "$WS/q308" """, + checks=[ + C("all four lines present and correctly formatted", 3, r""" +f="$WS/q308/resources.txt"; [ -f "$f" ] || exit 1 +for r in networkpolicies persistentvolumes storageclasses certificatesigningrequests; do grep -q "^$r|" "$f" || exit 1; done +[ "$(grep -c '|' "$f")" -ge 4 ]"""), + C("shortnames and namespaced flags correct", 4, r""" +f="$WS/q308/resources.txt" +grep -Eq '^networkpolicies\|netpol\|networking.k8s.io/v1\|true$' "$f" || exit 1 +grep -Eq '^persistentvolumes\|pv\|v1\|false$' "$f" || exit 1 +grep -Eq '^storageclasses\|sc\|storage.k8s.io/v1\|false$' "$f" || exit 1 +grep -Eq '^certificatesigningrequests\|csr\|certificates.k8s.io/v1\|false$' "$f" || exit 1"""), + C("order matches the task", 2, r""" +f="$WS/q308/resources.txt" +[ "$(grep -o '^[a-z]*' "$f" | grep . | head -4 | tr '\n' ' ')" = "networkpolicies persistentvolumes storageclasses certificatesigningrequests " ]"""), + ], + solution=""" +k api-resources | grep -E '^(networkpolicies|persistentvolumes|storageclasses|certificatesigningrequests) ' +# NAME SHORTNAMES APIVERSION NAMESPACED KIND +cat > {WS}/q308/resources.txt <<'EOF' +networkpolicies|netpol|networking.k8s.io/v1|true +persistentvolumes|pv|v1|false +storageclasses|sc|storage.k8s.io/v1|false +certificatesigningrequests|csr|certificates.k8s.io/v1|false +EOF +# k api-resources --namespaced=false narrows to cluster-scoped resources; +# k explain shows the same apiVersion when you're unsure. +""", +) + +# ------------------------------------------------------------------ 309 ---- +task( + id=309, domain="WORK", title="A PodDisruptionBudget that blocks maintenance", breaks="w3-drain", + task=""" +Namespace `project-atlas` runs Deployment `atlas-api` with 2 replicas, and a PodDisruptionBudget +`atlas-api` that currently makes voluntary eviction impossible. + +Node `{W3}` must be drained for maintenance, and `atlas-api` must keep at least one Pod serving +at all times. + +1. Adjust the PodDisruptionBudget so the drain can complete while still guaranteeing one + available Pod. Keep the PDB — do not delete it. +2. Drain `{W3}` (DaemonSet Pods aside), then make it schedulable again. +3. Write the value of the PDB's `.status.disruptionsAllowed` after your change to + `{WS}/q309/allowed.txt`. +""", + setup=NS + r""" +mkns project-atlas +rm -rf "$WS/q309"; mkdir -p "$WS/q309" +$K uncordon "$W3" >/dev/null 2>&1 || true +$K -n project-atlas delete deploy atlas-api --ignore-not-found >/dev/null +$K -n project-atlas delete pdb atlas-api --ignore-not-found >/dev/null +$K apply -f - >/dev/null <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: {name: atlas-api, namespace: project-atlas} +spec: + replicas: 2 + selector: {matchLabels: {app: atlas-api}} + template: + metadata: {labels: {app: atlas-api}} + spec: + containers: [{name: api, image: nginx:1.27, resources: {requests: {cpu: 10m, memory: 16Mi}}}] +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: {name: atlas-api, namespace: project-atlas} +spec: + minAvailable: 2 + selector: {matchLabels: {app: atlas-api}} +EOF +$K -n project-atlas rollout status deploy atlas-api --timeout=120s >/dev/null 2>&1 +""", + checks=[ + C("PDB still exists and now permits exactly one disruption", 3, r""" +p="$K -n project-atlas get pdb atlas-api" +$p >/dev/null 2>&1 || exit 1 +ma=$($p -o jsonpath='{.spec.minAvailable}'); mu=$($p -o jsonpath='{.spec.maxUnavailable}') +{ [ "$ma" = 1 ] || [ "$mu" = 1 ]; }"""), + C("no atlas-api Pod left on the drained node", 3, r""" +n=$($K -n project-atlas get pod -l app=atlas-api -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' 2>/dev/null | grep -c "^$W3$") +eq "$n" 0"""), + C("both replicas Available and node schedulable again", 3, r""" +eq "$(j -n project-atlas deploy atlas-api '{.status.availableReplicas}')" 2 && [ "$(j node "$W3" '{.spec.unschedulable}')" != true ]"""), + C("allowed.txt records disruptionsAllowed", 1, r"""grep -Eq '^[0-9]+$' <(tr -dc '0-9\n' < "$WS/q309/allowed.txt" 2>/dev/null | grep .) """), + ], + solution=""" +k -n project-atlas get pdb atlas-api +# ALLOWED DISRUPTIONS: 0 -> minAvailable 2 with 2 replicas means nothing may be evicted; +# a drain would hang forever waiting for the eviction API. +k -n project-atlas patch pdb atlas-api -p '{"spec":{"minAvailable":1}}' +k -n project-atlas get pdb atlas-api # ALLOWED DISRUPTIONS: 1 +k drain {W3} --ignore-daemonsets --delete-emptydir-data +k -n project-atlas rollout status deploy atlas-api +k uncordon {W3} +k -n project-atlas get pdb atlas-api -o jsonpath='{.status.disruptionsAllowed}' > {WS}/q309/allowed.txt +""", +) + +# ------------------------------------------------------------------ 310 ---- +task( + id=310, domain="NET", title="Route to something outside the cluster", + task=""" +An external database lives at `10.190.7.25` port 5432. Applications should reach it through a +normal in-cluster Service name. + +In Namespace `project-orbit`: +1. Create a Service `external-db` (ClusterIP, port 5432) that has **no selector**. +2. Make it resolve to that external address by creating the matching `EndpointSlice` + named `external-db-1` (port 5432, TCP, addressType IPv4) with the right labels so the + Service adopts it. +3. Write the ClusterIP of the Service to `{WS}/q310/ip.txt`. + +Do not use an ExternalName Service. +""", + setup=NS + r""" +mkns project-orbit +rm -rf "$WS/q310"; mkdir -p "$WS/q310" +$K -n project-orbit delete svc external-db --ignore-not-found >/dev/null +$K -n project-orbit delete endpointslice external-db-1 --ignore-not-found >/dev/null 2>&1 +""", + checks=[ + C("Service external-db is ClusterIP on 5432 with no selector", 3, r""" +s="$K -n project-orbit get svc external-db" +eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" ClusterIP && eq "$($s -o jsonpath='{.spec.ports[0].port}')" 5432 && [ -z "$($s -o jsonpath='{.spec.selector}')" ]"""), + C("EndpointSlice external-db-1 points at 10.190.7.25:5432", 4, r""" +e="$K -n project-orbit get endpointslice external-db-1" +eq "$($e -o jsonpath='{.addressType}' 2>/dev/null)" IPv4 && $e -o jsonpath='{.endpoints[*].addresses[*]}' | grep -q '10.190.7.25' \ + && eq "$($e -o jsonpath='{.ports[0].port}')" 5432"""), + C("the slice is linked to the Service by label", 2, r"""eq "$(j -n project-orbit endpointslice external-db-1 '{.metadata.labels.kubernetes\.io/service-name}')" external-db"""), + C("ip.txt records the Service ClusterIP", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q310/ip.txt" 2>/dev/null)" "$(j -n project-orbit svc external-db '{.spec.clusterIP}')" """), + ], + solution=""" +cat < {WS}/q310/ip.txt +k -n project-orbit get endpointslice +# ExternalName would return a CNAME instead and gives you no ClusterIP or port mapping. +""", +) + +# ------------------------------------------------------------------ 311 ---- +task( + id=311, domain="ARCH", title="Image inventory across the cluster", + task=""" +Produce `{WS}/q311/images.txt`: for every Deployment in every Namespace, one line + +``` +/ [,...] +``` + +with the images of its Pod template (init containers included), comma-separated in the order they +appear, and the file sorted by the first column. + +Then write the number of **distinct** images in use across those Deployments to +`{WS}/q311/distinct.txt`. +""", + setup=r"""rm -rf "$WS/q311"; mkdir -p "$WS/q311" """, + checks=[ + C("images.txt matches the cluster exactly", 5, r""" +want=$($K get deploy -A -o json 2>/dev/null | python3 -c ' +import sys,json +out=[] +for d in json.load(sys.stdin)["items"]: + s=d["spec"]["template"]["spec"] + imgs=[c["image"] for c in s.get("initContainers",[])]+[c["image"] for c in s["containers"]] + out.append("%s/%s %s"%(d["metadata"]["namespace"],d["metadata"]["name"],",".join(imgs))) +print("\n".join(sorted(out)))') +[ "$(grep . "$WS/q311/images.txt" 2>/dev/null | sort)" = "$want" ]"""), + C("distinct.txt has the right count", 3, r""" +want=$($K get deploy -A -o json 2>/dev/null | python3 -c ' +import sys,json +s=set() +for d in json.load(sys.stdin)["items"]: + sp=d["spec"]["template"]["spec"] + for c in sp.get("initContainers",[])+sp["containers"]: s.add(c["image"]) +print(len(s))') +eq "$(tr -dc '0-9' < "$WS/q311/distinct.txt" 2>/dev/null)" "$want" """), + ], + solution=""" +k get deploy -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" "}{range .spec.template.spec.initContainers[*]}{.image}{","}{end}{range .spec.template.spec.containers[*]}{.image}{","}{end}{"\\n"}{end}' \\ + | sed 's/,$//' | sort > {WS}/q311/images.txt +k get deploy -A -o jsonpath='{range .items[*]}{range .spec.template.spec.initContainers[*]}{.image}{"\\n"}{end}{range .spec.template.spec.containers[*]}{.image}{"\\n"}{end}{end}' \\ + | grep . | sort -u | wc -l > {WS}/q311/distinct.txt +# Two `range` blocks over the same pod spec keeps init and normal containers in order. +# A trailing comma is easiest to strip afterwards with sed. +""", +) + +# ------------------------------------------------------------------ 312 ---- +task( + id=312, domain="WORK", title="A Job that fails, then one that doesn't", + task=""" +In Namespace `project-orion`: + +1. Job `flaky` exists and keeps failing. Let it reach its terminal state, then write the name of + the condition that marks it failed and its reason, as `/`, to `{WS}/q312/why.txt`. +2. Create a **new** Job `steady` that does the same work correctly: image `busybox:1.36`, + 6 completions, 3 in parallel, `backoffLimit: 2`, restartPolicy Never, each Pod printing + `ok` and exiting 0. +3. `steady` must complete all 6. +""", + setup=NS + r""" +mkns project-orion +rm -rf "$WS/q312"; mkdir -p "$WS/q312" +$K -n project-orion delete job flaky steady --ignore-not-found >/dev/null +$K apply -f - >/dev/null <<'EOF' +apiVersion: batch/v1 +kind: Job +metadata: {name: flaky, namespace: project-orion} +spec: + completions: 2 + parallelism: 1 + backoffLimit: 1 + template: + spec: + restartPolicy: Never + containers: [{name: c, image: busybox:1.36, command: [sh, -c, "echo starting; exit 7"]}] +EOF +""", + checks=[ + C("why.txt names the Failed condition and BackoffLimitExceeded", 3, r""" +f="$WS/q312/why.txt"; grep -qi 'Failed' "$f" 2>/dev/null && grep -qi 'BackoffLimitExceeded' "$f" """), + C("Job steady: 6 completions, parallelism 3, backoffLimit 2, restartPolicy Never", 3, r""" +j1="$K -n project-orion get job steady" +eq "$($j1 -o jsonpath='{.spec.completions}' 2>/dev/null)" 6 && eq "$($j1 -o jsonpath='{.spec.parallelism}')" 3 \ + && eq "$($j1 -o jsonpath='{.spec.backoffLimit}')" 2 && eq "$($j1 -o jsonpath='{.spec.template.spec.restartPolicy}')" Never"""), + C("steady completed all 6", 4, r"""for i in $(seq 1 20); do [ "$(j -n project-orion job steady '{.status.succeeded}')" = 6 ] && exit 0; sleep 3; done; exit 1"""), + ], + solution=""" +k -n project-orion get job flaky -o yaml | grep -A6 conditions +# type: Failed / reason: BackoffLimitExceeded +k -n project-orion describe job flaky +k -n project-orion logs job/flaky # exit 7 +echo Failed/BackoffLimitExceeded > {WS}/q312/why.txt +k -n project-orion create job steady --image=busybox:1.36 $do -- sh -c "echo ok" > steady.yaml +# set spec.completions: 6, spec.parallelism: 3, spec.backoffLimit: 2 +k apply -f steady.yaml +k -n project-orion wait --for=condition=complete job/steady --timeout=120s +""", +) + +# ------------------------------------------------------------------ 313 ---- +task( + id=313, domain="NET", title="Two hosts, TLS, and a default backend", + task=""" +Namespace `project-r700` has Services `web-classic` and `web-next` (port 80). + +Create Ingress `multi` in that Namespace with class `nginx`: + +- host `classic.r700.io`, path `/` (Prefix) → `web-classic:80`, TLS with secret `classic-tls` +- host `next.r700.io`, path `/` (Prefix) → `web-next:80`, TLS with the same secret +- a **default backend** (requests matching no rule) → `web-classic:80` + +Also create the TLS secret `classic-tls` from a self-signed certificate whose CN is `classic.r700.io`. +""", + setup=NS + r""" +mkns project-r700 +for s in web-classic web-next; do + $K -n project-r700 create deploy $s --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null + $K -n project-r700 expose deploy $s --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +done +$K -n project-r700 delete ingress multi --ignore-not-found >/dev/null +$K -n project-r700 delete secret classic-tls --ignore-not-found >/dev/null +""", + checks=[ + C("Secret classic-tls is a TLS secret with CN classic.r700.io", 3, r""" +eq "$(j -n project-r700 secret classic-tls '{.type}')" kubernetes.io/tls \ + && j -n project-r700 secret classic-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'CN *= *classic.r700.io'"""), + C("both host rules route to the right Services on /", 3, r""" +$K -n project-r700 get ingress multi -o json 2>/dev/null | python3 -c ' +import sys,json +rs=json.load(sys.stdin)["spec"]["rules"] +m={r["host"]:r["http"]["paths"][0] for r in rs if r.get("host")} +ok=all(h in m and m[h]["pathType"]=="Prefix" and m[h]["path"]=="/" for h in ("classic.r700.io","next.r700.io")) +ok=ok and m["classic.r700.io"]["backend"]["service"]["name"]=="web-classic" and m["next.r700.io"]["backend"]["service"]["name"]=="web-next" +sys.exit(0 if ok else 1)'"""), + C("TLS covers both hosts with classic-tls", 2, r""" +$K -n project-r700 get ingress multi -o json 2>/dev/null | python3 -c ' +import sys,json +t=json.load(sys.stdin)["spec"].get("tls",[]) +hosts={h for e in t for h in e.get("hosts",[])}; secs={e.get("secretName") for e in t} +sys.exit(0 if {"classic.r700.io","next.r700.io"}<=hosts and secs=={"classic-tls"} else 1)'"""), + C("defaultBackend points at web-classic:80, class is nginx", 3, r""" +i="$K -n project-r700 get ingress multi" +eq "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" nginx \ + && eq "$($i -o jsonpath='{.spec.defaultBackend.service.name}')" web-classic && eq "$($i -o jsonpath='{.spec.defaultBackend.service.port.number}')" 80"""), + ], + solution=""" +openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=classic.r700.io" -keyout t.key -out t.crt +k -n project-r700 create secret tls classic-tls --cert=t.crt --key=t.key +cat < "$d/edge-cache.yaml" < is Running with nginx:1.27", 5, r""" +for i in $(seq 1 12); do + [ "$(j -n default pod "edge-cache-$W3" '{.status.phase}')" = Running ] && [ "$(j -n default pod "edge-cache-$W3" '{.spec.containers[0].image}')" = nginx:1.27 ] && exit 0 + sleep 5 +done; exit 1"""), + C("the hostPath the manifest references now exists on the node", 3, r""" +p=$(nx "$W3" 'grep -A2 hostPath /etc/kubernetes/manifests/edge-cache.yaml /etc/kubernetes/static/edge-cache.yaml 2>/dev/null | grep "path:" | head -1' | awk '{print $2}') +[ -n "$p" ] && nx "$W3" "test -d $p" """), + C("reason.txt mentions the missing host directory", 2, r"""grep -Eqi 'director|hostpath|/opt/edge-cache|not exist|missing' "$WS/q314/reason.txt" 2>/dev/null"""), + ], + solution=""" +k get pod -A | grep edge-cache # nothing +{SHELL_W3} +journalctl -u kubelet -n 40 --no-pager | grep -i edge-cache +# ... failed to start ... hostPath type check failed: /opt/edge-cache/conf is not a directory +grep staticPodPath /var/lib/kubelet/config.yaml +cat /edge-cache.yaml # hostPath /opt/edge-cache/conf with type: Directory +mkdir -p /opt/edge-cache/conf # type: Directory means the kubelet will NOT create it +exit +k get pod edge-cache-{W3} +echo 'hostPath /opt/edge-cache/conf did not exist and type was Directory' > {WS}/q314/reason.txt +# The alternative fix is type: DirectoryOrCreate, but the task asks for the pod to run as written. +""", +) + +# ------------------------------------------------------------------ 315 ---- +task( + id=315, domain="STOR", title="Keep the data when the claim goes", + task=""" +In Namespace `project-orbit`, Pod `keeper` writes to PVC `keeper-claim`, which was provisioned +dynamically by the default StorageClass — whose reclaim policy is `Delete`. + +The data must survive the claim being deleted: + +1. Without recreating anything, make sure the bound PersistentVolume is **retained** when its claim + goes away. +2. Write the PV's name to `{WS}/q315/pv.txt`. +3. Delete Pod `keeper` and PVC `keeper-claim`. +4. The PV must still exist afterwards, in phase `Released`. +""", + setup=NS + r""" +mkns project-orbit +rm -rf "$WS/q315"; mkdir -p "$WS/q315" +$K -n project-orbit delete pod keeper --ignore-not-found --wait=true >/dev/null +$K -n project-orbit delete pvc keeper-claim --ignore-not-found >/dev/null +$K apply -f - >/dev/null <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: {name: keeper-claim, namespace: project-orbit} +spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 200Mi}}} +--- +apiVersion: v1 +kind: Pod +metadata: {name: keeper, namespace: project-orbit} +spec: + volumes: [{name: d, persistentVolumeClaim: {claimName: keeper-claim}}] + containers: [{name: c, image: busybox:1.36, command: [sh, -c, "echo important > /data/keep.txt; sleep 86400"], volumeMounts: [{name: d, mountPath: /data}]}] +EOF +for i in $(seq 1 30); do [ "$(j -n project-orbit pod keeper '{.status.phase}')" = Running ] && break; sleep 2; done +""", + checks=[ + C("pv.txt names the PV that was bound to keeper-claim", 3, r""" +p=$(tr -d '[:space:]' < "$WS/q315/pv.txt" 2>/dev/null); [ -n "$p" ] || exit 1 +eq "$(j pv "$p" '{.spec.claimRef.name}')" keeper-claim"""), + C("that PV has reclaim policy Retain", 4, r"""eq "$(j pv "$(tr -d '[:space:]' < "$WS/q315/pv.txt")" '{.spec.persistentVolumeReclaimPolicy}')" Retain"""), + C("Pod and PVC are gone", 2, r"""! $K -n project-orbit get pod keeper >/dev/null 2>&1 && ! $K -n project-orbit get pvc keeper-claim >/dev/null 2>&1"""), + C("the PV still exists and is Released", 3, r""" +p=$(tr -d '[:space:]' < "$WS/q315/pv.txt"); for i in $(seq 1 10); do [ "$(j pv "$p" '{.status.phase}')" = Released ] && exit 0; sleep 3; done; exit 1"""), + ], + solution=""" +k -n project-orbit get pvc keeper-claim -o jsonpath='{.spec.volumeName}' | tee {WS}/q315/pv.txt +PV=$(cat {WS}/q315/pv.txt) +k patch pv $PV -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +k get pv $PV # RECLAIM POLICY: Retain +k -n project-orbit delete pod keeper --now +k -n project-orbit delete pvc keeper-claim +k get pv $PV # STATUS: Released - data still on disk +# Patch the PV, not the StorageClass: the class only sets the policy for volumes created later. +# To reuse it, clear .spec.claimRef and it becomes Available again. +""", +) + +# ------------------------------------------------------------------ 316 ---- +task( + id=316, domain="ARCH", title="Plan a control-plane upgrade", + task=""" +The control-plane node must be upgraded to the next patch release. + +1. Write the cluster's current control-plane version to `{WS}/q316/current.txt` (as `kubectl` reports it). +2. Run the kubeadm command that shows what an upgrade would do, and save its output to + `{WS}/q316/plan.txt`. +3. Write to `{WS}/q316/steps.sh` the ordered commands you would run to upgrade the control-plane + node — package install of kubeadm, the kubeadm upgrade step, draining, upgrading kubelet and + kubectl, restarting the kubelet, and uncordoning — one command per line, in the order they must + be executed. Do **not** execute them. +""", + setup=r"""rm -rf "$WS/q316"; mkdir -p "$WS/q316" """, + checks=[ + C("current.txt matches the server version", 2, r""" +v=$($K version -o json 2>/dev/null | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])') +grep -qF "$v" "$WS/q316/current.txt" 2>/dev/null"""), + C("plan.txt is real kubeadm upgrade plan output", 3, r"""grep -Eqi 'upgrade|component|version' "$WS/q316/plan.txt" 2>/dev/null && [ "$(wc -l < "$WS/q316/plan.txt")" -ge 5 ]"""), + C("steps.sh has the commands in the correct order", 5, r""" +f="$WS/q316/steps.sh"; [ -f "$f" ] || exit 1 +grep -q 'kubeadm upgrade apply' "$f" || exit 1 +grep -Eq '(apt|yum|dnf).*(install).*kubeadm' "$f" || exit 1 +grep -q 'drain' "$f" && grep -q 'uncordon' "$f" || exit 1 +grep -Eq '(apt|yum|dnf).*(install).*kubelet' "$f" || exit 1 +grep -Eq 'systemctl (restart|daemon-reload)' "$f" || exit 1 +n=$(grep -n 'kubeadm upgrade apply' "$f" | head -1 | cut -d: -f1) +ka=$(grep -nE '(apt|yum|dnf).*install.*kubeadm' "$f" | head -1 | cut -d: -f1) +kl=$(grep -nE '(apt|yum|dnf).*install.*kubelet' "$f" | head -1 | cut -d: -f1) +dr=$(grep -n 'drain' "$f" | head -1 | cut -d: -f1) +un=$(grep -n 'uncordon' "$f" | head -1 | cut -d: -f1) +[ "$ka" -lt "$n" ] && [ "$n" -lt "$kl" ] && [ "$dr" -lt "$kl" ] && [ "$kl" -lt "$un" ]"""), + ], + solution=""" +k version -o json | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])' > {WS}/q316/current.txt +{SHELL_CP} +kubeadm upgrade plan | tee /tmp/plan.txt # copy into {WS}/q316/plan.txt +cat > {WS}/q316/steps.sh <<'EOF' +apt-get update && apt-get install -y kubeadm=1.34.1-1.1 +kubeadm upgrade plan +kubeadm upgrade apply v1.34.1 +kubectl drain --ignore-daemonsets +apt-get install -y kubelet=1.34.1-1.1 kubectl=1.34.1-1.1 +systemctl daemon-reload +systemctl restart kubelet +kubectl uncordon +EOF +# Order matters and is the point of this task: kubeadm binary first, then `kubeadm upgrade apply` +# (it upgrades the static pod control plane), then drain, then kubelet/kubectl packages, restart, +# uncordon. Workers use `kubeadm upgrade node` instead of `upgrade apply`. +""", +) + +# ------------------------------------------------------------------ 317 ---- +task( + id=317, domain="TSHOOT", title="Certificates stop being issued", breaks="controller-manager", + task=""" +Certificate signing requests are approved but never get a certificate: `kubectl get csr` shows +them `Approved` with an empty `CONDITION`-issued certificate forever. + +Find the responsible control-plane component, repair it on the control-plane node, and prove it +works: a CertificateSigningRequest named `issue-probe` (CN `probe`, signer +`kubernetes.io/kube-apiserver-client`, usage `client auth`) that you create and approve must +receive a certificate. +""", + setup=r""" +$K delete csr issue-probe --ignore-not-found >/dev/null 2>&1 +nx "$CP" ' +f=/etc/kubernetes/manifests/kube-controller-manager.yaml +grep -q "cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt" $f && sed -i "s|--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt|--cluster-signing-cert-file=/etc/kubernetes/pki/ca-signing.crt|" $f +' +""", + checks=[ + C("kube-controller-manager points at a signing certificate that exists", 4, r""" +p=$(nx "$CP" 'grep -o "\-\-cluster-signing-cert-file=[^ ]*" /etc/kubernetes/manifests/kube-controller-manager.yaml' | cut -d= -f2) +[ -n "$p" ] && nx "$CP" "test -f $p" """), + C("kube-controller-manager pod is Running", 3, r""" +for i in $(seq 1 12); do [ "$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].status.phase}')" = Running ] && exit 0; sleep 5; done; exit 1"""), + C("an approved CSR now receives a certificate", 5, r""" +$K get csr issue-probe >/dev/null 2>&1 || exit 1 +$K get csr issue-probe -o jsonpath='{.status.conditions[*].type}' | grep -q Approved || exit 1 +for i in $(seq 1 15); do [ -n "$(j csr issue-probe '{.status.certificate}')" ] && exit 0; sleep 4; done; exit 1"""), + ], + solution=""" +k get csr # Approved but never Issued +k -n kube-system logs kube-controller-manager-{CP} | tail -20 +# ... unable to load client CA file .../ca-signing.crt: no such file or directory +{SHELL_CP} +grep signing /etc/kubernetes/manifests/kube-controller-manager.yaml +ls /etc/kubernetes/pki/ca.crt # the real file +vi /etc/kubernetes/manifests/kube-controller-manager.yaml +# --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt +exit +openssl genrsa -out probe.key 2048 +openssl req -new -key probe.key -subj "/CN=probe" -out probe.csr +cat </dev/null; } +""" + +def C(name, pts, script): + return {"name": name, "pts": pts, "script": script} + +def ns_pick(r, base): + """Namespace name with a random suffix so the same task never lands on the same namespace twice.""" + return f"{base}-{r.choice(['alpha', 'bravo', 'delta', 'echo', 'kilo', 'nova', 'omega', 'zulu'])}" + +def name_pick(r, base): + return f"{base}-{r.randint(10, 99)}" diff --git a/tracks/cka/questions.py b/tracks/cka/questions.py new file mode 100644 index 0000000..aca4e4d --- /dev/null +++ b/tracks/cka/questions.py @@ -0,0 +1,652 @@ +""" +CKA track for the exam simulator. + +TRACK - metadata (name, duration, pass mark, domain table) +QUESTIONS - list of tasks. Each task: + id, domain, title, task (markdown shown to candidate), + setup : bash that seeds the scenario + checks : list of (name, points, bash) - each check is graded independently + (exit 0 = pass). Points earned = sum of passing checks, so partial + credit works the way it does on the real exam. + disruptive: True => setup runs last, after everything else is scheduled. + +Scripts get a prelude: $K (kubectl), $WS (workspace), $CP/$W1/$W2/$W3 (node +names), nx NODE 'cmd' (run a command on a node). {WS}, {CP}, {W2}, {W3}, +{SHELL_CP} etc. are substituted in task text. +""" + +TRACK = { + "id": "cka", + "name": "Certified Kubernetes Administrator", + "minutes": 120, + "pass_pct": 66, + # tasks per domain drawn into a session (weights follow the exam blueprint) + "quota": {"ARCH": 5, "WORK": 4, "NET": 4, "STOR": 2, "TSHOOT": 5}, # 20 tasks + "quota_hard": {"ARCH": 6, "WORK": 5, "NET": 5, "STOR": 3, "TSHOOT": 6}, # 25 tasks, hard ones preferred + # fixed simulator sets (./exam sim a|b) - excluded from random sampling + "sets": {"a": "Simulator Set A", "b": "Simulator Set B"}, + "domains": { + "ARCH": "Cluster Architecture, Installation & Configuration (25%)", + "WORK": "Workloads & Scheduling (15%)", + "NET": "Services & Networking (20%)", + "STOR": "Storage (10%)", + "TSHOOT": "Troubleshooting (30%)", + }, +} + +from tracks.cka.common import C, NS + +QUESTIONS = [ +# ================================================================ ARCH ==== +{ + "id": 1, "domain": "ARCH", "title": "RBAC for a deployment bot", + "task": """ +Namespace `rbac-app` already exists. + +1. Create a ServiceAccount `deploy-bot` in namespace `rbac-app`. +2. Create a Role `deploy-manager` in `rbac-app` that grants + get, list, watch, create, update, patch, delete on **deployments** (apps API group) only. +3. Bind it to the ServiceAccount with a RoleBinding named `deploy-bot-binding`. + +The ServiceAccount must NOT be able to delete pods, and must have no access +outside the `rbac-app` namespace. Verify with `kubectl auth can-i`. +""", + "setup": NS + r""" +mkns rbac-app +$K -n rbac-app delete sa deploy-bot --ignore-not-found >/dev/null +$K -n rbac-app delete role deploy-manager --ignore-not-found >/dev/null +$K -n rbac-app delete rolebinding deploy-bot-binding --ignore-not-found >/dev/null +""", + "checks": [ + C("ServiceAccount rbac-app/deploy-bot exists", 1, r"""$K -n rbac-app get sa deploy-bot >/dev/null 2>&1"""), + C("Role deploy-manager and RoleBinding deploy-bot-binding exist", 1, + r"""$K -n rbac-app get role deploy-manager >/dev/null 2>&1 && $K -n rbac-app get rolebinding deploy-bot-binding >/dev/null 2>&1"""), + C("deploy-bot can get/list/watch/create/update/patch/delete deployments", 2, r""" +for v in get list watch create update patch delete; do + [ "$($K auth can-i $v deployments -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot 2>/dev/null)" = yes ] || exit 1 +done"""), + C("deploy-bot cannot delete pods", 1, r"""[ "$($K auth can-i delete pods -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot 2>/dev/null || true)" = no ]"""), + C("deploy-bot has no access in namespace default", 1, r"""[ "$($K auth can-i create deployments -n default --as=system:serviceaccount:rbac-app:deploy-bot 2>/dev/null || true)" = no ]"""), + ], +}, +{ + "id": 2, "domain": "ARCH", "title": "etcd snapshot", + "task": """ +Create a snapshot of the cluster's etcd database and save it on the +**control-plane node** at `/opt/cka/etcd-snapshot.db`. + +Details you need: +- etcd listens on `https://127.0.0.1:2379` +- certificates are under `/etc/kubernetes/pki/etcd/` (`ca.crt`, `server.crt`, `server.key`) +- "ssh to the control-plane node" is: `{SHELL_CP}` +- Setup tried to place `etcdctl` at `/usr/local/bin/etcdctl` on that node. + If it is missing, run etcdctl inside the etcd pod + (`kubectl -n kube-system exec etcd-{CP} -- etcdctl ...`) and copy the file to the node afterwards. + +Verify the snapshot with `etcdutl snapshot status` (or `etcdctl snapshot status`). +""", + "setup": r""" +nx "$CP" 'mkdir -p /opt/cka; rm -f /opt/cka/etcd-snapshot.db' +POD=$($K -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].metadata.name}') +for bin in etcdctl etcdutl; do + if $K -n kube-system exec "$POD" -- cat /usr/local/bin/$bin > /tmp/cka-$bin 2>/dev/null && [ -s /tmp/cka-$bin ]; then + nx "$CP" "cat > /usr/local/bin/$bin && chmod +x /usr/local/bin/$bin" < /tmp/cka-$bin + fi + rm -f /tmp/cka-$bin +done +""", + "checks": [ + C("/opt/cka/etcd-snapshot.db exists on the control-plane node", 3, r"""nx "$CP" 'test -f /opt/cka/etcd-snapshot.db'"""), + C("snapshot is a real etcd backup (> 100 KB)", 4, r"""[ "$(nx "$CP" 'stat -c %s /opt/cka/etcd-snapshot.db' 2>/dev/null || echo 0)" -gt 100000 ]"""), + ], +}, +{ + "id": 3, "domain": "ARCH", "title": "Kustomize overlay", + "task": """ +A Kustomize base lives in `{WS}/q03/base` (Deployment `store-api` + Service `store-api`). + +Create a production overlay at `{WS}/q03/overlays/prod` that: +- deploys everything into namespace `store-prod` (create the Namespace as part of the overlay), +- adds the common label `env=prod` to all resources, +- sets the `store-api` Deployment to **3** replicas. + +Apply it with `kubectl apply -k`. +""", + "setup": r""" +rm -rf "$WS/q03"; mkdir -p "$WS/q03/base" +cat > "$WS/q03/base/kustomization.yaml" <<'EOF' +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - deployment.yaml + - service.yaml +EOF +cat > "$WS/q03/base/deployment.yaml" <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: store-api +spec: + replicas: 1 + selector: + matchLabels: + app: store-api + template: + metadata: + labels: + app: store-api + spec: + containers: + - name: api + image: nginx:1.25 + ports: + - containerPort: 80 +EOF +cat > "$WS/q03/base/service.yaml" <<'EOF' +apiVersion: v1 +kind: Service +metadata: + name: store-api +spec: + selector: + app: store-api + ports: + - port: 80 + targetPort: 80 +EOF +$K delete ns store-prod --ignore-not-found --wait=false >/dev/null +""", + "checks": [ + C("overlay kustomization.yaml exists", 1, r"""[ -f "$WS/q03/overlays/prod/kustomization.yaml" ]"""), + C("resources deployed into namespace store-prod", 1, r"""$K -n store-prod get deploy store-api >/dev/null 2>&1 && $K -n store-prod get svc store-api >/dev/null 2>&1"""), + C("store-api has 3 replicas", 1, r"""[ "$($K -n store-prod get deploy store-api -o jsonpath='{.spec.replicas}')" = 3 ]"""), + C("Deployment and Service carry label env=prod", 2, r"""[ "$($K -n store-prod get deploy store-api -o jsonpath='{.metadata.labels.env}')" = prod ] && [ "$($K -n store-prod get svc store-api -o jsonpath='{.metadata.labels.env}')" = prod ]"""), + ], +}, +{ + "id": 4, "domain": "ARCH", "title": "CustomResourceDefinition", + "task": """ +1. Create a CRD with: + - group `data.example.com`, version `v1` (served + storage) + - kind `Backup`, plural `backups`, singular `backup`, shortName `bk` + - scope `Namespaced` + - an OpenAPI v3 schema where `spec` has string fields `schedule` and `target` +2. Create a `Backup` named `nightly` in namespace `default` with + `spec.schedule: "0 2 * * *"` and `spec.target: "s3://backups/prod"`. + +`kubectl get bk` must list it. +""", + "setup": r"""$K delete crd backups.data.example.com --ignore-not-found >/dev/null""", + "checks": [ + C("CRD backups.data.example.com exists and is Namespaced", 1, r"""[ "$($K get crd backups.data.example.com -o jsonpath='{.spec.scope}' 2>/dev/null)" = Namespaced ]"""), + C("shortName bk works", 1, r"""$K get crd backups.data.example.com -o jsonpath='{.spec.names.shortNames}' | grep -q '"bk"'"""), + C("Backup nightly has the right schedule", 1, r"""[ "$($K -n default get bk nightly -o jsonpath='{.spec.schedule}' 2>/dev/null)" = "0 2 * * *" ]"""), + C("Backup nightly has the right target", 1, r"""[ "$($K -n default get bk nightly -o jsonpath='{.spec.target}' 2>/dev/null)" = "s3://backups/prod" ]"""), + ], +}, +{ + "id": 5, "domain": "ARCH", "title": "Static pod", + "task": """ +Create a **static pod** named `static-web` running image `nginx:1.27` +on node `{W3}` (namespace `default`). + +You must place the manifest in the kubelet's static pod path on that node. +Reach the node with: `{SHELL_W3}` +(check `/var/lib/kubelet/config.yaml` for `staticPodPath`). + +The mirror pod `static-web-{W3}` must reach Running. +""", + "setup": r"""nx "$W3" 'rm -f /etc/kubernetes/manifests/static-web*.yaml' || true""", + "checks": [ + C("manifest containing static-web is in /etc/kubernetes/manifests on the node", 2, r"""nx "$W3" 'grep -ls static-web /etc/kubernetes/manifests/*.yaml' >/dev/null 2>&1"""), + C("mirror pod static-web- is Running", 2, r"""[ "$($K -n default get pod "static-web-$W3" -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ]"""), + C("image is nginx:1.27", 1, r"""[ "$($K -n default get pod "static-web-$W3" -o jsonpath='{.spec.containers[0].image}' 2>/dev/null)" = nginx:1.27 ]"""), + ], +}, +{ + "id": 6, "domain": "ARCH", "title": "Certificates: expiry check and user CSR", + "task": """ +Part A — On the control-plane node (`{SHELL_CP}`) determine the expiry date of the +**kube-apiserver** serving certificate (use `kubeadm certs check-expiration` or `openssl`). +Write the expiry line to `{WS}/q06/apiserver-expiry.txt` on your machine. + +Part B — Onboard a new user `jane`: +1. Generate a private key and a CSR with CN=jane (openssl). +2. Create a Kubernetes `CertificateSigningRequest` named `jane` + (signer `kubernetes.io/kube-apiserver-client`, usage `client auth`). +3. Approve it so a certificate is issued. +4. Create a RoleBinding `jane-view` in namespace `default` giving user `jane` + the built-in ClusterRole `view`. +""", + "setup": r""" +rm -rf "$WS/q06"; mkdir -p "$WS/q06" +$K delete csr jane --ignore-not-found >/dev/null +$K -n default delete rolebinding jane-view --ignore-not-found >/dev/null +""", + "checks": [ + C("apiserver-expiry.txt records the real expiry month and year", 2, r""" +[ -s "$WS/q06/apiserver-expiry.txt" ] || exit 1 +exp=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate' | sed 's/notAfter=//') +yr=$(echo "$exp" | awk '{print $4}'); mon=$(echo "$exp" | awk '{print $1}') +grep -q "$yr" "$WS/q06/apiserver-expiry.txt" && grep -qi "$mon" "$WS/q06/apiserver-expiry.txt" """), + C("CSR jane exists with signer kubernetes.io/kube-apiserver-client", 1, r"""[ "$($K get csr jane -o jsonpath='{.spec.signerName}' 2>/dev/null)" = kubernetes.io/kube-apiserver-client ]"""), + C("CSR jane approved and certificate issued", 2, r"""$K get csr jane -o jsonpath='{.status.conditions[*].type}' 2>/dev/null | grep -q Approved && [ -n "$($K get csr jane -o jsonpath='{.status.certificate}')" ]"""), + C("jane can list pods in default (view) but not delete them", 1, r"""[ "$($K auth can-i list pods -n default --as=jane 2>/dev/null || true)" = yes ] && [ "$($K auth can-i delete pods -n default --as=jane 2>/dev/null || true)" = no ]"""), + ], +}, +# ================================================================ WORK ==== +{ + "id": 7, "domain": "WORK", "title": "Rolling update & scaling", + "task": """ +Deployment `web-frontend` in namespace `apps` runs `nginx:1.25` with 2 replicas. + +- Update the image to `nginx:1.27`. +- Scale to **4** replicas. +- Configure the rolling update strategy: `maxSurge: 1`, `maxUnavailable: 0`. +- All 4 replicas must be available when you are done. +""", + "setup": NS + r""" +mkns apps +$K -n apps delete deploy web-frontend --ignore-not-found >/dev/null +$K apply -f - >/dev/null <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: web-frontend + namespace: apps +spec: + replicas: 2 + selector: + matchLabels: {app: web-frontend} + template: + metadata: + labels: {app: web-frontend} + spec: + containers: + - name: nginx + image: nginx:1.25 + resources: + requests: {cpu: 50m, memory: 32Mi} +EOF +""", + "checks": [ + C("image is nginx:1.27", 1, r"""[ "$($K -n apps get deploy web-frontend -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null)" = nginx:1.27 ]"""), + C("replicas = 4", 1, r"""[ "$($K -n apps get deploy web-frontend -o jsonpath='{.spec.replicas}' 2>/dev/null)" = 4 ]"""), + C("strategy maxSurge=1 maxUnavailable=0", 1, r"""[ "$($K -n apps get deploy web-frontend -o jsonpath='{.spec.strategy.rollingUpdate.maxSurge}')" = 1 ] && [ "$($K -n apps get deploy web-frontend -o jsonpath='{.spec.strategy.rollingUpdate.maxUnavailable}')" = 0 ]"""), + C("all 4 replicas available", 1, r"""[ "$($K -n apps get deploy web-frontend -o jsonpath='{.status.availableReplicas}' 2>/dev/null)" = 4 ]"""), + ], +}, +{ + "id": 8, "domain": "WORK", "title": "Multi-container pod with shared volume", + "task": """ +Create a pod `log-shipper` in namespace `apps` with two containers: + +- `app` (image `busybox:1.36`): appends the current date to `/var/log/app.log` every 2 seconds. +- `shipper` (image `busybox:1.36`): runs `tail -f /var/log/app.log`. + +Both containers share an `emptyDir` volume named `logs` mounted at `/var/log`. +`kubectl logs log-shipper -c shipper` must show the dates written by `app`. +""", + "setup": NS + r""" +mkns apps +$K -n apps delete pod log-shipper --ignore-not-found --wait=false >/dev/null +""", + "checks": [ + C("pod log-shipper Running with containers app and shipper", 1, r""" +p="$K -n apps get pod log-shipper" +[ "$($p -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] || exit 1 +n=$($p -o jsonpath='{.spec.containers[*].name}'); echo "$n" | grep -qw app && echo "$n" | grep -qw shipper"""), + C("volume logs is an emptyDir mounted at /var/log in both containers", 2, r""" +p="$K -n apps get pod log-shipper" +[ -n "$($p -o jsonpath='{.spec.volumes[?(@.name=="logs")].emptyDir}' 2>/dev/null)" ] || exit 1 +[ "$($p -o jsonpath='{.spec.containers[?(@.name=="app")].volumeMounts[?(@.name=="logs")].mountPath}')" = /var/log ] || exit 1 +[ "$($p -o jsonpath='{.spec.containers[?(@.name=="shipper")].volumeMounts[?(@.name=="logs")].mountPath}')" = /var/log ]"""), + C("shipper's logs show lines written by app", 1, r"""sleep 3; [ -n "$($K -n apps logs log-shipper -c shipper --tail=2 2>/dev/null)" ]"""), + ], +}, +{ + "id": 9, "domain": "WORK", "title": "Scheduling with node affinity and resources", + "task": """ +One node in the cluster is labelled `disktype=ssd`. + +Create a Deployment `cache` in namespace `apps`, 2 replicas, image `redis:7-alpine`, that: +- uses **required** node affinity (`requiredDuringSchedulingIgnoredDuringExecution`) + so pods only run on nodes with `disktype=ssd` (do not use nodeSelector), +- sets resource requests `cpu: 100m`, `memory: 64Mi` and limits `cpu: 200m`, `memory: 128Mi`. + +Both pods must be Running. +""", + "setup": NS + r""" +mkns apps +$K label node "$W3" disktype=ssd --overwrite >/dev/null +$K label node "$W1" disktype- >/dev/null 2>&1 || true +$K label node "$W2" disktype- >/dev/null 2>&1 || true +$K -n apps delete deploy cache --ignore-not-found >/dev/null +""", + "checks": [ + C("deployment cache with 2 replicas", 1, r"""[ "$($K -n apps get deploy cache -o jsonpath='{.spec.replicas}' 2>/dev/null)" = 2 ]"""), + C("required node affinity set, no nodeSelector", 1, r"""d="$K -n apps get deploy cache"; [ -n "$($d -o jsonpath='{.spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution}' 2>/dev/null)" ] && [ -z "$($d -o jsonpath='{.spec.template.spec.nodeSelector}')" ]"""), + C("requests 100m/64Mi and limits 200m/128Mi", 1, r""" +c='{.spec.template.spec.containers[0].resources'; d="$K -n apps get deploy cache" +[ "$($d -o jsonpath="$c.requests.cpu}")" = 100m ] && [ "$($d -o jsonpath="$c.requests.memory}")" = 64Mi ] && [ "$($d -o jsonpath="$c.limits.cpu}")" = 200m ] && [ "$($d -o jsonpath="$c.limits.memory}")" = 128Mi ]"""), + C("both pods available and on the ssd node", 1, r""" +[ "$($K -n apps get deploy cache -o jsonpath='{.status.availableReplicas}')" = 2 ] || exit 1 +for n in $($K -n apps get pod -l app=cache -o jsonpath='{.items[*].spec.nodeName}'); do [ "$n" = "$W3" ] || exit 1; done"""), + ], +}, +{ + "id": 10, "domain": "WORK", "title": "ConfigMap and Secret consumption", + "task": """ +In namespace `apps`: +1. Create ConfigMap `app-config` with keys `APP_MODE=production` and `LOG_LEVEL=debug`. +2. Create Secret `db-creds` with keys `username=admin` and `password=S3cr3t!`. +3. Create pod `config-pod` (image `busybox:1.36`, command `sleep 3600`) that: + - exposes `APP_MODE` as an env var sourced from the ConfigMap key, + - exposes **all** keys of `db-creds` as env vars (envFrom), + - mounts the ConfigMap as a volume at `/etc/config`. +""", + "setup": NS + r""" +mkns apps +$K -n apps delete pod config-pod --ignore-not-found --wait=false >/dev/null +$K -n apps delete cm app-config --ignore-not-found >/dev/null +$K -n apps delete secret db-creds --ignore-not-found >/dev/null +""", + "checks": [ + C("ConfigMap app-config and Secret db-creds with correct values", 1, r"""[ "$($K -n apps get cm app-config -o jsonpath='{.data.LOG_LEVEL}' 2>/dev/null)" = debug ] && [ "$($K -n apps get secret db-creds -o jsonpath='{.data.password}' 2>/dev/null | base64 -d)" = 'S3cr3t!' ]"""), + C("APP_MODE injected via configMapKeyRef", 1, r"""[ "$($K -n apps get pod config-pod -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && [ -n "$($K -n apps get pod config-pod -o jsonpath='{.spec.containers[0].env[?(@.name=="APP_MODE")].valueFrom.configMapKeyRef}')" ] && $K -n apps exec config-pod -- env 2>/dev/null | grep -q '^APP_MODE=production$'"""), + C("all db-creds keys injected via envFrom", 1, r"""[ -n "$($K -n apps get pod config-pod -o jsonpath='{.spec.containers[0].envFrom[*].secretRef}' 2>/dev/null)" ] && $K -n apps exec config-pod -- env 2>/dev/null | grep -q '^password=S3cr3t!$'"""), + C("ConfigMap mounted at /etc/config", 1, r"""[ "$($K -n apps exec config-pod -- cat /etc/config/LOG_LEVEL 2>/dev/null)" = debug ]"""), + ], +}, +# ================================================================= NET ==== +{ + "id": 11, "domain": "NET", "title": "NetworkPolicy", + "task": """ +Namespace `netpol` has deployments `frontend`, `backend` and `db` +(labels `app=frontend|backend|db`). + +Create a NetworkPolicy `backend-policy` in `netpol` so that pods with `app=backend` +- accept ingress **only** from pods labelled `app=frontend` in the same namespace, + on TCP port **8080**, +- reject all other ingress. + +Do not restrict egress. (The default kind CNI does not enforce policies; +you are graded on the policy spec.) +""", + "setup": NS + r""" +mkns netpol +for a in frontend backend db; do + $K -n netpol create deploy $a --image=nginx:1.25 --dry-run=client -o yaml | $K apply -f - >/dev/null +done +$K -n netpol delete netpol backend-policy --ignore-not-found >/dev/null +""", + "checks": [ + C("policy selects app=backend, policyTypes Ingress only", 2, r"""n="$K -n netpol get netpol backend-policy"; [ "$($n -o jsonpath='{.spec.podSelector.matchLabels.app}' 2>/dev/null)" = backend ] && $n -o jsonpath='{.spec.policyTypes}' | grep -q Ingress && ! $n -o jsonpath='{.spec.policyTypes}' | grep -q Egress"""), + C("single ingress rule from podSelector app=frontend only", 2, r"""n="$K -n netpol get netpol backend-policy"; [ "$($n -o jsonpath='{.spec.ingress[0].from[0].podSelector.matchLabels.app}' 2>/dev/null)" = frontend ] && [ -z "$($n -o jsonpath='{.spec.ingress[1]}')" ] && [ -z "$($n -o jsonpath='{.spec.ingress[0].from[1]}')" ]"""), + C("allowed port is TCP 8080", 2, r"""n="$K -n netpol get netpol backend-policy"; [ "$($n -o jsonpath='{.spec.ingress[0].ports[0].port}' 2>/dev/null)" = 8080 ] && { p=$($n -o jsonpath='{.spec.ingress[0].ports[0].protocol}'); [ -z "$p" ] || [ "$p" = TCP ]; }"""), + ], +}, +{ + "id": 12, "domain": "NET", "title": "NodePort Service", + "task": """ +Deployment `shop` (namespace `svc`) serves HTTP on container port 80. + +Expose it with a Service `shop-svc` of type **NodePort**: +port 80, targetPort 80, nodePort **30080**. +The Service must have ready endpoints. +""", + "setup": NS + r""" +mkns svc +$K -n svc create deploy shop --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n svc delete svc shop-svc --ignore-not-found >/dev/null +""", + "checks": [ + C("Service shop-svc is NodePort, port 80 -> targetPort 80", 1, r"""s="$K -n svc get svc shop-svc"; [ "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" = NodePort ] && [ "$($s -o jsonpath='{.spec.ports[0].port}')" = 80 ] && [ "$($s -o jsonpath='{.spec.ports[0].targetPort}')" = 80 ]"""), + C("nodePort is 30080", 1, r"""[ "$($K -n svc get svc shop-svc -o jsonpath='{.spec.ports[0].nodePort}' 2>/dev/null)" = 30080 ]"""), + C("Service has endpoints", 2, r"""[ -n "$($K -n svc get endpointslices -l kubernetes.io/service-name=shop-svc -o jsonpath='{.items[*].endpoints[*].addresses[*]}' 2>/dev/null)" ]"""), + ], +}, +{ + "id": 13, "domain": "NET", "title": "Ingress routing", + "task": """ +Namespace `ingress-app` has Services `web` (port 80) and `api` (port 8080). + +Create an Ingress `app-ingress` in `ingress-app`: +- `ingressClassName: nginx` +- host `shop.example.com` +- path `/api` (Prefix) -> service `api` port 8080 +- path `/` (Prefix) -> service `web` port 80 + +(No ingress controller is installed; you are graded on the object.) +""", + "setup": NS + r""" +mkns ingress-app +$K -n ingress-app create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n ingress-app create deploy api --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n ingress-app expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n ingress-app expose deploy api --port=8080 --target-port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n ingress-app delete ingress app-ingress --ignore-not-found >/dev/null +""", + "checks": [ + C("ingressClassName nginx and host shop.example.com", 1, r"""i="$K -n ingress-app get ingress app-ingress"; [ "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" = nginx ] && [ "$($i -o jsonpath='{.spec.rules[0].host}')" = shop.example.com ]"""), + C("/api (Prefix) -> api:8080", 2, r"""i="$K -n ingress-app get ingress app-ingress"; P='{.spec.rules[0].http.paths[?(@.path=="/api")]'; [ "$($i -o jsonpath="$P.backend.service.name}" 2>/dev/null)" = api ] && [ "$($i -o jsonpath="$P.backend.service.port.number}")" = 8080 ] && [ "$($i -o jsonpath="$P.pathType}")" = Prefix ]"""), + C("/ (Prefix) -> web:80", 2, r"""i="$K -n ingress-app get ingress app-ingress"; P='{.spec.rules[0].http.paths[?(@.path=="/")]'; [ "$($i -o jsonpath="$P.backend.service.name}" 2>/dev/null)" = web ] && [ "$($i -o jsonpath="$P.backend.service.port.number}")" = 80 ] && [ "$($i -o jsonpath="$P.pathType}")" = Prefix ]"""), + ], +}, +{ + "id": 14, "domain": "NET", "title": "Cluster DNS", + "task": """ +Service `hello` exists in namespace `dns-check`. + +1. Run a pod `dns-test` (image `busybox:1.36`) in namespace `dns-check`. +2. From inside it, resolve the Service using its **fully-qualified** DNS name. +3. Save the FQDN you used (one line) to `{WS}/q14/svc-fqdn.txt`. +4. Save the full `nslookup` output to `{WS}/q14/nslookup.txt`. +""", + "setup": NS + r""" +mkns dns-check +rm -rf "$WS/q14"; mkdir -p "$WS/q14" +$K -n dns-check create deploy hello --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n dns-check expose deploy hello --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n dns-check delete pod dns-test --ignore-not-found --wait=false >/dev/null +""", + "checks": [ + C("pod dns-test exists in dns-check", 1, r"""$K -n dns-check get pod dns-test >/dev/null 2>&1"""), + C("svc-fqdn.txt contains hello.dns-check.svc.cluster.local", 2, r"""grep -qx 'hello.dns-check.svc.cluster.local\.\?' "$WS/q14/svc-fqdn.txt" 2>/dev/null"""), + C("nslookup.txt contains the service ClusterIP", 1, r"""ip=$($K -n dns-check get svc hello -o jsonpath='{.spec.clusterIP}'); grep -q "$ip" "$WS/q14/nslookup.txt" 2>/dev/null"""), + ], +}, +# ================================================================ STOR ==== +{ + "id": 15, "domain": "STOR", "title": "PersistentVolume, claim and pod", + "task": """ +1. Create a PersistentVolume `pv-data`: 1Gi, `hostPath` `/mnt/data`, + accessMode `ReadWriteOnce`, `storageClassName: manual`, reclaim policy `Retain`. +2. Create a PersistentVolumeClaim `pvc-data` in namespace `storage` requesting 500Mi + with `storageClassName: manual` so it binds to `pv-data`. +3. Create pod `data-pod` (image `nginx:1.25`) in `storage` mounting the claim at `/usr/share/data`. +""", + "setup": NS + r""" +mkns storage +$K -n storage delete pod data-pod --ignore-not-found --wait=false >/dev/null +$K -n storage delete pvc pvc-data --ignore-not-found >/dev/null +$K delete pv pv-data --ignore-not-found >/dev/null +""", + "checks": [ + C("PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", 2, r""" +pv="$K get pv pv-data" +[ "$($pv -o jsonpath='{.spec.capacity.storage}' 2>/dev/null)" = 1Gi ] && [ "$($pv -o jsonpath='{.spec.hostPath.path}')" = /mnt/data ] && [ "$($pv -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')" = Retain ] && [ "$($pv -o jsonpath='{.spec.storageClassName}')" = manual ] && $pv -o jsonpath='{.spec.accessModes}' | grep -q ReadWriteOnce"""), + C("PVC pvc-data Bound to pv-data", 2, r"""[ "$($K -n storage get pvc pvc-data -o jsonpath='{.status.phase}' 2>/dev/null)" = Bound ] && [ "$($K -n storage get pvc pvc-data -o jsonpath='{.spec.volumeName}')" = pv-data ]"""), + C("data-pod Running with claim mounted at /usr/share/data", 2, r"""[ "$($K -n storage get pod data-pod -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && [ -n "$($K -n storage get pod data-pod -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim.claimName=="pvc-data")].name}')" ] && $K -n storage get pod data-pod -o jsonpath='{.spec.containers[0].volumeMounts[*].mountPath}' | grep -q /usr/share/data"""), + ], +}, +{ + "id": 16, "domain": "STOR", "title": "Default StorageClass and dynamic provisioning", + "task": """ +1. Create a StorageClass `fast-local`: + provisioner `rancher.io/local-path`, `volumeBindingMode: WaitForFirstConsumer`, + `reclaimPolicy: Delete`. +2. Make `fast-local` the **default** StorageClass and ensure the existing + `standard` class is no longer default. +3. Create PVC `fast-claim` (200Mi, RWO) in namespace `storage` **without** specifying a storageClassName. +4. Create pod `fast-pod` (image `nginx:1.25`) in `storage` that mounts `fast-claim` at `/data`. + The claim must end up Bound. +""", + "setup": NS + r""" +mkns storage +$K -n storage delete pod fast-pod --ignore-not-found --wait=false >/dev/null +$K -n storage delete pvc fast-claim --ignore-not-found >/dev/null +$K delete sc fast-local --ignore-not-found >/dev/null +$K annotate sc standard storageclass.kubernetes.io/is-default-class=true --overwrite >/dev/null +""", + "checks": [ + C("StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", 1, r"""sc="$K get sc fast-local"; [ "$($sc -o jsonpath='{.provisioner}' 2>/dev/null)" = rancher.io/local-path ] && [ "$($sc -o jsonpath='{.volumeBindingMode}')" = WaitForFirstConsumer ] && [ "$($sc -o jsonpath='{.reclaimPolicy}')" = Delete ]"""), + C("fast-local is default and standard is not", 2, r"""[ "$($K get sc fast-local -o jsonpath='{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}' 2>/dev/null)" = true ] && [ "$($K get sc standard -o jsonpath='{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}')" != true ]"""), + C("fast-claim picked up fast-local and is Bound", 1, r"""[ "$($K -n storage get pvc fast-claim -o jsonpath='{.spec.storageClassName}' 2>/dev/null)" = fast-local ] && [ "$($K -n storage get pvc fast-claim -o jsonpath='{.status.phase}')" = Bound ]"""), + C("fast-pod Running with claim at /data", 1, r"""[ "$($K -n storage get pod fast-pod -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && $K -n storage get pod fast-pod -o jsonpath='{.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /data"""), + ], +}, +# ============================================================== TSHOOT ==== +{ + "id": 17, "domain": "TSHOOT", "breaks": "w2-kubelet", "title": "Node NotReady", "disruptive": True, + "task": """ +Node `{W2}` is `NotReady`. + +Investigate and fix it so the node returns to `Ready`. The fix must survive a +reboot of the node (i.e. the responsible service must be enabled). + +Reach the node with: `{SHELL_W2}` +""", + "setup": r"""nx "$W2" 'systemctl disable kubelet >/dev/null 2>&1; systemctl stop kubelet'""", + "checks": [ + C("node is Ready", 4, r"""[ "$($K get node "$W2" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null)" = True ]"""), + C("kubelet is active", 2, r"""[ "$(nx "$W2" 'systemctl is-active kubelet')" = active ]"""), + C("kubelet is enabled (survives reboot)", 2, r"""[ "$(nx "$W2" 'systemctl is-enabled kubelet')" = enabled ]"""), + ], +}, +{ + "id": 18, "domain": "TSHOOT", "title": "Broken deployment and service", + "task": """ +The `payments` application in namespace `troubleshoot` is down. +Users report that the Service `payments` returns no backends and the pods are not healthy. + +Fix everything necessary so that: +- the `payments` Deployment has all replicas Available (the app is a plain nginx web server), and +- the `payments` Service has endpoints. + +Do not delete the Deployment or the Service; fix them in place. +""", + "setup": NS + r""" +mkns troubleshoot +$K apply -f - >/dev/null <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: payments + namespace: troubleshoot +spec: + replicas: 2 + selector: + matchLabels: {app: payments} + template: + metadata: + labels: {app: payments} + spec: + containers: + - name: payments + image: nginx:1.99-nope + ports: [{containerPort: 80}] +--- +apiVersion: v1 +kind: Service +metadata: + name: payments + namespace: troubleshoot +spec: + selector: + app: payment + ports: + - port: 80 + targetPort: 8080 +EOF +""", + "checks": [ + C("all payments replicas available", 3, r"""d="$K -n troubleshoot get deploy payments"; r=$($d -o jsonpath='{.spec.replicas}' 2>/dev/null); a=$($d -o jsonpath='{.status.availableReplicas}'); [ -n "$a" ] && [ "$a" = "$r" ]"""), + C("Service payments has endpoints (selector fixed)", 2, r"""[ -n "$($K -n troubleshoot get endpointslices -l kubernetes.io/service-name=payments -o jsonpath='{.items[*].endpoints[*].addresses[*]}' 2>/dev/null)" ]"""), + C("Service targetPort matches container port 80", 2, r"""[ "$($K -n troubleshoot get svc payments -o jsonpath='{.spec.ports[0].targetPort}' 2>/dev/null)" = 80 ]"""), + ], +}, +{ + "id": 19, "domain": "TSHOOT", "title": "CrashLoopBackOff diagnosis", + "task": """ +Pod `crasher` in namespace `troubleshoot` keeps restarting. + +1. Find out why. Save the pod's (current or previous) container logs to + `{WS}/q19/crasher.log`. +2. Fix the pod so it stays Running. The application must keep the same name, + namespace, image and command; it only needs the configuration it is complaining about + (use the value `db.internal` if a value is required). +""", + "setup": NS + r""" +mkns troubleshoot +rm -rf "$WS/q19"; mkdir -p "$WS/q19" +$K -n troubleshoot delete pod crasher --ignore-not-found --wait=true >/dev/null +$K apply -f - >/dev/null </dev/null"""), + C("pod crasher Running with DB_HOST set", 3, r"""[ "$($K -n troubleshoot get pod crasher -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ] && [ -n "$($K -n troubleshoot exec crasher -- sh -c 'echo $DB_HOST' 2>/dev/null)" ]"""), + C("image unchanged (busybox:1.36)", 1, r"""[ "$($K -n troubleshoot get pod crasher -o jsonpath='{.spec.containers[0].image}' 2>/dev/null)" = busybox:1.36 ]"""), + ], +}, +{ + "id": 20, "domain": "TSHOOT", "breaks": "scheduler", "title": "Control plane component failure", "disruptive": True, + "task": """ +New pods in the cluster are stuck in `Pending` and are never assigned to a node. + +Find the failing control-plane component and repair it on the control-plane node +(`{SHELL_CP}`). Afterwards, newly created pods must be scheduled normally. +""", + "setup": r""" +nx "$CP" ' +f=/etc/kubernetes/manifests/kube-scheduler.yaml +grep -q "^ *- kube-scheduler$" $f && sed -i "s|^\( *\)- kube-scheduler$|\1- kube-schedulerr|" $f +' || true +""", + "checks": [ + C("kube-scheduler static pod manifest repaired", 3, r"""nx "$CP" 'grep -q "^ *- kube-scheduler$" /etc/kubernetes/manifests/kube-scheduler.yaml'"""), + C("new pods get scheduled", 5, r""" +$K -n default delete pod sched-probe --ignore-not-found --wait=true >/dev/null 2>&1 +$K -n default run sched-probe --image=busybox:1.36 --restart=Never -- sleep 300 >/dev/null 2>&1 || exit 1 +n=""; for i in $(seq 1 30); do n=$($K -n default get pod sched-probe -o jsonpath='{.spec.nodeName}' 2>/dev/null); [ -n "$n" ] && break; sleep 2; done +$K -n default delete pod sched-probe --wait=false >/dev/null 2>&1 +[ -n "$n" ]"""), + ], +}, +] + +# ---- aggregate the banks: 1-20 here, 21-43 in bank_extra, 101-122 (hard) in bank_hard +from tracks.cka import bank_extra, bank_hard, bank_sim_a, bank_sim_b +QUESTIONS = QUESTIONS + bank_extra.QUESTIONS + bank_hard.QUESTIONS + bank_sim_a.QUESTIONS + bank_sim_b.QUESTIONS +QUESTIONS.sort(key=lambda q: q["id"]) +_ids = [q["id"] for q in QUESTIONS] +assert len(_ids) == len(set(_ids)), "duplicate question ids" +from tracks.cka.solutions import S as _SOL +for _q in QUESTIONS: + _q["solution"] = (_q.get("solution") or _SOL.get(_q["id"], "")).strip() + _q.setdefault("set", None) +_missing = [q["id"] for q in QUESTIONS if not q["solution"]] +assert not _missing, f"tasks without a solution: {_missing}" diff --git a/tracks/cka/solutions.py b/tracks/cka/solutions.py new file mode 100644 index 0000000..d968118 --- /dev/null +++ b/tracks/cka/solutions.py @@ -0,0 +1,906 @@ +"""Reference solutions, keyed by task id. Rendered with the same %{param} / {WS} / {W1} substitution +as the task text. Shown only after a task has been graded (drills) or the session submitted (exams). +These are one correct way, written the way you'd type them under time pressure: imperative first, +YAML only where imperative can't express it.""" + +S = {} + +# ================================================================= ARCH ==== +S[1] = """ +k -n rbac-app create sa deploy-bot +k -n rbac-app create role deploy-manager --verb=get,list,watch,create,update,patch,delete --resource=deployments.apps +k -n rbac-app create rolebinding deploy-bot-binding --role=deploy-manager --serviceaccount=rbac-app:deploy-bot +k auth can-i create deployments -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot # yes +k auth can-i delete pods -n rbac-app --as=system:serviceaccount:rbac-app:deploy-bot # no +""" +S[2] = """ +{SHELL_CP} +ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \\ + --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \\ + snapshot save /opt/cka/etcd-snapshot.db +etcdutl snapshot status /opt/cka/etcd-snapshot.db -w table +# If etcdctl is only inside the pod: run the same command via +# kubectl -n kube-system exec etcd-{CP} -- etcdctl ... snapshot save /var/lib/etcd/snap.db +# then on the node: mv /var/lib/etcd/snap.db /opt/cka/etcd-snapshot.db (the pod's /var/lib/etcd is the node's /var/lib/etcd) +""" +S[3] = """ +mkdir -p {WS}/q03/overlays/prod && cd {WS}/q03/overlays/prod +cat > namespace.yaml < kustomization.yaml < /etc/kubernetes/manifests/static-web.yaml < (copy the line into {WS}/q06/apiserver-expiry.txt) +# Part B +openssl genrsa -out jane.key 2048 +openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr +cat < jane.crt +k -n default create rolebinding jane-view --clusterrole=view --user=jane +""" +S[21] = """ +k create ns %{ns} +helm install %{rel} {WS}/q21/chart -n %{ns} --set replicaCount=%{n} +helm upgrade %{rel} {WS}/q21/chart -n %{ns} --reuse-values --set service.type=NodePort +helm -n %{ns} history %{rel} # revision 2 +""" +S[33] = """ +cat < {WS}/q34/path.txt +cat < apiserver= +k get node %{node} -o jsonpath='{.status.nodeInfo.kubeletVersion}{"\\n"}{.status.nodeInfo.containerRuntimeVersion}{"\\n"}' +k -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].spec.containers[0].image}' # tag after the colon +""" +# --------------------------------------------------------------- hard ARCH +S[101] = """ +k get clusterrolebinding -o wide | grep %{sa} # find the cluster-admin binding -> delete it +k delete clusterrolebinding %{sa}-legacy-admin +k -n %{ns} create role %{sa}-role --verb=get,list,watch,patch --resource=deployments.apps,replicasets.apps +k -n %{ns} patch role %{sa}-role --type=json -p='[{"op":"add","path":"/rules/-","value":{"apiGroups":[""],"resources":["pods/exec"],"verbs":["create"]}}]' +k -n %{ns} create rolebinding %{sa}-rb --role=%{sa}-role --serviceaccount=%{ns}:%{sa} +k create clusterrole %{sa}-nodes --verb=list --resource=nodes +k create clusterrolebinding %{sa}-crb --clusterrole=%{sa}-nodes --serviceaccount=%{ns}:%{sa} +k auth can-i --list -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} +""" +S[102] = """ +{SHELL_CP} +etcdutl snapshot restore /opt/cka/backup.db --data-dir /var/lib/etcd-restore +# point the etcd static pod at the new dir: the hostPath volume, NOT just --data-dir +vi /etc/kubernetes/manifests/etcd.yaml # volumes: - hostPath: path: /var/lib/etcd -> /var/lib/etcd-restore +# kubelet restarts etcd; apiserver reconnects in ~30-60s +watch crictl ps # etcd + kube-apiserver come back +kubectl -n default get cm %{cm} # restored +""" +S[103] = """ +k get nodes -l tier=edge # -> the node +{SHELL_W3} +grep staticPodPath /var/lib/kubelet/config.yaml # NOT /etc/kubernetes/manifests here +cat > /%{pod}.yaml < %{user}.crt +k -n %{ns} create rolebinding %{user}-edit --clusterrole=edit --user=%{user} +# self-contained kubeconfig (--embed-certs) +KC={WS}/q104/%{user}.kubeconfig +k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt +SERVER=$(k config view --minify -o jsonpath='{.clusters[0].cluster.server}') +kubectl --kubeconfig $KC config set-cluster lab --server=$SERVER --certificate-authority=ca.crt --embed-certs +kubectl --kubeconfig $KC config set-credentials %{user} --client-certificate=%{user}.crt --client-key=%{user}.key --embed-certs +kubectl --kubeconfig $KC config set-context %{user}@lab --cluster=lab --user=%{user} --namespace=%{ns} +kubectl --kubeconfig $KC config use-context %{user}@lab +kubectl --kubeconfig $KC get pods +""" +S[105] = """ +mkdir -p {WS}/q105/overlays/prod && cd {WS}/q105/overlays/prod +cat > replicas.yaml < envfrom.yaml < kustomization.yaml < web-config- in the envFrom too +""" +S[106] = """ +cat < {WS}/q106/rejected.txt +printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: defaulted}\\nspec: {replicas: 2, image: nginx:1.27}\\n' | k apply -f - +""" +S[122] = """ +{SHELL_CP} +kubeadm certs renew all +# static pods do NOT reload certs by themselves: bounce them +mv /etc/kubernetes/manifests/*.yaml /tmp/ && sleep 20 && mv /tmp/*.yaml /etc/kubernetes/manifests/ +# admin.conf was regenerated by `renew all`; use it: +cp /etc/kubernetes/admin.conf ~/.kube/config +kubeadm certs check-expiration | grep '^apiserver ' # -> {WS}/q122/expiry.txt +""" + +# ================================================================= WORK ==== +S[7] = """ +k -n apps set image deploy web-frontend nginx=nginx:1.27 +k -n apps scale deploy web-frontend --replicas=4 +k -n apps patch deploy web-frontend -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' +k -n apps rollout status deploy web-frontend +""" +S[8] = """ +cat <> /var/log/app.log; sleep 2; done"] + volumeMounts: [{name: logs, mountPath: /var/log}] + - name: shipper + image: busybox:1.36 + command: [sh, -c, "tail -f /var/log/app.log"] + volumeMounts: [{name: logs, mountPath: /var/log}] +EOF +""" +S[9] = """ +k -n apps create deploy cache --image=redis:7-alpine --replicas=2 $do > cache.yaml +# edit: add under spec.template.spec +# affinity: +# nodeAffinity: +# requiredDuringSchedulingIgnoredDuringExecution: +# nodeSelectorTerms: +# - matchExpressions: [{key: disktype, operator: In, values: [ssd]}] +# and under the container: resources: {requests: {cpu: 100m, memory: 64Mi}, limits: {cpu: 200m, memory: 128Mi}} +k apply -f cache.yaml +""" +S[10] = """ +k -n apps create cm app-config --from-literal=APP_MODE=production --from-literal=LOG_LEVEL=debug +k -n apps create secret generic db-creds --from-literal=username=admin --from-literal='password=S3cr3t!' +cat < d.yaml +# add under spec.template.spec: +# nodeSelector: {kubernetes.io/hostname: {W1}} +# tolerations: [{key: %{key}, operator: Equal, value: %{val}, effect: NoSchedule}] +k apply -f d.yaml +""" +S[26] = """ +cat < cj.yaml +# set: spec.concurrencyPolicy: Forbid, successfulJobsHistoryLimit: 3, failedJobsHistoryLimit: 1, +# spec.jobTemplate.spec.activeDeadlineSeconds: 30, ...template.spec.restartPolicy: Never +k apply -f cj.yaml +k -n %{ns} create job %{cj}-manual --from=cronjob/%{cj} +""" +S[28] = """ +cat < /work/index.html"] + volumeMounts: [{name: content, mountPath: /work}] + containers: + - name: web + image: nginx:1.25 + volumeMounts: [{name: content, mountPath: /usr/share/nginx/html}] +EOF +""" +S[29] = """ +k -n %{ns} create cm %{cm}-v2 --from-literal=mode=new +k -n %{ns} patch cm %{cm}-v2 -p '{"immutable":true}' # optional, keeps the pattern +k -n %{ns} set volume deploy %{dep} --add=false 2>/dev/null; \\ +k -n %{ns} patch deploy %{dep} --type=json -p='[{"op":"replace","path":"/spec/template/spec/volumes/0/configMap/name","value":"%{cm}-v2"}]' +k -n %{ns} rollout status deploy %{dep} +""" +S[31] = """ +k -n %{ns} create job %{job} --image=busybox:1.36 $do -- sh -c "echo done; sleep 2" > job.yaml +# spec: completions: %{c}, parallelism: %{p}, backoffLimit: 2 (restartPolicy Never is the default from create job) +k apply -f job.yaml && k -n %{ns} wait --for=condition=complete job/%{job} --timeout=120s +""" +S[32] = """ +k -n %{ns} create quota %{ns}-quota --hard=pods=%{pods},requests.cpu=1,limits.memory=2Gi +cat < d.yaml +# spec.template.spec: +# priorityClassName: %{dep}-prio +# tolerations: [{key: %{key}, operator: Equal, value: %{val}, effect: NoSchedule}] +# affinity: +# podAntiAffinity: +# requiredDuringSchedulingIgnoredDuringExecution: +# - labelSelector: {matchLabels: {app: %{dep}}} +# topologyKey: kubernetes.io/hostname +# container resources: {requests: {cpu: 100m, memory: 64Mi}} +k apply -f d.yaml +""" +S[108] = """ +k -n %{ns} rollout history deploy %{dep} # find the revision whose image is nginx:1.25 +k -n %{ns} rollout history deploy %{dep} --revision=1 # confirm +k -n %{ns} rollout undo deploy %{dep} --to-revision=1 +k -n %{ns} scale deploy %{dep} --replicas=3 && k -n %{ns} rollout status deploy %{dep} +k -n %{ns} rollout pause deploy %{dep} +k -n %{ns} set image deploy %{dep} nginx=nginx:1.27 +k -n %{ns} set env deploy %{dep} RELEASE=next +k -n %{ns} get rs # no new RS while paused +""" +S[109] = """ +cat <> /var/log/agent.log; sleep 1; done"] + volumeMounts: [{name: logs, mountPath: /var/log}] + containers: + - name: app + image: nginx:1.27 + volumeMounts: [{name: logs, mountPath: /var/log}] + readinessProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5} + livenessProbe: {tcpSocket: {port: 80}} +EOF +k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name} {.spec.containers[*].name}' | tr ' ' '\\n' | sort > {WS}/q109/containers.txt +k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' > {WS}/q109/ip.txt +""" +S[110] = """ +k get pods -A -o json | jq -r '.items[] | select(.metadata.namespace!="kube-system") | select(.spec.containers[0].readinessProbe==null) | "\\(.metadata.namespace)/\\(.metadata.name)"' | sort > {WS}/q110/no-readiness.txt +k get pods -A --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\\n"}{end}' | sort | uniq -c | awk '{print $2, $1}' > {WS}/q110/per-node.txt +k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\\n' | sort -u > {WS}/q110/images.txt +# no jq? use: k get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,RP:.spec.containers[0].readinessProbe --no-headers | awk '$3=="" && $1!="kube-system"{print $1"/"$2}' | sort +""" +S[121] = """ +k config use-context k8s-c2-AC +k config view --minify -o jsonpath='{..namespace}' # -> %{ns}, write it to {WS}/q121/ns.txt +k create deploy %{dep} --image=busybox:1.36 --replicas=2 -- sleep 3600 +k label deploy %{dep} owner=ac && k patch deploy %{dep} -p '{"spec":{"template":{"metadata":{"labels":{"owner":"ac"}}}}}' +k expose deploy %{dep} --port=9000 --target-port=9000 --name=%{dep} +""" + +# ================================================================== NET ==== +S[11] = """ +cat < svc.yaml +# set spec.ports[0].nodePort: 30080 +k apply -f svc.yaml && k -n svc get endpointslices +""" +S[13] = """ +k -n ingress-app create ingress app-ingress --class=nginx \\ + --rule="shop.example.com/api*=api:8080" --rule="shop.example.com/*=web:80" +# `*` suffix -> pathType Prefix. Verify: k -n ingress-app get ingress app-ingress -o yaml +""" +S[14] = """ +k -n dns-check run dns-test --image=busybox:1.36 -- sleep 3600 +k -n dns-check exec dns-test -- nslookup hello.dns-check.svc.cluster.local | tee {WS}/q14/nslookup.txt +echo hello.dns-check.svc.cluster.local > {WS}/q14/svc-fqdn.txt +""" +S[22] = """ +k apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml +cat < d.yaml +# container: ports: [{name: %{pname}, containerPort: 80}] +# readinessProbe: {httpGet: {path: /, port: %{pname}}} +k apply -f d.yaml +k -n %{ns} expose deploy %{dep} --port=8080 --target-port=%{pname} +""" +S[42] = """ +cat < h.yaml # fix selector: app: %{sts} +k apply -f h.yaml +k -n %{ns} create svc externalname legacy --external-name=%{ext} +k -n %{ns} exec %{sts}-0 -- nslookup %{sts}-0.%{sts} > {WS}/q43/pod0.txt +""" +# --------------------------------------------------------------- hard NET +S[111] = """ +k -n %{ns} delete netpol temp-allow-all +k label ns %{from} team=%{from} +cat < {WS}/q112/fp.txt +""" +S[113] = """ +cat < scale it +k -n kube-system scale deploy coredns --replicas=2 +k -n kube-system edit cm coredns # forward . 10.255.255.254 -> forward . /etc/resolv.conf +k -n kube-system rollout restart deploy coredns +k run t --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default +""" + +# ================================================================= STOR ==== +S[15] = """ +cat < remove it +k patch pv %{pv} --type=json -p='[{"op":"remove","path":"/spec/claimRef"}]' +k get pv %{pv} # Available +cat < /data/marker.txt"], volumeMounts: [{name: d, mountPath: /data}]}] +EOF +k -n %{ns} wait --for=jsonpath='{.status.phase}'=Succeeded pod/writer +cat < add nodeName +k -n %{ns} wait --for=condition=complete job/migrate --timeout=120s +k -n %{ns} get pod legacy -o yaml > legacy.yaml # change claimName old-data -> new-data, strip status/uid/resourceVersion +k -n %{ns} delete pod legacy --now && k apply -f legacy.yaml +k get pv $(k -n %{ns} get pvc old-data -o jsonpath='{.spec.volumeName}') -o jsonpath='{.spec.persistentVolumeReclaimPolicy}' # Retain +k -n %{ns} delete pvc old-data +""" + +# =============================================================== TSHOOT ==== +S[17] = """ +k get node {W2} # NotReady +{SHELL_W2} +systemctl status kubelet # inactive (dead), disabled +systemctl enable --now kubelet +journalctl -u kubelet -f # until "Successfully registered node" +""" +S[18] = """ +k -n troubleshoot describe deploy payments | grep -i image # nginx:1.99-nope -> ErrImagePull +k -n troubleshoot set image deploy payments payments=nginx:1.27 +k -n troubleshoot get svc payments -o yaml # selector app: payment (typo), targetPort 8080 (container is 80) +k -n troubleshoot patch svc payments -p '{"spec":{"selector":{"app":"payments"},"ports":[{"port":80,"targetPort":80}]}}' +k -n troubleshoot get endpointslices +""" +S[19] = """ +k -n troubleshoot logs crasher --previous | tee {WS}/q19/crasher.log # FATAL: ... DB_HOST is not set +k -n troubleshoot get pod crasher -o yaml > crasher.yaml +# add under the container: env: [{name: DB_HOST, value: db.internal}] (strip status/uid/resourceVersion) +k -n troubleshoot delete pod crasher --now && k apply -f crasher.yaml +""" +S[20] = """ +k -n kube-system get pods # kube-scheduler-... CrashLoopBackOff / Error +k -n kube-system logs kube-scheduler-{CP} # exec: "kube-schedulerr": not found +{SHELL_CP} +vi /etc/kubernetes/manifests/kube-scheduler.yaml # command: - kube-schedulerr -> - kube-scheduler +# kubelet restarts it; k -n kube-system get pods -w +""" +S[37] = """ +k -n kube-system get pods -l k8s-app=kube-dns; k -n kube-system logs -l k8s-app=kube-dns --tail=20 +k -n kube-system get cm coredns -o yaml # forward . 10.255.255.1 -> unreachable upstream +k -n kube-system edit cm coredns # forward . /etc/resolv.conf +k -n kube-system rollout restart deploy coredns +k run t --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default +""" +S[38] = """ +k -n kube-system get pods # kube-controller-manager Error/CrashLoop +k -n kube-system logs kube-controller-manager-{CP} # flag error: lease duration ... invalid +{SHELL_CP} +vi /etc/kubernetes/manifests/kube-controller-manager.yaml # remove the bogus --leader-elect-lease-duration=1s line +k create deploy probe --image=busybox:1.36 -- sleep 60 && k get rs +""" +S[39] = """ +{SHELL_W1} +systemctl status kubelet; journalctl -u kubelet -n 30 --no-pager # "open /etc/kubernetes/pki/ca-bundle.crt: no such file" +grep -n clientCAFile /var/lib/kubelet/config.yaml # -> /etc/kubernetes/pki/ca.crt +systemctl restart kubelet && systemctl enable kubelet +""" +# -------------------------------------------------------------- hard TSHOOT +S[117] = """ +k get node {W2} # NotReady,SchedulingDisabled +{SHELL_W2} +systemctl status kubelet # disabled + dead +journalctl -u kubelet -n 30 --no-pager # after enable: "ca-wrong.crt: no such file" +sed -i 's|ca-wrong.crt|ca.crt|' /var/lib/kubelet/config.yaml +systemctl enable --now kubelet +exit +k uncordon {W2} # the third thing +k run t --image=busybox:1.36 --restart=Never --overrides='{"spec":{"nodeName":"{W2}"}}' -- sleep 60 +""" +S[118] = """ +k -n kube-system get pods # scheduler CrashLoop, controller-manager ImagePullBackOff +{SHELL_CP} +vi /etc/kubernetes/manifests/kube-scheduler.yaml # kube-schedulerr -> kube-scheduler +vi /etc/kubernetes/manifests/kube-controller-manager.yaml # image tag ...-broken -> remove suffix (match the others: crictl images) +k -n kube-system get pods -w +k create deploy probe --image=busybox:1.36 -- sleep 60 && k get deploy probe -w +""" +S[119] = """ +k -n %{ns} get all; k -n %{ns} describe deploy %{dep}; k -n %{ns} describe rs -l app=%{dep} # quota: pods 1 +k -n %{ns} patch quota tight -p '{"spec":{"hard":{"pods":"3"}}}' +k -n %{ns} set image deploy %{dep} web=nginx:1.27 # image tag typo +k -n %{ns} patch secret %{dep}-env -p '{"stringData":{"DATABASE_URL":"postgres://db"}}' # missing key referenced by env +k -n %{ns} rollout restart deploy %{dep} # picks up the secret key +k -n %{ns} patch svc %{dep} -p '{"spec":{"selector":{"app":"%{dep}"},"ports":[{"port":80,"targetPort":80}]}}' +""" +S[120] = """ +k -n kube-system get pod -l component=kube-controller-manager -o yaml | grep cluster-cidr +k -n kube-system get pod -l component=kube-apiserver -o yaml | grep service-cluster-ip-range +k -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}' +k -n kube-system get cm kube-proxy -o yaml | grep '^ *mode:' # "" -> iptables +k get node %{node} -o jsonpath='{.spec.podCIDR}' +""" diff --git a/tracks/ckad/questions.py b/tracks/ckad/questions.py new file mode 100644 index 0000000..b1c3c41 --- /dev/null +++ b/tracks/ckad/questions.py @@ -0,0 +1,563 @@ +"""CKAD track. Starter bank - same format as the CKA banks, extend it the same way. + +CKAD is a build-things exam: less cluster surgery, more "produce the right object, fast". +The clock is the adversary, so these are mostly single-object tasks graded exactingly. +""" +from tracks.cka.common import C, NS, ns_pick, name_pick + +TRACK = { + "id": "ckad", + "name": "Certified Kubernetes Application Developer", + "minutes": 120, + "pass_pct": 66, + "domains": { + "DESIGN": "Application Design and Build (20%)", + "DEPLOY": "Application Deployment (20%)", + "OBS": "Application Observability and Maintenance (15%)", + "ENV": "Application Environment, Configuration and Security (25%)", + "NET": "Services and Networking (20%)", + }, + "quota": {"DESIGN": 3, "DEPLOY": 3, "OBS": 2, "ENV": 2, "NET": 2}, + "quota_hard": {"DESIGN": 3, "DEPLOY": 3, "OBS": 2, "ENV": 2, "NET": 2}, +} + +QUESTIONS = [ +# ---------------------------------------------------------------- DESIGN ---- +{ + "id": 1, "domain": "DESIGN", "title": "Adapter sidecar", + "params": lambda r: {"ns": ns_pick(r, "app"), "pod": name_pick(r, "adapter")}, + "task": """ +Create Pod `%{pod}` in Namespace `%{ns}` with two containers sharing an `emptyDir` named `spool` +mounted at `/spool` in both: + +- `writer` (busybox:1.36): every 2 seconds appends a line `metric 1` to `/spool/raw.txt` +- `adapter` (busybox:1.36): runs `tail -f /spool/raw.txt` + +`kubectl logs %{pod} -c adapter` must show the writer's lines. +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""", + "checks": [ + C("pod Running with containers writer and adapter", 2, r""" +p="$K -n %{ns} get pod %{pod}" +eq "$($p -o jsonpath='{.status.phase}' 2>/dev/null)" Running && $p -o jsonpath='{.spec.containers[*].name}' | grep -qw writer && $p -o jsonpath='{.spec.containers[*].name}' | grep -qw adapter"""), + C("emptyDir 'spool' mounted at /spool in both containers", 2, r""" +p="$K -n %{ns} get pod %{pod}" +[ -n "$($p -o jsonpath='{.spec.volumes[?(@.name=="spool")].emptyDir}')" ] \ + && eq "$($p -o jsonpath='{.spec.containers[?(@.name=="writer")].volumeMounts[?(@.name=="spool")].mountPath}')" /spool \ + && eq "$($p -o jsonpath='{.spec.containers[?(@.name=="adapter")].volumeMounts[?(@.name=="spool")].mountPath}')" /spool"""), + C("adapter logs show the writer's output", 2, r"""sleep 3; $K -n %{ns} logs %{pod} -c adapter --tail=3 2>/dev/null | grep -q 'metric 1'"""), + ], + "solution": """ +cat <> /spool/raw.txt; sleep 2; done"] + volumeMounts: [{name: spool, mountPath: /spool}] + - name: adapter + image: busybox:1.36 + command: [sh, -c, "tail -f /spool/raw.txt"] + volumeMounts: [{name: spool, mountPath: /spool}] +EOF +k -n %{ns} logs %{pod} -c adapter +""", +}, +{ + "id": 2, "domain": "DESIGN", "title": "Init container waits for a dependency", + "params": lambda r: {"ns": ns_pick(r, "boot"), "pod": name_pick(r, "consumer")}, + "task": """ +Namespace `%{ns}` has a Service `backend` with no endpoints yet. + +1. Create Pod `%{pod}` with an init container `wait-for-backend` (busybox:1.36) that loops until + the DNS name `backend` resolves, and a main container `app` (nginx:1.27). + It will sit in `Init:0/1`. +2. Then create Deployment `backend` (nginx:1.27, 1 replica, Pod label `app=backend`) so the + Service gets an endpoint and your Pod starts. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null +$K -n %{ns} delete deploy backend --ignore-not-found >/dev/null +$K apply -f - >/dev/null </dev/null""", + "checks": [ + C("schedule and history limits", 2, r""" +c="$K -n %{ns} get cj %{cj}" +eq "$($c -o jsonpath='{.spec.schedule}' 2>/dev/null)" "*/%{m} * * * *" && eq "$($c -o jsonpath='{.spec.successfulJobsHistoryLimit}')" 2 && eq "$($c -o jsonpath='{.spec.failedJobsHistoryLimit}')" 2"""), + C("startingDeadlineSeconds 10 and activeDeadlineSeconds 20", 3, r""" +c="$K -n %{ns} get cj %{cj}" +eq "$($c -o jsonpath='{.spec.startingDeadlineSeconds}' 2>/dev/null)" 10 && eq "$($c -o jsonpath='{.spec.jobTemplate.spec.activeDeadlineSeconds}')" 20"""), + C("restartPolicy Never", 1, r"""eq "$(j -n %{ns} cj %{cj} '{.spec.jobTemplate.spec.template.spec.restartPolicy}')" Never"""), + ], + "solution": """ +k -n %{ns} create cj %{cj} --image=busybox:1.36 --schedule='*/%{m} * * * *' $do -- sh -c "echo digest" > cj.yaml +# spec.startingDeadlineSeconds: 10 (skip a late start) +# spec.successfulJobsHistoryLimit: 2 / failedJobsHistoryLimit: 2 +# spec.jobTemplate.spec.activeDeadlineSeconds: 20 (kill an overrunning run) +# spec.jobTemplate.spec.template.spec.restartPolicy: Never +k apply -f cj.yaml +# startingDeadlineSeconds lives on the CronJob, activeDeadlineSeconds on the Job template. +""", +}, +# ---------------------------------------------------------------- DEPLOY ---- +{ + "id": 4, "domain": "DEPLOY", "title": "Roll forward, then back", + "params": lambda r: {"ns": ns_pick(r, "rel"), "dep": name_pick(r, "frontend")}, + "task": """ +Deployment `%{dep}` in `%{ns}` runs `nginx:1.25` with 3 replicas. + +1. Update the image to `nginx:1.27` and record a change-cause annotation of `pick 1.27`. +2. Then update it to `nginx:9.99-nope`, which will never become ready. +3. Roll back so all 3 replicas are Available on `nginx:1.27` again. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} delete deploy %{dep} --ignore-not-found --wait=true >/dev/null +$K -n %{ns} create deploy %{dep} --image=nginx:1.25 --replicas=3 >/dev/null +$K -n %{ns} rollout status deploy %{dep} --timeout=120s >/dev/null 2>&1 +""", + "checks": [ + C("running image is nginx:1.27 with 3 available", 4, r""" +eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.containers[0].image}')" nginx:1.27 && eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3"""), + C("rollout history has at least three revisions", 2, r"""[ "$($K -n %{ns} rollout history deploy %{dep} 2>/dev/null | grep -c '^[0-9]')" -ge 3 ]"""), + C("a change-cause annotation is recorded", 2, r"""j -n %{ns} deploy %{dep} '{.metadata.annotations.kubernetes\.io/change-cause}' | grep -q ."""), + ], + "solution": """ +k -n %{ns} set image deploy %{dep} nginx=nginx:1.27 +k -n %{ns} annotate deploy %{dep} kubernetes.io/change-cause='pick 1.27' --overwrite +k -n %{ns} rollout status deploy %{dep} +k -n %{ns} set image deploy %{dep} nginx=nginx:9.99-nope +k -n %{ns} rollout status deploy %{dep} --timeout=30s # never completes +k -n %{ns} rollout history deploy %{dep} +k -n %{ns} rollout undo deploy %{dep} +k -n %{ns} rollout status deploy %{dep} +""", +}, +{ + "id": 5, "domain": "DEPLOY", "title": "Blue/green cutover", + "params": lambda r: {"ns": ns_pick(r, "bg"), "svc": name_pick(r, "shop")}, + "task": """ +Namespace `%{ns}` has Deployment `%{svc}-blue` (Pod labels `app=%{svc}`, `version=blue`) behind +Service `%{svc}`. + +1. Create Deployment `%{svc}-green`: 2 replicas, image `nginx:1.27`, Pod labels `app=%{svc}` and + `version=green`. +2. When it is fully Available, switch Service `%{svc}` to serve green only. +3. Leave blue running untouched so the switch can be reversed. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} delete deploy %{svc}-blue %{svc}-green --ignore-not-found >/dev/null +$K -n %{ns} delete svc %{svc} --ignore-not-found >/dev/null +$K apply -f - >/dev/null < green.yaml +# rename to %{svc}-green, change every version: blue -> green, image -> nginx:1.27, +# strip status/uid/resourceVersion/creationTimestamp +k apply -f green.yaml +k -n %{ns} rollout status deploy %{svc}-green +k -n %{ns} patch svc %{svc} -p '{"spec":{"selector":{"app":"%{svc}","version":"green"}}}' +k -n %{ns} get endpointslices -l kubernetes.io/service-name=%{svc} +# Rolling back is the same patch with version: blue - which is why blue stays up. +""", +}, +{ + "id": 6, "domain": "DEPLOY", "title": "Helm release lifecycle", + "params": lambda r: {"ns": ns_pick(r, "helm"), "rel": name_pick(r, "site")}, + "task": """ +A chart is at `{WS}/q6/chart`. + +1. Install it into Namespace `%{ns}` (create it) as release `%{rel}` with `replicaCount=2`. +2. Upgrade the release to `replicaCount=4`. +3. Roll the release back so the Deployment is at 2 replicas again — without uninstalling it. +4. Write the release's current revision number to `{WS}/q6/revision.txt`. +""", + "setup": r""" +rm -rf "$WS/q6"; mkdir -p "$WS/q6/chart/templates" +cat > "$WS/q6/chart/Chart.yaml" <<'EOF' +apiVersion: v2 +name: site +version: 0.1.0 +EOF +cat > "$WS/q6/chart/values.yaml" <<'EOF' +replicaCount: 1 +image: nginx:1.27 +EOF +cat > "$WS/q6/chart/templates/deploy.yaml" <<'EOF' +apiVersion: apps/v1 +kind: Deployment +metadata: {name: {{ .Release.Name }}-site} +spec: + replicas: {{ .Values.replicaCount }} + selector: {matchLabels: {app: {{ .Release.Name }}-site}} + template: + metadata: {labels: {app: {{ .Release.Name }}-site}} + spec: {containers: [{name: web, image: "{{ .Values.image }}"}]} +EOF +$K create ns %{ns} --dry-run=client -o yaml | $K apply -f - >/dev/null +helm -n %{ns} uninstall %{rel} >/dev/null 2>&1 || true +command -v helm >/dev/null || echo "warn: helm not installed" +""", + "checks": [ + C("release deployed with at least three revisions of history", 3, r""" +helm -n %{ns} status %{rel} -o json 2>/dev/null | python3 -c 'import sys,json;sys.exit(0 if json.load(sys.stdin)["info"]["status"]=="deployed" else 1)' || exit 1 +[ "$(helm -n %{ns} history %{rel} -o json 2>/dev/null | python3 -c 'import sys,json;print(len(json.load(sys.stdin)))')" -ge 3 ]"""), + C("Deployment is back to 2 replicas", 3, r"""eq "$(j -n %{ns} deploy %{rel}-site '{.spec.replicas}')" 2"""), + C("revision.txt matches the release's current revision", 2, r""" +cur=$(helm -n %{ns} history %{rel} -o json 2>/dev/null | python3 -c 'import sys,json;print(max(r["revision"] for r in json.load(sys.stdin)))' 2>/dev/null) +[ -n "$cur" ] && eq "$(tr -dc '0-9' < "$WS/q6/revision.txt" 2>/dev/null)" "$cur" """), + ], + "solution": """ +k create ns %{ns} +helm -n %{ns} install %{rel} {WS}/q6/chart --set replicaCount=2 +helm -n %{ns} upgrade %{rel} {WS}/q6/chart --set replicaCount=4 +helm -n %{ns} history %{rel} +helm -n %{ns} rollback %{rel} 1 # creates a NEW revision identical to revision 1 +helm -n %{ns} history %{rel} | grep '^[0-9]' | tail -1 | awk '{print $1}' > {WS}/q6/revision.txt +""", +}, +# ------------------------------------------------------------------- OBS ---- +{ + "id": 7, "domain": "OBS", "title": "Three kinds of probe", + "params": lambda r: {"ns": ns_pick(r, "probe"), "dep": name_pick(r, "api")}, + "task": """ +Deployment `%{dep}` in `%{ns}` (nginx:1.27, 2 replicas) needs health checks: + +- **startup**: HTTP GET `/` on port 80, `periodSeconds: 5`, allowed a total of 60 seconds to come up +- **readiness**: HTTP GET `/` on port 80, every 5 seconds, a single failure removes it from endpoints +- **liveness**: TCP on port 80, every 10 seconds, initial delay 15 seconds + +All replicas must end up Ready. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null +$K -n %{ns} create deploy %{dep} --image=nginx:1.27 --replicas=2 >/dev/null +""", + "checks": [ + C("startupProbe: httpGet / :80, period 5, failureThreshold 12", 3, r""" +c='{.spec.template.spec.containers[0].startupProbe' +eq "$(j -n %{ns} deploy %{dep} "$c.httpGet.path}")" / && eq "$(j -n %{ns} deploy %{dep} "$c.periodSeconds}")" 5 && eq "$(j -n %{ns} deploy %{dep} "$c.failureThreshold}")" 12"""), + C("readinessProbe: httpGet, period 5, failureThreshold 1", 3, r""" +c='{.spec.template.spec.containers[0].readinessProbe' +eq "$(j -n %{ns} deploy %{dep} "$c.httpGet.path}")" / && eq "$(j -n %{ns} deploy %{dep} "$c.periodSeconds}")" 5 && eq "$(j -n %{ns} deploy %{dep} "$c.failureThreshold}")" 1"""), + C("livenessProbe: tcpSocket 80, period 10, initialDelay 15", 3, r""" +c='{.spec.template.spec.containers[0].livenessProbe' +eq "$(j -n %{ns} deploy %{dep} "$c.tcpSocket.port}")" 80 && eq "$(j -n %{ns} deploy %{dep} "$c.periodSeconds}")" 10 && eq "$(j -n %{ns} deploy %{dep} "$c.initialDelaySeconds}")" 15"""), + C("all replicas Ready", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.readyReplicas}')" 2"""), + ], + "solution": """ +k -n %{ns} edit deploy %{dep} # add to the container: +# startupProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5, failureThreshold: 12} +# readinessProbe: {httpGet: {path: /, port: 80}, periodSeconds: 5, failureThreshold: 1} +# livenessProbe: {tcpSocket: {port: 80}, periodSeconds: 10, initialDelaySeconds: 15} +k -n %{ns} rollout status deploy %{dep} +# 12 x 5s = 60s. While a startupProbe runs, liveness and readiness are suspended, which is why +# it beats a long initialDelaySeconds on liveness for slow starters. +""", +}, +{ + "id": 8, "domain": "OBS", "title": "Diagnose and fix a broken Pod", + "params": lambda r: {"ns": ns_pick(r, "debug"), "pod": name_pick(r, "worker")}, + "task": """ +Pod `%{pod}` in `%{ns}` is not running. + +1. Write the container's waiting **reason**, as reported in the Pod's status, to `{WS}/q8/reason.txt`. +2. Fix the Pod so it runs. Keep the name, Namespace and command — correct only what is wrong. +""", + "setup": NS + r""" +mkns %{ns}; rm -rf "$WS/q8"; mkdir -p "$WS/q8" +$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=true >/dev/null +$K apply -f - >/dev/null </dev/null"""), + C("pod Running with its original command", 4, r""" +eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && j -n %{ns} pod %{pod} '{.spec.containers[0].command}' | grep -q sleep"""), + ], + "solution": """ +k -n %{ns} get pod %{pod} +k -n %{ns} get pod %{pod} -o jsonpath='{.status.containerStatuses[0].state.waiting.reason}' +# ImagePullBackOff -> {WS}/q8/reason.txt +k -n %{ns} describe pod %{pod} | tail -15 # "Failed to pull image ... not found" +k -n %{ns} get pod %{pod} -o yaml > fix.yaml # set image: busybox:1.36, strip status/uid/resourceVersion +k -n %{ns} delete pod %{pod} --now && k apply -f fix.yaml +# A Pod's image can't be changed in place, so delete + recreate is the fix (a Deployment could +# have been patched with `set image`). +""", +}, +# ------------------------------------------------------------------- ENV ---- +{ + "id": 9, "domain": "ENV", "title": "SecurityContext and capabilities", + "params": lambda r: {"ns": ns_pick(r, "sec"), "pod": name_pick(r, "hardened"), "uid": r.choice([1001, 2000, 10001])}, + "task": """ +Create Pod `%{pod}` in `%{ns}` (image `busybox:1.36`, command `sleep 86400`) that: + +- runs as user %{uid} and group %{uid}, and must not run as root, +- has a read-only root filesystem, +- drops **all** capabilities and adds back only `NET_BIND_SERVICE`, +- cannot gain new privileges. +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""", + "checks": [ + C("runs as UID %{uid} with runAsNonRoot", 3, r""" +p="$K -n %{ns} get pod %{pod}" +u=$($p -o jsonpath='{.spec.securityContext.runAsUser}' 2>/dev/null); [ -z "$u" ] && u=$($p -o jsonpath='{.spec.containers[0].securityContext.runAsUser}') +eq "$u" %{uid} && $p -o jsonpath='{.spec.securityContext.runAsNonRoot}{.spec.containers[0].securityContext.runAsNonRoot}' | grep -q true"""), + C("readOnlyRootFilesystem and allowPrivilegeEscalation=false", 3, r""" +c='{.spec.containers[0].securityContext' +eq "$(j -n %{ns} pod %{pod} "$c.readOnlyRootFilesystem}")" true && eq "$(j -n %{ns} pod %{pod} "$c.allowPrivilegeEscalation}")" false"""), + C("capabilities drop ALL, add NET_BIND_SERVICE", 3, r""" +c='{.spec.containers[0].securityContext.capabilities' +j -n %{ns} pod %{pod} "$c.drop[*]}" | grep -q ALL && j -n %{ns} pod %{pod} "$c.add[*]}" | grep -q NET_BIND_SERVICE"""), + C("pod Running as that UID", 2, r""" +eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && $K -n %{ns} exec %{pod} -- id 2>/dev/null | grep -q 'uid=%{uid}'"""), + ], + "solution": """ +cat </dev/null +$K -n %{ns} delete cm app-cfg --ignore-not-found >/dev/null; $K -n %{ns} delete secret app-sec --ignore-not-found >/dev/null +""", + "checks": [ + C("ConfigMap and Secret hold the right values", 2, r""" +eq "$(j -n %{ns} cm app-cfg '{.data.mode}')" live && eq "$(j -n %{ns} secret app-sec '{.data.apikey}' | base64 -d)" xyz789"""), + C("one projected volume 'bundle' with all three sources", 3, r""" +s=$(j -n %{ns} pod %{pod} '{.spec.volumes[?(@.name=="bundle")].projected.sources}') +echo "$s" | grep -q configMap && echo "$s" | grep -q secret && echo "$s" | grep -q serviceAccountToken"""), + C("token source: audience vault, expiry 3600", 3, r""" +s=$(j -n %{ns} pod %{pod} '{.spec.volumes[?(@.name=="bundle")].projected.sources[*].serviceAccountToken}') +echo "$s" | grep -q '"audience":"vault"' && echo "$s" | grep -q '"expirationSeconds":3600'"""), + C("all three files readable at /etc/bundle", 3, r""" +eq "$($K -n %{ns} exec %{pod} -- cat /etc/bundle/mode 2>/dev/null)" live \ + && eq "$($K -n %{ns} exec %{pod} -- cat /etc/bundle/apikey 2>/dev/null)" xyz789 \ + && [ -n "$($K -n %{ns} exec %{pod} -- cat /etc/bundle/token 2>/dev/null)" ]"""), + ], + "solution": """ +k -n %{ns} create cm app-cfg --from-literal=mode=live +k -n %{ns} create secret generic app-sec --from-literal=apikey=xyz789 +cat </dev/null +$K -n %{ns} delete svc %{dep} %{dep}-ext --ignore-not-found >/dev/null +$K -n %{ns} delete ingress %{dep} --ignore-not-found >/dev/null +$K -n %{ns} create deploy %{dep} --image=nginx:1.27 --replicas=2 --port=80 >/dev/null +""", + "checks": [ + C("ClusterIP Service 8080 -> 80 with endpoints", 3, r""" +s="$K -n %{ns} get svc %{dep}" +eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" ClusterIP && eq "$($s -o jsonpath='{.spec.ports[0].port}')" 8080 && eq "$($s -o jsonpath='{.spec.ports[0].targetPort}')" 80 \ + && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[*].addresses[*]}')" ]"""), + C("NodePort Service on %{np}", 2, r""" +s="$K -n %{ns} get svc %{dep}-ext" +eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" NodePort && eq "$($s -o jsonpath='{.spec.ports[0].nodePort}')" %{np}"""), + C("Ingress routes the host to the ClusterIP Service on 8080", 3, r""" +i="$K -n %{ns} get ingress %{dep}" +eq "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" nginx && eq "$($i -o jsonpath='{.spec.rules[0].host}')" %{dep}.example.com \ + && eq "$($i -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}')" %{dep} \ + && eq "$($i -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.port.number}')" 8080"""), + ], + "solution": """ +k -n %{ns} expose deploy %{dep} --port=8080 --target-port=80 +k -n %{ns} expose deploy %{dep} --name=%{dep}-ext --type=NodePort --port=80 --target-port=80 $do > ext.yaml +# set spec.ports[0].nodePort: %{np} +k apply -f ext.yaml +k -n %{ns} create ingress %{dep} --class=nginx --rule="%{dep}.example.com/*=%{dep}:8080" +""", +}, +{ + "id": 12, "domain": "NET", "title": "Restrict a namespace with NetworkPolicy", + "params": lambda r: {"ns": ns_pick(r, "np"), "client": r.choice(["frontend", "gateway", "portal"])}, + "task": """ +Namespace `%{ns}` runs Pods labelled `app=db` and `app=%{client}`. + +Create NetworkPolicy `db-allow` so `app=db` Pods accept ingress **only** from `app=%{client}` +Pods in the same Namespace on TCP 5432, and nothing else. Do not restrict egress. + +(kindnet doesn't enforce policies here, so you are graded on the object.) +""", + "setup": NS + r""" +mkns %{ns} +for a in db %{client}; do + $K -n %{ns} create deploy $a --image=nginx:1.27 --dry-run=client -o yaml | $K apply -f - >/dev/null + $K -n %{ns} patch deploy $a -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app\":\"$a\"}}}}}" >/dev/null +done +$K -n %{ns} delete netpol db-allow --ignore-not-found >/dev/null +""", + "checks": [ + C("selects app=db, Ingress only", 2, r""" +n="$K -n %{ns} get netpol db-allow" +eq "$($n -o jsonpath='{.spec.podSelector.matchLabels.app}' 2>/dev/null)" db && eq "$($n -o jsonpath='{.spec.policyTypes}')" '["Ingress"]'"""), + C("exactly one rule: from app=%{client} on TCP 5432", 4, r""" +$K -n %{ns} get netpol db-allow -o json 2>/dev/null | python3 -c ' +import sys,json +i=json.load(sys.stdin)["spec"].get("ingress",[]) +ok=len(i)==1 and len(i[0].get("from",[]))==1 and i[0]["from"][0].get("podSelector",{}).get("matchLabels",{}).get("app")=="%{client}" \ + and len(i[0].get("ports",[]))==1 and str(i[0]["ports"][0]["port"])=="5432" +sys.exit(0 if ok else 1)'"""), + ], + "solution": """ +cat </dev/null + $K -n %{ns} patch deploy $a -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app\":\"$a\"}}}}}" >/dev/null +done +$K -n %{ns} delete netpol --all >/dev/null 2>&1 +""", + "checks": [ + C("default-deny covers all pods, both directions, with no allow rules", 3, r""" +n="$K -n %{ns} get netpol default-deny" +eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' && $n -o jsonpath='{.spec.policyTypes}' | grep -q Ingress \ + && $n -o jsonpath='{.spec.policyTypes}' | grep -q Egress && [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ]"""), + C("allow-dns permits UDP and TCP 53 to kube-system", 3, r""" +$K -n %{ns} get netpol allow-dns -o json 2>/dev/null | python3 -c ' +import sys,json +s=json.load(sys.stdin)["spec"] +ok=any({p.get("protocol") for p in r.get("ports",[]) if str(p.get("port"))=="53"}>={"UDP","TCP"} + and r.get("to") and all("kube-system" in json.dumps(t.get("namespaceSelector",{})) for t in r["to"]) + for r in s.get("egress",[])) +sys.exit(0 if ok else 1)'"""), + C("web-to-db pairs web egress and db ingress on 5432", 4, r""" +$K -n %{ns} get netpol -o json 2>/dev/null | python3 -c ' +import sys,json +eg=ing=False +for n in json.load(sys.stdin)["items"]: + s=n["spec"]; app=s["podSelector"].get("matchLabels",{}).get("app") + for r in s.get("egress",[]): + if app=="web" and any(t.get("podSelector",{}).get("matchLabels",{}).get("app")=="db" for t in r.get("to",[])) \ + and any(str(p.get("port"))=="5432" for p in r.get("ports",[])): eg=True + for r in s.get("ingress",[]): + if app=="db" and any(t.get("podSelector",{}).get("matchLabels",{}).get("app")=="web" for t in r.get("from",[])) \ + and any(str(p.get("port"))=="5432" for p in r.get("ports",[])): ing=True +sys.exit(0 if eg and ing else 1)'"""), + ], + "solution": """ +cat </dev/null"""), + C("the manifest now sets --profiling=false", 4, r"""nx "$CP" 'grep -q -- "--profiling=false" /etc/kubernetes/manifests/kube-apiserver.yaml'"""), + C("apiserver is healthy again", 4, r""" +for i in $(seq 1 20); do $K get --raw=/healthz >/dev/null 2>&1 && [ "$(j -n kube-system pod -l component=kube-apiserver '{.items[0].status.phase}')" = Running ] && exit 0; sleep 6; done; exit 1"""), + C("after.txt records false", 1, r"""grep -q 'false' "$WS/q2/after.txt" 2>/dev/null"""), + ], + "solution": """ +{SHELL_CP} +grep profiling /etc/kubernetes/manifests/kube-apiserver.yaml || echo absent # -> {WS}/q2/before.txt +vi /etc/kubernetes/manifests/kube-apiserver.yaml +# under command:, add - --profiling=false +# the kubelet notices the manifest change and restarts the static pod - watch with: +crictl ps | grep apiserver +exit +k get --raw=/healthz +echo false > {WS}/q2/after.txt +# Editing the manifest is what makes it persistent; `kubectl edit` on the mirror pod would be lost. +# Mind the indentation: the flag goes in the container's command list, not under spec. +""", +}, +# ---------------------------------------------------------------- HARDEN ---- +{ + "id": 3, "domain": "HARDEN", "title": "Replace a wildcard ClusterRole", + "params": lambda r: {"ns": ns_pick(r, "team"), "sa": name_pick(r, "runner")}, + "task": """ +ServiceAccount `%{sa}` in `%{ns}` is bound to a ClusterRole granting `*` on `*` cluster-wide. + +Reduce it to exactly what it needs: +- read-only (`get`, `list`, `watch`) on `pods` and `pods/log` **in `%{ns}` only**. + +Use Role `%{sa}-view` and RoleBinding `%{sa}-view`. Remove the over-privileged binding. +The ServiceAccount must have no other permissions anywhere. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} create sa %{sa} --dry-run=client -o yaml | $K apply -f - >/dev/null +$K delete clusterrole %{sa}-god --ignore-not-found >/dev/null; $K delete clusterrolebinding %{sa}-god --ignore-not-found >/dev/null +$K -n %{ns} delete role,rolebinding --all >/dev/null 2>&1 +$K create clusterrole %{sa}-god --verb='*' --resource='*.*' >/dev/null +$K create clusterrolebinding %{sa}-god --clusterrole=%{sa}-god --serviceaccount=%{ns}:%{sa} >/dev/null +""", + "checks": [ + C("the wildcard binding is gone", 3, r"""! $K get clusterrolebinding %{sa}-god >/dev/null 2>&1"""), + C("can get/list/watch pods and pods/log in %{ns}", 3, r""" +for res in pods pods/log; do for v in get list watch; do + eq "$($K auth can-i $v $res -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null)" yes || exit 1 +done; done"""), + C("nothing else: no delete pods, no secrets, nothing in other namespaces", 4, r""" +for t in "delete pods -n %{ns}" "get secrets -n %{ns}" "create pods -n %{ns}" "list pods -n default" "list nodes"; do + eq "$($K auth can-i $t --as=system:serviceaccount:%{ns}:%{sa} 2>/dev/null || true)" no || exit 1 +done"""), + ], + "solution": """ +k get clusterrolebinding -o wide | grep %{sa} +k delete clusterrolebinding %{sa}-god +k -n %{ns} create role %{sa}-view --verb=get,list,watch --resource=pods,pods/log +k -n %{ns} create rolebinding %{sa}-view --role=%{sa}-view --serviceaccount=%{ns}:%{sa} +k auth can-i --list -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} +# Deleting the ClusterRole itself is optional - the binding is what grants the access. +""", +}, +{ + "id": 4, "domain": "HARDEN", "title": "Stop handing out API tokens", + "params": lambda r: {"ns": ns_pick(r, "apps"), "dep": name_pick(r, "worker")}, + "task": """ +In Namespace `%{ns}`: + +1. Make the **default** ServiceAccount stop mounting its API token into Pods automatically. +2. Deployment `%{dep}` genuinely needs API access: give it a dedicated ServiceAccount `%{dep}-sa` + that does mount its token, and make the Deployment use it. +3. Confirm: a new Pod using the default ServiceAccount has no token mounted, while `%{dep}`'s Pods do. +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} patch sa default -p '{"automountServiceAccountToken":null}' >/dev/null 2>&1 || true +$K -n %{ns} delete sa %{dep}-sa --ignore-not-found >/dev/null +$K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null +$K -n %{ns} delete pod plain --ignore-not-found --wait=false >/dev/null +$K -n %{ns} create deploy %{dep} --image=nginx:1.27 >/dev/null +""", + "checks": [ + C("default ServiceAccount has automountServiceAccountToken: false", 3, r"""eq "$(j -n %{ns} sa default '{.automountServiceAccountToken}')" false"""), + C("%{dep} uses %{dep}-sa, which does mount a token", 3, r""" +eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.serviceAccountName}')" %{dep}-sa \ + && [ "$(j -n %{ns} sa %{dep}-sa '{.automountServiceAccountToken}')" != false ]"""), + C("%{dep} pods have a projected token mounted", 3, r""" +p=$(j -n %{ns} pod -l app=%{dep} '{.items[0].metadata.name}'); [ -n "$p" ] || exit 1 +$K -n %{ns} exec "$p" -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null"""), + C("a pod on the default ServiceAccount has no token", 3, r""" +$K -n %{ns} delete pod plain --ignore-not-found --wait=true >/dev/null 2>&1 +$K -n %{ns} run plain --image=busybox:1.36 --restart=Never -- sleep 120 >/dev/null 2>&1 +for i in $(seq 1 15); do [ "$(j -n %{ns} pod plain '{.status.phase}')" = Running ] && break; sleep 3; done +r=1; $K -n %{ns} exec plain -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null || r=0 +$K -n %{ns} delete pod plain --wait=false >/dev/null 2>&1; exit $r"""), + ], + "solution": """ +k -n %{ns} patch sa default -p '{"automountServiceAccountToken": false}' +k -n %{ns} create sa %{dep}-sa +k -n %{ns} set serviceaccount deploy %{dep} %{dep}-sa +k -n %{ns} rollout status deploy %{dep} +# The setting can also be put on the Pod spec (automountServiceAccountToken: false), which +# overrides the ServiceAccount - useful when only one workload should differ. +""", +}, +# ------------------------------------------------------------------- SYS ---- +{ + "id": 5, "domain": "SYS", "title": "seccomp and a minimal container", + "params": lambda r: {"ns": ns_pick(r, "sys"), "pod": name_pick(r, "confined")}, + "task": """ +Create Pod `%{pod}` in `%{ns}` (image `nginx:1.27`) that: + +- uses the **RuntimeDefault** seccomp profile at Pod level, +- runs as a non-root user (UID 101, nginx's own user) with `allowPrivilegeEscalation: false`, +- drops all capabilities and adds back only `NET_BIND_SERVICE` and `CHOWN`, +- mounts an `emptyDir` at `/var/cache/nginx` and one at `/var/run` so nginx can still write + while the root filesystem is read-only. + +The Pod must reach Running. +""", + "setup": NS + r"""mkns %{ns}; $K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null""", + "checks": [ + C("seccompProfile RuntimeDefault at pod level", 3, r"""eq "$(j -n %{ns} pod %{pod} '{.spec.securityContext.seccompProfile.type}')" RuntimeDefault"""), + C("non-root UID 101, no privilege escalation, read-only root fs", 3, r""" +p="$K -n %{ns} get pod %{pod}" +u=$($p -o jsonpath='{.spec.securityContext.runAsUser}' 2>/dev/null); [ -z "$u" ] && u=$($p -o jsonpath='{.spec.containers[0].securityContext.runAsUser}') +eq "$u" 101 && eq "$($p -o jsonpath='{.spec.containers[0].securityContext.allowPrivilegeEscalation}')" false \ + && eq "$($p -o jsonpath='{.spec.containers[0].securityContext.readOnlyRootFilesystem}')" true"""), + C("capabilities drop ALL, add NET_BIND_SERVICE and CHOWN", 3, r""" +c='{.spec.containers[0].securityContext.capabilities' +j -n %{ns} pod %{pod} "$c.drop[*]}" | grep -q ALL && j -n %{ns} pod %{pod} "$c.add[*]}" | grep -q NET_BIND_SERVICE && j -n %{ns} pod %{pod} "$c.add[*]}" | grep -q CHOWN"""), + C("pod Running with both writable mounts", 3, r""" +eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running \ + && j -n %{ns} pod %{pod} '{.spec.containers[0].volumeMounts[*].mountPath}' | grep -q '/var/cache/nginx' \ + && j -n %{ns} pod %{pod} '{.spec.containers[0].volumeMounts[*].mountPath}' | grep -q '/var/run'"""), + ], + "solution": """ +cat </dev/null 2>&1 || true +$K -n %{ns} delete deploy %{dep} --ignore-not-found --wait=true >/dev/null +$K apply -f - >/dev/null </dev/null 2>&1 +""", + "checks": [ + C("namespace enforces, warns and audits restricted", 3, r""" +l=$(j ns %{ns} '{.metadata.labels}') +echo "$l" | grep -q '"pod-security.kubernetes.io/enforce":"restricted"' && echo "$l" | grep -q '"pod-security.kubernetes.io/warn":"restricted"' && echo "$l" | grep -q '"pod-security.kubernetes.io/audit":"restricted"'"""), + C("deployment no longer asks for privilege", 3, r""" +t=$(j -n %{ns} deploy %{dep} '{.spec.template.spec}') +! echo "$t" | grep -q '"privileged":true' && echo "$t" | grep -q '"allowPrivilegeEscalation":false' && echo "$t" | grep -q 'RuntimeDefault'"""), + C("it satisfies restricted: non-root, drop ALL", 3, r""" +t=$(j -n %{ns} deploy %{dep} '{.spec.template.spec}') +echo "$t" | grep -q '"runAsNonRoot":true' && echo "$t" | grep -q '"drop":\["ALL"\]'"""), + C("2 replicas Available under enforcement, image unchanged", 4, r""" +eq "$(j -n %{ns} deploy %{dep} '{.spec.template.spec.containers[0].image}')" nginx:1.27 +for i in $(seq 1 12); do [ "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" = 2 ] && exit 0; sleep 5; done; exit 1"""), + ], + "solution": """ +k label ns %{ns} \\ + pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/enforce-version=latest \\ + pod-security.kubernetes.io/warn=restricted pod-security.kubernetes.io/audit=restricted --overwrite +k -n %{ns} rollout restart deploy %{dep} # now blocked, see the events for the exact reasons +k -n %{ns} get events --field-selector reason=FailedCreate +k -n %{ns} edit deploy %{dep} # pod template needs, at minimum: +# securityContext: {runAsNonRoot: true, runAsUser: 101, seccompProfile: {type: RuntimeDefault}} +# container securityContext: {allowPrivilegeEscalation: false, capabilities: {drop: [ALL]}} +# and remove privileged: true +k -n %{ns} rollout status deploy %{dep} +# restricted requires all four: runAsNonRoot, allowPrivilegeEscalation=false, drop ALL, +# and a RuntimeDefault (or Localhost) seccomp profile. +""", +}, +{ + "id": 7, "domain": "MICRO", "title": "Isolate a workload's ServiceAccount and namespace", + "params": lambda r: {"ns": ns_pick(r, "tenant"), "pod": name_pick(r, "isolated")}, + "task": """ +Create Pod `%{pod}` in Namespace `%{ns}` (image `busybox:1.36`, `sleep 86400`) that: + +- uses ServiceAccount `%{pod}-sa` (create it), +- does **not** get an API token mounted, set on the **Pod**, not the ServiceAccount, +- runs with `hostNetwork`, `hostPID` and `hostIPC` all explicitly disabled, +- has both CPU and memory requests and limits set (any sane values). +""", + "setup": NS + r""" +mkns %{ns} +$K -n %{ns} delete pod %{pod} --ignore-not-found --wait=false >/dev/null +$K -n %{ns} delete sa %{pod}-sa --ignore-not-found >/dev/null +""", + "checks": [ + C("uses its own ServiceAccount", 2, r"""$K -n %{ns} get sa %{pod}-sa >/dev/null 2>&1 && eq "$(j -n %{ns} pod %{pod} '{.spec.serviceAccountName}')" %{pod}-sa"""), + C("token not mounted, set on the pod spec", 3, r""" +eq "$(j -n %{ns} pod %{pod} '{.spec.automountServiceAccountToken}')" false \ + && ! $K -n %{ns} exec %{pod} -- test -f /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null"""), + C("host namespaces explicitly disabled", 3, r""" +p="$K -n %{ns} get pod %{pod}" +eq "$($p -o jsonpath='{.spec.hostNetwork}' 2>/dev/null)" false && eq "$($p -o jsonpath='{.spec.hostPID}')" false && eq "$($p -o jsonpath='{.spec.hostIPC}')" false"""), + C("requests and limits set for cpu and memory", 2, r""" +c='{.spec.containers[0].resources' +for f in requests.cpu requests.memory limits.cpu limits.memory; do [ -n "$(j -n %{ns} pod %{pod} "$c.$f}")" ] || exit 1; done"""), + ], + "solution": """ +k -n %{ns} create sa %{pod}-sa +cat <&1 || echo "no token, as intended" +""", +}, +# ---------------------------------------------------------------- SUPPLY ---- +{ + "id": 8, "domain": "SUPPLY", "title": "Admission policy for allowed registries", "difficulty": "hard", + "params": lambda r: {"ns": ns_pick(r, "trusted"), "reg": r.choice(["registry.internal.io", "harbor.corp.io"])}, + "task": """ +Only images from `%{reg}/` may run in Namespace `%{ns}`. + +Using **ValidatingAdmissionPolicy** (no third-party admission controller): +1. Create a `ValidatingAdmissionPolicy` named `allowed-registry` that denies any Pod whose + containers are not all from `%{reg}/`. +2. Bind it with a `ValidatingAdmissionPolicyBinding` named `allowed-registry` that enforces + (`validationActions: [Deny]`) on Namespace `%{ns}` only. +3. Prove it: creating a Pod from `nginx:1.27` in `%{ns}` must be rejected. Save the rejection + message to `{WS}/q8/denied.txt`. +""", + "setup": NS + r""" +mkns %{ns} +rm -rf "$WS/q8"; mkdir -p "$WS/q8" +$K label ns %{ns} registry-policy=enforced --overwrite >/dev/null +$K delete validatingadmissionpolicybinding allowed-registry --ignore-not-found >/dev/null 2>&1 +$K delete validatingadmissionpolicy allowed-registry --ignore-not-found >/dev/null 2>&1 +""", + "checks": [ + C("ValidatingAdmissionPolicy allowed-registry exists and checks image prefixes", 3, r""" +v=$(j validatingadmissionpolicy allowed-registry '{.spec.validations[*].expression}' 2>/dev/null) +[ -n "$v" ] && echo "$v" | grep -q '%{reg}' && j validatingadmissionpolicy allowed-registry '{.spec.matchConstraints.resourceRules[*].resources[*]}' | grep -q pods"""), + C("binding enforces Deny and is scoped to %{ns}", 3, r""" +b="$K get validatingadmissionpolicybinding allowed-registry" +eq "$($b -o jsonpath='{.spec.policyName}' 2>/dev/null)" allowed-registry && $b -o jsonpath='{.spec.validationActions[*]}' | grep -q Deny \ + && $b -o jsonpath='{.spec.matchResources}' | grep -Eq '%{ns}|registry-policy'"""), + C("a disallowed image is actually rejected", 4, r""" +out=$($K -n %{ns} run vap-probe --image=nginx:1.27 --restart=Never 2>&1); rc=$? +$K -n %{ns} delete pod vap-probe --ignore-not-found --wait=false >/dev/null 2>&1 +[ $rc -ne 0 ] && echo "$out" | grep -qi 'denied\|not allowed\|failed'"""), + C("denied.txt captures the rejection", 2, r"""grep -Eqi 'denied|not allowed|admission' "$WS/q8/denied.txt" 2>/dev/null"""), + ], + "solution": """ +cat <&1 | tee {WS}/q8/denied.txt # rejected +# CEL runs in-process in the apiserver - no webhook to deploy, no certificate to manage. +# Add initContainers to the expression too if the exam asks for every container. +""", +}, +# --------------------------------------------------------------- RUNTIME ---- +{ + "id": 9, "domain": "RUNTIME", "title": "Turn on API audit logging", "breaks": "apiserver-flag", "difficulty": "hard", + "task": """ +The cluster has no audit trail. + +On the control-plane node: +1. Write an audit policy to `/etc/kubernetes/audit-policy.yaml` that logs Secret access at + `Metadata` level and everything else at `Metadata` too, but drops the noisy + `RequestReceived` stage. +2. Configure the kube-apiserver to use it, writing to `/var/log/kubernetes/audit.log`, keeping + 7 days of logs and at most 5 rotated files. +3. Bring the API server back healthy, then generate an event (`kubectl get secrets -A`) and copy + the last audit line mentioning `secrets` to `{WS}/q9/audit-line.json`. +""", + "setup": r""" +rm -rf "$WS/q9"; mkdir -p "$WS/q9" +nx "$CP" 'sed -i "/--audit-/d" /etc/kubernetes/manifests/kube-apiserver.yaml; rm -f /etc/kubernetes/audit-policy.yaml' || true +""", + "checks": [ + C("audit policy file exists and omits RequestReceived", 3, r""" +nx "$CP" 'test -f /etc/kubernetes/audit-policy.yaml' && nx "$CP" 'grep -q "omitStages" /etc/kubernetes/audit-policy.yaml' \ + && nx "$CP" 'grep -q "RequestReceived" /etc/kubernetes/audit-policy.yaml'"""), + C("apiserver flags: policy file, log path, maxage 7, maxbackup 5", 4, r""" +m=$(nx "$CP" 'cat /etc/kubernetes/manifests/kube-apiserver.yaml') +echo "$m" | grep -q -- '--audit-policy-file=/etc/kubernetes/audit-policy.yaml' \ + && echo "$m" | grep -q -- '--audit-log-path=/var/log/kubernetes/audit.log' \ + && echo "$m" | grep -q -- '--audit-log-maxage=7' && echo "$m" | grep -q -- '--audit-log-maxbackup=5'"""), + C("apiserver healthy and writing the log", 4, r""" +for i in $(seq 1 20); do $K get --raw=/healthz >/dev/null 2>&1 && break; sleep 6; done +$K get --raw=/healthz >/dev/null 2>&1 || exit 1 +$K get secrets -A >/dev/null 2>&1 +for i in $(seq 1 6); do nx "$CP" 'test -s /var/log/kubernetes/audit.log' && exit 0; sleep 5; done; exit 1"""), + C("audit-line.json is a real audit event about secrets", 3, r""" +python3 -c ' +import json,sys +d=json.load(open("'"$WS"'/q9/audit-line.json")) +sys.exit(0 if d.get("kind")=="Event" and "secrets" in json.dumps(d) else 1)' 2>/dev/null"""), + ], + "solution": """ +{SHELL_CP} +mkdir -p /var/log/kubernetes +cat > /etc/kubernetes/audit-policy.yaml <<'EOF' +apiVersion: audit.k8s.io/v1 +kind: Policy +omitStages: + - RequestReceived +rules: + - level: Metadata + resources: + - group: "" + resources: ["secrets"] + - level: Metadata +EOF +vi /etc/kubernetes/manifests/kube-apiserver.yaml +# command: +# - --audit-policy-file=/etc/kubernetes/audit-policy.yaml +# - --audit-log-path=/var/log/kubernetes/audit.log +# - --audit-log-maxage=7 +# - --audit-log-maxbackup=5 +# volumeMounts: +# - {name: audit-policy, mountPath: /etc/kubernetes/audit-policy.yaml, readOnly: true} +# - {name: audit-log, mountPath: /var/log/kubernetes} +# volumes: +# - {name: audit-policy, hostPath: {path: /etc/kubernetes/audit-policy.yaml, type: File}} +# - {name: audit-log, hostPath: {path: /var/log/kubernetes, type: DirectoryOrCreate}} +exit +k get secrets -A +# grab the last secrets line: +# grep secrets /var/log/kubernetes/audit.log | tail -1 -> {WS}/q9/audit-line.json +# Forgetting the two hostPath mounts is the usual reason the apiserver crashloops here: +# the flags point at paths that don't exist inside the container. +""", +}, +] diff --git a/web/index.html b/web/index.html new file mode 100644 index 0000000..d9f0a2a --- /dev/null +++ b/web/index.html @@ -0,0 +1,468 @@ + + + + + +Exam simulator + + + +
+ Exam simulator + + + --:--:-- + + + +
+ +
+ +
+
+
+

+
+
+
+ + +
+
+ + + + + +
+ +
+ +
+
+ + +
+
+

Exam simulator

+
+
+
+
Drill a domain (✓ = drilled at least once):
+
+ +
+ + + + + + + +
+ + + + +
+
+ + + + + + + + + + + + + diff --git a/workspace/q21/chart/Chart.yaml b/workspace/q21/chart/Chart.yaml new file mode 100644 index 0000000..90cc637 --- /dev/null +++ b/workspace/q21/chart/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: chart +description: A small web chart for the exam simulator +type: application +version: 0.1.0 +appVersion: "1.27" diff --git a/workspace/q21/chart/templates/_helpers.tpl b/workspace/q21/chart/templates/_helpers.tpl new file mode 100644 index 0000000..ea0abb8 --- /dev/null +++ b/workspace/q21/chart/templates/_helpers.tpl @@ -0,0 +1,3 @@ +{{- define "chart.fullname" -}} +{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}} +{{- end -}} diff --git a/workspace/q21/chart/templates/deployment.yaml b/workspace/q21/chart/templates/deployment.yaml new file mode 100644 index 0000000..9531231 --- /dev/null +++ b/workspace/q21/chart/templates/deployment.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "chart.fullname" . }} + labels: {app: {{ include "chart.fullname" . }}} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: {app: {{ include "chart.fullname" . }}} + template: + metadata: + labels: {app: {{ include "chart.fullname" . }}} + spec: + containers: + - name: web + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + ports: [{containerPort: 80}] diff --git a/workspace/q21/chart/templates/service.yaml b/workspace/q21/chart/templates/service.yaml new file mode 100644 index 0000000..9bf5162 --- /dev/null +++ b/workspace/q21/chart/templates/service.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "chart.fullname" . }} +spec: + type: {{ .Values.service.type }} + selector: {app: {{ include "chart.fullname" . }}} + ports: [{port: {{ .Values.service.port }}, targetPort: 80}] diff --git a/workspace/q21/chart/values.yaml b/workspace/q21/chart/values.yaml new file mode 100644 index 0000000..1f82b1e --- /dev/null +++ b/workspace/q21/chart/values.yaml @@ -0,0 +1,7 @@ +replicaCount: 1 +image: + repository: nginx + tag: "1.27" +service: + type: ClusterIP + port: 80