diff --git a/CKA_SOLUTIONS_EXPLAINED.md b/CKA_SOLUTIONS_EXPLAINED.md index fd6194a..f8f1a2a 100644 --- a/CKA_SOLUTIONS_EXPLAINED.md +++ b/CKA_SOLUTIONS_EXPLAINED.md @@ -10,36 +10,36 @@ This bank mirrors the real CKA curriculum weights (25/15/20/10/30% across the fi |---|---|---| | StorageClass and Dynamic Provisioning | Storage | Q16 | | PV Access Modes | Storage | Q15 | -| Volume Can't Be Mounted | Storage | Q35, Q115 (troubleshooting a binding/mount failure) | -| PVC Data Migration | Storage | Q116 | -| Deployment and StatefulSet Rollouts | Workloads & Scheduling | Q07, Q108 | +| Volume Can't Be Mounted | Storage | Q35, Q58 (troubleshooting a binding/mount failure) | +| PVC Data Migration | Storage | Q59 | +| Deployment and StatefulSet Rollouts | Workloads & Scheduling | Q07, Q51 | | Init Container and Secret | Workloads & Scheduling | Q28 + Q10/Q29 (init container *and* a broken Secret reference, combined) | | DaemonSet HostPath Configurator | Workloads & Scheduling | Q26 | | Node Selector and Scheduling | Workloads & Scheduling | Q09 (this bank uses required nodeAffinity instead of nodeSelector — worth doing both) | -| Node Taints and Tolerations | Workloads & Scheduling | Q25, Q107 | -| Scheduling Pod Affinity | Workloads & Scheduling | *no direct equivalent here* — this bank only exercises anti-affinity (Q107); do this one to cover positive affinity too | -| Scheduling Pod Anti Affinity | Workloads & Scheduling | Q107 | -| Scheduling Priority | Workloads & Scheduling | Q24, Q107 | -| NetworkPolicy Isolation | Services & Networking | Q11, Q111 | -| Troubleshoot NetworkPolicies | Services & Networking | Q114 (a blocking NetworkPolicy is one of its three faults) | +| Node Taints and Tolerations | Workloads & Scheduling | Q25, Q50 | +| Scheduling Pod Affinity | Workloads & Scheduling | *no direct equivalent here* — this bank only exercises anti-affinity (Q50); do this one to cover positive affinity too | +| Scheduling Pod Anti Affinity | Workloads & Scheduling | Q50 | +| Scheduling Priority | Workloads & Scheduling | Q24, Q50 | +| NetworkPolicy Isolation | Services & Networking | Q11, Q54 | +| Troubleshoot NetworkPolicies | Services & Networking | Q57 (a blocking NetworkPolicy is one of its three faults) | | App Goes Public | Services & Networking | Q12 (NodePort here; LoadBalancer+autoscale adds Q23's HPA angle) | -| Gateway API Setup | Services & Networking | Q22, Q113 | -| CoreDNS Misconfigured | Services & Networking | Q37, Q114 | -| TLS Secret and HTTPS | Services & Networking | Q112 | -| Apiserver Crash | Troubleshooting | Q123 | -| Apiserver Misconfigured | Troubleshooting | Q123 | +| Gateway API Setup | Services & Networking | Q22, Q56 | +| CoreDNS Misconfigured | Services & Networking | Q37, Q57 | +| TLS Secret and HTTPS | Services & Networking | Q55 | +| Apiserver Crash | Troubleshooting | Q66 | +| Apiserver Misconfigured | Troubleshooting | Q66 | | Deployment Scaling Issue | Troubleshooting | Q23 | -| Application Multi Container Issue | Troubleshooting | Q08, Q109 (troubleshooting angle on a multi-container pod) | -| Applications Misconfigured | Troubleshooting | Q18, Q119 | -| Components Misconfigured | Troubleshooting | Q20, Q38, Q118 | -| RBAC for CronJob | Cluster Architecture | Q01/Q101's RBAC pattern, applied to a CronJob's ServiceAccount specifically | -| RBAC User Permissions | Cluster Architecture | Q06, Q33, Q104 | -| Operator Setup and CRD | Cluster Architecture | Q04, Q34, Q106 (CRD half only — this bank never writes an actual operator/controller) | +| Application Multi Container Issue | Troubleshooting | Q08, Q52 (troubleshooting angle on a multi-container pod) | +| Applications Misconfigured | Troubleshooting | Q18, Q62 | +| Components Misconfigured | Troubleshooting | Q20, Q38, Q61 | +| RBAC for CronJob | Cluster Architecture | Q01/Q44's RBAC pattern, applied to a CronJob's ServiceAccount specifically | +| RBAC User Permissions | Cluster Architecture | Q06, Q33, Q47 | +| Operator Setup and CRD | Cluster Architecture | Q04, Q34, Q49 (CRD half only — this bank never writes an actual operator/controller) | | Operator Namespace Creator | Cluster Architecture | *no equivalent here* — this bank doesn't cover writing controller logic (arguably CKAD/dev territory more than CKA) | -| Static Pod Move | Cluster Architecture | Q05, Q103 | -| Cluster Setup and Node Join | Cluster Architecture | Q40 (token) + Q124 (independently verifying the CA discovery hash and real in-cluster API endpoint) — the token/crypto half only, see gap note | +| Static Pod Move | Cluster Architecture | Q05, Q46 | +| Cluster Setup and Node Join | Cluster Architecture | Q40 (token) + Q67 (independently verifying the CA discovery hash and real in-cluster API endpoint) — the token/crypto half only, see gap note | | Cluster Upgrade | Cluster Architecture | *no equivalent here* — see gap note below | -| Cluster Certificate Management | Cluster Architecture | Q06, Q122 | +| Cluster Certificate Management | Cluster Architecture | Q06, Q65 | ### Real gaps worth knowing about @@ -50,8 +50,8 @@ Two topics from Killercoda's list still have **no faithful equivalent in this si Two other gaps this bank previously had are now closed: -- **Apiserver Crash / Apiserver Misconfigured** → **Q123**. Same diagnostic method as the existing scheduler/controller-manager failures (Q20, Q38, Q118), just applied to kube-apiserver itself — with the added twist that *every* `kubectl` command is unusable until you fix it, so you have to work from the node directly (`crictl`, `journalctl`) rather than leaning on `kubectl describe`. -- **Cluster Setup and Node Join** → partially closed by **Q124**. A full `kubeadm init`/`join` end-to-end on a fresh node isn't safe to automate here (it would mean deleting a live Node object on your shared 3-worker cluster, which several other questions pin pods to by label/taint) — but Q124 drills the cryptographic half a real join actually depends on: computing the CA's discovery hash yourself and finding the real in-cluster API endpoint, rather than trusting whatever `kubeadm token create --print-join-command` or your local kubeconfig hands you. +- **Apiserver Crash / Apiserver Misconfigured** → **Q66**. Same diagnostic method as the existing scheduler/controller-manager failures (Q20, Q38, Q61), just applied to kube-apiserver itself — with the added twist that *every* `kubectl` command is unusable until you fix it, so you have to work from the node directly (`crictl`, `journalctl`) rather than leaning on `kubectl describe`. +- **Cluster Setup and Node Join** → partially closed by **Q67**. A full `kubeadm init`/`join` end-to-end on a fresh node isn't safe to automate here (it would mean deleting a live Node object on your shared 3-worker cluster, which several other questions pin pods to by label/taint) — but Q67 drills the cryptographic half a real join actually depends on: computing the CA's discovery hash yourself and finding the real in-cluster API endpoint, rather than trusting whatever `kubeadm token create --print-join-command` or your local kubeconfig hands you. The exam-day skill this table is really pointing at, regardless of which platform you practice on: the diagnostic *method* is identical everywhere — read the actual error (`describe`, `logs`, `journalctl -u kubelet`), don't guess from memory, and verify with the tool the grader would use (`auth can-i`, `get endpointslices`, `rollout status`) rather than assuming a manifest applied cleanly means it's actually working. @@ -308,7 +308,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q101 · [HARD] Exact RBAC with an Existing Over-Grant +### Q44 · [HARD] Exact RBAC with an Existing Over-Grant - **Concept**: This is Q01's lesson turned adversarial: someone already bound `cluster-admin` to the ServiceAccount "temporarily" — a stand-in for the all-too-common real-world over-grant. Least privilege means actively auditing and revoking excess access, not just adding what's missing; the grader checks both the positive grants **and** that nothing broader survives. - **Solution**: ```bash @@ -329,7 +329,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q102 · [HARD] etcd Restore +### Q45 · [HARD] etcd Restore - **Concept**: Restoring etcd doesn't overwrite the live data directory in place — `etcdutl snapshot restore` materializes a **brand-new** data directory from the snapshot, and you then repoint the etcd static pod's `hostPath` volume at that new directory. The kubelet notices the manifest change and recreates the static pod, which comes up serving whatever state existed at snapshot time — so anything created after the snapshot (like the ConfigMap this task deletes and recreates) reverts, and this is the entire point of the exercise. - **Solution**: ```bash @@ -349,7 +349,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q103 · [HARD] Static Pod on a Relocated Manifest Path +### Q46 · [HARD] Static Pod on a Relocated Manifest Path - **Concept**: Same mechanism as Q05, with the deliberate trap that `staticPodPath` has been changed on this node — dropping a manifest into the default `/etc/kubernetes/manifests` here does nothing, because that kubelet isn't watching it anymore. This tests whether you actually check the config instead of assuming the default. - **Solution**: ```bash @@ -372,7 +372,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q104 · [HARD] Self-Contained Kubeconfig Creation +### Q47 · [HARD] Self-Contained Kubeconfig Creation - **Concept**: A kubeconfig can reference certs either by file path (`certificate-authority`, `client-certificate`) or by embedding their base64 contents directly (`certificate-authority-data`, `client-certificate-data`). Only the embedded form is truly **portable** — a path-based kubeconfig breaks the moment it's copied to a machine without those exact files at those exact paths, which is exactly what "self-contained" is testing. - **Solution**: ```bash @@ -390,7 +390,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor k -n %{ns} create rolebinding %{user}-edit --clusterrole=edit --user=%{user} - KC={WS}/q104/%{user}.kubeconfig + KC={WS}/q47/%{user}.kubeconfig k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt SERVER=$(k config view --minify -o jsonpath='{.clusters[0].cluster.server}') @@ -405,11 +405,11 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q105 · [HARD] Kustomize with Generators and Patches +### Q48 · [HARD] Kustomize with Generators and Patches - **Concept**: This combines every non-trivial Kustomize feature at once: a **strategic-merge patch** (`patches: [{path: replicas.yaml}]`) for a simple field override, a **JSON 6902 patch** for a structural addition that strategic-merge can't express well (injecting a whole `envFrom` array element), and a `configMapGenerator`, which — critically — appends a content hash to the generated name and then **automatically rewrites every reference to that ConfigMap** elsewhere in the same kustomization. That's what makes hashed-ConfigMap rollouts safe: change the literal, get a new hash, get a new name, and Kubernetes treats it as a completely new object — pods roll to pick it up instead of silently reading stale data from a ConfigMap edited in place. - **Solution**: ```bash - mkdir -p {WS}/q105/overlays/prod && cd {WS}/q105/overlays/prod + mkdir -p {WS}/q48/overlays/prod && cd {WS}/q48/overlays/prod cat > replicas.yaml < {WS}/q106/rejected.txt + | k apply -f - 2> {WS}/q49/rejected.txt # Step 4 — confirm defaulting printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: defaulted}\nspec: {replicas: 2, image: nginx:1.27}\n' \ @@ -481,7 +481,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q122 · [HARD] Renew Control Plane Certificates +### Q65 · [HARD] Renew Control Plane Certificates - **Concept**: kubeadm tracks every certificate it issued and can regenerate all of them from the existing cluster CA in one shot with `kubeadm certs renew all`. New certificate files on disk don't automatically get picked up by already-running static-pod processes — you have to force the kubelet to recreate those pods (moving manifests out and back is the classic trick) so each component re-reads its cert files on startup. - **Solution**: ```bash @@ -498,7 +498,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q124 · [HARD] Prepare a Node Join Without Shell Access to the Control Plane +### Q67 · [HARD] Prepare a Node Join Without Shell Access to the Control Plane - **Concept**: `kubeadm join` doesn't trust the control plane on faith — it verifies the server it's talking to by hashing the CA certificate's SubjectPublicKeyInfo with SHA-256 and comparing it against the `--discovery-token-ca-cert-hash` you supply, which is exactly the computation this task has you reproduce by hand instead of trusting whatever `kubeadm token create --print-join-command` prints. Separately, the address a *joining node* needs is the API server's real address on the cluster network — not the `127.0.0.1:` your own kubeconfig uses, which only works because kind port-forwards that specifically for your host machine. - **Solution**: ```bash @@ -727,7 +727,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- ### Q28 · Init Containers -- **Concept**: `initContainers` run to completion, in order, before any regular container starts — the standard pattern for "prepare something the main container needs," here using a shared `emptyDir` to hand off generated content from a setup step to the long-running server. Unlike native sidecars (Q109), a plain init container fully exits before the main container's process even begins. +- **Concept**: `initContainers` run to completion, in order, before any regular container starts — the standard pattern for "prepare something the main container needs," here using a shared `emptyDir` to hand off generated content from a setup step to the long-running server. Unlike native sidecars (Q52), a plain init container fully exits before the main container's process even begins. - **Solution**: ```yaml apiVersion: v1 @@ -810,7 +810,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q107 · [HARD] Spread, Tolerate, Prioritise — Under a Quota +### Q50 · [HARD] Spread, Tolerate, Prioritise — Under a Quota - **Concept**: This stacks four scheduling primitives that each solve a different problem: **pod anti-affinity** (`requiredDuringSchedulingIgnoredDuringExecution` on `topologyKey: kubernetes.io/hostname`) forces one replica per node by refusing to co-locate pods matching a label selector; a **toleration** lets pods land on an otherwise-repelling tainted node; **PriorityClass** affects preemption, not placement directly; and a **ResourceQuota** caps the namespace regardless of what the scheduler would otherwise allow — so the requests must be small enough to fit inside it. - **Solution**: ```yaml @@ -845,7 +845,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q108 · [HARD] Rollback to a Specific Revision, Then Pause +### Q51 · [HARD] Rollback to a Specific Revision, Then Pause - **Concept**: Every `kubectl rollout undo --to-revision=N` targets a specific historical **ReplicaSet** by its stored revision annotation, not just "the previous one" — useful when several bad releases have shipped in a row. `rollout pause` freezes a Deployment's controller from acting on template changes: you can edit `spec.template` afterward all you like (image, env, whatever) and the controller will **not** create a new ReplicaSet or touch running pods until you `resume` — a safe way to stage a change for later. - **Solution**: ```bash @@ -863,7 +863,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q109 · [HARD] Native Sidecar with Probes and an Exact Report +### Q52 · [HARD] Native Sidecar with Probes and an Exact Report - **Concept**: Kubernetes 1.28+ added **native sidecars**: an `initContainers` entry with `restartPolicy: Always` starts before the main containers (like any init container) but then keeps running for the pod's whole lifetime and is restarted on its own if it crashes — unlike a legacy init container, it never blocks pod readiness by "completing," and unlike a regular sidecar container it's guaranteed to start first. Combined with `terminationGracePeriodSeconds`, this also governs shutdown ordering: main containers get the grace period to exit first, and native sidecars are stopped only after. - **Solution**: ```yaml @@ -888,14 +888,14 @@ The exam-day skill this table is really pointing at, regardless of which platfor ``` ```bash k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name}{"\n"}{.spec.containers[*].name}' \ - | tr ' ' '\n' | grep . | sort > {WS}/q109/containers.txt - k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' | tr -d '\n' > {WS}/q109/ip.txt + | tr ' ' '\n' | grep . | sort > {WS}/q52/containers.txt + k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' | tr -d '\n' > {WS}/q52/ip.txt ``` - **Gotchas**: never mount an emptyDir over `/var/log` in an nginx container — it hides nginx's own `/var/log/nginx` and nginx dies with `could not open error log file`, crash-looping forever. A volume mount **replaces** the directory's contents; mount a subdirectory instead. `restartPolicy: Always` goes **inside the initContainer entry itself**, not at the pod level — that per-container field is exactly what flips a normal init container into a native sidecar. The report file must list container names sorted, one per line, no header — matching the check's own `sort` byte-for-byte. --- -### Q110 · [HARD] Cluster Audit to Exact Files +### Q53 · [HARD] Cluster Audit to Exact Files - **Concept**: "Produce exact files" tasks are a scripting exercise disguised as a Kubernetes question — the grader diffs your output against a canonical, sorted, header-free rendering of live cluster state, so the reliable path is always `-o json` piped through `jq`/`python3`, never hand-transcribing what `kubectl get` prints to the terminal. - **Solution**: ```bash @@ -905,28 +905,28 @@ The exam-day skill this table is really pointing at, regardless of which platfor for p in json.load(sys.stdin)["items"]: if p["metadata"]["namespace"]=="kube-system": continue if not p["spec"]["containers"][0].get("readinessProbe"): - print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort > {WS}/q110/no-readiness.txt + print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort > {WS}/q53/no-readiness.txt # of Running pods per node, all namespaces k get pods -A -o json | python3 -c ' import sys,json,collections c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"] if p["status"]["phase"]=="Running" and p["spec"].get("nodeName")) - for n in sorted(c): print(n, c[n])' > {WS}/q110/per-node.txt + for n in sorted(c): print(n, c[n])' > {WS}/q53/per-node.txt # every distinct image running in kube-system k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' \ - | tr ' ' '\n' | sort -u > {WS}/q110/images.txt + | tr ' ' '\n' | sort -u > {WS}/q53/images.txt ``` - **Gotchas**: no header lines, no trailing commentary — the checker does a literal `sort`-then-compare against each file's contents. Excluding `kube-system` from the first file but restricting *to* `kube-system` in the third is easy to flip by accident; re-read exactly which file wants which scope before running the one-liners. --- -### Q121 · [HARD] The Context's Namespace +### Q64 · [HARD] The Context's Namespace - **Concept**: A kubeconfig **context** bundles a cluster, a user, and a default namespace (`contexts[].context.namespace`) — any command run without an explicit `-n`/`--namespace` inherits that default, falling back to literal `default` only if the context doesn't set one. `kubectl config view --minify` collapses the kubeconfig down to just the active context, which is the safe way to read that default without assuming it's `default`. - **Solution**: ```bash - k config view --minify -o jsonpath='{..namespace}' > {WS}/q121/ns.txt + k config view --minify -o jsonpath='{..namespace}' > {WS}/q64/ns.txt # No -n anywhere below — everything inherits the context's namespace k create deploy %{dep} --image=busybox:1.36 --replicas=2 -- sleep 3600 @@ -1102,7 +1102,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q111 · [HARD] Zero-Trust Namespace +### Q54 · [HARD] Zero-Trust Namespace - **Concept**: True zero-trust starts from **default-deny both directions**, then layers narrow allow rules back in — order of construction matters conceptually even though NetworkPolicies are all additive/OR'd together: without the deny-all baseline first, you're just adding allow rules on top of an implicitly-open namespace and nothing is actually restricted. Every allow-egress-to-DNS rule is non-negotiable in a default-deny-egress namespace, exactly as in Q42, or literally nothing else works either. - **Solution**: ```yaml @@ -1152,7 +1152,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q112 · [HARD] TLS Ingress with a Generated Certificate +### Q55 · [HARD] TLS Ingress with a Generated Certificate - **Concept**: TLS termination at the Ingress layer needs a `kubernetes.io/tls` Secret holding `tls.crt`/`tls.key`, referenced from `spec.tls[].secretName`; for a self-signed lab cert, `openssl req -x509` generates both the key and a self-signed certificate in one step (no separate CA involved) with the `CN` acting as the hostname it's valid for. - **Solution**: ```bash @@ -1175,13 +1175,13 @@ The exam-day skill this table is really pointing at, regardless of which platfor paths: [{path: /, pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}] ``` ```bash - openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > {WS}/q112/fp.txt + openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > {WS}/q55/fp.txt ``` - **Gotchas**: `-nodes` on `openssl req` means "no DES" — i.e. don't encrypt the private key with a passphrase, which you'd otherwise be prompted for and which `kubectl create secret tls` can't supply interactively. The fingerprint recorded in `fp.txt` must match the certificate **actually stored in the Secret**, not just the local `tls.crt` file — if you regenerate the cert after creating the Secret, they'll drift apart. --- -### Q113 · [HARD] Gateway API Canary with Header Match +### Q56 · [HARD] Gateway API Canary with Header Match - **Concept**: `HTTPRoute` rules are evaluated top-to-bottom for a match; a rule with a header `matches` clause only applies to requests carrying that header, so stacking a header-matched 100%-to-`v2` rule *before* a header-less weighted-split rule implements a canary override — anyone sending the magic header always gets the canary, everyone else gets the normal weighted split. `parentRefs[].sectionName` scopes the route's attachment to one specific listener on a multi-listener Gateway, rather than every listener. - **Solution**: ```yaml @@ -1215,7 +1215,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q114 · [HARD] DNS Down: Three Faults +### Q57 · [HARD] DNS Down: Three Faults - **Concept**: A single symptom ("DNS is broken") can have multiple independent causes stacked on top of each other — this task combines a bad upstream `forward` target in the Corefile, CoreDNS scaled to zero, and a stray NetworkPolicy blocking ingress to the CoreDNS pods, all at once. The exam-realistic lesson is to check **each** layer (config, replica count, network policy) rather than stopping the moment the first plausible cause is found. - **Solution**: ```bash @@ -1338,7 +1338,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q115 · [HARD] Bind the Right Local Volume +### Q58 · [HARD] Bind the Right Local Volume - **Concept**: `local` PersistentVolumes are tied to a specific node via `nodeAffinity` — the scheduler must place any pod using that volume on exactly that node, which is why choosing the *wrong* one of two same-class PVs quietly schedules your pod onto the wrong hardware. A PVC's `spec.selector` (label match) is the tool for picking a specific PV among several of the same StorageClass **without editing the PVs themselves** — exactly the constraint this task imposes. - **Solution**: ```yaml @@ -1363,7 +1363,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q116 · [HARD] Migrate Data Between Claims +### Q59 · [HARD] Migrate Data Between Claims - **Concept**: There's no built-in "resize/migrate a volume to a different backend" primitive — the standard pattern is a throwaway `Job` that mounts both the old and new PVCs simultaneously and copies data between their mount paths, then you cut the live workload over to the new claim. Setting the old PV's `persistentVolumeReclaimPolicy` to `Retain` *before* deleting its PVC is what keeps the original data recoverable (`Released`, not gone) in case the migration needs to be redone. - **Solution**: ```bash @@ -1534,7 +1534,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q117 · [HARD] Worker Node: Three Things Wrong +### Q60 · [HARD] Worker Node: Three Things Wrong - **Concept**: This stacks three unrelated failure modes onto one node — a cordon (`spec.unschedulable: true`, which stops new scheduling but is unrelated to `Ready` status), a wrong `clientCAFile` path (kubelet can't start, per Q39), and the kubelet service being stopped/disabled (per Q17) — to test that you don't stop investigating after the first fix "seems to work." - **Solution**: ```bash @@ -1556,7 +1556,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q118 · [HARD] Two Control-Plane Components Down +### Q61 · [HARD] Two Control-Plane Components Down - **Concept**: The scheduler-down (Q20) and controller-manager-down (Q38) failure signatures combine here — pods that do get created stay `Pending` (scheduler dead) *and* brand-new Deployments never even get a ReplicaSet (controller-manager dead), for two entirely unrelated static-pod manifest problems. Fixing only one still leaves the cluster non-functional for new workloads. - **Solution**: ```bash @@ -1578,7 +1578,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q119 · [HARD] Application Down for Four Reasons +### Q62 · [HARD] Application Down for Four Reasons - **Concept**: Four independent faults are stacked on one app: a `ResourceQuota` capping pods at 1 (blocking scale-up entirely), a broken image tag, a Secret missing the key the container's `env` references (`CreateContainerConfigError`), and a Service `selector`/`targetPort` mismatch (Q18's pattern again) — and critically, the fix must happen **in place**: the Deployment's UID and the Service's ClusterIP must not change, ruling out delete-and-recreate as a shortcut. - **Solution**: ```bash @@ -1604,7 +1604,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q120 · [HARD] Cluster Facts, Exactly +### Q63 · [HARD] Cluster Facts, Exactly - **Concept**: This is a "read the cluster's actual configuration, don't guess" exercise — pod/service CIDRs live in the static pod command-line flags of `kube-controller-manager`/`kube-apiserver` (not in any single friendly `kubectl get` field), the cluster DNS address is just the `kube-dns` Service's ClusterIP, kube-proxy's mode is buried in its ConfigMap (and an *empty* mode value means the default, `iptables` — not "not configured"), and per-node pod CIDR lives on the Node object itself. - **Solution**: ```bash @@ -1625,8 +1625,8 @@ The exam-day skill this table is really pointing at, regardless of which platfor --- -### Q123 · [HARD] Nothing Works: kube-apiserver Itself Is Down -- **Concept**: Every other control-plane failure in this bank (scheduler, controller-manager, even etcd mid-restore) leaves `kubectl` itself usable, because kube-apiserver is still up to talk to. A broken kube-apiserver is categorically different: it's the single component every `kubectl` command depends on, so once it's down, `kubectl describe`/`logs`/`get` all stop working simultaneously and give you nothing to work with. The kubelet, however, doesn't need the API server to manage **static pods** — it reads the manifest directly off disk — so the fix is the same class of move as Q20/Q118 (repair a static pod manifest), just reached without `kubectl` in hand. +### Q66 · [HARD] Nothing Works: kube-apiserver Itself Is Down +- **Concept**: Every other control-plane failure in this bank (scheduler, controller-manager, even etcd mid-restore) leaves `kubectl` itself usable, because kube-apiserver is still up to talk to. A broken kube-apiserver is categorically different: it's the single component every `kubectl` command depends on, so once it's down, `kubectl describe`/`logs`/`get` all stop working simultaneously and give you nothing to work with. The kubelet, however, doesn't need the API server to manage **static pods** — it reads the manifest directly off disk — so the fix is the same class of move as Q20/Q61 (repair a static pod manifest), just reached without `kubectl` in hand. - **Solution**: ```bash kubectl get nodes # connection refused / timeout -- confirms it's the apiserver, not your network diff --git a/README.md b/README.md index 6c26cc2..2ee30ad 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ set -x CKA_CP cp1; set -x CKA_W1 node1; set -x CKA_W2 node2; set -x CKA_W3 no `/opt/cka` and `/etc/kubernetes/manifests`, so the login must be able to. Managed clusters (EKS/AKS/GKE) won't work — no node or control-plane access. Startup preflights all of this. -The etcd tasks (2 and 102) need `etcdctl`/`etcdutl` **on the control-plane node**, as the real +The etcd tasks (2 and 45) need `etcdctl`/`etcdutl` **on the control-plane node**, as the real exam has them — they cannot be copied out of the running pod, since the etcd image is distroless. `setup.sh` installs them on kind; on any cluster, seeding those tasks installs them if they are missing (from the etcd image in the node's containerd, else the upstream release, else apt) and @@ -127,7 +127,7 @@ fails loudly if it cannot. `./etcd-tools.sh` does the same by hand. Disruptive tasks (anything that breaks a node or control-plane component) are seeded last and can block other tasks until fixed — read everything first, as in the real exam. Tasks that damage or depend on the same shared state declare it in `breaks` and never share a session, and the etcd -restore (Q102) is `solo`: restoring rolls the whole cluster back, so it only runs on its own. +restore (Q45) is `solo`: restoring rolls the whole cluster back, so it only runs on its own. Before every session a baseline pass repairs anything a previous, unfinished session left broken. The kindnet CNI here **enforces** NetworkPolicy (it embeds a policy engine), so a wrong policy @@ -145,7 +145,7 @@ were retired in favour of these two; their last versions are in git history.) Banks live in `tracks/cka/{questions,bank_extra,bank_hard,bank_sim_*}.py`; a task is: ```python -{"id": 44, "domain": "NET", "difficulty": "normal", # or "hard" +{"id": 103, "domain": "NET", "difficulty": "normal", # or "hard" - ids stay contiguous "params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])}, "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...", "setup": "bash that seeds the scenario (may use %{ns})", diff --git a/exam b/exam index 1ca62b1..7ae0513 100755 --- a/exam +++ b/exam @@ -212,8 +212,8 @@ def q_by_id(n: int) -> dict[str, Any]: for q in QUESTIONS: if q["id"] == n: return q - # ids are not contiguous (1-43 base, 101-124 hard, 401-418 / 501-517 sim sets), so print the - # real ranges rather than "1-", which named ids that do not exist. + # ids run 1..N without gaps today, but print the real ranges rather than "1-" so a + # future gap can never make this message name ids that do not exist. ids = sorted(q["id"] for q in QUESTIONS) ranges, start, prev = [], ids[0], ids[0] for i in ids[1:] + [None]: @@ -374,14 +374,14 @@ def drilled_domains(): return out def breaks_of(q): - """A task's "breaks" as a set. It is a string or a list: Q118 damages the scheduler AND the + """A task's "breaks" as a set. It is a string or a list: Q61 damages the scheduler AND the controller-manager, and declaring only one let it share a session with Q38.""" b = q.get("breaks") return set() if not b else ({b} if isinstance(b, str) else set(b)) def no_conflicts(cands, n, taken=()): """Take up to n tasks, never two that break the same thing (or clash with tasks already taken). - A "solo" task (Q102's etcd restore rolls the WHOLE cluster back to its snapshot, wiping every + A "solo" task (Q45's etcd restore rolls the WHOLE cluster back to its snapshot, wiping every other task's seed and answers) is only ever taken on its own.""" used = set().union(*(breaks_of(q) for q in taken)) if taken else set() out = [] @@ -635,7 +635,7 @@ def _log_msg(*args: Any, **kwargs: Any) -> None: # restores every known "breaks" target to a good baseline before each session seeds, so # leftover damage never leaks across sessions regardless of which questions get sampled. RESTORE_BASELINE = r""" -# apiserver FIRST: Q123 renames the command to "kube-apiserverr", which takes the whole API +# apiserver FIRST: Q66 renames the command to "kube-apiserverr", which takes the whole API # down - every kubectl-based step below would fail until this is undone. The repair goes over # podman/ssh (nx), not kubectl, so it works precisely when the cluster is unreachable. if nx "$CP" 'grep -q "^ *- kube-apiserverr$" /etc/kubernetes/manifests/kube-apiserver.yaml' /dev/null; then @@ -644,8 +644,8 @@ if nx "$CP" 'grep -q "^ *- kube-apiserverr$" /etc/kubernetes/manifests/kube-apis for _ in $(seq 1 30); do kubectl --context "${CKA_CONTEXT:-kind-cka-sim}" get --raw=/readyz >/dev/null 2>&1 && break; sleep 2; done fi -# scheduler: undo the "kube-scheduler" -> "kube-schedulerr" command typo (Q20/Q118), and put the -# manifest back if a task had the candidate stop the scheduler by moving it out (Q512) and they +# scheduler: undo the "kube-scheduler" -> "kube-schedulerr" command typo (Q20/Q61), and put the +# manifest back if a task had the candidate stop the scheduler by moving it out (Q97) and they # never moved it back - that setup keeps a pristine copy outside the static pod directory. nx "$CP" ' f=/etc/kubernetes/manifests/kube-scheduler.yaml @@ -653,7 +653,7 @@ f=/etc/kubernetes/manifests/kube-scheduler.yaml sed -i "s|^\( *\)- kube-schedulerr$|\1- kube-scheduler|" $f 2>/dev/null ' /dev/null ' /dev/null) if [ -n "$cf" ] && ! printf '%s' "$cf" | grep -Eq '^[[:space:]]*forward \. /etc/resolv\.conf'; then printf '%s' "$cf" | sed -E 's|^([[:space:]]*forward \. )[^[:space:]{]+|\1/etc/resolv.conf|' > /tmp/Corefile.restore $K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.restore --dry-run=client -o yaml | $K apply -f - >/dev/null 2>&1 $K -n kube-system rollout restart deploy coredns >/dev/null 2>&1 fi -# ...and Q114's other two faults. kindnet ENFORCES NetworkPolicy, so a leftover lockdown policy on +# ...and Q57's other two faults. kindnet ENFORCES NetworkPolicy, so a leftover lockdown policy on # the CoreDNS pods kills name resolution for the whole cluster in every later session, and so does # a Deployment left at 0 replicas. $K -n kube-system delete netpol maintenance-lockdown --ignore-not-found >/dev/null 2>&1 @@ -680,14 +680,14 @@ r=$($K -n kube-system get deploy coredns -o jsonpath='{.spec.replicas}' 2>/dev/n if [ -n "$r" ] && [ "$r" -lt 2 ]; then $K -n kube-system scale deploy coredns --replicas=2 >/dev/null 2>&1; fi # w1/w2/w3 kubelet: undo Q17/Q39-style tampering, then make sure it's enabled and running. -# staticPodPath is normalised unconditionally: Q103 relocates it to /etc/kubernetes/static, and +# staticPodPath is normalised unconditionally: Q46 relocates it to /etc/kubernetes/static, and # the config.yaml.orig copy below only helps if something made that backup first - nothing does, # so a relocated path used to survive every later session and silently break Q05 (whose check # looks in /etc/kubernetes/manifests). Only restart the kubelet if we actually changed the file. -# clientCAFile gets the same treatment: Q39 points it at ca-bundle.crt and Q117 at ca-wrong.crt, and -# only W1 ever gets a config.yaml.orig (Q39 makes it), so an unfinished Q117 left W2's kubelet +# clientCAFile gets the same treatment: Q39 points it at ca-bundle.crt and Q60 at ca-wrong.crt, and +# only W1 ever gets a config.yaml.orig (Q39 makes it), so an unfinished Q60 left W2's kubelet # crash-looping in every later session - enable/start below cannot help a kubelet that exits. -# The systemd drop-in is checked too: Q507 points ExecStart at a kubelet binary that does not exist. +# The systemd drop-in is checked too: Q92 points ExecStart at a kubelet binary that does not exist. for n in "$W1" "$W2" "$W3"; do nx "$n" ' c=/var/lib/kubelet/config.yaml; d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf @@ -710,7 +710,7 @@ for n in "$W1" "$W2" "$W3"; do ' /dev/null 2>&1; then fi # w1-taint: clear every randomized demo taint key the bank can apply. Q25 draws from -# dedicated/team/workload and Q107 from pool/lane; `maintenance` was the retired set B's and may +# dedicated/team/workload and Q50 from pool/lane; `maintenance` was the retired set B's and may # still be on a cluster that ran it. Miss one and that node silently stops accepting pods. # One key per call: `kubectl taint node N a- b-` aborts on the first key that is not present and # removes NOTHING, so the single taint that is actually set survives the "cleanup" and W1 keeps @@ -750,13 +750,13 @@ if $K -n kube-system get deploy metrics-server >/dev/null 2>&1 \ -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' >/dev/null 2>&1 || true fi -# cordons: Q117 cordons W2, and a drain task left half-done leaves a node cordoned. A cordon left -# behind fails any later task that needs every worker (Q26's DaemonSet coverage, Q107's spread). +# cordons: Q60 cordons W2, and a drain task left half-done leaves a node cordoned. A cordon left +# behind fails any later task that needs every worker (Q26's DaemonSet coverage, Q50's spread). for n in "$W1" "$W2" "$W3"; do $K uncordon "$n" >/dev/null 2>&1 || true; done # Deliberately NOT restored, and why: -# etcd (Q102) and certs (Q122). Neither setup damages anything - Q102 only takes a snapshot -# and deletes a ConfigMap, Q122 only records the current notBefore. Any damage comes from a +# etcd (Q45) and certs (Q65). Neither setup damages anything - Q45 only takes a snapshot +# and deletes a ConfigMap, Q65 only records the current notBefore. Any damage comes from a # half-finished candidate attempt (etcd.yaml left pointing at a restored data-dir, static pod # manifests moved to /tmp and never moved back), which is not deterministic: there is no way # to tell "mid-attempt" from "correctly completed" without guessing, and guessing wrong here diff --git a/history-cka.jsonl b/history-cka.jsonl index ade7c4d..3d64e46 100644 --- a/history-cka.jsonl +++ b/history-cka.jsonl @@ -1,11 +1,11 @@ {"at": "2026-09-06T22:07", "mode": "drill", "hard": false, "ids": [1, 5, 6, 34, 41], "score_pct": 8.0, "earned": 2, "total": 25, "time_used": 432, "results": {"1": {"pass": false, "earned": 2, "total": 6, "failed": ["ServiceAccount rbac-app/deploy-bot exists", "Role deploy-manager and RoleBinding deploy-bot-binding exist", "deploy-bot can get/list/watch/create/update/patch/delete deployments"]}, "5": {"pass": false, "earned": 0, "total": 5, "failed": ["manifest containing static-web is in /etc/kubernetes/manifests on the node", "mirror pod static-web- is Running", "image is nginx:1.27"]}, "6": {"pass": false, "earned": 0, "total": 6, "failed": ["apiserver-expiry.txt records the real expiry month and year", "CSR jane exists with signer kubernetes.io/kube-apiserver-client", "CSR jane approved and certificate issued", "jane can list pods in default (view) but not delete them"]}, "34": {"pass": false, "earned": 0, "total": 4, "failed": ["path.txt has the correct field path", "Gadget/sample created with the field = 4"]}, "41": {"pass": false, "earned": 0, "total": 4, "failed": ["apiserver and kubelet versions correct", "etcd and runtime versions correct"]}}} {"at": "2026-09-08T19:59", "mode": "exam", "hard": false, "ids": [2, 3, 4, 8, 12, 13, 15, 16, 17, 18, 19, 20, 25, 28, 29, 30, 33, 37, 40, 42], "score_pct": 14.3, "earned": 15, "total": 105, "time_used": 2902, "results": {"2": {"pass": true, "earned": 7, "total": 7, "failed": []}, "3": {"pass": false, "earned": 0, "total": 5, "failed": ["overlay kustomization.yaml exists", "resources deployed into namespace store-prod", "store-api has 3 replicas", "Deployment and Service carry label env=prod"]}, "4": {"pass": true, "earned": 4, "total": 4, "failed": []}, "8": {"pass": false, "earned": 0, "total": 4, "failed": ["pod log-shipper Running with containers app and shipper", "volume logs is an emptyDir mounted at /var/log in both containers", "shipper's logs show lines written by app"]}, "12": {"pass": false, "earned": 0, "total": 4, "failed": ["Service shop-svc is NodePort, port 80 -> targetPort 80", "nodePort is 30080", "Service has endpoints"]}, "13": {"pass": false, "earned": 0, "total": 5, "failed": ["ingressClassName nginx and host shop.example.com", "/api (Prefix) -> api:8080", "/ (Prefix) -> web:80"]}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "17": {"pass": false, "earned": 0, "total": 8, "failed": ["node is Ready", "kubelet is active", "kubelet is enabled (survives reboot)"]}, "18": {"pass": false, "earned": 0, "total": 7, "failed": ["all payments replicas available", "Service payments has endpoints (selector fixed)", "Service targetPort matches container port 80"]}, "19": {"pass": false, "earned": 1, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure", "pod crasher Running with DB_HOST set"]}, "20": {"pass": false, "earned": 0, "total": 8, "failed": ["kube-scheduler static pod manifest repaired", "new pods get scheduled"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries workload=batch:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "30": {"pass": false, "earned": 0, "total": 4, "failed": ["container port 80 named app-port", "Service port 8080 -> targetPort app-port (by name)", "readiness probe uses the named port; endpoints ready"]}, "33": {"pass": false, "earned": 1, "total": 5, "failed": ["ops-44 has an aggregationRule with the right label", "aggregation populated rules from the two labelled roles", "lee can list pods and services cluster-wide"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~2h"]}, "42": {"pass": false, "earned": 0, "total": 4, "failed": ["applies to app=client-47, Egress only", "DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP", "TCP 443 to 192.168.100.0/24 only"]}}} -{"at": "2026-09-10T21:07", "mode": "drill", "hard": true, "ids": [106, 112, 115, 120, 121], "score_pct": 6.5, "earned": 2, "total": 31, "time_used": 30, "results": {"106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "115": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "120": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "121": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-14 in ac-omega (the context's namespace), 2 available, owner=ac", "Service ledger-14 port 9000 in the same namespace", "ns.txt has the namespace"]}}} +{"at": "2026-09-10T21:07", "mode": "drill", "hard": true, "ids": [49, 55, 58, 63, 64], "score_pct": 6.5, "earned": 2, "total": 31, "time_used": 30, "results": {"49": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "55": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "58": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "63": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "64": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-14 in ac-omega (the context's namespace), 2 available, owner=ac", "Service ledger-14 port 9000 in the same namespace", "ns.txt has the namespace"]}}} {"at": "2026-09-12T22:17", "mode": "drill", "hard": false, "ids": [1, 6, 21, 34, 41], "score_pct": 87.5, "earned": 21, "total": 24, "time_used": 534, "results": {"1": {"pass": true, "earned": 6, "total": 6, "failed": []}, "6": {"pass": false, "earned": 3, "total": 6, "failed": ["CSR jane exists with signer kubernetes.io/kube-apiserver-client", "CSR jane approved and certificate issued"]}, "21": {"pass": true, "earned": 4, "total": 4, "failed": []}, "34": {"pass": true, "earned": 4, "total": 4, "failed": []}, "41": {"pass": true, "earned": 4, "total": 4, "failed": []}}} {"at": "2026-09-12T22:59", "mode": "drill", "hard": false, "ids": [17, 19, 37, 38, 39], "score_pct": 29.4, "earned": 10, "total": 34, "time_used": 1987, "results": {"17": {"pass": false, "earned": 4, "total": 8, "failed": ["kubelet is active", "kubelet is enabled (survives reboot)"]}, "19": {"pass": false, "earned": 1, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure", "pod crasher Running with DB_HOST set"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "38": {"pass": false, "earned": 0, "total": 7, "failed": ["kube-controller-manager manifest repaired and pod Running", "a new Deployment gets a ReplicaSet and running pods"]}, "39": {"pass": false, "earned": 4, "total": 7, "failed": ["kubelet config points at a CA file that exists", "kubelet active and enabled"]}}} {"at": "2026-09-13T01:06", "mode": "drill", "hard": false, "ids": [17, 19, 37, 38, 39], "score_pct": 29.4, "earned": 10, "total": 34, "time_used": 6649, "results": {"17": {"pass": false, "earned": 4, "total": 8, "failed": ["kubelet is active", "kubelet is enabled (survives reboot)"]}, "19": {"pass": false, "earned": 1, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure", "pod crasher Running with DB_HOST set"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "38": {"pass": false, "earned": 0, "total": 7, "failed": ["kube-controller-manager manifest repaired and pod Running", "a new Deployment gets a ReplicaSet and running pods"]}, "39": {"pass": false, "earned": 4, "total": 7, "failed": ["kubelet config points at a CA file that exists", "kubelet active and enabled"]}}} -{"at": "2026-09-14T16:49", "mode": "drill", "hard": false, "ids": [3, 5, 6, 8, 12, 13, 15, 16, 17, 18, 19, 20, 25, 28, 29, 30, 33, 37, 38, 39, 40, 42, 106, 112, 115, 120, 121], "score_pct": 42.0, "earned": 63, "total": 150, "time_used": 510, "results": {"17": {"pass": true, "earned": 8, "total": 8, "failed": []}, "20": {"pass": true, "earned": 8, "total": 8, "failed": []}, "3": {"pass": true, "earned": 5, "total": 5, "failed": []}, "5": {"pass": true, "earned": 5, "total": 5, "failed": []}, "8": {"pass": true, "earned": 4, "total": 4, "failed": []}, "12": {"pass": true, "earned": 4, "total": 4, "failed": []}, "38": {"pass": true, "earned": 7, "total": 7, "failed": []}, "18": {"pass": true, "earned": 7, "total": 7, "failed": []}, "19": {"pass": false, "earned": 4, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure"]}, "6": {"pass": true, "earned": 6, "total": 6, "failed": []}, "13": {"pass": false, "earned": 0, "total": 5, "failed": ["ingressClassName nginx and host shop.example.com", "/api (Prefix) -> api:8080", "/ (Prefix) -> web:80"]}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries workload=batch:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "30": {"pass": false, "earned": 0, "total": 4, "failed": ["container port 80 named svc-http", "Service port 8080 -> targetPort svc-http (by name)", "readiness probe uses the named port; endpoints ready"]}, "33": {"pass": false, "earned": 1, "total": 5, "failed": ["ops-19 has an aggregationRule with the right label", "aggregation populated rules from the two labelled roles", "sam can list pods and services cluster-wide"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "39": {"pass": false, "earned": 0, "total": 7, "failed": ["kubelet config points at a CA file that exists", "node Ready", "kubelet active and enabled"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~2h"]}, "42": {"pass": false, "earned": 0, "total": 4, "failed": ["applies to app=client-64, Egress only", "DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP", "TCP 443 to 172.16.0.0/16 only"]}, "106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret admin.lab.local-tls is kubernetes.io/tls with CN admin.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "115": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "120": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "121": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-73 in ac-bravo (the context's namespace), 2 available, owner=ac", "Service ledger-73 port 9000 in the same namespace", "ns.txt has the namespace"]}}} -{"at": "2026-09-15T20:55", "mode": "drill", "hard": false, "ids": [13, 15, 16, 19, 25, 28, 29, 30, 33, 37, 39, 40, 42, 106, 112, 115, 120, 121], "score_pct": 41.7, "earned": 40, "total": 96, "time_used": 608, "results": {"13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "19": {"pass": true, "earned": 6, "total": 6, "failed": []}, "33": {"pass": true, "earned": 5, "total": 5, "failed": []}, "30": {"pass": true, "earned": 4, "total": 4, "failed": []}, "37": {"pass": true, "earned": 6, "total": 6, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~6h"]}, "42": {"pass": true, "earned": 4, "total": 4, "failed": []}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries team=finance:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "115": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "120": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "121": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-21 in ac-echo (the context's namespace), 2 available, owner=ac", "Service ledger-21 port 9000 in the same namespace", "ns.txt has the namespace"]}}} +{"at": "2026-09-14T16:49", "mode": "drill", "hard": false, "ids": [3, 5, 6, 8, 12, 13, 15, 16, 17, 18, 19, 20, 25, 28, 29, 30, 33, 37, 38, 39, 40, 42, 49, 55, 58, 63, 64], "score_pct": 42.0, "earned": 63, "total": 150, "time_used": 510, "results": {"17": {"pass": true, "earned": 8, "total": 8, "failed": []}, "20": {"pass": true, "earned": 8, "total": 8, "failed": []}, "3": {"pass": true, "earned": 5, "total": 5, "failed": []}, "5": {"pass": true, "earned": 5, "total": 5, "failed": []}, "8": {"pass": true, "earned": 4, "total": 4, "failed": []}, "12": {"pass": true, "earned": 4, "total": 4, "failed": []}, "38": {"pass": true, "earned": 7, "total": 7, "failed": []}, "18": {"pass": true, "earned": 7, "total": 7, "failed": []}, "19": {"pass": false, "earned": 4, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure"]}, "6": {"pass": true, "earned": 6, "total": 6, "failed": []}, "13": {"pass": false, "earned": 0, "total": 5, "failed": ["ingressClassName nginx and host shop.example.com", "/api (Prefix) -> api:8080", "/ (Prefix) -> web:80"]}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries workload=batch:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "30": {"pass": false, "earned": 0, "total": 4, "failed": ["container port 80 named svc-http", "Service port 8080 -> targetPort svc-http (by name)", "readiness probe uses the named port; endpoints ready"]}, "33": {"pass": false, "earned": 1, "total": 5, "failed": ["ops-19 has an aggregationRule with the right label", "aggregation populated rules from the two labelled roles", "sam can list pods and services cluster-wide"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "39": {"pass": false, "earned": 0, "total": 7, "failed": ["kubelet config points at a CA file that exists", "node Ready", "kubelet active and enabled"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~2h"]}, "42": {"pass": false, "earned": 0, "total": 4, "failed": ["applies to app=client-64, Egress only", "DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP", "TCP 443 to 172.16.0.0/16 only"]}, "49": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "55": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret admin.lab.local-tls is kubernetes.io/tls with CN admin.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "58": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "63": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "64": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-73 in ac-bravo (the context's namespace), 2 available, owner=ac", "Service ledger-73 port 9000 in the same namespace", "ns.txt has the namespace"]}}} +{"at": "2026-09-15T20:55", "mode": "drill", "hard": false, "ids": [13, 15, 16, 19, 25, 28, 29, 30, 33, 37, 39, 40, 42, 49, 55, 58, 63, 64], "score_pct": 41.7, "earned": 40, "total": 96, "time_used": 608, "results": {"13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "19": {"pass": true, "earned": 6, "total": 6, "failed": []}, "33": {"pass": true, "earned": 5, "total": 5, "failed": []}, "30": {"pass": true, "earned": 4, "total": 4, "failed": []}, "37": {"pass": true, "earned": 6, "total": 6, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~6h"]}, "42": {"pass": true, "earned": 4, "total": 4, "failed": []}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries team=finance:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "49": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "55": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "58": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "63": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "64": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-21 in ac-echo (the context's namespace), 2 available, owner=ac", "Service ledger-21 port 9000 in the same namespace", "ns.txt has the namespace"]}}} {"at": "2026-09-16T19:40", "mode": "drill", "hard": false, "ids": [15, 16, 25, 28, 29, 40], "score_pct": 50.0, "earned": 14, "total": 28, "time_used": 1258, "results": {"15": {"pass": true, "earned": 6, "total": 6, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "29": {"pass": true, "earned": 4, "total": 4, "failed": []}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~24h"]}}} -{"at": "2026-09-19T12:27", "mode": "drill", "hard": false, "ids": [106, 217, 34, 202, 102, 301, 39, 13, 32, 204, 25, 2, 8, 112, 4, 109, 313, 9], "score_pct": 39.7, "earned": 48, "total": 121, "time_used": 4197, "results": {"8": {"pass": true, "earned": 4, "total": 4, "failed": []}, "34": {"pass": true, "earned": 4, "total": 4, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "301": {"pass": true, "earned": 9, "total": 9, "failed": []}, "4": {"pass": true, "earned": 4, "total": 4, "failed": []}, "9": {"pass": true, "earned": 4, "total": 4, "failed": []}, "106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "217": {"pass": false, "earned": 0, "total": 10, "failed": ["Pod tiger-probe Running with both labels", "container.txt holds a real container ID for that Pod", "container.txt records the runtimeType", "container.log holds the httpd startup output"]}, "202": {"pass": false, "earned": 3, "total": 8, "failed": ["watchAllNamespaces=true reached the pod env", "Ledger main-ledger has currency USD and retentionDays 30"]}, "102": {"pass": false, "earned": 2, "total": 10, "failed": ["ConfigMap restore-proof-67 exists again with its original data", "etcd uses a restored data directory (manifest changed)"]}, "32": {"pass": false, "earned": 0, "total": 4, "failed": ["ResourceQuota limits pods/6, requests.cpu 1, limits.memory 2Gi", "LimitRange defaults set", "worker pods Running with defaulted requests"]}, "204": {"pass": false, "earned": 0, "total": 8, "failed": ["evicted-first.txt lists exactly the BestEffort pods", "api-qos.txt says Burstable", "guaranteed.txt names orion-ledger"]}, "2": {"pass": false, "earned": 0, "total": 7, "failed": ["/opt/cka/etcd-snapshot.db exists on the control-plane node", "snapshot is a real etcd backup (> 100 KB)"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret secure.lab.local-tls is kubernetes.io/tls with CN secure.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "109": {"pass": false, "earned": 2, "total": 8, "failed": ["native sidecar: initContainer agent with restartPolicy Always", "probes on app as specified", "Running, shared log written by agent, grace period 5"]}, "313": {"pass": false, "earned": 0, "total": 11, "failed": ["Secret classic-tls is a TLS secret with CN classic.r700.io", "both host rules route to the right Services on /", "TLS covers both hosts with classic-tls", "defaultBackend points at web-classic:80, class is nginx"]}}} -{"at": "2026-09-20T17:32", "mode": "exam", "hard": true, "ids": [401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418], "score_pct": 100.0, "earned": 163, "total": 163, "time_used": 2909, "results": {"402": {"pass": true, "earned": 9, "total": 9, "failed": []}, "404": {"pass": true, "earned": 9, "total": 9, "failed": []}, "405": {"pass": true, "earned": 11, "total": 11, "failed": []}, "401": {"pass": true, "earned": 8, "total": 8, "failed": []}, "403": {"pass": true, "earned": 9, "total": 9, "failed": []}, "406": {"pass": true, "earned": 7, "total": 7, "failed": []}, "407": {"pass": true, "earned": 10, "total": 10, "failed": []}, "408": {"pass": true, "earned": 9, "total": 9, "failed": []}, "409": {"pass": true, "earned": 9, "total": 9, "failed": []}, "410": {"pass": true, "earned": 8, "total": 8, "failed": []}, "411": {"pass": true, "earned": 7, "total": 7, "failed": []}, "412": {"pass": true, "earned": 11, "total": 11, "failed": []}, "413": {"pass": true, "earned": 10, "total": 10, "failed": []}, "414": {"pass": true, "earned": 9, "total": 9, "failed": []}, "415": {"pass": true, "earned": 9, "total": 9, "failed": []}, "416": {"pass": true, "earned": 10, "total": 10, "failed": []}, "417": {"pass": true, "earned": 10, "total": 10, "failed": []}, "418": {"pass": true, "earned": 8, "total": 8, "failed": []}}, "synthetic": true, "note": "Scripted end-to-end verification of Set C (tasks solved programmatically), not a person sitting it"} +{"at": "2026-09-19T12:27", "mode": "drill", "hard": false, "ids": [49, 217, 34, 202, 45, 301, 39, 13, 32, 204, 25, 2, 8, 55, 4, 52, 313, 9], "score_pct": 39.7, "earned": 48, "total": 121, "time_used": 4197, "results": {"8": {"pass": true, "earned": 4, "total": 4, "failed": []}, "34": {"pass": true, "earned": 4, "total": 4, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "301": {"pass": true, "earned": 9, "total": 9, "failed": []}, "4": {"pass": true, "earned": 4, "total": 4, "failed": []}, "9": {"pass": true, "earned": 4, "total": 4, "failed": []}, "49": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "217": {"pass": false, "earned": 0, "total": 10, "failed": ["Pod tiger-probe Running with both labels", "container.txt holds a real container ID for that Pod", "container.txt records the runtimeType", "container.log holds the httpd startup output"]}, "202": {"pass": false, "earned": 3, "total": 8, "failed": ["watchAllNamespaces=true reached the pod env", "Ledger main-ledger has currency USD and retentionDays 30"]}, "45": {"pass": false, "earned": 2, "total": 10, "failed": ["ConfigMap restore-proof-67 exists again with its original data", "etcd uses a restored data directory (manifest changed)"]}, "32": {"pass": false, "earned": 0, "total": 4, "failed": ["ResourceQuota limits pods/6, requests.cpu 1, limits.memory 2Gi", "LimitRange defaults set", "worker pods Running with defaulted requests"]}, "204": {"pass": false, "earned": 0, "total": 8, "failed": ["evicted-first.txt lists exactly the BestEffort pods", "api-qos.txt says Burstable", "guaranteed.txt names orion-ledger"]}, "2": {"pass": false, "earned": 0, "total": 7, "failed": ["/opt/cka/etcd-snapshot.db exists on the control-plane node", "snapshot is a real etcd backup (> 100 KB)"]}, "55": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret secure.lab.local-tls is kubernetes.io/tls with CN secure.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "52": {"pass": false, "earned": 2, "total": 8, "failed": ["native sidecar: initContainer agent with restartPolicy Always", "probes on app as specified", "Running, shared log written by agent, grace period 5"]}, "313": {"pass": false, "earned": 0, "total": 11, "failed": ["Secret classic-tls is a TLS secret with CN classic.r700.io", "both host rules route to the right Services on /", "TLS covers both hosts with classic-tls", "defaultBackend points at web-classic:80, class is nginx"]}}} +{"at": "2026-09-20T17:32", "mode": "exam", "hard": true, "ids": [68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85], "score_pct": 100.0, "earned": 163, "total": 163, "time_used": 2909, "results": {"69": {"pass": true, "earned": 9, "total": 9, "failed": []}, "71": {"pass": true, "earned": 9, "total": 9, "failed": []}, "72": {"pass": true, "earned": 11, "total": 11, "failed": []}, "68": {"pass": true, "earned": 8, "total": 8, "failed": []}, "70": {"pass": true, "earned": 9, "total": 9, "failed": []}, "73": {"pass": true, "earned": 7, "total": 7, "failed": []}, "74": {"pass": true, "earned": 10, "total": 10, "failed": []}, "75": {"pass": true, "earned": 9, "total": 9, "failed": []}, "76": {"pass": true, "earned": 9, "total": 9, "failed": []}, "77": {"pass": true, "earned": 8, "total": 8, "failed": []}, "78": {"pass": true, "earned": 7, "total": 7, "failed": []}, "79": {"pass": true, "earned": 11, "total": 11, "failed": []}, "80": {"pass": true, "earned": 10, "total": 10, "failed": []}, "81": {"pass": true, "earned": 9, "total": 9, "failed": []}, "82": {"pass": true, "earned": 9, "total": 9, "failed": []}, "83": {"pass": true, "earned": 10, "total": 10, "failed": []}, "84": {"pass": true, "earned": 10, "total": 10, "failed": []}, "85": {"pass": true, "earned": 8, "total": 8, "failed": []}}, "synthetic": true, "note": "Scripted end-to-end verification of Set C (tasks solved programmatically), not a person sitting it"} diff --git a/setup.sh b/setup.sh index 263d863..743e0ed 100755 --- a/setup.sh +++ b/setup.sh @@ -88,7 +88,7 @@ fi # copied out of the running etcd pod (that image is distroless), so install them properly. echo "==> Installing etcdctl/etcdutl on the control-plane node..." printf ' ' -CONTAINER_RUNTIME="$RT" bash ./etcd-tools.sh || echo " warn: no etcd client on the control-plane node; Q02/Q102 will say so when they seed" +CONTAINER_RUNTIME="$RT" bash ./etcd-tools.sh || echo " warn: no etcd client on the control-plane node; Q02/Q45 will say so when they seed" # remember the runtime so the CLI uses the matching exec command mkdir -p .state && echo "$RT" > .state/runtime diff --git a/tracks/cka/bank_extra.py b/tracks/cka/bank_extra.py index c67a8c8..3f67291 100644 --- a/tracks/cka/bank_extra.py +++ b/tracks/cka/bank_extra.py @@ -282,7 +282,7 @@ $K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null }, { "id": 26, "domain": "WORK", "title": "DaemonSet on every node", - # needs every node schedulable: Q25/Q107 taint W1 and must keep the taint + # needs every node schedulable: Q25/Q50 taint W1 and must keep the taint "breaks": ["w1-taint"], "params": lambda r: {"ns": ns_pick(r, "agents"), "ds": name_pick(r, "node-agent")}, "task": """ diff --git a/tracks/cka/bank_hard.py b/tracks/cka/bank_hard.py index f44ad3b..611f325 100644 --- a/tracks/cka/bank_hard.py +++ b/tracks/cka/bank_hard.py @@ -1,4 +1,4 @@ -"""Hard-difficulty CKA tasks (ids 101-122): multi-part, no hints, traps, exact output, +"""Hard-difficulty CKA tasks (ids 44-67): multi-part, no hints, traps, exact output, compound failures, collateral-damage checks, exam-style contexts. Modelled on the kind of difficulty people report from killer.sh, not on its content.""" from tracks.cka.common import C, NS, ns_pick, name_pick @@ -11,7 +11,7 @@ def ctx_pick(r): QUESTIONS = [ # ================================================================ ARCH ==== { - "id": 101, "domain": "ARCH", "difficulty": "hard", "title": "Exact RBAC with an existing over-grant", + "id": 44, "domain": "ARCH", "difficulty": "hard", "title": "Exact RBAC with an existing over-grant", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "ci"), "sa": name_pick(r, "ci-bot")}, "task": CTX + """ ServiceAccount `%{sa}` in namespace `%{ns}` is used by a CI system. Its permissions must be @@ -41,7 +41,7 @@ eq "$($K auth can-i create pods --subresource=exec -n %{ns} --as=system:servicea ], }, { - "id": 102, "domain": "ARCH", "breaks": "etcd", "difficulty": "hard", "title": "etcd restore", + "id": 45, "domain": "ARCH", "breaks": "etcd", "difficulty": "hard", "title": "etcd restore", # solo: restoring rolls EVERY API object back to the snapshot - other tasks' seeds and answers too "solo": True, "params": lambda r: {"ctx": "k8s-c1-H", "cm": name_pick(r, "restore-proof")}, @@ -65,13 +65,13 @@ $K -n default create cm %{cm} --from-literal=proof=$(date +%s)-%{cm} >/dev/null out=$(nx "$CP" 'etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/cka/backup.db' 2>&1) \ || { echo "snapshot save on $CP failed: $out"; exit 1; } sleep 2; $K -n default delete cm %{cm} >/dev/null -nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' "$WS/.q102-datadir" -date +%s > "$WS/.q102-t0" +nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' "$WS/.q45-datadir" +date +%s > "$WS/.q45-t0" """, "checks": [ C("ConfigMap %{cm} exists again with its original data", 6, r"""j -n default cm %{cm} '{.data.proof}' | grep -q -- '-%{cm}$'"""), C("etcd runs on a data directory restored from the snapshot", 2, r""" -was=$(cat "$WS/.q102-datadir" 2>/dev/null); t0=$(cat "$WS/.q102-t0" 2>/dev/null); [ -n "$was" ] && [ -n "$t0" ] || exit 1 +was=$(cat "$WS/.q45-datadir" 2>/dev/null); t0=$(cat "$WS/.q45-t0" 2>/dev/null); [ -n "$was" ] && [ -n "$t0" ] || exit 1 now=$(nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' /dev/null; $K -n %{ns} delete rolebinding --all >/dev/null 2>&1""", + "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q47"; mkdir -p "$WS/q47"; $K delete csr %{user} --ignore-not-found >/dev/null; $K -n %{ns} delete rolebinding --all >/dev/null 2>&1""", "checks": [ C("CSR %{user} approved with certificate, O=developers", 2, r"""[ -n "$(j csr %{user} '{.status.certificate}')" ] && j csr %{user} '{.status.certificate}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'O *= *developers'"""), - C("kubeconfig is self-contained and works without flags", 3, r"""f="$WS/q104/%{user}.kubeconfig"; [ -f "$f" ] || exit 1; grep -q 'client-certificate-data' "$f" && grep -q 'certificate-authority-data' "$f" && ! grep -q 'client-certificate:' "$f" && kubectl --kubeconfig "$f" get pods >/dev/null 2>&1"""), - C("context %{user}@lab defaults to %{ns}", 1, r"""eq "$(kubectl --kubeconfig "$WS/q104/%{user}.kubeconfig" config view -o jsonpath='{.contexts[?(@.name=="%{user}@lab")].context.namespace}')" %{ns}"""), + C("kubeconfig is self-contained and works without flags", 3, r"""f="$WS/q47/%{user}.kubeconfig"; [ -f "$f" ] || exit 1; grep -q 'client-certificate-data' "$f" && grep -q 'certificate-authority-data' "$f" && ! grep -q 'client-certificate:' "$f" && kubectl --kubeconfig "$f" get pods >/dev/null 2>&1"""), + C("context %{user}@lab defaults to %{ns}", 1, r"""eq "$(kubectl --kubeconfig "$WS/q47/%{user}.kubeconfig" config view -o jsonpath='{.contexts[?(@.name=="%{user}@lab")].context.namespace}')" %{ns}"""), C("edit rights in %{ns}, none elsewhere", 2, r"""eq "$($K auth can-i create deployments -n %{ns} --as=%{user} --as-group=developers 2>/dev/null)" yes && eq "$($K auth can-i delete pods -n %{ns} --as=%{user} --as-group=developers 2>/dev/null)" yes && eq "$($K auth can-i get pods -n default --as=%{user} --as-group=developers 2>/dev/null || true)" no && eq "$($K auth can-i get secrets -n kube-system --as=%{user} --as-group=developers 2>/dev/null || true)" no"""), ], }, { - "id": 105, "domain": "ARCH", "difficulty": "hard", "title": "Kustomize with generators and patches", + "id": 48, "domain": "ARCH", "difficulty": "hard", "title": "Kustomize with generators and patches", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "kz"), "rep": r.choice([3, 4]), "lvl": r.choice(["debug", "trace", "warn"])}, "task": CTX + """ -Base at `{WS}/q105/base` (Deployment `web`, Service `web`). Build overlay `{WS}/q105/overlays/prod` that: +Base at `{WS}/q48/base` (Deployment `web`, Service `web`). Build overlay `{WS}/q48/overlays/prod` that: - targets namespace `%{ns}` and adds label `tier=prod` to all resources (labels only, not selectors), - sets replicas to %{rep} via a **strategic merge patch** file `replicas.yaml`, - adds a `configMapGenerator` named `web-config` with literal `LOG_LEVEL=%{lvl}` (hash suffix enabled), @@ -140,11 +140,11 @@ Base at `{WS}/q105/base` (Deployment `web`, Service `web`). Build overlay `{WS}/ Apply with `kubectl apply -k`. The Deployment must reference the hashed ConfigMap name. """, "setup": r""" -rm -rf "$WS/q105"; mkdir -p "$WS/q105/base" -cat > "$WS/q105/base/kustomization.yaml" <<'EOF' +rm -rf "$WS/q48"; mkdir -p "$WS/q48/base" +cat > "$WS/q48/base/kustomization.yaml" <<'EOF' resources: [deployment.yaml, service.yaml] EOF -cat > "$WS/q105/base/deployment.yaml" <<'EOF' +cat > "$WS/q48/base/deployment.yaml" <<'EOF' apiVersion: apps/v1 kind: Deployment metadata: {name: web} @@ -156,7 +156,7 @@ spec: spec: containers: [{name: web, image: nginx:1.25, ports: [{containerPort: 80}]}] EOF -cat > "$WS/q105/base/service.yaml" <<'EOF' +cat > "$WS/q48/base/service.yaml" <<'EOF' apiVersion: v1 kind: Service metadata: {name: web} @@ -165,14 +165,14 @@ EOF $K delete ns %{ns} --ignore-not-found --wait=false >/dev/null """, "checks": [ - C("overlay has replicas.yaml (SMP) and envfrom.yaml (JSON6902)", 1, r"""[ -f "$WS/q105/overlays/prod/replicas.yaml" ] && [ -f "$WS/q105/overlays/prod/envfrom.yaml" ] && grep -q 'op:' "$WS/q105/overlays/prod/envfrom.yaml" """), + C("overlay has replicas.yaml (SMP) and envfrom.yaml (JSON6902)", 1, r"""[ -f "$WS/q48/overlays/prod/replicas.yaml" ] && [ -f "$WS/q48/overlays/prod/envfrom.yaml" ] && grep -q 'op:' "$WS/q48/overlays/prod/envfrom.yaml" """), C("deployed to %{ns}: %{rep} replicas, tier=prod label, selector untouched", 2, r"""d="$K -n %{ns} get deploy web"; eq "$($d -o jsonpath='{.spec.replicas}' 2>/dev/null)" %{rep} && eq "$($d -o jsonpath='{.metadata.labels.tier}')" prod && [ -z "$($d -o jsonpath='{.spec.selector.matchLabels.tier}')" ]"""), C("hashed ConfigMap with LOG_LEVEL=%{lvl}, referenced via envFrom", 3, r"""cm=$(j -n %{ns} deploy web '{.spec.template.spec.containers[0].envFrom[0].configMapRef.name}'); [ -n "$cm" ] && echo "$cm" | grep -Eq '^web-config-[a-z0-9]{6,}$' && eq "$(j -n %{ns} cm "$cm" '{.data.LOG_LEVEL}')" %{lvl}"""), C("pods available and env visible", 1, r"""eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" %{rep} && $K -n %{ns} exec deploy/web -- env 2>/dev/null | grep -q '^LOG_LEVEL=%{lvl}$'"""), ], }, { - "id": 106, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults", + "id": 49, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults", "params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])}, "task": CTX + """ Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the kind in lowercase plus @@ -182,44 +182,44 @@ Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the k - `spec.image` string, required. Then: 1. Create `%{kind}` `good` in `default` with replicas 3, image `nginx:1.27`, environment `%{env}`. -2. Save the API server's rejection message for a `%{kind}` with `replicas: 0` to `{WS}/q106/rejected.txt`. +2. Save the API server's rejection message for a `%{kind}` with `replicas: 0` to `{WS}/q49/rejected.txt`. 3. Create `%{kind}` `defaulted` in `default` with only replicas 2 and image set; its environment must be `dev` after creation. This context's default namespace is not necessarily `default` — say `-n default` explicitly. """, - "setup": r"""rm -rf "$WS/q106"; mkdir -p "$WS/q106"; k=$(echo %{kind} | tr A-Z a-z); $K delete crd ${k}s.deploy.example.com --ignore-not-found >/dev/null""", + "setup": r"""rm -rf "$WS/q49"; mkdir -p "$WS/q49"; k=$(echo %{kind} | tr A-Z a-z); $K delete crd ${k}s.deploy.example.com --ignore-not-found >/dev/null""", "checks": [ C("CRD exists, Namespaced, replicas has min/max and is required", 2, r"""s=""; k=$(echo %{kind} | tr A-Z a-z); c="$K get crd ${k}s.deploy.example.com"; eq "$($c -o jsonpath='{.spec.scope}' 2>/dev/null)" Namespaced && s=$($c -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec}'); echo "$s" | grep -q '"minimum":1' && echo "$s" | grep -q '"maximum":10' && echo "$s" | grep -q '"required":\[' """), C("good created with environment %{env}", 1, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com good '{.spec.environment}')" %{env}"""), - C("replicas 0 is rejected and the message was saved", 2, r"""grep -qiE 'minimum|greater than|>= 1|invalid' "$WS/q106/rejected.txt" 2>/dev/null"""), + C("replicas 0 is rejected and the message was saved", 2, r"""grep -qiE 'minimum|greater than|>= 1|invalid' "$WS/q49/rejected.txt" 2>/dev/null"""), C("defaulting works: defaulted.spec.environment == dev", 2, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com defaulted '{.spec.environment}')" dev"""), ], }, { - "id": 122, "domain": "ARCH", "breaks": "certs", "difficulty": "hard", "title": "Renew control-plane certificates", + "id": 65, "domain": "ARCH", "breaks": "certs", "difficulty": "hard", "title": "Renew control-plane certificates", "params": lambda r: {"ctx": "k8s-c1-H"}, "task": CTX + """ Security policy requires the control-plane certificates to be renewed now. On the control-plane node, renew **all** kubeadm-managed certificates, make every control-plane component pick up the new certificates, and update the admin kubeconfig on the node (`/etc/kubernetes/admin.conf`) so it uses the renewed client certificate. -Write the new `apiserver` certificate's expiry (as printed by `kubeadm certs check-expiration`) to `{WS}/q122/expiry.txt`. +Write the new `apiserver` certificate's expiry (as printed by `kubeadm certs check-expiration`) to `{WS}/q65/expiry.txt`. """, - "setup": r"""rm -rf "$WS/q122"; mkdir -p "$WS/q122"; nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' > "$WS/.q122-before" """, + "setup": r"""rm -rf "$WS/q65"; mkdir -p "$WS/q65"; nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' > "$WS/.q65-before" """, "checks": [ - C("apiserver certificate was renewed (newer notBefore)", 3, r"""b=$(sed 's/notBefore=//' "$WS/.q122-before"); a=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' | sed 's/notBefore=//'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""), + C("apiserver certificate was renewed (newer notBefore)", 3, r"""b=$(sed 's/notBefore=//' "$WS/.q65-before"); a=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' | sed 's/notBefore=//'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""), C("all components restarted and Running with the new certs (apiserver serving cert matches file)", 3, r""" for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done srv=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial -startdate'); f=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial -startdate'); [ -n "$srv" ] && eq "$srv" "$f" || exit 1 # ...and it is the RENEWED certificate: an untouched cluster also serves the file it has on disk -b=$(sed 's/notBefore=//' "$WS/.q122-before"); a=$(echo "$srv" | sed -n 's/notBefore=//p'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""), - C("admin.conf uses the renewed client cert", 2, r"""nx "$CP" 'grep client-certificate-data /etc/kubernetes/admin.conf | awk "{print \$2}" | base64 -d | openssl x509 -noout -startdate' | sed 's/notBefore=//' > /tmp/q122a; b=$(sed 's/notBefore=//' "$WS/.q122-before"); [ "$(date -d "$(cat /tmp/q122a)" +%s)" -gt "$(date -d "$b" +%s)" ]"""), - C("expiry.txt matches kubeadm's report for apiserver", 1, r"""exp=$(nx "$CP" 'kubeadm certs check-expiration 2>/dev/null' | awk '$1=="apiserver"{print $2,$3,$4,$5,$6}'); [ -n "$exp" ] && grep -qF "$exp" "$WS/q122/expiry.txt" 2>/dev/null"""), +b=$(sed 's/notBefore=//' "$WS/.q65-before"); a=$(echo "$srv" | sed -n 's/notBefore=//p'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""), + C("admin.conf uses the renewed client cert", 2, r"""nx "$CP" 'grep client-certificate-data /etc/kubernetes/admin.conf | awk "{print \$2}" | base64 -d | openssl x509 -noout -startdate' | sed 's/notBefore=//' > /tmp/q65a; b=$(sed 's/notBefore=//' "$WS/.q65-before"); [ "$(date -d "$(cat /tmp/q65a)" +%s)" -gt "$(date -d "$b" +%s)" ]"""), + C("expiry.txt matches kubeadm's report for apiserver", 1, r"""exp=$(nx "$CP" 'kubeadm certs check-expiration 2>/dev/null' | awk '$1=="apiserver"{print $2,$3,$4,$5,$6}'); [ -n "$exp" ] && grep -qF "$exp" "$WS/q65/expiry.txt" 2>/dev/null"""), ], }, { - "id": 124, "domain": "ARCH", "difficulty": "hard", "title": "Prepare a node join without shell access to the control plane", + "id": 67, "domain": "ARCH", "difficulty": "hard", "title": "Prepare a node join without shell access to the control plane", "params": lambda r: {"ctx": "k8s-c1-H"}, "task": CTX + """ A new worker node is about to join this cluster, but you can only hand the networking team a @@ -232,7 +232,7 @@ A new worker node is about to join this cluster, but you can only hand the netwo 3. Determine the API server's real address as seen **from inside the cluster network** — not the `127.0.0.1:` your local kubeconfig uses to reach it from this machine, which a joining node cannot use. -4. Write exactly these three lines to `{WS}/q124/node-join.env`: +4. Write exactly these three lines to `{WS}/q67/node-join.env`: ``` JOIN_TOKEN= CA_HASH=sha256: @@ -242,27 +242,27 @@ Every value must be genuine: plugged into `kubeadm join $API_ENDPOINT --token $J --discovery-token-ca-cert-hash $CA_HASH` right now, it would work. """, "setup": r""" -rm -rf "$WS/q124"; mkdir -p "$WS/q124" +rm -rf "$WS/q67"; mkdir -p "$WS/q67" for t in $($K -n kube-system get secrets -o jsonpath='{.items[?(@.type=="bootstrap.kubernetes.io/token")].metadata.name}'); do $K -n kube-system delete secret $t >/dev/null; done """, "checks": [ C("a real, unexpired bootstrap token", 2, r""" -tok=$(grep -oE '^JOIN_TOKEN=[a-z0-9]{6}\.[a-z0-9]{16}' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2); [ -n "$tok" ] || exit 1 +tok=$(grep -oE '^JOIN_TOKEN=[a-z0-9]{6}\.[a-z0-9]{16}' "$WS/q67/node-join.env" 2>/dev/null | cut -d= -f2); [ -n "$tok" ] || exit 1 id=${tok%%.*}; exp=$(j -n kube-system secret bootstrap-token-$id '{.data.expiration}' | base64 -d 2>/dev/null); [ -n "$exp" ] || exit 1 [ "$(date -d "$exp" +%s)" -gt "$(date +%s)" ]"""), C("CA_HASH computed correctly (matches the real CA's SPKI SHA-256)", 3, r""" want=$(nx "$CP" "openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'") -got=$(grep -oE '^CA_HASH=sha256:[0-9a-f]{64}' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2 | cut -d: -f2) +got=$(grep -oE '^CA_HASH=sha256:[0-9a-f]{64}' "$WS/q67/node-join.env" 2>/dev/null | cut -d= -f2 | cut -d: -f2) [ -n "$want" ] && eq "$got" "$want" """), C("API_ENDPOINT is the real in-cluster address, not the local kubeconfig's", 2, r""" want=$($K get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}' 2>/dev/null) -got=$(grep -oE '^API_ENDPOINT=.*' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2 | tr -d '[:space:]') +got=$(grep -oE '^API_ENDPOINT=.*' "$WS/q67/node-join.env" 2>/dev/null | cut -d= -f2 | tr -d '[:space:]') [ -n "$want" ] && eq "$got" "$want" """), ], }, # ================================================================ WORK ==== { - "id": 107, "domain": "WORK", "breaks": "w1-taint", "difficulty": "hard", "title": "Spread, tolerate, prioritise — under a quota", + "id": 50, "domain": "WORK", "breaks": "w1-taint", "difficulty": "hard", "title": "Spread, tolerate, prioritise — under a quota", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "spread"), "dep": name_pick(r, "gateway"), "key": r.choice(["pool", "lane"]), "val": r.choice(["ingress", "edge"])}, "task": CTX + """ Node `{W1}` is tainted `%{key}=%{val}:NoSchedule`. In namespace `%{ns}` create Deployment `%{dep}` @@ -294,7 +294,7 @@ EOF ], }, { - "id": 108, "domain": "WORK", "difficulty": "hard", "title": "Rollback to a specific revision, then pause", + "id": 51, "domain": "WORK", "difficulty": "hard", "title": "Rollback to a specific revision, then pause", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "rel"), "dep": name_pick(r, "portal")}, "task": CTX + """ Deployment `%{dep}` in `%{ns}` has gone through several releases; the current one is broken. @@ -330,7 +330,7 @@ sys.exit(1)'"""), ], }, { - "id": 109, "domain": "WORK", "difficulty": "hard", "title": "Native sidecar with probes and an exact report", + "id": 52, "domain": "WORK", "difficulty": "hard", "title": "Native sidecar with probes and an exact report", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "side"), "pod": name_pick(r, "collector")}, "task": CTX + """ Create pod `%{pod}` in `%{ns}`: @@ -342,33 +342,33 @@ Create pod `%{pod}` in `%{ns}`: own logs under `/var/log/nginx`, and an emptyDir mounted over `/var/log` hides that directory), - `terminationGracePeriodSeconds: 5`. When it is Running, write the names of **all** containers of the pod (init and regular), sorted, -one per line, to `{WS}/q109/containers.txt`, and the pod's `.status.podIP` to `{WS}/q109/ip.txt`. +one per line, to `{WS}/q52/containers.txt`, and the pod's `.status.podIP` to `{WS}/q52/ip.txt`. """, - "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q109"; mkdir -p "$WS/q109"; $K -n %{ns} delete pod %{pod} --ignore-not-found --force --grace-period=0 --wait=true >/dev/null 2>&1""", + "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q52"; mkdir -p "$WS/q52"; $K -n %{ns} delete pod %{pod} --ignore-not-found --force --grace-period=0 --wait=true >/dev/null 2>&1""", "checks": [ C("native sidecar: initContainer agent with restartPolicy Always", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.spec.initContainers[?(@.name=="agent")].restartPolicy}')" Always"""), C("probes on app as specified", 2, r"""c='{.spec.containers[?(@.name=="app")]'; eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.httpGet.path}")" / && eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.periodSeconds}")" 5 && eq "$(j -n %{ns} pod %{pod} "$c.livenessProbe.tcpSocket.port}")" 80"""), C("Running, shared log written by agent, grace period 5", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.terminationGracePeriodSeconds}')" 5 && [ -n "$($K -n %{ns} exec %{pod} -c app -- sh -c 'tail -1 /var/log/agent/agent.log' 2>/dev/null)" ]"""), - C("report files exact", 2, r"""want=$( (j -n %{ns} pod %{pod} '{.spec.initContainers[*].name}'; echo; j -n %{ns} pod %{pod} '{.spec.containers[*].name}') | tr ' ' '\n' | grep . | sort); [ -n "$want" ] && [ "$(grep . "$WS/q109/containers.txt" 2>/dev/null | sort)" = "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q109/ip.txt" 2>/dev/null)" "$(j -n %{ns} pod %{pod} '{.status.podIP}')" """), + C("report files exact", 2, r"""want=$( (j -n %{ns} pod %{pod} '{.spec.initContainers[*].name}'; echo; j -n %{ns} pod %{pod} '{.spec.containers[*].name}') | tr ' ' '\n' | grep . | sort); [ -n "$want" ] && [ "$(grep . "$WS/q52/containers.txt" 2>/dev/null | sort)" = "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q52/ip.txt" 2>/dev/null)" "$(j -n %{ns} pod %{pod} '{.status.podIP}')" """), ], }, { # Scoped to its own namespace: the cluster-wide version compared a file written mid-exam with the # whole cluster at submit time, so every pod another task (or a grader probe) created or deleted # in between failed it. killer.sh-style exact-output tasks work on data the task owns. - "id": 110, "domain": "WORK", "difficulty": "hard", "title": "Namespace audit to exact files", + "id": 53, "domain": "WORK", "difficulty": "hard", "title": "Namespace audit to exact files", "params": lambda r: {"ctx": "k8s-c1-H", "ns": ns_pick(r, "audit")}, "task": CTX + """ Namespace `%{ns}` runs a mix of workloads. Produce three files with contents **exactly** as specified — no headers, one entry per line, sorted with `sort`: -1. `{WS}/q110/no-readiness.txt` — the name of every Pod in `%{ns}` whose **first** container has +1. `{WS}/q53/no-readiness.txt` — the name of every Pod in `%{ns}` whose **first** container has no readiness probe (whatever the Pod's phase). -2. `{WS}/q110/per-node.txt` — ` `: how many Pods in `%{ns}` are **Running** on each +2. `{WS}/q53/per-node.txt` — ` `: how many Pods in `%{ns}` are **Running** on each node, listing only nodes that have at least one. -3. `{WS}/q110/images.txt` — every distinct container image used by Pods in `kube-system`. +3. `{WS}/q53/images.txt` — every distinct container image used by Pods in `kube-system`. """, "setup": NS + r""" -mkns %{ns}; rm -rf "$WS/q110"; mkdir -p "$WS/q110" +mkns %{ns}; rm -rf "$WS/q53"; mkdir -p "$WS/q53" $K -n %{ns} delete pod --all --force --grace-period=0 --wait=true >/dev/null 2>&1 $K apply -f - >/dev/null </dev/null)" = "$want" ]"""), + if not p["spec"]["containers"][0].get("readinessProbe"): print(p["metadata"]["name"])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q53/no-readiness.txt" 2>/dev/null)" = "$want" ]"""), C("per-node.txt exact (Running pods only)", 2, r"""want=$($K -n %{ns} get pods -o json 2>/dev/null | python3 -c ' import sys,json,collections c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"] if p["status"]["phase"]=="Running" and p["spec"].get("nodeName")) -for n in c: print(n, c[n])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q110/per-node.txt" 2>/dev/null)" = "$want" ]"""), - C("images.txt exact", 2, r"""want=$(j -n kube-system pods '{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u); [ -n "$want" ] && [ "$(grep . "$WS/q110/images.txt" 2>/dev/null)" = "$want" ]"""), +for n in c: print(n, c[n])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q53/per-node.txt" 2>/dev/null)" = "$want" ]"""), + C("images.txt exact", 2, r"""want=$(j -n kube-system pods '{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u); [ -n "$want" ] && [ "$(grep . "$WS/q53/images.txt" 2>/dev/null)" = "$want" ]"""), ], }, { - "id": 121, "domain": "WORK", "difficulty": "hard", "title": "The context's namespace", + "id": 64, "domain": "WORK", "difficulty": "hard", "title": "The context's namespace", # ctx_ns: this task reads the context's own default namespace instead of being told one, so # k8s-c2-AC must be created pointing at *this* task's ns (see context_namespace in ./exam). "ctx_ns": True, @@ -440,19 +440,19 @@ for n in c: print(n, c[n])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q110/per In the **current namespace of that context** (do not pass `-n`), create Deployment `%{dep}` (busybox:1.36, `sleep 3600`, 2 replicas, label `owner=ac`) and a ClusterIP Service `%{dep}` on port 9000 targeting port 9000. Then write the namespace name you deployed into, as reported -by `kubectl config view --minify`, to `{WS}/q121/ns.txt`. +by `kubectl config view --minify`, to `{WS}/q64/ns.txt`. """, - "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q121"; mkdir -p "$WS/q121"; $K -n %{ns} delete deploy,svc %{dep} --ignore-not-found >/dev/null; $K -n default delete deploy,svc %{dep} --ignore-not-found >/dev/null""", + "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q64"; mkdir -p "$WS/q64"; $K -n %{ns} delete deploy,svc %{dep} --ignore-not-found >/dev/null; $K -n default delete deploy,svc %{dep} --ignore-not-found >/dev/null""", "checks": [ C("Deployment %{dep} in %{ns} (the context's namespace), 2 available, owner=ac", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && { eq "$(j -n %{ns} deploy %{dep} '{.metadata.labels.owner}')" ac || eq "$(j -n %{ns} deploy %{dep} '{.spec.template.metadata.labels.owner}')" ac; }"""), C("Service %{dep} port 9000 in the same namespace", 1, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].port}')" 9000"""), C("nothing was created in namespace default by mistake", 1, r"""! $K -n default get deploy %{dep} >/dev/null 2>&1"""), - C("ns.txt has the namespace", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q121/ns.txt" 2>/dev/null)" %{ns}"""), + C("ns.txt has the namespace", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q64/ns.txt" 2>/dev/null)" %{ns}"""), ], }, # ================================================================= NET ==== { - "id": 111, "domain": "NET", "difficulty": "hard", "title": "Zero-trust namespace", + "id": 54, "domain": "NET", "difficulty": "hard", "title": "Zero-trust namespace", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "zt"), "from": ns_pick(r, "front")}, "task": CTX + """ Namespace `%{ns}` runs `api` (label `app=api`, port 8080) and `db` (label `app=db`, port 5432). @@ -499,7 +499,7 @@ sys.exit(0 if eg and ing else 1)'"""), ], }, { - "id": 112, "domain": "NET", "difficulty": "hard", "title": "TLS Ingress with a generated certificate", + "id": 55, "domain": "NET", "difficulty": "hard", "title": "TLS Ingress with a generated certificate", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "tls"), "host": r.choice(["secure.lab.local", "portal.lab.local", "admin.lab.local"])}, "task": CTX + """ Namespace `%{ns}` has Deployment+Service `web` (port 80). @@ -507,17 +507,17 @@ Namespace `%{ns}` has Deployment+Service `web` (port 80). Secret `%{host}-tls` of type `kubernetes.io/tls`. 2. Create Ingress `web` (class `nginx`) serving host `%{host}` with that TLS secret, path `/` Prefix → `web:80`, and the annotation `nginx.ingress.kubernetes.io/ssl-redirect: "true"`. -3. Write the certificate's SHA-256 fingerprint (`openssl x509 -fingerprint -sha256`, the value only) to `{WS}/q112/fp.txt`. +3. Write the certificate's SHA-256 fingerprint (`openssl x509 -fingerprint -sha256`, the value only) to `{WS}/q55/fp.txt`. """, - "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q112"; mkdir -p "$WS/q112"; $K -n %{ns} create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} delete secret %{host}-tls --ignore-not-found >/dev/null; $K -n %{ns} delete ingress web --ignore-not-found >/dev/null""", + "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q55"; mkdir -p "$WS/q55"; $K -n %{ns} create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} delete secret %{host}-tls --ignore-not-found >/dev/null; $K -n %{ns} delete ingress web --ignore-not-found >/dev/null""", "checks": [ C("Secret %{host}-tls is kubernetes.io/tls with CN %{host}", 2, r"""eq "$(j -n %{ns} secret %{host}-tls '{.type}')" kubernetes.io/tls && j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'CN *= *%{host}'"""), C("Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", 3, r"""i="$K -n %{ns} get ingress web"; eq "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" nginx && eq "$($i -o jsonpath='{.spec.tls[0].secretName}')" %{host}-tls && eq "$($i -o jsonpath='{.spec.tls[0].hosts[0]}')" %{host} && eq "$($i -o jsonpath='{.spec.rules[0].host}')" %{host} && eq "$($i -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}')" web && eq "$($i -o jsonpath='{.metadata.annotations.nginx\.ingress\.kubernetes\.io/ssl-redirect}')" true"""), - C("fp.txt matches the secret's certificate fingerprint", 2, r"""fp=$(j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//'); [ -n "$fp" ] && grep -qiF "$fp" "$WS/q112/fp.txt" 2>/dev/null"""), + C("fp.txt matches the secret's certificate fingerprint", 2, r"""fp=$(j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//'); [ -n "$fp" ] && grep -qiF "$fp" "$WS/q55/fp.txt" 2>/dev/null"""), ], }, { - "id": 113, "domain": "NET", "difficulty": "hard", "title": "Gateway API canary with header match", + "id": 56, "domain": "NET", "difficulty": "hard", "title": "Gateway API canary with header match", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "canary"), "host": r.choice(["shop", "pay", "cart"]), "w": r.choice([10, 20, 25])}, "task": CTX + """ Gateway API CRDs are installed. Namespace `%{ns}` has Services `v1` and `v2` (port 80). @@ -549,7 +549,7 @@ sys.exit(0 if ok else 1)'"""), ], }, { - "id": 114, "domain": "NET", "breaks": "coredns", "difficulty": "hard", "title": "DNS down: three faults", "disruptive": True, + "id": 57, "domain": "NET", "breaks": "coredns", "difficulty": "hard", "title": "DNS down: three faults", "disruptive": True, "params": lambda r: {"ctx": "k8s-c1-H"}, "task": CTX + """ Nothing in the cluster resolves DNS. Restore it. There is more than one cause. @@ -559,9 +559,9 @@ succeed from a pod in `default` (use the FQDN: busybox's nslookup does not apply `kube-system`. """, "setup": r""" -$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > /tmp/Corefile.q114 -sed -i 's|forward \. /etc/resolv.conf|forward . 10.255.255.254|' /tmp/Corefile.q114 -$K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.q114 --dry-run=client -o yaml | $K apply -f - >/dev/null +$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > /tmp/Corefile.q57 +sed -i 's|forward \. /etc/resolv.conf|forward . 10.255.255.254|' /tmp/Corefile.q57 +$K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.q57 --dry-run=client -o yaml | $K apply -f - >/dev/null $K -n kube-system scale deploy coredns --replicas=0 >/dev/null $K apply -f - >/dev/null </dev/null; exit 1"""), }, # ================================================================ STOR ==== { - "id": 115, "domain": "STOR", "difficulty": "hard", "title": "Bind the right local volume", + "id": 58, "domain": "STOR", "difficulty": "hard", "title": "Bind the right local volume", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "lv"), "pvc": name_pick(r, "fast-data")}, "task": CTX + """ Two PersistentVolumes of class `local-fast` exist. Only one of them lives on the node labelled @@ -631,7 +631,7 @@ done ], }, { - "id": 116, "domain": "STOR", "difficulty": "hard", "title": "Migrate data between claims", + "id": 59, "domain": "STOR", "difficulty": "hard", "title": "Migrate data between claims", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "mig"), "n": r.choice([7, 11, 13])}, "task": CTX + """ In `%{ns}`, PVC `old-data` holds files a running pod `legacy` wrote (`/data`). Storage is being @@ -660,17 +660,17 @@ spec: EOF for i in $(seq 1 30); do [ "$(j -n %{ns} pod legacy '{.status.phase}')" = Running ] && break; sleep 2; done pv=$(j -n %{ns} pvc old-data '{.spec.volumeName}'); [ -n "$pv" ] && $K patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' >/dev/null -echo "$pv" > "$WS/.q116-pv" +echo "$pv" > "$WS/.q59-pv" """, "checks": [ C("Job migrate completed", 2, r"""eq "$(j -n %{ns} job migrate '{.status.succeeded}')" 1"""), C("legacy Running on new-data with all %{n} files intact", 3, r"""eq "$(j -n %{ns} pod legacy '{.status.phase}')" Running && eq "$(j -n %{ns} pod legacy '{.spec.volumes[?(@.name!="kube-api-access")].persistentVolumeClaim.claimName}' | tr ' ' '\n' | grep -v '^$' | head -1)" new-data && eq "$($K -n %{ns} exec legacy -- sh -c 'ls /data/file-*.txt | wc -l' 2>/dev/null | tr -d ' ')" %{n} && eq "$($K -n %{ns} exec legacy -- cat /data/file-%{n}.txt 2>/dev/null)" record-%{n}"""), - C("old-data deleted; its PV Released (not deleted)", 2, r"""! $K -n %{ns} get pvc old-data >/dev/null 2>&1 && pv=$(cat "$WS/.q116-pv" 2>/dev/null) && eq "$(j pv $pv '{.status.phase}')" Released"""), + C("old-data deleted; its PV Released (not deleted)", 2, r"""! $K -n %{ns} get pvc old-data >/dev/null 2>&1 && pv=$(cat "$WS/.q59-pv" 2>/dev/null) && eq "$(j pv $pv '{.status.phase}')" Released"""), ], }, # ============================================================== TSHOOT ==== { - "id": 117, "domain": "TSHOOT", "breaks": "w2-kubelet", "difficulty": "hard", "title": "Worker node: three things wrong", "disruptive": True, + "id": 60, "domain": "TSHOOT", "breaks": "w2-kubelet", "difficulty": "hard", "title": "Worker node: three things wrong", "disruptive": True, "params": lambda r: {"ctx": "k8s-c1-H"}, "task": CTX + """ Node `{W2}` is unusable. Make it fully usable again: `Ready`, schedulable, kubelet running and @@ -694,7 +694,7 @@ $K -n default delete pod node-probe --wait=false >/dev/null 2>&1; exit $ok"""), ], }, { - "id": 118, "domain": "TSHOOT", "breaks": ["scheduler", "controller-manager"], "difficulty": "hard", "title": "Two control-plane components down", "disruptive": True, + "id": 61, "domain": "TSHOOT", "breaks": ["scheduler", "controller-manager"], "difficulty": "hard", "title": "Two control-plane components down", "disruptive": True, "params": lambda r: {"ctx": "k8s-c1-H"}, "task": CTX + """ New Deployments create no pods, and pods that do get created stay `Pending`. Two different @@ -718,7 +718,7 @@ $K -n default delete deploy cp-probe --wait=false >/dev/null 2>&1; exit $ok"""), ], }, { - "id": 119, "domain": "TSHOOT", "difficulty": "hard", "title": "Application down for four reasons", + "id": 62, "domain": "TSHOOT", "difficulty": "hard", "title": "Application down for four reasons", "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "checkout"), "dep": name_pick(r, "checkout")}, "task": CTX + """ Deployment `%{dep}` in `%{ns}` should have 3 Available replicas and Service `%{dep}` should route @@ -762,21 +762,21 @@ spec: selector: {app: %{dep}-web} ports: [{port: 80, targetPort: 8080}] EOF -j -n %{ns} deploy %{dep} '{.metadata.uid}' > "$WS/.q119-uid"; j -n %{ns} svc %{dep} '{.spec.clusterIP}' > "$WS/.q119-ip" +j -n %{ns} deploy %{dep} '{.metadata.uid}' > "$WS/.q62-uid"; j -n %{ns} svc %{dep} '{.spec.clusterIP}' > "$WS/.q62-ip" """, "checks": [ C("3 replicas Available", 3, r"""for i in $(seq 1 6); do [ "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" = 3 ] && exit 0; sleep 5; done; exit 1"""), C("Service routes: endpoints ready, targetPort 80", 2, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].targetPort}')" 80 && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""), C("Deployment UID and Service ClusterIP unchanged (fixed in place)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 || exit 1 # only once it is actually fixed -eq "$(j -n %{ns} deploy %{dep} '{.metadata.uid}')" "$(cat "$WS/.q119-uid")" && eq "$(j -n %{ns} svc %{dep} '{.spec.clusterIP}')" "$(cat "$WS/.q119-ip")" """), +eq "$(j -n %{ns} deploy %{dep} '{.metadata.uid}')" "$(cat "$WS/.q62-uid")" && eq "$(j -n %{ns} svc %{dep} '{.spec.clusterIP}')" "$(cat "$WS/.q62-ip")" """), C("quota still present and permits the 3 pods", 1, r"""p=$(j -n %{ns} quota tight '{.spec.hard.pods}'); [ -n "$p" ] && [ "$p" -ge 3 ]"""), ], }, { - "id": 120, "domain": "TSHOOT", "difficulty": "hard", "title": "Cluster facts, exactly", + "id": 63, "domain": "TSHOOT", "difficulty": "hard", "title": "Cluster facts, exactly", "params": lambda r: {"ctx": "k8s-c1-H", "node": r.choice(["{W1}", "{W2}", "{W3}"])}, "task": CTX + """ -Write `{WS}/q120/facts.txt` with these lines, values taken from the running cluster: +Write `{WS}/q63/facts.txt` with these lines, values taken from the running cluster: ``` pod-cidr= service-cidr= @@ -785,15 +785,15 @@ proxy-mode= ``` """, - "setup": r"""rm -rf "$WS/q120"; mkdir -p "$WS/q120" """, + "setup": r"""rm -rf "$WS/q63"; mkdir -p "$WS/q63" """, "checks": [ - C("pod-cidr and service-cidr", 2, r"""f="$WS/q120/facts.txt"; pc=$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'cluster-cidr=[^"]*' | cut -d= -f2); sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2); grep -qx "pod-cidr=$pc" "$f" 2>/dev/null && grep -qx "service-cidr=$sc" "$f" """), - C("dns-ip and proxy-mode", 2, r"""f="$WS/q120/facts.txt"; d=$(j -n kube-system svc kube-dns '{.spec.clusterIP}'); m=$(j -n kube-system cm kube-proxy '{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"'); [ -n "$m" ] || m=iptables; grep -qx "dns-ip=$d" "$f" 2>/dev/null && grep -qx "proxy-mode=$m" "$f" """), - C("%{node} pod CIDR", 1, r"""grep -qx "%{node}-pod-cidr=$(j node %{node} '{.spec.podCIDR}')" "$WS/q120/facts.txt" 2>/dev/null"""), + C("pod-cidr and service-cidr", 2, r"""f="$WS/q63/facts.txt"; pc=$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'cluster-cidr=[^"]*' | cut -d= -f2); sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2); grep -qx "pod-cidr=$pc" "$f" 2>/dev/null && grep -qx "service-cidr=$sc" "$f" """), + C("dns-ip and proxy-mode", 2, r"""f="$WS/q63/facts.txt"; d=$(j -n kube-system svc kube-dns '{.spec.clusterIP}'); m=$(j -n kube-system cm kube-proxy '{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"'); [ -n "$m" ] || m=iptables; grep -qx "dns-ip=$d" "$f" 2>/dev/null && grep -qx "proxy-mode=$m" "$f" """), + C("%{node} pod CIDR", 1, r"""grep -qx "%{node}-pod-cidr=$(j node %{node} '{.spec.podCIDR}')" "$WS/q63/facts.txt" 2>/dev/null"""), ], }, { - "id": 123, "domain": "TSHOOT", "breaks": "apiserver", "difficulty": "hard", "title": "Nothing works: kube-apiserver itself is down", "disruptive": True, + "id": 66, "domain": "TSHOOT", "breaks": "apiserver", "difficulty": "hard", "title": "Nothing works: kube-apiserver itself is down", "disruptive": True, "params": lambda r: {"ctx": "k8s-c1-H"}, "task": CTX + """ Every `kubectl` command from every context times out or refuses to connect. This is not a @@ -824,6 +824,6 @@ $K -n default delete pod api-probe --wait=false >/dev/null 2>&1; exit $ok"""), # derived params: w2 = 100 - w for the canary split for _q in QUESTIONS: - if _q["id"] == 113: + if _q["id"] == 56: _orig = _q["params"] _q["params"] = lambda r, _o=_orig: (lambda p: {**p, "w2": 100 - p["w"]})(_o(r)) diff --git a/tracks/cka/bank_sim_c.py b/tracks/cka/bank_sim_c.py index 0c6b8d8..221e2bc 100644 --- a/tracks/cka/bank_sim_c.py +++ b/tracks/cka/bank_sim_c.py @@ -7,8 +7,8 @@ repair, PSA-free securityContext work, HPA behavior blocks, and "write the answe data-extraction tasks. Two scenarios are deliberately adapted so they cannot wreck the practice cluster: - - 410 audits the installed CNI instead of installing Calico over kindnet. - - 413 plans a control-plane upgrade instead of running `kubeadm upgrade apply` for real. + - 77 audits the installed CNI instead of installing Calico over kindnet. + - 80 plans a control-plane upgrade instead of running `kubeadm upgrade apply` for real. Run with: ./exam sim c """ @@ -23,9 +23,9 @@ def task(**kw): Q.append(kw) -# ------------------------------------------------------------------ 401 ---- +# ------------------------------------------------------------------ 68 ---- task( - id=401, domain="STOR", title="Provision geo-fence persistent storage", + id=68, domain="STOR", title="Provision geo-fence persistent storage", task=""" In Namespace `geo-fence`: @@ -100,31 +100,31 @@ k -n geo-fence get pvc """, ) -# ------------------------------------------------------------------ 402 ---- +# ------------------------------------------------------------------ 69 ---- task( - id=402, domain="ARCH", title="Render a chart, then install it", + id=69, domain="ARCH", title="Render a chart, then install it", task=""" -A chart is at `{WS}/q402/chart`. +A chart is at `{WS}/q69/chart`. 1. Render the chart **without installing it** as release name `ride-argocd`, namespace `ride-gitops`, with `replicaCount=3`, saving the rendered manifests to - `{WS}/q402/manifest.yaml`. + `{WS}/q69/manifest.yaml`. 2. Then install that same chart into Namespace `ride-gitops` (create it) as release `ride-argocd` with `replicaCount=3`. -3. Save the output of listing the releases in that Namespace to `{WS}/q402/releases.txt`. +3. Save the output of listing the releases in that Namespace to `{WS}/q69/releases.txt`. """, setup=r""" -rm -rf "$WS/q402"; mkdir -p "$WS/q402/chart/templates" -cat > "$WS/q402/chart/Chart.yaml" <<'EOF' +rm -rf "$WS/q69"; mkdir -p "$WS/q69/chart/templates" +cat > "$WS/q69/chart/Chart.yaml" <<'EOF' apiVersion: v2 name: ride version: 0.2.0 EOF -cat > "$WS/q402/chart/values.yaml" <<'EOF' +cat > "$WS/q69/chart/values.yaml" <<'EOF' replicaCount: 1 image: nginx:1.27 EOF -cat > "$WS/q402/chart/templates/deploy.yaml" <<'EOF' +cat > "$WS/q69/chart/templates/deploy.yaml" <<'EOF' apiVersion: apps/v1 kind: Deployment metadata: {name: {{ .Release.Name }}-server, namespace: {{ .Release.Namespace }}} @@ -141,29 +141,29 @@ command -v helm >/dev/null || echo "warn: helm is not installed on this machine" """, checks=[ C("manifest.yaml holds rendered manifests, not an installed release", 3, r""" -f="$WS/q402/manifest.yaml"; [ -s "$f" ] || exit 1 +f="$WS/q69/manifest.yaml"; [ -s "$f" ] || exit 1 grep -q 'kind: Deployment' "$f" && grep -q 'ride-argocd-server' "$f" && grep -Eq 'replicas: 3' "$f" """), C("release ride-argocd is deployed in ride-gitops with 3 replicas", 3, r""" helm -n ride-gitops status ride-argocd -o json 2>/dev/null \ | python3 -c 'import sys,json;sys.exit(0 if json.load(sys.stdin)["info"]["status"]=="deployed" else 1)' || exit 1 for i in $(seq 1 20); do [ "$(j -n ride-gitops deploy ride-argocd-server '{.status.availableReplicas}')" = 3 ] && exit 0; sleep 5; done; exit 1"""), C("releases.txt lists the ride-argocd release", 3, r""" -grep -q 'ride-argocd' "$WS/q402/releases.txt" 2>/dev/null && grep -qi 'deployed' "$WS/q402/releases.txt" """), +grep -q 'ride-argocd' "$WS/q69/releases.txt" 2>/dev/null && grep -qi 'deployed' "$WS/q69/releases.txt" """), ], solution=""" -helm template ride-argocd {WS}/q402/chart -n ride-gitops --set replicaCount=3 > {WS}/q402/manifest.yaml +helm template ride-argocd {WS}/q69/chart -n ride-gitops --set replicaCount=3 > {WS}/q69/manifest.yaml k create ns ride-gitops -helm -n ride-gitops install ride-argocd {WS}/q402/chart --set replicaCount=3 -helm -n ride-gitops list > {WS}/q402/releases.txt +helm -n ride-gitops install ride-argocd {WS}/q69/chart --set replicaCount=3 +helm -n ride-gitops list > {WS}/q69/releases.txt # `helm template` renders locally and talks to no cluster - it is how you review what a chart # would do, and how GitOps pipelines produce committed YAML. `helm install` is the one that # creates a release. --create-namespace on install saves the separate `k create ns`. """, ) -# ------------------------------------------------------------------ 403 ---- +# ------------------------------------------------------------------ 70 ---- task( - id=403, domain="TSHOOT", title="Fix the driver onboarding app", + id=70, domain="TSHOOT", title="Fix the driver onboarding app", task=""" Deployment `onboard-app` in Namespace `driver-onboard` never becomes Ready. It has **three** separate faults. @@ -225,21 +225,21 @@ k -n driver-onboard rollout status deploy onboard-app """, ) -# ------------------------------------------------------------------ 404 ---- +# ------------------------------------------------------------------ 71 ---- task( - id=404, domain="NET", title="Expose and verify the dispatch API", + id=71, domain="NET", title="Expose and verify the dispatch API", task=""" Deployment `dispatch-api` is running in Namespace `dispatch-hub` and serves on container port 80. 1. Expose it as a **ClusterIP** Service named `dispatch-service`, port 80 to target port 80. It must actually have endpoints. 2. From inside the cluster, fetch `http://dispatch-service.dispatch-hub.svc:80` and save the - response body to `{WS}/q404/test-result.txt`. -3. Write the Service's ClusterIP — the address only, nothing else — to `{WS}/q404/service-ip.txt`. + response body to `{WS}/q71/test-result.txt`. +3. Write the Service's ClusterIP — the address only, nothing else — to `{WS}/q71/service-ip.txt`. """, setup=NS + r""" mkns dispatch-hub -rm -rf "$WS/q404"; mkdir -p "$WS/q404" +rm -rf "$WS/q71"; mkdir -p "$WS/q71" $K -n dispatch-hub delete svc dispatch-service --ignore-not-found >/dev/null 2>&1 $K -n dispatch-hub delete deploy dispatch-api --ignore-not-found --wait=true >/dev/null 2>&1 $K -n dispatch-hub create deploy dispatch-api --image=nginx:1.27 --replicas=2 --port=80 >/dev/null @@ -254,26 +254,26 @@ eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" ClusterIP \ C("the Service has active endpoints", 3, r""" [ -n "$(j -n dispatch-hub endpointslices -l kubernetes.io/service-name=dispatch-service '{.items[*].endpoints[*].addresses[*]}')" ]"""), C("test-result.txt holds the response from nginx", 2, r""" -grep -qi 'nginx\|/dev/null"""), +grep -qi 'nginx\|/dev/null"""), C("service-ip.txt contains exactly the Service ClusterIP", 2, r""" want=$(j -n dispatch-hub svc dispatch-service '{.spec.clusterIP}') -got=$(tr -d '[:space:]' < "$WS/q404/service-ip.txt" 2>/dev/null) +got=$(tr -d '[:space:]' < "$WS/q71/service-ip.txt" 2>/dev/null) [ -n "$want" ] && eq "$got" "$want" """), ], solution=""" k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80 k -n dispatch-hub get endpointslices -l kubernetes.io/service-name=dispatch-service # must not be empty k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- \\ - wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q404/test-result.txt -k -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q404/service-ip.txt + wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt +k -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt # `expose` copies the Deployment's selector for you, which is what makes the endpoints appear. # Write the IP with jsonpath, not by eyeballing `get svc` - the grader compares it literally. """, ) -# ------------------------------------------------------------------ 405 ---- +# ------------------------------------------------------------------ 72 ---- task( - id=405, domain="STOR", title="Migrate ride history to a Retain volume", + id=72, domain="STOR", title="Migrate ride history to a Retain volume", task=""" In Namespace `ride-history`, Deployment `ride-archive` stores data on PVC `ride-archive-pvc`, whose PersistentVolume has reclaim policy `Delete` — losing the claim would destroy the data. @@ -407,9 +407,9 @@ k -n ride-history scale deploy ride-archive --replicas=1 """, ) -# ------------------------------------------------------------------ 406 ---- +# ------------------------------------------------------------------ 73 ---- task( - id=406, domain="TSHOOT", title="Fix ETA service connectivity", + id=73, domain="TSHOOT", title="Fix ETA service connectivity", task=""" In Namespace `eta-service`, Deployment `eta-calculator` is healthy but Service `eta-backend` returns nothing. There are **two** faults in the Service. @@ -457,9 +457,9 @@ k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend """, ) -# ------------------------------------------------------------------ 407 ---- +# ------------------------------------------------------------------ 74 ---- task( - id=407, domain="NET", title="Fleet dashboard canary with Gateway API", + id=74, domain="NET", title="Fleet dashboard canary with Gateway API", task=""" Namespace `fleet-ops` has Services `fleet-dashboard-v1` and `fleet-dashboard-v2` (both port 80) and a GatewayClass `fleet-gateway-class` already exists. @@ -551,9 +551,9 @@ EOF """, ) -# ------------------------------------------------------------------ 408 ---- +# ------------------------------------------------------------------ 75 ---- task( - id=408, domain="ARCH", title="Cluster-wide read access for the auth team", + id=75, domain="ARCH", title="Cluster-wide read access for the auth team", task=""" The rider auth team needs a read-only identity that works in **every** Namespace. @@ -562,11 +562,11 @@ The rider auth team needs a read-only identity that works in **every** Namespace `pods`, `deployments`, `services` and `events`. 3. Bind it cluster-wide to that ServiceAccount with ClusterRoleBinding `auth-viewer-binding`. 4. Verify the ServiceAccount can list pods in `kube-system` and write the single-word answer to - `{WS}/q408/auth-check.txt`. + `{WS}/q75/auth-check.txt`. """, setup=NS + r""" mkns rider-auth -rm -rf "$WS/q408"; mkdir -p "$WS/q408" +rm -rf "$WS/q75"; mkdir -p "$WS/q75" $K delete clusterrolebinding auth-viewer-binding --ignore-not-found >/dev/null 2>&1 $K delete clusterrole auth-cluster-viewer --ignore-not-found >/dev/null 2>&1 $K -n rider-auth delete sa auth-auditor --ignore-not-found >/dev/null 2>&1 @@ -587,22 +587,22 @@ eq "$($b -o jsonpath='{.roleRef.name}' 2>/dev/null)" auth-cluster-viewer \ && $b -o jsonpath='{.subjects[*].name}' | grep -qw auth-auditor \ && eq "$($K auth can-i list pods -n kube-system --as=system:serviceaccount:rider-auth:auth-auditor 2>/dev/null)" yes"""), C("auth-check.txt records the verification result", 2, r""" -grep -qx 'yes' "$WS/q408/auth-check.txt" 2>/dev/null"""), +grep -qx 'yes' "$WS/q75/auth-check.txt" 2>/dev/null"""), ], solution=""" k -n rider-auth create sa auth-auditor k create clusterrole auth-cluster-viewer --verb=get,list,watch --resource=pods,deployments,services,events k create clusterrolebinding auth-viewer-binding --clusterrole=auth-cluster-viewer \\ --serviceaccount=rider-auth:auth-auditor -k auth can-i list pods -n kube-system --as=system:serviceaccount:rider-auth:auth-auditor > {WS}/q408/auth-check.txt +k auth can-i list pods -n kube-system --as=system:serviceaccount:rider-auth:auth-auditor > {WS}/q75/auth-check.txt # A ClusterRole bound with a RoleBinding only applies in that one Namespace - it is the # ClusterRoleBinding that makes it cluster-wide. The --serviceaccount flag takes ns:name. """, ) -# ------------------------------------------------------------------ 409 ---- +# ------------------------------------------------------------------ 76 ---- task( - id=409, domain="WORK", title="Add an init container to the trip logger", + id=76, domain="WORK", title="Add an init container to the trip logger", task=""" Deployment `trip-logger` in Namespace `trip-tracker` must be given a startup step. @@ -669,15 +669,15 @@ k -n trip-tracker rollout status deploy trip-logger """, ) -# ------------------------------------------------------------------ 410 ---- +# ------------------------------------------------------------------ 77 ---- task( - id=410, domain="NET", title="Audit the CNI, then lock a namespace down", + id=77, domain="NET", title="Audit the CNI, then lock a namespace down", task=""" Before rolling out NetworkPolicies, confirm what the cluster's CNI can actually enforce. 1. Write the name of the DaemonSet in `kube-system` that provides pod networking to - `{WS}/q410/cni.txt`. -2. Write `yes` or `no` to `{WS}/q410/enforced.txt`: does this CNI enforce NetworkPolicy? + `{WS}/q77/cni.txt`. +2. Write `yes` or `no` to `{WS}/q77/enforced.txt`: does this CNI enforce NetworkPolicy? 3. In Namespace `zero-trust`, create NetworkPolicy `default-deny-all` that denies **all** ingress and egress for every Pod in the Namespace, with no allow rules of any kind. @@ -686,14 +686,14 @@ happily and never enforce them, and a CNI's behaviour can change between release """, setup=NS + r""" mkns zero-trust -rm -rf "$WS/q410"; mkdir -p "$WS/q410" +rm -rf "$WS/q77"; mkdir -p "$WS/q77" $K -n zero-trust delete netpol default-deny-all --ignore-not-found >/dev/null 2>&1 """, checks=[ C("cni.txt names the CNI DaemonSet in kube-system", 2, r""" want=$($K -n kube-system get ds -o name 2>/dev/null | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1) [ -n "$want" ] || want=kindnet -grep -qi "$want" "$WS/q410/cni.txt" 2>/dev/null"""), +grep -qi "$want" "$WS/q77/cni.txt" 2>/dev/null"""), # The right answer depends on the build, so derive it: recent kindnet embeds a policy engine # (nftables table `kindnet-network-policies` on every node - verified enforcing on this # cluster), calico/cilium/weave enforce, flannel does not. @@ -704,7 +704,7 @@ case "$ds" in *kindnet*) nx "$W1" 'nft list table inet kindnet-network-policies >/dev/null 2>&1' /dev/null"""), +grep -qix "$want" "$WS/q77/enforced.txt" 2>/dev/null"""), C("default-deny-all selects every pod in both directions", 4, r""" n="$K -n zero-trust get netpol default-deny-all" eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \ @@ -713,7 +713,7 @@ eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \ && [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ]"""), ], solution=""" -k -n kube-system get ds # kindnet -> {WS}/q410/cni.txt +k -n kube-system get ds # kindnet -> {WS}/q77/cni.txt # Don't answer from memory - prove it. Two pods, one deny policy, one wget: k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv; k -n zero-trust expose pod srv --port 80 k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600 @@ -727,7 +727,7 @@ spec: policyTypes: [Ingress, Egress] EOF k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null || echo BLOCKED # -> BLOCKED here -echo yes > {WS}/q410/enforced.txt +echo yes > {WS}/q77/enforced.txt k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv # leave only the policy # podSelector {} means every pod in the namespace; naming both policyTypes with no ingress/egress # blocks listed means "deny everything". Recent kindnet builds (this cluster's included) embed a @@ -737,9 +737,9 @@ k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv """, ) -# ------------------------------------------------------------------ 411 ---- +# ------------------------------------------------------------------ 78 ---- task( - id=411, domain="WORK", title="Priority for the rider support portal", + id=78, domain="WORK", title="Priority for the rider support portal", task=""" 1. Create PriorityClass `rider-support-priority` with value `1000000`, **not** the global default, and description `Priority for rider support services`. @@ -784,9 +784,9 @@ k -n rider-support get pod -l app=support-portal -o jsonpath='{.items[0].spec.pr """, ) -# ------------------------------------------------------------------ 412 ---- +# ------------------------------------------------------------------ 79 ---- task( - id=412, domain="WORK", title="Surge pricing configuration", + id=79, domain="WORK", title="Surge pricing configuration", task=""" In Namespace `surge-pricing`: @@ -849,33 +849,33 @@ k -n surge-pricing rollout status deploy surge-engine """, ) -# ------------------------------------------------------------------ 413 ---- +# ------------------------------------------------------------------ 80 ---- task( - id=413, domain="ARCH", title="Plan the control-plane upgrade", + id=80, domain="ARCH", title="Plan the control-plane upgrade", task=""" The control-plane node is scheduled for a minor-version upgrade. Prepare the change — **do not execute an upgrade**. -1. Write the cluster's current server version, as `kubectl` reports it, to `{WS}/q413/current.txt`. -2. Write every node's name and kubelet version, one node per line, to `{WS}/q413/kubelets.txt`. -3. Write to `{WS}/q413/steps.sh` the ordered commands to upgrade the **control-plane** node — the +1. Write the cluster's current server version, as `kubectl` reports it, to `{WS}/q80/current.txt`. +2. Write every node's name and kubelet version, one node per line, to `{WS}/q80/kubelets.txt`. +3. Write to `{WS}/q80/steps.sh` the ordered commands to upgrade the **control-plane** node — the kubeadm package install, the kubeadm upgrade step, draining, the kubelet/kubectl package install, restarting the kubelet, and uncordoning — one command per line, in execution order. -4. Write to `{WS}/q413/worker.txt` the single kubeadm subcommand that upgrades a **worker** node +4. Write to `{WS}/q80/worker.txt` the single kubeadm subcommand that upgrades a **worker** node instead (it is not `upgrade apply`). """, - setup=r"""rm -rf "$WS/q413"; mkdir -p "$WS/q413" """, + setup=r"""rm -rf "$WS/q80"; mkdir -p "$WS/q80" """, checks=[ C("current.txt matches the live server version", 2, r""" v=$($K version -o json 2>/dev/null | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])') -[ -n "$v" ] && grep -qF "$v" "$WS/q413/current.txt" 2>/dev/null"""), +[ -n "$v" ] && grep -qF "$v" "$WS/q80/current.txt" 2>/dev/null"""), C("kubelets.txt lists every node with its kubelet version", 2, r""" -f="$WS/q413/kubelets.txt"; [ -s "$f" ] || exit 1 +f="$WS/q80/kubelets.txt"; [ -s "$f" ] || exit 1 for n in $($K get nodes -o jsonpath='{.items[*].metadata.name}'); do grep -q "$n" "$f" || exit 1; done v=$($K get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}') grep -qF "$v" "$f" """), C("steps.sh has the upgrade commands in a workable order", 4, r""" -f="$WS/q413/steps.sh"; [ -f "$f" ] || exit 1 +f="$WS/q80/steps.sh"; [ -f "$f" ] || exit 1 grep -q 'kubeadm upgrade apply' "$f" || exit 1 grep -Eq '(apt|yum|dnf).*install.*kubeadm' "$f" || exit 1 grep -Eq '(apt|yum|dnf).*install.*kubelet' "$f" || exit 1 @@ -887,13 +887,13 @@ kl=$(grep -nE '(apt|yum|dnf).*install.*kubelet' "$f" | head -1 | cut -d: -f1) un=$(grep -n 'uncordon' "$f" | head -1 | cut -d: -f1) [ "$ka" -lt "$ap" ] && [ "$ap" -lt "$kl" ] && [ "$kl" -lt "$un" ]"""), C("worker.txt names `kubeadm upgrade node`", 2, r""" -grep -Eq 'kubeadm[[:space:]]+upgrade[[:space:]]+node' "$WS/q413/worker.txt" 2>/dev/null \ - && ! grep -q 'upgrade apply' "$WS/q413/worker.txt" """), +grep -Eq 'kubeadm[[:space:]]+upgrade[[:space:]]+node' "$WS/q80/worker.txt" 2>/dev/null \ + && ! grep -q 'upgrade apply' "$WS/q80/worker.txt" """), ], solution=""" -k version -o json | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])' > {WS}/q413/current.txt -k get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion --no-headers > {WS}/q413/kubelets.txt -cat > {WS}/q413/steps.sh <<'EOF' +k version -o json | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])' > {WS}/q80/current.txt +k get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion --no-headers > {WS}/q80/kubelets.txt +cat > {WS}/q80/steps.sh <<'EOF' apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.34.1-1.1 && apt-mark hold kubeadm kubeadm upgrade plan kubeadm upgrade apply v1.34.1 -y @@ -903,7 +903,7 @@ systemctl daemon-reload systemctl restart kubelet kubectl uncordon EOF -echo 'kubeadm upgrade node' > {WS}/q413/worker.txt +echo 'kubeadm upgrade node' > {WS}/q80/worker.txt # kubeadm goes first because it drives the whole thing; the control plane is upgraded by # `upgrade apply`, and only then do the node binaries (kubelet/kubectl) follow, ending with # uncordon. Workers run `kubeadm upgrade node` - they host no control-plane static pods, so @@ -912,9 +912,9 @@ echo 'kubeadm upgrade node' > {WS}/q413/worker.txt """, ) -# ------------------------------------------------------------------ 414 ---- +# ------------------------------------------------------------------ 81 ---- task( - id=414, domain="TSHOOT", title="Fix the fare calculation pods", + id=81, domain="TSHOOT", title="Fix the fare calculation pods", task=""" Deployment `fare-calculator` in Namespace `fare-engine` has no running pods. There are **two** independent reasons, one storage and one scheduling. @@ -1001,9 +1001,9 @@ k -n fare-engine rollout status deploy fare-calculator """, ) -# ------------------------------------------------------------------ 415 ---- +# ------------------------------------------------------------------ 82 ---- task( - id=415, domain="WORK", title="Autoscale the route planner on memory", + id=82, domain="WORK", title="Autoscale the route planner on memory", task=""" Create HorizontalPodAutoscaler `route-planner-hpa` in Namespace `route-planner` for Deployment `route-planner`: @@ -1073,9 +1073,9 @@ k -n route-planner get hpa route-planner-hpa """, ) -# ------------------------------------------------------------------ 416 ---- +# ------------------------------------------------------------------ 83 ---- task( - id=416, domain="NET", title="NetworkPolicy for the safety monitor", + id=83, domain="NET", title="NetworkPolicy for the safety monitor", task=""" In Namespace `safety-monitor`, create NetworkPolicy `safety-api-policy` for the Pods labelled `app=safety-api`: @@ -1146,9 +1146,9 @@ EOF """, ) -# ------------------------------------------------------------------ 417 ---- +# ------------------------------------------------------------------ 84 ---- task( - id=417, domain="ARCH", title="Enforce TLS 1.3 on the booking portal", + id=84, domain="ARCH", title="Enforce TLS 1.3 on the booking portal", task=""" Namespace `booking-tls` runs Deployment `booking-nginx`, which serves HTTPS using the nginx configuration in ConfigMap `booking-nginx-config`. It currently accepts TLS 1.2 **and** 1.3. @@ -1156,13 +1156,13 @@ configuration in ConfigMap `booking-nginx-config`. It currently accepts TLS 1.2 1. Change the ConfigMap so only TLS 1.3 is accepted. 2. Make the running pods pick up the change, without editing the Deployment's pod template by hand. 3. Write the nginx directive line you ended up with, as it appears in the ConfigMap, to - `{WS}/q417/ssl-line.txt`. + `{WS}/q84/ssl-line.txt`. A TLS 1.2 handshake against the service must be refused afterwards. """, setup=NS + r""" mkns booking-tls -rm -rf "$WS/q417"; mkdir -p "$WS/q417" +rm -rf "$WS/q84"; mkdir -p "$WS/q84" $K -n booking-tls delete deploy booking-nginx --ignore-not-found --wait=true >/dev/null 2>&1 $K -n booking-tls delete cm booking-nginx-config --ignore-not-found >/dev/null 2>&1 $K -n booking-tls delete secret booking-tls-cert --ignore-not-found >/dev/null 2>&1 @@ -1223,7 +1223,7 @@ cm_t=$($K -n booking-tls get cm booking-nginx-config -o json --show-managed-fiel ts=$(j -n booking-tls pod -l app=booking-nginx '{.items[*].metadata.creationTimestamp}'); [ -n "$cm_t" ] && [ -n "$ts" ] || exit 1 for t in $ts; do [[ "$t" > "$cm_t" ]] || exit 1; done"""), C("ssl-line.txt records the directive", 2, r""" -f="$WS/q417/ssl-line.txt" +f="$WS/q84/ssl-line.txt" grep -q 'ssl_protocols' "$f" 2>/dev/null && grep -q 'TLSv1\.3' "$f" && ! grep -q 'TLSv1\.2' "$f" """), C("TLS 1.3 is accepted and TLS 1.2 is refused", 3, r""" for i in $(seq 1 20); do [ "$(j -n booking-tls deploy booking-nginx '{.status.readyReplicas}')" = 1 ] && break; sleep 5; done @@ -1240,7 +1240,7 @@ k -n booking-tls edit cm booking-nginx-config # ssl_protocols TLSv1.2 TLSv1.3; -> ssl_protocols TLSv1.3; k -n booking-tls rollout restart deploy booking-nginx k -n booking-tls rollout status deploy booking-nginx -echo 'ssl_protocols TLSv1.3;' > {WS}/q417/ssl-line.txt +echo 'ssl_protocols TLSv1.3;' > {WS}/q84/ssl-line.txt # verify from ON a node (the Service is published as NodePort 30443): # {SHELL_W1} # echo | openssl s_client -connect 127.0.0.1:30443 -tls1_3 # connects @@ -1253,20 +1253,20 @@ echo 'ssl_protocols TLSv1.3;' > {WS}/q417/ssl-line.txt """, ) -# ------------------------------------------------------------------ 418 ---- +# ------------------------------------------------------------------ 85 ---- task( - id=418, domain="TSHOOT", title="Investigate vehicle tracker health", + id=85, domain="TSHOOT", title="Investigate vehicle tracker health", task=""" Namespace `vehicle-track` has a misbehaving workload. Collect evidence: 1. List **all** pods in **all** namespaces sorted by restart count, and save the output to - `{WS}/q418/most-restarts.txt`. -2. Save the last 50 log lines from the `app=vehicle-tracker` pods to `{WS}/q418/logs.txt`. -3. Save the Warning events from `vehicle-track`, sorted by time, to `{WS}/q418/warnings.txt`. + `{WS}/q85/most-restarts.txt`. +2. Save the last 50 log lines from the `app=vehicle-tracker` pods to `{WS}/q85/logs.txt`. +3. Save the Warning events from `vehicle-track`, sorted by time, to `{WS}/q85/warnings.txt`. """, setup=NS + r""" mkns vehicle-track -rm -rf "$WS/q418"; mkdir -p "$WS/q418" +rm -rf "$WS/q85"; mkdir -p "$WS/q85" $K -n vehicle-track delete deploy vehicle-tracker --ignore-not-found --wait=true >/dev/null 2>&1 $K apply -f - >/dev/null <<'EOF' apiVersion: apps/v1 @@ -1286,19 +1286,19 @@ EOF """, checks=[ C("most-restarts.txt is a sorted all-namespace pod listing", 3, r""" -f="$WS/q418/most-restarts.txt"; [ -s "$f" ] || exit 1 +f="$WS/q85/most-restarts.txt"; [ -s "$f" ] || exit 1 grep -qi 'NAMESPACE' "$f" && grep -q 'kube-system' "$f" && grep -q 'vehicle-track' "$f" \ && [ "$(wc -l < "$f")" -ge 5 ]"""), C("logs.txt holds the tracker's log output", 2, r""" -grep -q 'tracker starting\|gps fix lost' "$WS/q418/logs.txt" 2>/dev/null"""), +grep -q 'tracker starting\|gps fix lost' "$WS/q85/logs.txt" 2>/dev/null"""), C("warnings.txt holds Warning events from the namespace", 3, r""" -f="$WS/q418/warnings.txt"; [ -s "$f" ] || exit 1 +f="$WS/q85/warnings.txt"; [ -s "$f" ] || exit 1 grep -q 'Warning' "$f" && ! grep -q '^Normal' "$f" """), ], solution=""" -k get pods -A --sort-by='.status.containerStatuses[0].restartCount' > {WS}/q418/most-restarts.txt -k -n vehicle-track logs -l app=vehicle-tracker --tail=50 > {WS}/q418/logs.txt -k -n vehicle-track get events --field-selector type=Warning --sort-by='.lastTimestamp' > {WS}/q418/warnings.txt +k get pods -A --sort-by='.status.containerStatuses[0].restartCount' > {WS}/q85/most-restarts.txt +k -n vehicle-track logs -l app=vehicle-tracker --tail=50 > {WS}/q85/logs.txt +k -n vehicle-track get events --field-selector type=Warning --sort-by='.lastTimestamp' > {WS}/q85/warnings.txt # --sort-by takes a JSONPath into the object and sorts ascending, so the worst offenders land at # the BOTTOM of that first file. `logs -l` reads every pod matching the label, which is what you # want for a Deployment whose pod name keeps changing under a CrashLoopBackOff. diff --git a/tracks/cka/bank_sim_d.py b/tracks/cka/bank_sim_d.py index e27bb4b..fe73e49 100644 --- a/tracks/cka/bank_sim_d.py +++ b/tracks/cka/bank_sim_d.py @@ -30,12 +30,12 @@ def task(**kw): Q.append(kw) -# ------------------------------------------------------------------ 501 ---- +# ------------------------------------------------------------------ 86 ---- task( - id=501, domain="ARCH", title="How the control plane is run", + id=86, domain="ARCH", title="How the control plane is run", task=""" The operations handbook has to say how each part of this cluster's control plane is run. Write -`{WS}/q501/controlplane.txt` with exactly these six lines, each value one of `process`, +`{WS}/q86/controlplane.txt` with exactly these six lines, each value one of `process`, `static-pod`, `pod`, `deployment` or `daemonset`: ``` @@ -49,21 +49,21 @@ dns: `dns` is whatever serves cluster DNS, for example `dns: daemonset some-name`. """, - setup=r"""rm -rf "$WS/q501"; mkdir -p "$WS/q501" """, + setup=r"""rm -rf "$WS/q86"; mkdir -p "$WS/q86" """, checks=[ C("kubelet is a process", 1, r""" -f="$WS/q501/controlplane.txt"; [ -s "$f" ] || exit 1 +f="$WS/q86/controlplane.txt"; [ -s "$f" ] || exit 1 want=process; [ "$(nx "$CP" 'systemctl is-active kubelet' /dev/null 2>&1 || exit 1 for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do eq "$(sed -n "s/^ *$c *: *//p" "$f" | head -1 | tr -d '[:space:]' | tr A-Z a-z)" static-pod || exit 1 done"""), C("dns names the right kind of object and its name", 2, r""" -f="$WS/q501/controlplane.txt" +f="$WS/q86/controlplane.txt" sel=$(j -n kube-system svc kube-dns '{.spec.selector.k8s-app}') own=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].kind}') name=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].name}') @@ -79,7 +79,7 @@ exit k -n kube-system get pod -o wide # static pods carry the node name as a suffix... k -n kube-system get pod kube-apiserver-{CP} -o jsonpath='{.metadata.ownerReferences[0].kind}{"\\n"}' # ...and are owned by the Node k -n kube-system get deploy,ds # coredns is a Deployment (kube-proxy, kindnet are DaemonSets) -cat > {WS}/q501/controlplane.txt <<'EOF' +cat > {WS}/q86/controlplane.txt <<'EOF' kubelet: process kube-apiserver: static-pod kube-scheduler: static-pod @@ -92,11 +92,11 @@ EOF """, ) -# ------------------------------------------------------------------ 502 ---- +# ------------------------------------------------------------------ 87 ---- task( - id=502, domain="TSHOOT", title="Cluster facts to an exact file", + id=87, domain="TSHOOT", title="Cluster facts to an exact file", task=""" -Write `{WS}/q502/cluster-info.txt` — answers only, exactly this format: +Write `{WS}/q87/cluster-info.txt` — answers only, exactly this format: ``` 1: @@ -106,22 +106,22 @@ Write `{WS}/q502/cluster-info.txt` — answers only, exactly this format: 5: ``` """, - setup=r"""rm -rf "$WS/q502"; mkdir -p "$WS/q502" """, + setup=r"""rm -rf "$WS/q87"; mkdir -p "$WS/q87" """, checks=[ C("node counts (lines 1 and 2)", 2, r""" -f="$WS/q502/cluster-info.txt"; [ -s "$f" ] || exit 1 +f="$WS/q87/cluster-info.txt"; [ -s "$f" ] || exit 1 cp=$($K get nodes -l node-role.kubernetes.io/control-plane --no-headers 2>/dev/null | wc -l); all=$($K get nodes --no-headers | wc -l) eq "$(sed -n 's/^ *1 *: *//p' "$f" | tr -d '[:space:]')" "$cp" && eq "$(sed -n 's/^ *2 *: *//p' "$f" | tr -d '[:space:]')" "$((all - cp))" """), C("Service CIDR (line 3)", 2, r""" sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2) -[ -n "$sc" ] && eq "$(sed -n 's/^ *3 *: *//p' "$WS/q502/cluster-info.txt" | tr -d '[:space:]')" "$sc" """), +[ -n "$sc" ] && eq "$(sed -n 's/^ *3 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]')" "$sc" """), C("CNI plugin and its config file (line 4)", 2, r""" -l=$(sed -n 's/^ *4 *: *//p' "$WS/q502/cluster-info.txt" | tr -d '[:space:]') +l=$(sed -n 's/^ *4 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]') cni=$($K -n kube-system get ds -o name | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1) cfg=$(nx "$W1" 'ls /etc/cni/net.d/*.conflist /etc/cni/net.d/*.conf 2>/dev/null | head -1' {WS}/q502/cluster-info.txt <<'EOF' +cat > {WS}/q87/cluster-info.txt <<'EOF' 1: 1 2: 3 3: 10.96.0.0/16 @@ -142,9 +142,9 @@ EOF """, ) -# ------------------------------------------------------------------ 503 ---- +# ------------------------------------------------------------------ 88 ---- task( - id=503, domain="NET", title="Ready only when its dependency answers", + id=88, domain="NET", title="Ready only when its dependency answers", task=""" In Namespace `oslo-web`: @@ -191,125 +191,125 @@ k -n oslo-web get pod gatekeeper -w # 1/1 after the next probe """, ) -# ------------------------------------------------------------------ 504 ---- +# ------------------------------------------------------------------ 89 ---- task( - id=504, domain="WORK", title="Sorting done by kubectl", + id=89, domain="WORK", title="Sorting done by kubectl", task=""" Write two executable scripts that call `kubectl` (spelled out — aliases do not exist in a fresh shell) and let **kubectl itself** do the sorting with `--sort-by`: -1. `{WS}/q504/pods-by-age.sh` — every Pod in every Namespace, oldest first (by creation time). -2. `{WS}/q504/pods-by-uid.sh` — the Pods in Namespace `kube-system`, sorted by `metadata.uid`. +1. `{WS}/q89/pods-by-age.sh` — every Pod in every Namespace, oldest first (by creation time). +2. `{WS}/q89/pods-by-uid.sh` — the Pods in Namespace `kube-system`, sorted by `metadata.uid`. """, - setup=r"""rm -rf "$WS/q504"; mkdir -p "$WS/q504" """, + setup=r"""rm -rf "$WS/q89"; mkdir -p "$WS/q89" """, checks=[ C("pods-by-age.sh: all namespaces, --sort-by creationTimestamp, runs", 2, r""" -f="$WS/q504/pods-by-age.sh"; [ -x "$f" ] || exit 1 +f="$WS/q89/pods-by-age.sh"; [ -x "$f" ] || exit 1 grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \ && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.creationTimestamp" "$f" || exit 1 out=$(timeout 60 bash "$f" 2>/dev/null); [ "$(echo "$out" | wc -l)" -ge 5 ] && echo "$out" | grep -q kube-system"""), C("pods-by-uid.sh: kube-system, --sort-by metadata.uid, runs", 2, r""" -f="$WS/q504/pods-by-uid.sh"; [ -x "$f" ] || exit 1 +f="$WS/q89/pods-by-uid.sh"; [ -x "$f" ] || exit 1 grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-n|--namespace)[= ]*kube-system' "$f" \ && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.uid" "$f" || exit 1 timeout 60 bash "$f" 2>/dev/null | grep -q etcd"""), - C("no aliases in either script", 1, r"""[ -s "$WS/q504/pods-by-age.sh" ] && [ -s "$WS/q504/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q504/pods-by-age.sh" "$WS/q504/pods-by-uid.sh" """), + C("no aliases in either script", 1, r"""[ -s "$WS/q89/pods-by-age.sh" ] && [ -s "$WS/q89/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q89/pods-by-age.sh" "$WS/q89/pods-by-uid.sh" """), ], solution=""" -cat > {WS}/q504/pods-by-age.sh <<'EOF' +cat > {WS}/q89/pods-by-age.sh <<'EOF' kubectl get pods -A --sort-by=.metadata.creationTimestamp EOF -cat > {WS}/q504/pods-by-uid.sh <<'EOF' +cat > {WS}/q89/pods-by-uid.sh <<'EOF' kubectl -n kube-system get pods --sort-by=.metadata.uid EOF -chmod +x {WS}/q504/*.sh -bash {WS}/q504/pods-by-age.sh | head -3 # the control-plane pods come first - they are oldest +chmod +x {WS}/q89/*.sh +bash {WS}/q89/pods-by-age.sh | head -3 # the control-plane pods come first - they are oldest # --sort-by takes a JSONPath into each item; ascending only, so "newest first" is `| tac` or # sorting by a field that grows the other way. """, ) -# ------------------------------------------------------------------ 505 ---- +# ------------------------------------------------------------------ 90 ---- task( - id=505, domain="TSHOOT", title="Event forensics around kube-proxy", breaks="w2-kube-proxy", + id=90, domain="TSHOOT", title="Event forensics around kube-proxy", breaks="w2-kube-proxy", task=""" -1. Write `{WS}/q505/cluster-events.sh`: a script that prints the latest events of the **whole** +1. Write `{WS}/q90/cluster-events.sh`: a script that prints the latest events of the **whole** cluster (all Namespaces) sorted by creation time, using `kubectl`. Make it executable. 2. Delete the kube-proxy Pod that runs on node `{W2}`. Write the events this caused to - `{WS}/q505/pod-deleted.log`. + `{WS}/q90/pod-deleted.log`. 3. On node `{W2}`, stop **and remove** the kube-proxy **container** with `crictl` — leave the Pod - object alone. Write the events this caused to `{WS}/q505/container-killed.log`. + object alone. Write the events this caused to `{WS}/q90/container-killed.log`. 4. Which of the two actions made the Pod's `RESTARTS` counter go up? Write `pod` or `container` to - `{WS}/q505/restarts.txt`. + `{WS}/q90/restarts.txt`. Do step 2 before step 3. """, setup=r""" -rm -rf "$WS/q505"; mkdir -p "$WS/q505" -$K -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 -o jsonpath='{.items[0].metadata.uid}' > "$WS/.q505-uid" +rm -rf "$WS/q90"; mkdir -p "$WS/q90" +$K -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 -o jsonpath='{.items[0].metadata.uid}' > "$WS/.q90-uid" """, checks=[ C("cluster-events.sh: all namespaces, sorted by time, runs", 2, r""" -f="$WS/q505/cluster-events.sh"; [ -x "$f" ] || exit 1 +f="$WS/q90/cluster-events.sh"; [ -x "$f" ] || exit 1 grep -Eq 'kubectl\b.*\bget\b.*\b(ev|events?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \ && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?(metadata\.creationTimestamp|lastTimestamp|eventTime|firstTimestamp)" "$f" || exit 1 timeout 60 bash "$f" >/dev/null 2>&1"""), C("the kube-proxy Pod on {W2} was deleted and replaced", 2, r""" -old=$(cat "$WS/.q505-uid" 2>/dev/null); [ -n "$old" ] || exit 1 +old=$(cat "$WS/.q90-uid" 2>/dev/null); [ -n "$old" ] || exit 1 now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}') [ -n "$now" ] && [ "$now" != "$old" ] && eq "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.phase}')" Running"""), # counted on the REPLACEMENT pod, which starts at 0 - after a node reboot the original pod # often already shows a restart, which would otherwise hand these points out for free C("then its container was killed (restart counter >= 1)", 2, r""" -old=$(cat "$WS/.q505-uid" 2>/dev/null); now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}') +old=$(cat "$WS/.q90-uid" 2>/dev/null); now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}') [ -n "$old" ] && [ -n "$now" ] && [ "$now" != "$old" ] || exit 1 [ "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null"""), C("pod-deleted.log and container-killed.log hold the events", 2, r""" for f in pod-deleted.log container-killed.log; do - grep -q 'kube-proxy' "$WS/q505/$f" 2>/dev/null && grep -Eq 'Killing|SuccessfulCreate|SuccessfulDelete|Scheduled|Pulled|Created|Started|BackOff' "$WS/q505/$f" || exit 1 + grep -q 'kube-proxy' "$WS/q90/$f" 2>/dev/null && grep -Eq 'Killing|SuccessfulCreate|SuccessfulDelete|Scheduled|Pulled|Created|Started|BackOff' "$WS/q90/$f" || exit 1 done"""), - C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q505/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""), + C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q90/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""), ], solution=""" -cat > {WS}/q505/cluster-events.sh <<'EOF' +cat > {WS}/q90/cluster-events.sh <<'EOF' kubectl get events -A --sort-by=.metadata.creationTimestamp EOF -chmod +x {WS}/q505/cluster-events.sh +chmod +x {WS}/q90/cluster-events.sh k -n kube-system get pod -l k8s-app=kube-proxy -o wide # the one on {W2} k -n kube-system delete pod -k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q505/pod-deleted.log +k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log # Killing (old pod), SuccessfulCreate (DaemonSet), Scheduled, Pulled, Created, Started (new pod) {SHELL_W2} crictl ps --name kube-proxy # container id crictl stop && crictl rm exit -k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q505/container-killed.log +k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log # Pulled, Created, Started - same Pod, new container k -n kube-system get pod -l k8s-app=kube-proxy -o wide # RESTARTS 1 on {W2} -echo container > {WS}/q505/restarts.txt +echo container > {WS}/q90/restarts.txt # Deleting the Pod makes the DaemonSet create a brand-new Pod (RESTARTS 0). Killing only the # container makes the kubelet start a new container inside the SAME Pod - that is a restart. """, ) -# ------------------------------------------------------------------ 506 ---- +# ------------------------------------------------------------------ 91 ---- def _census(r): n = r.sample(range(2, 10), 4) return {"ctx": "k8s-c1-H", "north": n[0], "south": n[1], "east": n[2], "west": n[3]} task( - id=506, domain="ARCH", title="API resources and a Role census", params=_census, + id=91, domain="ARCH", title="API resources and a Role census", params=_census, task=""" 1. Write the names of every **namespaced** resource type this API server offers to - `{WS}/q506/namespaced.txt`, one per line — plural names, as `kubectl api-resources` prints them. + `{WS}/q91/namespaced.txt`, one per line — plural names, as `kubectl api-resources` prints them. 2. Of the Namespaces `census-north`, `census-south`, `census-east` and `census-west`, which one holds - the most Roles? Write ` ` to `{WS}/q506/most-roles.txt`. + the most Roles? Write ` ` to `{WS}/q91/most-roles.txt`. """, setup=NS + r""" -rm -rf "$WS/q506"; mkdir -p "$WS/q506" +rm -rf "$WS/q91"; mkdir -p "$WS/q91" for pair in north:%{north} south:%{south} east:%{east} west:%{west}; do ns=census-${pair%%:*}; n=${pair##*:} mkns $ns @@ -319,37 +319,37 @@ done """, checks=[ C("namespaced.txt lists exactly the namespaced resource types", 3, r""" -f="$WS/q506/namespaced.txt"; [ -s "$f" ] || exit 1 +f="$WS/q91/namespaced.txt"; [ -s "$f" ] || exit 1 want=$($K api-resources --namespaced=true -o name 2>/dev/null | sed 's/\..*//' | sort -u) got=$(awk '{print $1}' "$f" | sed 's/\..*//' | grep . | sort -u) [ -n "$want" ] && [ "$got" = "$want" ]"""), C("most-roles.txt names the namespace and its Role count", 3, r""" best=$(for ns in census-north census-south census-east census-west; do echo "$ns $($K -n $ns get role --no-headers 2>/dev/null | wc -l)"; done | sort -k2 -n | tail -1) -eq "$(tr -s ' ' < "$WS/q506/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """), +eq "$(tr -s ' ' < "$WS/q91/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """), ], solution=""" k api-resources --namespaced=true -o name > /tmp/r.txt # deployments.apps, ... (group suffix) -k api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q506/namespaced.txt +k api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q91/namespaced.txt for ns in census-north census-south census-east census-west; do echo "$ns $(k -n $ns get role --no-headers | wc -l)"; done -# pick the largest, e.g. echo "census-east 8" > {WS}/q506/most-roles.txt +# pick the largest, e.g. echo "census-east 8" > {WS}/q91/most-roles.txt # --namespaced=false gives the cluster-scoped half (nodes, pv, storageclasses, clusterroles...). # `k get role -A --no-headers | awk '{print $1}' | sort | uniq -c | sort -n` counts every # namespace at once. """, ) -# ------------------------------------------------------------------ 507 ---- +# ------------------------------------------------------------------ 92 ---- task( - id=507, domain="TSHOOT", title="Kubelet unit points at nothing", breaks="w1-kubelet", disruptive=True, + id=92, domain="TSHOOT", title="Kubelet unit points at nothing", breaks="w1-kubelet", disruptive=True, task=""" Node `{W1}` went `NotReady` after someone "tidied up" its kubelet installation. Bring it back to `Ready`; the kubelet has to run as the systemd service it always was. Then write the full path of the -file you had to fix to `{WS}/q507/fixed-file.txt`. +file you had to fix to `{WS}/q92/fixed-file.txt`. Reach the node with `{SHELL_W1}`. Pods on a node that stays NotReady are evicted after five minutes. """, setup=r""" -rm -rf "$WS/q507"; mkdir -p "$WS/q507" +rm -rf "$WS/q92"; mkdir -p "$WS/q92" nx "$W1" ' d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf grep -q "^ExecStart=/usr/bin/kubelet " $d && sed -i "s|^ExecStart=/usr/bin/kubelet |ExecStart=/usr/local/bin/kubelet |" $d @@ -360,7 +360,7 @@ exit 0 C("{W1} is Ready", 4, r"""for i in $(seq 1 15); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 4; done; exit 1"""), C("kubelet active, and its unit runs a kubelet binary that exists", 2, r""" nx "$W1" 'systemctl is-active --quiet kubelet && b=$(sed -n "s|^ExecStart=\([^ ]*kubelet\) .*|\1|p" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | tail -1) && [ -x "$b" ]' /dev/null"""), + C("fixed-file.txt names the drop-in", 2, r"""grep -q '/etc/systemd/system/kubelet.service.d/10-kubeadm.conf' "$WS/q92/fixed-file.txt" 2>/dev/null"""), ], solution=""" k get nodes # {W1} NotReady @@ -375,21 +375,21 @@ systemctl daemon-reload # systemd caches unit files - with systemctl restart kubelet systemctl status kubelet # active (running) exit -echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q507/fixed-file.txt +echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q92/fixed-file.txt k get nodes -w # status=203/EXEC means systemd could not exec the binary at all - a path problem, not a kubelet # configuration problem. `ExecStart=` (empty) in a drop-in clears the unit's own ExecStart first. """, ) -# ------------------------------------------------------------------ 508 ---- +# ------------------------------------------------------------------ 93 ---- task( - id=508, domain="WORK", title="Secrets as a read-only volume and renamed env vars", + id=93, domain="WORK", title="Secrets as a read-only volume and renamed env vars", params=lambda r: {"ctx": "k8s-c1-H", "pw": "ledger-" + "".join(r.choice("abcdefghjkmnpqrstuvwxyz23456789") for _ in range(10))}, task=""" In Namespace `oslo-secure`: -1. Create Secret `bundle` from the file `{WS}/q508/tls-bundle.pem`; the key must be the file's name. +1. Create Secret `bundle` from the file `{WS}/q93/tls-bundle.pem`; the key must be the file's name. 2. Create Secret `db-login` with `user=svc-ledger` and `pass=%{pw}`. 3. Create Pod `vault-reader` (image `busybox:1.36`, command `sleep 86400`) that - mounts Secret `bundle` **read-only** at `/etc/bundle`, @@ -397,14 +397,14 @@ In Namespace `oslo-secure`: """, setup=NS + r""" mkns oslo-secure -rm -rf "$WS/q508"; mkdir -p "$WS/q508" -openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=ledger.oslo.internal" -keyout /dev/null -out "$WS/q508/tls-bundle.pem" >/dev/null 2>&1 +rm -rf "$WS/q93"; mkdir -p "$WS/q93" +openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=ledger.oslo.internal" -keyout /dev/null -out "$WS/q93/tls-bundle.pem" >/dev/null 2>&1 $K -n oslo-secure delete pod vault-reader --ignore-not-found --force --grace-period=0 >/dev/null 2>&1 $K -n oslo-secure delete secret bundle db-login --ignore-not-found >/dev/null """, checks=[ C("Secrets bundle (key tls-bundle.pem, the file's content) and db-login", 2, r""" -[ "$(j -n oslo-secure secret bundle '{.data.tls-bundle\.pem}' | base64 -d 2>/dev/null)" = "$(cat "$WS/q508/tls-bundle.pem")" ] \ +[ "$(j -n oslo-secure secret bundle '{.data.tls-bundle\.pem}' | base64 -d 2>/dev/null)" = "$(cat "$WS/q93/tls-bundle.pem")" ] \ && eq "$(j -n oslo-secure secret db-login '{.data.user}' | base64 -d)" svc-ledger && eq "$(j -n oslo-secure secret db-login '{.data.pass}' | base64 -d)" '%{pw}'"""), C("vault-reader Running with bundle mounted read-only at /etc/bundle", 3, r""" p="$K -n oslo-secure get pod vault-reader" @@ -420,7 +420,7 @@ env=$($K -n oslo-secure exec vault-reader -- env 2>/dev/null) echo "$env" | grep -qx 'APP_USER=svc-ledger' && echo "$env" | grep -qx 'APP_PASS=%{pw}'"""), ], solution=""" -k -n oslo-secure create secret generic bundle --from-file={WS}/q508/tls-bundle.pem # key = file name +k -n oslo-secure create secret generic bundle --from-file={WS}/q93/tls-bundle.pem # key = file name k -n oslo-secure create secret generic db-login --from-literal=user=svc-ledger --from-literal=pass='%{pw}' k -n oslo-secure run vault-reader --image=busybox:1.36 $do -- sleep 86400 > vr.yaml # in vr.yaml, pod level: @@ -440,18 +440,18 @@ k -n oslo-secure exec vault-reader -- ls /etc/bundle """, ) -# ------------------------------------------------------------------ 509 ---- +# ------------------------------------------------------------------ 94 ---- task( - id=509, domain="ARCH", title="Static Pod on the control plane behind a Service", + id=94, domain="ARCH", title="Static Pod on the control plane behind a Service", task=""" 1. On the control-plane node (`{SHELL_CP}`) create a **static Pod** `ops-beacon` in Namespace `default`: image `nginx:1.27`, requests `cpu: 10m` and `memory: 20Mi`, label `app=ops-beacon`. 2. Expose it with a Service `ops-beacon-svc` of type **NodePort** (port 80 → 80) whose endpoints contain that static Pod. -3. Write the node port the Service was given to `{WS}/q509/nodeport.txt`. +3. Write the node port the Service was given to `{WS}/q94/nodeport.txt`. """, setup=r""" -rm -rf "$WS/q509"; mkdir -p "$WS/q509" +rm -rf "$WS/q94"; mkdir -p "$WS/q94" nx "$CP" 'rm -f /etc/kubernetes/manifests/ops-beacon*.yaml' /dev/null # the kubelet drops the mirror pod on its next sync of the manifest directory @@ -469,7 +469,7 @@ s="$K -n default get svc ops-beacon-svc" eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" NodePort && eq "$($s -o jsonpath='{.spec.ports[0].port}')" 80 && eq "$($s -o jsonpath='{.spec.ports[0].targetPort}')" 80 || exit 1 ip=$(j -n default pod "ops-beacon-$CP" '{.status.podIP}'); [ -n "$ip" ] || exit 1 j -n default endpointslices -l kubernetes.io/service-name=ops-beacon-svc '{.items[*].endpoints[*].addresses[*]}' | tr ' ' '\n' | grep -qx "$ip" """), - C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q509/nodeport.txt" 2>/dev/null)" "$np" """), + C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q94/nodeport.txt" 2>/dev/null)" "$np" """), ], solution=""" k run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon $do > beacon.yaml @@ -481,19 +481,19 @@ exit k -n default get pod ops-beacon-{CP} -o wide # mirror pod, named - k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80 k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc # the mirror pod's IP -k -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q509/nodeport.txt +k -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt # The control-plane taint only keeps the SCHEDULER from placing pods there; the kubelet runs its # static pods regardless. `expose pod` copies the pod's labels into the selector. """, ) -# ------------------------------------------------------------------ 510 ---- +# ------------------------------------------------------------------ 95 ---- task( - id=510, domain="ARCH", title="The kubelet's two certificates", + id=95, domain="ARCH", title="The kubelet's two certificates", task=""" On node `{W2}` find the certificate the kubelet uses as a **client** (to talk to the API server) and the one it **serves** its own API with (port 10250). Write exactly this to -`{WS}/q510/kubelet-certs.txt`: +`{WS}/q95/kubelet-certs.txt`: ``` client-issuer: @@ -504,15 +504,15 @@ server-eku: Reach the node with `{SHELL_W2}`. """, - setup=r"""rm -rf "$WS/q510"; mkdir -p "$WS/q510" """, + setup=r"""rm -rf "$WS/q95"; mkdir -p "$WS/q95" """, checks=[ C("client certificate: issuer and extended key usage", 3, r""" -f="$WS/q510/kubelet-certs.txt"; [ -s "$f" ] || exit 1 +f="$WS/q95/kubelet-certs.txt"; [ -s "$f" ] || exit 1 iss=$(nx "$W2" 'openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -issuer' /dev/null); do helm -n oslo-helm uninstall "$r" >/dev/null 2>&1; done helm -n oslo-helm install web-legacy "$C/nginx-lite-1.0.0" >/dev/null helm -n oslo-helm install web-main "$C/nginx-lite-1.0.0" >/dev/null # helm records the release as pending-install BEFORE it starts waiting; kill it mid-wait and it # stays that way - the classic result of an interrupted install or a CI job that timed out -timeout -s KILL 6 helm -n oslo-helm install cache-broken /tmp/q513-stuck --wait --timeout 120s >/dev/null 2>&1 -rm -rf /tmp/q513-stuck +timeout -s KILL 6 helm -n oslo-helm install cache-broken /tmp/q98-stuck --wait --timeout 120s >/dev/null 2>&1 +rm -rf /tmp/q98-stuck helm -n oslo-helm status cache-broken -o json 2>/dev/null | grep -q '"status":"pending-install"' || echo "warn: cache-broken did not end up pending-install" """, checks=[ @@ -702,8 +702,8 @@ for i in $(seq 1 20); do [ "$(j -n oslo-helm deploy cache '{.status.availableRep helm -n oslo-helm list # Helm 4 shows every status; Helm 3 hides pending ones helm -n oslo-helm list --pending # works in both: cache-broken pending-install helm -n oslo-helm uninstall web-legacy -helm -n oslo-helm upgrade web-main {WS}/q513/charts/nginx-lite-1.1.0 -helm -n oslo-helm install cache {WS}/q513/charts/redis-lite --set replicaCount=2 +helm -n oslo-helm upgrade web-main {WS}/q98/charts/nginx-lite-1.1.0 +helm -n oslo-helm install cache {WS}/q98/charts/redis-lite --set replicaCount=2 helm -n oslo-helm uninstall cache-broken # also removes the Deployment it half-created helm -n oslo-helm list # web-main (1.1.0, rev 2) and cache, both deployed # A pending-install release blocks a plain re-install under the same name ("cannot re-use a name @@ -711,9 +711,9 @@ helm -n oslo-helm list # web-main (1.1.0, rev 2) and cac """, ) -# ------------------------------------------------------------------ 514 ---- +# ------------------------------------------------------------------ 99 ---- task( - id=514, domain="STOR", title="Per-replica storage for a StatefulSet", + id=99, domain="STOR", title="Per-replica storage for a StatefulSet", task=""" In Namespace `lima-data`: @@ -723,11 +723,11 @@ In Namespace `lima-data`: `/data`. 2. Once all 3 are Ready, scale the StatefulSet down to **2**. 3. Write the names of the PersistentVolumeClaims that belong to the StatefulSet and still exist - after the scale-down to `{WS}/q514/pvcs.txt`, one per line, sorted. + after the scale-down to `{WS}/q99/pvcs.txt`, one per line, sorted. """, setup=NS + r""" mkns lima-data -rm -rf "$WS/q514"; mkdir -p "$WS/q514" +rm -rf "$WS/q99"; mkdir -p "$WS/q99" $K -n lima-data delete sts ledger-db --ignore-not-found --wait=true >/dev/null 2>&1 $K -n lima-data delete svc ledger-db --ignore-not-found >/dev/null $K -n lima-data delete pvc -l app=ledger-db --ignore-not-found >/dev/null 2>&1 @@ -745,7 +745,7 @@ for i in 0 1; do eq "$($K -n lima-data exec ledger-db-$i -- cat /data/id.txt 2>/ [ "$(j -n lima-data pod ledger-db-0 '{.spec.volumes[?(@.name=="data")].persistentVolumeClaim.claimName}')" = data-ledger-db-0 ]"""), C("pvcs.txt lists every claim that survived the scale-down", 2, r""" want=$($K -n lima-data get pvc --no-headers -o custom-columns=N:.metadata.name 2>/dev/null | grep '^data-ledger-db-' | sort) -[ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q514/pvcs.txt" 2>/dev/null)" = "$want" ]"""), +[ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q99/pvcs.txt" 2>/dev/null)" = "$want" ]"""), ], solution=""" cat < {WS}/q514/pvcs.txt -cat {WS}/q514/pvcs.txt # all THREE: data-ledger-db-0, -1, -2 +k -n lima-data get pvc -o custom-columns=N:.metadata.name --no-headers | grep '^data-ledger-db-' | sort > {WS}/q99/pvcs.txt +cat {WS}/q99/pvcs.txt # all THREE: data-ledger-db-0, -1, -2 # Claims are named