vumast3r/cka-sim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: renumber the CKA bank 1-102 with no gaps

Browse Source 
Task ids used to jump 1-43 / 101-124 / 401-418 / 501-517. They now run
contiguously in the same order:

  1-43    unchanged (base and extra)
  44-67   hard pool          (was 101-124)
  68-85   Simulator Set C    (was 401-418)
  86-102  Simulator Set D    (was 501-517)

Every reference moved with its task: ids, solution keys, workspace paths
({WS}/qNN and the graders' hidden .qNN-* files), temp paths, the one
hardcoded id in code (Q56's derived canary weight), and Q-number
cross-references in the harness, README and the explainer. Each task's
paths were checked to carry only its own new number. History entries and
the local session state were remapped so past attempts stay attached to the
same tasks; retired set A/B ids keep their old numbers and match nothing.
weak-spots is regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mhxm4rsUfuE5HoytYbu2YW
This commit is contained in:
vumast3r 2026-09-22 22:05:37 -06:00
parent 228e4f790b
commit 43e2e1db2f
12 changed files with 499 additions and 499 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

122
CKA_SOLUTIONS_EXPLAINED.md
View File

@ -10,36 +10,36 @@ This bank mirrors the real CKA curriculum weights (25/15/20/10/30% across the fi
|---|---|---| |---|---|---|
| StorageClass and Dynamic Provisioning | Storage | Q16 | | StorageClass and Dynamic Provisioning | Storage | Q16 |
| PV Access Modes | Storage | Q15 | | PV Access Modes | Storage | Q15 |
| Volume Can't Be Mounted | Storage | Q35, Q115 (troubleshooting a binding/mount failure) | | Volume Can't Be Mounted | Storage | Q35, Q58 (troubleshooting a binding/mount failure) |
| PVC Data Migration | Storage | Q116 | | PVC Data Migration | Storage | Q59 |
| Deployment and StatefulSet Rollouts | Workloads & Scheduling | Q07, Q108 | | Deployment and StatefulSet Rollouts | Workloads & Scheduling | Q07, Q51 |
| Init Container and Secret | Workloads & Scheduling | Q28 + Q10/Q29 (init container *and* a broken Secret reference, combined) | | Init Container and Secret | Workloads & Scheduling | Q28 + Q10/Q29 (init container *and* a broken Secret reference, combined) |
| DaemonSet HostPath Configurator | Workloads & Scheduling | Q26 | | DaemonSet HostPath Configurator | Workloads & Scheduling | Q26 |
| Node Selector and Scheduling | Workloads & Scheduling | Q09 (this bank uses required nodeAffinity instead of nodeSelector — worth doing both) | | Node Selector and Scheduling | Workloads & Scheduling | Q09 (this bank uses required nodeAffinity instead of nodeSelector — worth doing both) |
| Node Taints and Tolerations | Workloads & Scheduling | Q25, Q107 | | Node Taints and Tolerations | Workloads & Scheduling | Q25, Q50 |
| Scheduling Pod Affinity | Workloads & Scheduling | *no direct equivalent here* — this bank only exercises anti-affinity (Q107); do this one to cover positive affinity too | | Scheduling Pod Affinity | Workloads & Scheduling | *no direct equivalent here* — this bank only exercises anti-affinity (Q50); do this one to cover positive affinity too |
| Scheduling Pod Anti Affinity | Workloads & Scheduling | Q107 | | Scheduling Pod Anti Affinity | Workloads & Scheduling | Q50 |
| Scheduling Priority | Workloads & Scheduling | Q24, Q107 | | Scheduling Priority | Workloads & Scheduling | Q24, Q50 |
| NetworkPolicy Isolation | Services & Networking | Q11, Q111 | | NetworkPolicy Isolation | Services & Networking | Q11, Q54 |
| Troubleshoot NetworkPolicies | Services & Networking | Q114 (a blocking NetworkPolicy is one of its three faults) | | Troubleshoot NetworkPolicies | Services & Networking | Q57 (a blocking NetworkPolicy is one of its three faults) |
| App Goes Public | Services & Networking | Q12 (NodePort here; LoadBalancer+autoscale adds Q23's HPA angle) | | App Goes Public | Services & Networking | Q12 (NodePort here; LoadBalancer+autoscale adds Q23's HPA angle) |
| Gateway API Setup | Services & Networking | Q22, Q113 | | Gateway API Setup | Services & Networking | Q22, Q56 |
| CoreDNS Misconfigured | Services & Networking | Q37, Q114 | | CoreDNS Misconfigured | Services & Networking | Q37, Q57 |
| TLS Secret and HTTPS | Services & Networking | Q112 | | TLS Secret and HTTPS | Services & Networking | Q55 |
| Apiserver Crash | Troubleshooting | Q123 | | Apiserver Crash | Troubleshooting | Q66 |
| Apiserver Misconfigured | Troubleshooting | Q123 | | Apiserver Misconfigured | Troubleshooting | Q66 |
| Deployment Scaling Issue | Troubleshooting | Q23 | | Deployment Scaling Issue | Troubleshooting | Q23 |
| Application Multi Container Issue | Troubleshooting | Q08, Q109 (troubleshooting angle on a multi-container pod) | | Application Multi Container Issue | Troubleshooting | Q08, Q52 (troubleshooting angle on a multi-container pod) |
| Applications Misconfigured | Troubleshooting | Q18, Q119 | | Applications Misconfigured | Troubleshooting | Q18, Q62 |
| Components Misconfigured | Troubleshooting | Q20, Q38, Q118 | | Components Misconfigured | Troubleshooting | Q20, Q38, Q61 |
| RBAC for CronJob | Cluster Architecture | Q01/Q101's RBAC pattern, applied to a CronJob's ServiceAccount specifically | | RBAC for CronJob | Cluster Architecture | Q01/Q44's RBAC pattern, applied to a CronJob's ServiceAccount specifically |
| RBAC User Permissions | Cluster Architecture | Q06, Q33, Q104 | | RBAC User Permissions | Cluster Architecture | Q06, Q33, Q47 |
| Operator Setup and CRD | Cluster Architecture | Q04, Q34, Q106 (CRD half only — this bank never writes an actual operator/controller) | | Operator Setup and CRD | Cluster Architecture | Q04, Q34, Q49 (CRD half only — this bank never writes an actual operator/controller) |
| Operator Namespace Creator | Cluster Architecture | *no equivalent here* — this bank doesn't cover writing controller logic (arguably CKAD/dev territory more than CKA) | | Operator Namespace Creator | Cluster Architecture | *no equivalent here* — this bank doesn't cover writing controller logic (arguably CKAD/dev territory more than CKA) |
| Static Pod Move | Cluster Architecture | Q05, Q103 | | Static Pod Move | Cluster Architecture | Q05, Q46 |
| Cluster Setup and Node Join | Cluster Architecture | Q40 (token) + Q124 (independently verifying the CA discovery hash and real in-cluster API endpoint) — the token/crypto half only, see gap note | | Cluster Setup and Node Join | Cluster Architecture | Q40 (token) + Q67 (independently verifying the CA discovery hash and real in-cluster API endpoint) — the token/crypto half only, see gap note |
| Cluster Upgrade | Cluster Architecture | *no equivalent here* — see gap note below | | Cluster Upgrade | Cluster Architecture | *no equivalent here* — see gap note below |
| Cluster Certificate Management | Cluster Architecture | Q06, Q122 | | Cluster Certificate Management | Cluster Architecture | Q06, Q65 |
### Real gaps worth knowing about ### Real gaps worth knowing about
@ -50,8 +50,8 @@ Two topics from Killercoda's list still have **no faithful equivalent in this si
Two other gaps this bank previously had are now closed: Two other gaps this bank previously had are now closed:
- **Apiserver Crash / Apiserver Misconfigured** → **Q123**. Same diagnostic method as the existing scheduler/controller-manager failures (Q20, Q38, Q118), just applied to kube-apiserver itself — with the added twist that *every* `kubectl` command is unusable until you fix it, so you have to work from the node directly (`crictl`, `journalctl`) rather than leaning on `kubectl describe`. - **Apiserver Crash / Apiserver Misconfigured** → **Q66**. Same diagnostic method as the existing scheduler/controller-manager failures (Q20, Q38, Q61), just applied to kube-apiserver itself — with the added twist that *every* `kubectl` command is unusable until you fix it, so you have to work from the node directly (`crictl`, `journalctl`) rather than leaning on `kubectl describe`.
- **Cluster Setup and Node Join** → partially closed by **Q124**. A full `kubeadm init`/`join` end-to-end on a fresh node isn't safe to automate here (it would mean deleting a live Node object on your shared 3-worker cluster, which several other questions pin pods to by label/taint) — but Q124 drills the cryptographic half a real join actually depends on: computing the CA's discovery hash yourself and finding the real in-cluster API endpoint, rather than trusting whatever `kubeadm token create --print-join-command` or your local kubeconfig hands you. - **Cluster Setup and Node Join** → partially closed by **Q67**. A full `kubeadm init`/`join` end-to-end on a fresh node isn't safe to automate here (it would mean deleting a live Node object on your shared 3-worker cluster, which several other questions pin pods to by label/taint) — but Q67 drills the cryptographic half a real join actually depends on: computing the CA's discovery hash yourself and finding the real in-cluster API endpoint, rather than trusting whatever `kubeadm token create --print-join-command` or your local kubeconfig hands you.
The exam-day skill this table is really pointing at, regardless of which platform you practice on: the diagnostic *method* is identical everywhere — read the actual error (`describe`, `logs`, `journalctl -u kubelet`), don't guess from memory, and verify with the tool the grader would use (`auth can-i`, `get endpointslices`, `rollout status`) rather than assuming a manifest applied cleanly means it's actually working. The exam-day skill this table is really pointing at, regardless of which platform you practice on: the diagnostic *method* is identical everywhere — read the actual error (`describe`, `logs`, `journalctl -u kubelet`), don't guess from memory, and verify with the tool the grader would use (`auth can-i`, `get endpointslices`, `rollout status`) rather than assuming a manifest applied cleanly means it's actually working.
@ -308,7 +308,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q101 · [HARD] Exact RBAC with an Existing Over-Grant ### Q44 · [HARD] Exact RBAC with an Existing Over-Grant
- **Concept**: This is Q01's lesson turned adversarial: someone already bound `cluster-admin` to the ServiceAccount "temporarily" — a stand-in for the all-too-common real-world over-grant. Least privilege means actively auditing and revoking excess access, not just adding what's missing; the grader checks both the positive grants **and** that nothing broader survives. - **Concept**: This is Q01's lesson turned adversarial: someone already bound `cluster-admin` to the ServiceAccount "temporarily" — a stand-in for the all-too-common real-world over-grant. Least privilege means actively auditing and revoking excess access, not just adding what's missing; the grader checks both the positive grants **and** that nothing broader survives.
- **Solution**: - **Solution**:
```bash ```bash
@ -329,7 +329,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q102 · [HARD] etcd Restore ### Q45 · [HARD] etcd Restore
- **Concept**: Restoring etcd doesn't overwrite the live data directory in place — `etcdutl snapshot restore` materializes a **brand-new** data directory from the snapshot, and you then repoint the etcd static pod's `hostPath` volume at that new directory. The kubelet notices the manifest change and recreates the static pod, which comes up serving whatever state existed at snapshot time — so anything created after the snapshot (like the ConfigMap this task deletes and recreates) reverts, and this is the entire point of the exercise. - **Concept**: Restoring etcd doesn't overwrite the live data directory in place — `etcdutl snapshot restore` materializes a **brand-new** data directory from the snapshot, and you then repoint the etcd static pod's `hostPath` volume at that new directory. The kubelet notices the manifest change and recreates the static pod, which comes up serving whatever state existed at snapshot time — so anything created after the snapshot (like the ConfigMap this task deletes and recreates) reverts, and this is the entire point of the exercise.
- **Solution**: - **Solution**:
```bash ```bash
@ -349,7 +349,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q103 · [HARD] Static Pod on a Relocated Manifest Path ### Q46 · [HARD] Static Pod on a Relocated Manifest Path
- **Concept**: Same mechanism as Q05, with the deliberate trap that `staticPodPath` has been changed on this node — dropping a manifest into the default `/etc/kubernetes/manifests` here does nothing, because that kubelet isn't watching it anymore. This tests whether you actually check the config instead of assuming the default. - **Concept**: Same mechanism as Q05, with the deliberate trap that `staticPodPath` has been changed on this node — dropping a manifest into the default `/etc/kubernetes/manifests` here does nothing, because that kubelet isn't watching it anymore. This tests whether you actually check the config instead of assuming the default.
- **Solution**: - **Solution**:
```bash ```bash
@ -372,7 +372,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q104 · [HARD] Self-Contained Kubeconfig Creation ### Q47 · [HARD] Self-Contained Kubeconfig Creation
- **Concept**: A kubeconfig can reference certs either by file path (`certificate-authority`, `client-certificate`) or by embedding their base64 contents directly (`certificate-authority-data`, `client-certificate-data`). Only the embedded form is truly **portable** — a path-based kubeconfig breaks the moment it's copied to a machine without those exact files at those exact paths, which is exactly what "self-contained" is testing. - **Concept**: A kubeconfig can reference certs either by file path (`certificate-authority`, `client-certificate`) or by embedding their base64 contents directly (`certificate-authority-data`, `client-certificate-data`). Only the embedded form is truly **portable** — a path-based kubeconfig breaks the moment it's copied to a machine without those exact files at those exact paths, which is exactly what "self-contained" is testing.
- **Solution**: - **Solution**:
```bash ```bash
@ -390,7 +390,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
k -n %{ns} create rolebinding %{user}-edit --clusterrole=edit --user=%{user} k -n %{ns} create rolebinding %{user}-edit --clusterrole=edit --user=%{user}
KC={WS}/q104/%{user}.kubeconfig KC={WS}/q47/%{user}.kubeconfig
k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
SERVER=$(k config view --minify -o jsonpath='{.clusters[0].cluster.server}') SERVER=$(k config view --minify -o jsonpath='{.clusters[0].cluster.server}')
@ -405,11 +405,11 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q105 · [HARD] Kustomize with Generators and Patches ### Q48 · [HARD] Kustomize with Generators and Patches
- **Concept**: This combines every non-trivial Kustomize feature at once: a **strategic-merge patch** (`patches: [{path: replicas.yaml}]`) for a simple field override, a **JSON 6902 patch** for a structural addition that strategic-merge can't express well (injecting a whole `envFrom` array element), and a `configMapGenerator`, which — critically — appends a content hash to the generated name and then **automatically rewrites every reference to that ConfigMap** elsewhere in the same kustomization. That's what makes hashed-ConfigMap rollouts safe: change the literal, get a new hash, get a new name, and Kubernetes treats it as a completely new object — pods roll to pick it up instead of silently reading stale data from a ConfigMap edited in place. - **Concept**: This combines every non-trivial Kustomize feature at once: a **strategic-merge patch** (`patches: [{path: replicas.yaml}]`) for a simple field override, a **JSON 6902 patch** for a structural addition that strategic-merge can't express well (injecting a whole `envFrom` array element), and a `configMapGenerator`, which — critically — appends a content hash to the generated name and then **automatically rewrites every reference to that ConfigMap** elsewhere in the same kustomization. That's what makes hashed-ConfigMap rollouts safe: change the literal, get a new hash, get a new name, and Kubernetes treats it as a completely new object — pods roll to pick it up instead of silently reading stale data from a ConfigMap edited in place.
- **Solution**: - **Solution**:
```bash ```bash
mkdir -p {WS}/q105/overlays/prod && cd {WS}/q105/overlays/prod mkdir -p {WS}/q48/overlays/prod && cd {WS}/q48/overlays/prod
cat > replicas.yaml <<EOF cat > replicas.yaml <<EOF
apiVersion: apps/v1 apiVersion: apps/v1
@ -446,7 +446,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q106 · [HARD] CRD with Validation and Defaults ### Q49 · [HARD] CRD with Validation and Defaults
- **Concept**: A CRD's OpenAPI v3 schema is enforced by the API server itself, before an object is ever persisted: `minimum`/`maximum` reject an out-of-range number at admission time, `required` forces a field to be present, and `default` retroactively fills in a value the caller left out. This is "structural schema" validation, and it's what separates a CRD from just being a loosely-typed blob in etcd. - **Concept**: A CRD's OpenAPI v3 schema is enforced by the API server itself, before an object is ever persisted: `minimum`/`maximum` reject an out-of-range number at admission time, `required` forces a field to be present, and `default` retroactively fills in a value the caller left out. This is "structural schema" validation, and it's what separates a CRD from just being a loosely-typed blob in etcd.
- **Solution**: - **Solution**:
```yaml ```yaml
@ -470,7 +470,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
# Step 3 — capture the rejection of an invalid one # Step 3 — capture the rejection of an invalid one
printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: bad}\nspec: {replicas: 0, image: x}\n' \ printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: bad}\nspec: {replicas: 0, image: x}\n' \
| k apply -f - 2> {WS}/q106/rejected.txt | k apply -f - 2> {WS}/q49/rejected.txt
# Step 4 — confirm defaulting # Step 4 — confirm defaulting
printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: defaulted}\nspec: {replicas: 2, image: nginx:1.27}\n' \ printf 'apiVersion: deploy.example.com/v1\nkind: %{kind}\nmetadata: {name: defaulted}\nspec: {replicas: 2, image: nginx:1.27}\n' \
@ -481,7 +481,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q122 · [HARD] Renew Control Plane Certificates ### Q65 · [HARD] Renew Control Plane Certificates
- **Concept**: kubeadm tracks every certificate it issued and can regenerate all of them from the existing cluster CA in one shot with `kubeadm certs renew all`. New certificate files on disk don't automatically get picked up by already-running static-pod processes — you have to force the kubelet to recreate those pods (moving manifests out and back is the classic trick) so each component re-reads its cert files on startup. - **Concept**: kubeadm tracks every certificate it issued and can regenerate all of them from the existing cluster CA in one shot with `kubeadm certs renew all`. New certificate files on disk don't automatically get picked up by already-running static-pod processes — you have to force the kubelet to recreate those pods (moving manifests out and back is the classic trick) so each component re-reads its cert files on startup.
- **Solution**: - **Solution**:
```bash ```bash
@ -498,7 +498,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q124 · [HARD] Prepare a Node Join Without Shell Access to the Control Plane ### Q67 · [HARD] Prepare a Node Join Without Shell Access to the Control Plane
- **Concept**: `kubeadm join` doesn't trust the control plane on faith — it verifies the server it's talking to by hashing the CA certificate's SubjectPublicKeyInfo with SHA-256 and comparing it against the `--discovery-token-ca-cert-hash` you supply, which is exactly the computation this task has you reproduce by hand instead of trusting whatever `kubeadm token create --print-join-command` prints. Separately, the address a *joining node* needs is the API server's real address on the cluster network — not the `127.0.0.1:<port>` your own kubeconfig uses, which only works because kind port-forwards that specifically for your host machine. - **Concept**: `kubeadm join` doesn't trust the control plane on faith — it verifies the server it's talking to by hashing the CA certificate's SubjectPublicKeyInfo with SHA-256 and comparing it against the `--discovery-token-ca-cert-hash` you supply, which is exactly the computation this task has you reproduce by hand instead of trusting whatever `kubeadm token create --print-join-command` prints. Separately, the address a *joining node* needs is the API server's real address on the cluster network — not the `127.0.0.1:<port>` your own kubeconfig uses, which only works because kind port-forwards that specifically for your host machine.
- **Solution**: - **Solution**:
```bash ```bash
@ -727,7 +727,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q28 · Init Containers ### Q28 · Init Containers
- **Concept**: `initContainers` run to completion, in order, before any regular container starts — the standard pattern for "prepare something the main container needs," here using a shared `emptyDir` to hand off generated content from a setup step to the long-running server. Unlike native sidecars (Q109), a plain init container fully exits before the main container's process even begins. - **Concept**: `initContainers` run to completion, in order, before any regular container starts — the standard pattern for "prepare something the main container needs," here using a shared `emptyDir` to hand off generated content from a setup step to the long-running server. Unlike native sidecars (Q52), a plain init container fully exits before the main container's process even begins.
- **Solution**: - **Solution**:
```yaml ```yaml
apiVersion: v1 apiVersion: v1
@ -810,7 +810,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q107 · [HARD] Spread, Tolerate, Prioritise — Under a Quota ### Q50 · [HARD] Spread, Tolerate, Prioritise — Under a Quota
- **Concept**: This stacks four scheduling primitives that each solve a different problem: **pod anti-affinity** (`requiredDuringSchedulingIgnoredDuringExecution` on `topologyKey: kubernetes.io/hostname`) forces one replica per node by refusing to co-locate pods matching a label selector; a **toleration** lets pods land on an otherwise-repelling tainted node; **PriorityClass** affects preemption, not placement directly; and a **ResourceQuota** caps the namespace regardless of what the scheduler would otherwise allow — so the requests must be small enough to fit inside it. - **Concept**: This stacks four scheduling primitives that each solve a different problem: **pod anti-affinity** (`requiredDuringSchedulingIgnoredDuringExecution` on `topologyKey: kubernetes.io/hostname`) forces one replica per node by refusing to co-locate pods matching a label selector; a **toleration** lets pods land on an otherwise-repelling tainted node; **PriorityClass** affects preemption, not placement directly; and a **ResourceQuota** caps the namespace regardless of what the scheduler would otherwise allow — so the requests must be small enough to fit inside it.
- **Solution**: - **Solution**:
```yaml ```yaml
@ -845,7 +845,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q108 · [HARD] Rollback to a Specific Revision, Then Pause ### Q51 · [HARD] Rollback to a Specific Revision, Then Pause
- **Concept**: Every `kubectl rollout undo --to-revision=N` targets a specific historical **ReplicaSet** by its stored revision annotation, not just "the previous one" — useful when several bad releases have shipped in a row. `rollout pause` freezes a Deployment's controller from acting on template changes: you can edit `spec.template` afterward all you like (image, env, whatever) and the controller will **not** create a new ReplicaSet or touch running pods until you `resume` — a safe way to stage a change for later. - **Concept**: Every `kubectl rollout undo --to-revision=N` targets a specific historical **ReplicaSet** by its stored revision annotation, not just "the previous one" — useful when several bad releases have shipped in a row. `rollout pause` freezes a Deployment's controller from acting on template changes: you can edit `spec.template` afterward all you like (image, env, whatever) and the controller will **not** create a new ReplicaSet or touch running pods until you `resume` — a safe way to stage a change for later.
- **Solution**: - **Solution**:
```bash ```bash
@ -863,7 +863,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q109 · [HARD] Native Sidecar with Probes and an Exact Report ### Q52 · [HARD] Native Sidecar with Probes and an Exact Report
- **Concept**: Kubernetes 1.28+ added **native sidecars**: an `initContainers` entry with `restartPolicy: Always` starts before the main containers (like any init container) but then keeps running for the pod's whole lifetime and is restarted on its own if it crashes — unlike a legacy init container, it never blocks pod readiness by "completing," and unlike a regular sidecar container it's guaranteed to start first. Combined with `terminationGracePeriodSeconds`, this also governs shutdown ordering: main containers get the grace period to exit first, and native sidecars are stopped only after. - **Concept**: Kubernetes 1.28+ added **native sidecars**: an `initContainers` entry with `restartPolicy: Always` starts before the main containers (like any init container) but then keeps running for the pod's whole lifetime and is restarted on its own if it crashes — unlike a legacy init container, it never blocks pod readiness by "completing," and unlike a regular sidecar container it's guaranteed to start first. Combined with `terminationGracePeriodSeconds`, this also governs shutdown ordering: main containers get the grace period to exit first, and native sidecars are stopped only after.
- **Solution**: - **Solution**:
```yaml ```yaml
@ -888,14 +888,14 @@ The exam-day skill this table is really pointing at, regardless of which platfor
``` ```
```bash ```bash
k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name}{"\n"}{.spec.containers[*].name}' \ k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name}{"\n"}{.spec.containers[*].name}' \
| tr ' ' '\n' | grep . | sort > {WS}/q109/containers.txt | tr ' ' '\n' | grep . | sort > {WS}/q52/containers.txt
k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' | tr -d '\n' > {WS}/q109/ip.txt k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' | tr -d '\n' > {WS}/q52/ip.txt
``` ```
- **Gotchas**: never mount an emptyDir over `/var/log` in an nginx container — it hides nginx's own `/var/log/nginx` and nginx dies with `could not open error log file`, crash-looping forever. A volume mount **replaces** the directory's contents; mount a subdirectory instead. `restartPolicy: Always` goes **inside the initContainer entry itself**, not at the pod level — that per-container field is exactly what flips a normal init container into a native sidecar. The report file must list container names sorted, one per line, no header — matching the check's own `sort` byte-for-byte. - **Gotchas**: never mount an emptyDir over `/var/log` in an nginx container — it hides nginx's own `/var/log/nginx` and nginx dies with `could not open error log file`, crash-looping forever. A volume mount **replaces** the directory's contents; mount a subdirectory instead. `restartPolicy: Always` goes **inside the initContainer entry itself**, not at the pod level — that per-container field is exactly what flips a normal init container into a native sidecar. The report file must list container names sorted, one per line, no header — matching the check's own `sort` byte-for-byte.
--- ---
### Q110 · [HARD] Cluster Audit to Exact Files ### Q53 · [HARD] Cluster Audit to Exact Files
- **Concept**: "Produce exact files" tasks are a scripting exercise disguised as a Kubernetes question — the grader diffs your output against a canonical, sorted, header-free rendering of live cluster state, so the reliable path is always `-o json` piped through `jq`/`python3`, never hand-transcribing what `kubectl get` prints to the terminal. - **Concept**: "Produce exact files" tasks are a scripting exercise disguised as a Kubernetes question — the grader diffs your output against a canonical, sorted, header-free rendering of live cluster state, so the reliable path is always `-o json` piped through `jq`/`python3`, never hand-transcribing what `kubectl get` prints to the terminal.
- **Solution**: - **Solution**:
```bash ```bash
@ -905,28 +905,28 @@ The exam-day skill this table is really pointing at, regardless of which platfor
for p in json.load(sys.stdin)["items"]: for p in json.load(sys.stdin)["items"]:
if p["metadata"]["namespace"]=="kube-system": continue if p["metadata"]["namespace"]=="kube-system": continue
if not p["spec"]["containers"][0].get("readinessProbe"): if not p["spec"]["containers"][0].get("readinessProbe"):
print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort > {WS}/q110/no-readiness.txt print(p["metadata"]["namespace"]+"/"+p["metadata"]["name"])' | sort > {WS}/q53/no-readiness.txt
# <node> <count> of Running pods per node, all namespaces # <node> <count> of Running pods per node, all namespaces
k get pods -A -o json | python3 -c ' k get pods -A -o json | python3 -c '
import sys,json,collections import sys,json,collections
c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"] c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"]
if p["status"]["phase"]=="Running" and p["spec"].get("nodeName")) if p["status"]["phase"]=="Running" and p["spec"].get("nodeName"))
for n in sorted(c): print(n, c[n])' > {WS}/q110/per-node.txt for n in sorted(c): print(n, c[n])' > {WS}/q53/per-node.txt
# every distinct image running in kube-system # every distinct image running in kube-system
k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' \ k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' \
| tr ' ' '\n' | sort -u > {WS}/q110/images.txt | tr ' ' '\n' | sort -u > {WS}/q53/images.txt
``` ```
- **Gotchas**: no header lines, no trailing commentary — the checker does a literal `sort`-then-compare against each file's contents. Excluding `kube-system` from the first file but restricting *to* `kube-system` in the third is easy to flip by accident; re-read exactly which file wants which scope before running the one-liners. - **Gotchas**: no header lines, no trailing commentary — the checker does a literal `sort`-then-compare against each file's contents. Excluding `kube-system` from the first file but restricting *to* `kube-system` in the third is easy to flip by accident; re-read exactly which file wants which scope before running the one-liners.
--- ---
### Q121 · [HARD] The Context's Namespace ### Q64 · [HARD] The Context's Namespace
- **Concept**: A kubeconfig **context** bundles a cluster, a user, and a default namespace (`contexts[].context.namespace`) — any command run without an explicit `-n`/`--namespace` inherits that default, falling back to literal `default` only if the context doesn't set one. `kubectl config view --minify` collapses the kubeconfig down to just the active context, which is the safe way to read that default without assuming it's `default`. - **Concept**: A kubeconfig **context** bundles a cluster, a user, and a default namespace (`contexts[].context.namespace`) — any command run without an explicit `-n`/`--namespace` inherits that default, falling back to literal `default` only if the context doesn't set one. `kubectl config view --minify` collapses the kubeconfig down to just the active context, which is the safe way to read that default without assuming it's `default`.
- **Solution**: - **Solution**:
```bash ```bash
k config view --minify -o jsonpath='{..namespace}' > {WS}/q121/ns.txt k config view --minify -o jsonpath='{..namespace}' > {WS}/q64/ns.txt
# No -n anywhere below — everything inherits the context's namespace # No -n anywhere below — everything inherits the context's namespace
k create deploy %{dep} --image=busybox:1.36 --replicas=2 -- sleep 3600 k create deploy %{dep} --image=busybox:1.36 --replicas=2 -- sleep 3600
@ -1102,7 +1102,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q111 · [HARD] Zero-Trust Namespace ### Q54 · [HARD] Zero-Trust Namespace
- **Concept**: True zero-trust starts from **default-deny both directions**, then layers narrow allow rules back in — order of construction matters conceptually even though NetworkPolicies are all additive/OR'd together: without the deny-all baseline first, you're just adding allow rules on top of an implicitly-open namespace and nothing is actually restricted. Every allow-egress-to-DNS rule is non-negotiable in a default-deny-egress namespace, exactly as in Q42, or literally nothing else works either. - **Concept**: True zero-trust starts from **default-deny both directions**, then layers narrow allow rules back in — order of construction matters conceptually even though NetworkPolicies are all additive/OR'd together: without the deny-all baseline first, you're just adding allow rules on top of an implicitly-open namespace and nothing is actually restricted. Every allow-egress-to-DNS rule is non-negotiable in a default-deny-egress namespace, exactly as in Q42, or literally nothing else works either.
- **Solution**: - **Solution**:
```yaml ```yaml
@ -1152,7 +1152,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q112 · [HARD] TLS Ingress with a Generated Certificate ### Q55 · [HARD] TLS Ingress with a Generated Certificate
- **Concept**: TLS termination at the Ingress layer needs a `kubernetes.io/tls` Secret holding `tls.crt`/`tls.key`, referenced from `spec.tls[].secretName`; for a self-signed lab cert, `openssl req -x509` generates both the key and a self-signed certificate in one step (no separate CA involved) with the `CN` acting as the hostname it's valid for. - **Concept**: TLS termination at the Ingress layer needs a `kubernetes.io/tls` Secret holding `tls.crt`/`tls.key`, referenced from `spec.tls[].secretName`; for a self-signed lab cert, `openssl req -x509` generates both the key and a self-signed certificate in one step (no separate CA involved) with the `CN` acting as the hostname it's valid for.
- **Solution**: - **Solution**:
```bash ```bash
@ -1175,13 +1175,13 @@ The exam-day skill this table is really pointing at, regardless of which platfor
paths: [{path: /, pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}] paths: [{path: /, pathType: Prefix, backend: {service: {name: web, port: {number: 80}}}}]
``` ```
```bash ```bash
openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > {WS}/q112/fp.txt openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > {WS}/q55/fp.txt
``` ```
- **Gotchas**: `-nodes` on `openssl req` means "no DES" — i.e. don't encrypt the private key with a passphrase, which you'd otherwise be prompted for and which `kubectl create secret tls` can't supply interactively. The fingerprint recorded in `fp.txt` must match the certificate **actually stored in the Secret**, not just the local `tls.crt` file — if you regenerate the cert after creating the Secret, they'll drift apart. - **Gotchas**: `-nodes` on `openssl req` means "no DES" — i.e. don't encrypt the private key with a passphrase, which you'd otherwise be prompted for and which `kubectl create secret tls` can't supply interactively. The fingerprint recorded in `fp.txt` must match the certificate **actually stored in the Secret**, not just the local `tls.crt` file — if you regenerate the cert after creating the Secret, they'll drift apart.
--- ---
### Q113 · [HARD] Gateway API Canary with Header Match ### Q56 · [HARD] Gateway API Canary with Header Match
- **Concept**: `HTTPRoute` rules are evaluated top-to-bottom for a match; a rule with a header `matches` clause only applies to requests carrying that header, so stacking a header-matched 100%-to-`v2` rule *before* a header-less weighted-split rule implements a canary override — anyone sending the magic header always gets the canary, everyone else gets the normal weighted split. `parentRefs[].sectionName` scopes the route's attachment to one specific listener on a multi-listener Gateway, rather than every listener. - **Concept**: `HTTPRoute` rules are evaluated top-to-bottom for a match; a rule with a header `matches` clause only applies to requests carrying that header, so stacking a header-matched 100%-to-`v2` rule *before* a header-less weighted-split rule implements a canary override — anyone sending the magic header always gets the canary, everyone else gets the normal weighted split. `parentRefs[].sectionName` scopes the route's attachment to one specific listener on a multi-listener Gateway, rather than every listener.
- **Solution**: - **Solution**:
```yaml ```yaml
@ -1215,7 +1215,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q114 · [HARD] DNS Down: Three Faults ### Q57 · [HARD] DNS Down: Three Faults
- **Concept**: A single symptom ("DNS is broken") can have multiple independent causes stacked on top of each other — this task combines a bad upstream `forward` target in the Corefile, CoreDNS scaled to zero, and a stray NetworkPolicy blocking ingress to the CoreDNS pods, all at once. The exam-realistic lesson is to check **each** layer (config, replica count, network policy) rather than stopping the moment the first plausible cause is found. - **Concept**: A single symptom ("DNS is broken") can have multiple independent causes stacked on top of each other — this task combines a bad upstream `forward` target in the Corefile, CoreDNS scaled to zero, and a stray NetworkPolicy blocking ingress to the CoreDNS pods, all at once. The exam-realistic lesson is to check **each** layer (config, replica count, network policy) rather than stopping the moment the first plausible cause is found.
- **Solution**: - **Solution**:
```bash ```bash
@ -1338,7 +1338,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q115 · [HARD] Bind the Right Local Volume ### Q58 · [HARD] Bind the Right Local Volume
- **Concept**: `local` PersistentVolumes are tied to a specific node via `nodeAffinity` — the scheduler must place any pod using that volume on exactly that node, which is why choosing the *wrong* one of two same-class PVs quietly schedules your pod onto the wrong hardware. A PVC's `spec.selector` (label match) is the tool for picking a specific PV among several of the same StorageClass **without editing the PVs themselves** — exactly the constraint this task imposes. - **Concept**: `local` PersistentVolumes are tied to a specific node via `nodeAffinity` — the scheduler must place any pod using that volume on exactly that node, which is why choosing the *wrong* one of two same-class PVs quietly schedules your pod onto the wrong hardware. A PVC's `spec.selector` (label match) is the tool for picking a specific PV among several of the same StorageClass **without editing the PVs themselves** — exactly the constraint this task imposes.
- **Solution**: - **Solution**:
```yaml ```yaml
@ -1363,7 +1363,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q116 · [HARD] Migrate Data Between Claims ### Q59 · [HARD] Migrate Data Between Claims
- **Concept**: There's no built-in "resize/migrate a volume to a different backend" primitive — the standard pattern is a throwaway `Job` that mounts both the old and new PVCs simultaneously and copies data between their mount paths, then you cut the live workload over to the new claim. Setting the old PV's `persistentVolumeReclaimPolicy` to `Retain` *before* deleting its PVC is what keeps the original data recoverable (`Released`, not gone) in case the migration needs to be redone. - **Concept**: There's no built-in "resize/migrate a volume to a different backend" primitive — the standard pattern is a throwaway `Job` that mounts both the old and new PVCs simultaneously and copies data between their mount paths, then you cut the live workload over to the new claim. Setting the old PV's `persistentVolumeReclaimPolicy` to `Retain` *before* deleting its PVC is what keeps the original data recoverable (`Released`, not gone) in case the migration needs to be redone.
- **Solution**: - **Solution**:
```bash ```bash
@ -1534,7 +1534,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q117 · [HARD] Worker Node: Three Things Wrong ### Q60 · [HARD] Worker Node: Three Things Wrong
- **Concept**: This stacks three unrelated failure modes onto one node — a cordon (`spec.unschedulable: true`, which stops new scheduling but is unrelated to `Ready` status), a wrong `clientCAFile` path (kubelet can't start, per Q39), and the kubelet service being stopped/disabled (per Q17) — to test that you don't stop investigating after the first fix "seems to work." - **Concept**: This stacks three unrelated failure modes onto one node — a cordon (`spec.unschedulable: true`, which stops new scheduling but is unrelated to `Ready` status), a wrong `clientCAFile` path (kubelet can't start, per Q39), and the kubelet service being stopped/disabled (per Q17) — to test that you don't stop investigating after the first fix "seems to work."
- **Solution**: - **Solution**:
```bash ```bash
@ -1556,7 +1556,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q118 · [HARD] Two Control-Plane Components Down ### Q61 · [HARD] Two Control-Plane Components Down
- **Concept**: The scheduler-down (Q20) and controller-manager-down (Q38) failure signatures combine here — pods that do get created stay `Pending` (scheduler dead) *and* brand-new Deployments never even get a ReplicaSet (controller-manager dead), for two entirely unrelated static-pod manifest problems. Fixing only one still leaves the cluster non-functional for new workloads. - **Concept**: The scheduler-down (Q20) and controller-manager-down (Q38) failure signatures combine here — pods that do get created stay `Pending` (scheduler dead) *and* brand-new Deployments never even get a ReplicaSet (controller-manager dead), for two entirely unrelated static-pod manifest problems. Fixing only one still leaves the cluster non-functional for new workloads.
- **Solution**: - **Solution**:
```bash ```bash
@ -1578,7 +1578,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q119 · [HARD] Application Down for Four Reasons ### Q62 · [HARD] Application Down for Four Reasons
- **Concept**: Four independent faults are stacked on one app: a `ResourceQuota` capping pods at 1 (blocking scale-up entirely), a broken image tag, a Secret missing the key the container's `env` references (`CreateContainerConfigError`), and a Service `selector`/`targetPort` mismatch (Q18's pattern again) — and critically, the fix must happen **in place**: the Deployment's UID and the Service's ClusterIP must not change, ruling out delete-and-recreate as a shortcut. - **Concept**: Four independent faults are stacked on one app: a `ResourceQuota` capping pods at 1 (blocking scale-up entirely), a broken image tag, a Secret missing the key the container's `env` references (`CreateContainerConfigError`), and a Service `selector`/`targetPort` mismatch (Q18's pattern again) — and critically, the fix must happen **in place**: the Deployment's UID and the Service's ClusterIP must not change, ruling out delete-and-recreate as a shortcut.
- **Solution**: - **Solution**:
```bash ```bash
@ -1604,7 +1604,7 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q120 · [HARD] Cluster Facts, Exactly ### Q63 · [HARD] Cluster Facts, Exactly
- **Concept**: This is a "read the cluster's actual configuration, don't guess" exercise — pod/service CIDRs live in the static pod command-line flags of `kube-controller-manager`/`kube-apiserver` (not in any single friendly `kubectl get` field), the cluster DNS address is just the `kube-dns` Service's ClusterIP, kube-proxy's mode is buried in its ConfigMap (and an *empty* mode value means the default, `iptables` — not "not configured"), and per-node pod CIDR lives on the Node object itself. - **Concept**: This is a "read the cluster's actual configuration, don't guess" exercise — pod/service CIDRs live in the static pod command-line flags of `kube-controller-manager`/`kube-apiserver` (not in any single friendly `kubectl get` field), the cluster DNS address is just the `kube-dns` Service's ClusterIP, kube-proxy's mode is buried in its ConfigMap (and an *empty* mode value means the default, `iptables` — not "not configured"), and per-node pod CIDR lives on the Node object itself.
- **Solution**: - **Solution**:
```bash ```bash
@ -1625,8 +1625,8 @@ The exam-day skill this table is really pointing at, regardless of which platfor
--- ---
### Q123 · [HARD] Nothing Works: kube-apiserver Itself Is Down ### Q66 · [HARD] Nothing Works: kube-apiserver Itself Is Down
- **Concept**: Every other control-plane failure in this bank (scheduler, controller-manager, even etcd mid-restore) leaves `kubectl` itself usable, because kube-apiserver is still up to talk to. A broken kube-apiserver is categorically different: it's the single component every `kubectl` command depends on, so once it's down, `kubectl describe`/`logs`/`get` all stop working simultaneously and give you nothing to work with. The kubelet, however, doesn't need the API server to manage **static pods** — it reads the manifest directly off disk — so the fix is the same class of move as Q20/Q118 (repair a static pod manifest), just reached without `kubectl` in hand. - **Concept**: Every other control-plane failure in this bank (scheduler, controller-manager, even etcd mid-restore) leaves `kubectl` itself usable, because kube-apiserver is still up to talk to. A broken kube-apiserver is categorically different: it's the single component every `kubectl` command depends on, so once it's down, `kubectl describe`/`logs`/`get` all stop working simultaneously and give you nothing to work with. The kubelet, however, doesn't need the API server to manage **static pods** — it reads the manifest directly off disk — so the fix is the same class of move as Q20/Q61 (repair a static pod manifest), just reached without `kubectl` in hand.
- **Solution**: - **Solution**:
```bash ```bash
kubectl get nodes # connection refused / timeout -- confirms it's the apiserver, not your network kubectl get nodes # connection refused / timeout -- confirms it's the apiserver, not your network

6
README.md
View File

@ -108,7 +108,7 @@ set -x CKA_CP cp1; set -x CKA_W1 node1; set -x CKA_W2 node2; set -x CKA_W3 no
`/opt/cka` and `/etc/kubernetes/manifests`, so the login must be able to. Managed clusters `/opt/cka` and `/etc/kubernetes/manifests`, so the login must be able to. Managed clusters
(EKS/AKS/GKE) won't work — no node or control-plane access. Startup preflights all of this. (EKS/AKS/GKE) won't work — no node or control-plane access. Startup preflights all of this.
The etcd tasks (2 and 102) need `etcdctl`/`etcdutl` **on the control-plane node**, as the real The etcd tasks (2 and 45) need `etcdctl`/`etcdutl` **on the control-plane node**, as the real
exam has them — they cannot be copied out of the running pod, since the etcd image is distroless. exam has them — they cannot be copied out of the running pod, since the etcd image is distroless.
`setup.sh` installs them on kind; on any cluster, seeding those tasks installs them if they are `setup.sh` installs them on kind; on any cluster, seeding those tasks installs them if they are
missing (from the etcd image in the node's containerd, else the upstream release, else apt) and missing (from the etcd image in the node's containerd, else the upstream release, else apt) and
@ -127,7 +127,7 @@ fails loudly if it cannot. `./etcd-tools.sh` does the same by hand.
Disruptive tasks (anything that breaks a node or control-plane component) are seeded last and Disruptive tasks (anything that breaks a node or control-plane component) are seeded last and
can block other tasks until fixed — read everything first, as in the real exam. Tasks that damage can block other tasks until fixed — read everything first, as in the real exam. Tasks that damage
or depend on the same shared state declare it in `breaks` and never share a session, and the etcd or depend on the same shared state declare it in `breaks` and never share a session, and the etcd
restore (Q102) is `solo`: restoring rolls the whole cluster back, so it only runs on its own. restore (Q45) is `solo`: restoring rolls the whole cluster back, so it only runs on its own.
Before every session a baseline pass repairs anything a previous, unfinished session left broken. Before every session a baseline pass repairs anything a previous, unfinished session left broken.
The kindnet CNI here **enforces** NetworkPolicy (it embeds a policy engine), so a wrong policy The kindnet CNI here **enforces** NetworkPolicy (it embeds a policy engine), so a wrong policy
@ -145,7 +145,7 @@ were retired in favour of these two; their last versions are in git history.)
Banks live in `tracks/cka/{questions,bank_extra,bank_hard,bank_sim_*}.py`; a task is: Banks live in `tracks/cka/{questions,bank_extra,bank_hard,bank_sim_*}.py`; a task is:
```python ```python
{"id": 44, "domain": "NET", "difficulty": "normal", # or "hard" {"id": 103, "domain": "NET", "difficulty": "normal", # or "hard" - ids stay contiguous
"params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])}, "params": lambda r: {"ns": ns_pick(r, "svc"), "port": r.choice([8080, 9090])},
"title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...", "title": "...", "task": "...markdown; %{ns} and %{port} are substituted; `code` copies on click...",
"setup": "bash that seeds the scenario (may use %{ns})", "setup": "bash that seeds the scenario (may use %{ns})",

40
exam
View File

@ -212,8 +212,8 @@ def q_by_id(n: int) -> dict[str, Any]:
for q in QUESTIONS: for q in QUESTIONS:
if q["id"] == n: if q["id"] == n:
return q return q
# ids are not contiguous (1-43 base, 101-124 hard, 401-418 / 501-517 sim sets), so print the # ids run 1..N without gaps today, but print the real ranges rather than "1-<count>" so a
# real ranges rather than "1-<count>", which named ids that do not exist. # future gap can never make this message name ids that do not exist.
ids = sorted(q["id"] for q in QUESTIONS) ids = sorted(q["id"] for q in QUESTIONS)
ranges, start, prev = [], ids[0], ids[0] ranges, start, prev = [], ids[0], ids[0]
for i in ids[1:] + [None]: for i in ids[1:] + [None]:
@ -374,14 +374,14 @@ def drilled_domains():
return out return out
def breaks_of(q): def breaks_of(q):
"""A task's "breaks" as a set. It is a string or a list: Q118 damages the scheduler AND the """A task's "breaks" as a set. It is a string or a list: Q61 damages the scheduler AND the
controller-manager, and declaring only one let it share a session with Q38.""" controller-manager, and declaring only one let it share a session with Q38."""
b = q.get("breaks") b = q.get("breaks")
return set() if not b else ({b} if isinstance(b, str) else set(b)) return set() if not b else ({b} if isinstance(b, str) else set(b))
def no_conflicts(cands, n, taken=()): def no_conflicts(cands, n, taken=()):
"""Take up to n tasks, never two that break the same thing (or clash with tasks already taken). """Take up to n tasks, never two that break the same thing (or clash with tasks already taken).
A "solo" task (Q102's etcd restore rolls the WHOLE cluster back to its snapshot, wiping every A "solo" task (Q45's etcd restore rolls the WHOLE cluster back to its snapshot, wiping every
other task's seed and answers) is only ever taken on its own.""" other task's seed and answers) is only ever taken on its own."""
used = set().union(*(breaks_of(q) for q in taken)) if taken else set() used = set().union(*(breaks_of(q) for q in taken)) if taken else set()
out = [] out = []
@ -635,7 +635,7 @@ def _log_msg(*args: Any, **kwargs: Any) -> None:
# restores every known "breaks" target to a good baseline before each session seeds, so # restores every known "breaks" target to a good baseline before each session seeds, so
# leftover damage never leaks across sessions regardless of which questions get sampled. # leftover damage never leaks across sessions regardless of which questions get sampled.
RESTORE_BASELINE = r""" RESTORE_BASELINE = r"""
# apiserver FIRST: Q123 renames the command to "kube-apiserverr", which takes the whole API # apiserver FIRST: Q66 renames the command to "kube-apiserverr", which takes the whole API
# down - every kubectl-based step below would fail until this is undone. The repair goes over # down - every kubectl-based step below would fail until this is undone. The repair goes over
# podman/ssh (nx), not kubectl, so it works precisely when the cluster is unreachable. # podman/ssh (nx), not kubectl, so it works precisely when the cluster is unreachable.
if nx "$CP" 'grep -q "^ *- kube-apiserverr$" /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null 2>/dev/null; then if nx "$CP" 'grep -q "^ *- kube-apiserverr$" /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null 2>/dev/null; then
@ -644,8 +644,8 @@ if nx "$CP" 'grep -q "^ *- kube-apiserverr$" /etc/kubernetes/manifests/kube-apis
for _ in $(seq 1 30); do kubectl --context "${CKA_CONTEXT:-kind-cka-sim}" get --raw=/readyz >/dev/null 2>&1 && break; sleep 2; done for _ in $(seq 1 30); do kubectl --context "${CKA_CONTEXT:-kind-cka-sim}" get --raw=/readyz >/dev/null 2>&1 && break; sleep 2; done
fi fi
# scheduler: undo the "kube-scheduler" -> "kube-schedulerr" command typo (Q20/Q118), and put the # scheduler: undo the "kube-scheduler" -> "kube-schedulerr" command typo (Q20/Q61), and put the
# manifest back if a task had the candidate stop the scheduler by moving it out (Q512) and they # manifest back if a task had the candidate stop the scheduler by moving it out (Q97) and they
# never moved it back - that setup keeps a pristine copy outside the static pod directory. # never moved it back - that setup keeps a pristine copy outside the static pod directory.
nx "$CP" ' nx "$CP" '
f=/etc/kubernetes/manifests/kube-scheduler.yaml f=/etc/kubernetes/manifests/kube-scheduler.yaml
@ -653,7 +653,7 @@ f=/etc/kubernetes/manifests/kube-scheduler.yaml
sed -i "s|^\( *\)- kube-schedulerr$|\1- kube-scheduler|" $f 2>/dev/null sed -i "s|^\( *\)- kube-schedulerr$|\1- kube-scheduler|" $f 2>/dev/null
' </dev/null ' </dev/null
# controller-manager: undo Q38's injected flag and Q118's broken image tag. The last two lines # controller-manager: undo Q38's injected flag and Q61's broken image tag. The last two lines
# undo the retired set B's Q317 (the CSR signing controller switched off in --controllers, or in # undo the retired set B's Q317 (the CSR signing controller switched off in --controllers, or in
# its older form --cluster-signing-cert-file pointed at a missing file, which crash-looped the WHOLE # its older form --cluster-signing-cert-file pointed at a missing file, which crash-looped the WHOLE
# controller-manager); a cluster that ever ran that set may still carry either. # controller-manager); a cluster that ever ran that set may still carry either.
@ -665,14 +665,14 @@ sed -i "s|--cluster-signing-cert-file=/etc/kubernetes/pki/ca-signing.crt|--clust
sed -i "s|,-certificatesigningrequest-signing-controller||; s|,-csrsigning||" $f 2>/dev/null sed -i "s|,-certificatesigningrequest-signing-controller||; s|,-csrsigning||" $f 2>/dev/null
' </dev/null ' </dev/null
# coredns: restore the default upstream forward (Q37/Q114 point it somewhere bogus) # coredns: restore the default upstream forward (Q37/Q57 point it somewhere bogus)
cf=$($K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' 2>/dev/null) cf=$($K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' 2>/dev/null)
if [ -n "$cf" ] && ! printf '%s' "$cf" | grep -Eq '^[[:space:]]*forward \. /etc/resolv\.conf'; then if [ -n "$cf" ] && ! printf '%s' "$cf" | grep -Eq '^[[:space:]]*forward \. /etc/resolv\.conf'; then
printf '%s' "$cf" | sed -E 's|^([[:space:]]*forward \. )[^[:space:]{]+|\1/etc/resolv.conf|' > /tmp/Corefile.restore printf '%s' "$cf" | sed -E 's|^([[:space:]]*forward \. )[^[:space:]{]+|\1/etc/resolv.conf|' > /tmp/Corefile.restore
$K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.restore --dry-run=client -o yaml | $K apply -f - >/dev/null 2>&1 $K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.restore --dry-run=client -o yaml | $K apply -f - >/dev/null 2>&1
$K -n kube-system rollout restart deploy coredns >/dev/null 2>&1 $K -n kube-system rollout restart deploy coredns >/dev/null 2>&1
fi fi
# ...and Q114's other two faults. kindnet ENFORCES NetworkPolicy, so a leftover lockdown policy on # ...and Q57's other two faults. kindnet ENFORCES NetworkPolicy, so a leftover lockdown policy on
# the CoreDNS pods kills name resolution for the whole cluster in every later session, and so does # the CoreDNS pods kills name resolution for the whole cluster in every later session, and so does
# a Deployment left at 0 replicas. # a Deployment left at 0 replicas.
$K -n kube-system delete netpol maintenance-lockdown --ignore-not-found >/dev/null 2>&1 $K -n kube-system delete netpol maintenance-lockdown --ignore-not-found >/dev/null 2>&1
@ -680,14 +680,14 @@ r=$($K -n kube-system get deploy coredns -o jsonpath='{.spec.replicas}' 2>/dev/n
if [ -n "$r" ] && [ "$r" -lt 2 ]; then $K -n kube-system scale deploy coredns --replicas=2 >/dev/null 2>&1; fi if [ -n "$r" ] && [ "$r" -lt 2 ]; then $K -n kube-system scale deploy coredns --replicas=2 >/dev/null 2>&1; fi
# w1/w2/w3 kubelet: undo Q17/Q39-style tampering, then make sure it's enabled and running. # w1/w2/w3 kubelet: undo Q17/Q39-style tampering, then make sure it's enabled and running.
# staticPodPath is normalised unconditionally: Q103 relocates it to /etc/kubernetes/static, and # staticPodPath is normalised unconditionally: Q46 relocates it to /etc/kubernetes/static, and
# the config.yaml.orig copy below only helps if something made that backup first - nothing does, # the config.yaml.orig copy below only helps if something made that backup first - nothing does,
# so a relocated path used to survive every later session and silently break Q05 (whose check # so a relocated path used to survive every later session and silently break Q05 (whose check
# looks in /etc/kubernetes/manifests). Only restart the kubelet if we actually changed the file. # looks in /etc/kubernetes/manifests). Only restart the kubelet if we actually changed the file.
# clientCAFile gets the same treatment: Q39 points it at ca-bundle.crt and Q117 at ca-wrong.crt, and # clientCAFile gets the same treatment: Q39 points it at ca-bundle.crt and Q60 at ca-wrong.crt, and
# only W1 ever gets a config.yaml.orig (Q39 makes it), so an unfinished Q117 left W2's kubelet # only W1 ever gets a config.yaml.orig (Q39 makes it), so an unfinished Q60 left W2's kubelet
# crash-looping in every later session - enable/start below cannot help a kubelet that exits. # crash-looping in every later session - enable/start below cannot help a kubelet that exits.
# The systemd drop-in is checked too: Q507 points ExecStart at a kubelet binary that does not exist. # The systemd drop-in is checked too: Q92 points ExecStart at a kubelet binary that does not exist.
for n in "$W1" "$W2" "$W3"; do for n in "$W1" "$W2" "$W3"; do
nx "$n" ' nx "$n" '
c=/var/lib/kubelet/config.yaml; d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf c=/var/lib/kubelet/config.yaml; d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
@ -710,7 +710,7 @@ for n in "$W1" "$W2" "$W3"; do
' </dev/null ' </dev/null
done done
# static pods the bank drops on nodes: Q05's static-web on W3, Q103's relocated copies, Q509's # static pods the bank drops on nodes: Q05's static-web on W3, Q46's relocated copies, Q94's
# ops-beacon on the control plane - and edge-cache, left on W3 by the retired set B (it sat in # ops-beacon on the control plane - and edge-cache, left on W3 by the retired set B (it sat in
# ContainerCreating for days, its hostPath deliberately missing). Each setup recreates what its # ContainerCreating for days, its hostPath deliberately missing). Each setup recreates what its
# own session needs. # own session needs.
@ -729,7 +729,7 @@ if $K get sc standard >/dev/null 2>&1; then
fi fi
# w1-taint: clear every randomized demo taint key the bank can apply. Q25 draws from # w1-taint: clear every randomized demo taint key the bank can apply. Q25 draws from
# dedicated/team/workload and Q107 from pool/lane; `maintenance` was the retired set B's and may # dedicated/team/workload and Q50 from pool/lane; `maintenance` was the retired set B's and may
# still be on a cluster that ran it. Miss one and that node silently stops accepting pods. # still be on a cluster that ran it. Miss one and that node silently stops accepting pods.
# One key per call: `kubectl taint node N a- b-` aborts on the first key that is not present and # One key per call: `kubectl taint node N a- b-` aborts on the first key that is not present and
# removes NOTHING, so the single taint that is actually set survives the "cleanup" and W1 keeps # removes NOTHING, so the single taint that is actually set survives the "cleanup" and W1 keeps
@ -750,13 +750,13 @@ if $K -n kube-system get deploy metrics-server >/dev/null 2>&1 \
-p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' >/dev/null 2>&1 || true -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]' >/dev/null 2>&1 || true
fi fi
# cordons: Q117 cordons W2, and a drain task left half-done leaves a node cordoned. A cordon left # cordons: Q60 cordons W2, and a drain task left half-done leaves a node cordoned. A cordon left
# behind fails any later task that needs every worker (Q26's DaemonSet coverage, Q107's spread). # behind fails any later task that needs every worker (Q26's DaemonSet coverage, Q50's spread).
for n in "$W1" "$W2" "$W3"; do $K uncordon "$n" >/dev/null 2>&1 || true; done for n in "$W1" "$W2" "$W3"; do $K uncordon "$n" >/dev/null 2>&1 || true; done
# Deliberately NOT restored, and why: # Deliberately NOT restored, and why:
# etcd (Q102) and certs (Q122). Neither setup damages anything - Q102 only takes a snapshot # etcd (Q45) and certs (Q65). Neither setup damages anything - Q45 only takes a snapshot
# and deletes a ConfigMap, Q122 only records the current notBefore. Any damage comes from a # and deletes a ConfigMap, Q65 only records the current notBefore. Any damage comes from a
# half-finished candidate attempt (etcd.yaml left pointing at a restored data-dir, static pod # half-finished candidate attempt (etcd.yaml left pointing at a restored data-dir, static pod
# manifests moved to /tmp and never moved back), which is not deterministic: there is no way # manifests moved to /tmp and never moved back), which is not deterministic: there is no way
# to tell "mid-attempt" from "correctly completed" without guessing, and guessing wrong here # to tell "mid-attempt" from "correctly completed" without guessing, and guessing wrong here

10
history-cka.jsonl
View File

@ -1,11 +1,11 @@
{"at": "2026-09-06T22:07", "mode": "drill", "hard": false, "ids": [1, 5, 6, 34, 41], "score_pct": 8.0, "earned": 2, "total": 25, "time_used": 432, "results": {"1": {"pass": false, "earned": 2, "total": 6, "failed": ["ServiceAccount rbac-app/deploy-bot exists", "Role deploy-manager and RoleBinding deploy-bot-binding exist", "deploy-bot can get/list/watch/create/update/patch/delete deployments"]}, "5": {"pass": false, "earned": 0, "total": 5, "failed": ["manifest containing static-web is in /etc/kubernetes/manifests on the node", "mirror pod static-web-<node> is Running", "image is nginx:1.27"]}, "6": {"pass": false, "earned": 0, "total": 6, "failed": ["apiserver-expiry.txt records the real expiry month and year", "CSR jane exists with signer kubernetes.io/kube-apiserver-client", "CSR jane approved and certificate issued", "jane can list pods in default (view) but not delete them"]}, "34": {"pass": false, "earned": 0, "total": 4, "failed": ["path.txt has the correct field path", "Gadget/sample created with the field = 4"]}, "41": {"pass": false, "earned": 0, "total": 4, "failed": ["apiserver and kubelet versions correct", "etcd and runtime versions correct"]}}} {"at": "2026-09-06T22:07", "mode": "drill", "hard": false, "ids": [1, 5, 6, 34, 41], "score_pct": 8.0, "earned": 2, "total": 25, "time_used": 432, "results": {"1": {"pass": false, "earned": 2, "total": 6, "failed": ["ServiceAccount rbac-app/deploy-bot exists", "Role deploy-manager and RoleBinding deploy-bot-binding exist", "deploy-bot can get/list/watch/create/update/patch/delete deployments"]}, "5": {"pass": false, "earned": 0, "total": 5, "failed": ["manifest containing static-web is in /etc/kubernetes/manifests on the node", "mirror pod static-web-<node> is Running", "image is nginx:1.27"]}, "6": {"pass": false, "earned": 0, "total": 6, "failed": ["apiserver-expiry.txt records the real expiry month and year", "CSR jane exists with signer kubernetes.io/kube-apiserver-client", "CSR jane approved and certificate issued", "jane can list pods in default (view) but not delete them"]}, "34": {"pass": false, "earned": 0, "total": 4, "failed": ["path.txt has the correct field path", "Gadget/sample created with the field = 4"]}, "41": {"pass": false, "earned": 0, "total": 4, "failed": ["apiserver and kubelet versions correct", "etcd and runtime versions correct"]}}}
{"at": "2026-09-08T19:59", "mode": "exam", "hard": false, "ids": [2, 3, 4, 8, 12, 13, 15, 16, 17, 18, 19, 20, 25, 28, 29, 30, 33, 37, 40, 42], "score_pct": 14.3, "earned": 15, "total": 105, "time_used": 2902, "results": {"2": {"pass": true, "earned": 7, "total": 7, "failed": []}, "3": {"pass": false, "earned": 0, "total": 5, "failed": ["overlay kustomization.yaml exists", "resources deployed into namespace store-prod", "store-api has 3 replicas", "Deployment and Service carry label env=prod"]}, "4": {"pass": true, "earned": 4, "total": 4, "failed": []}, "8": {"pass": false, "earned": 0, "total": 4, "failed": ["pod log-shipper Running with containers app and shipper", "volume logs is an emptyDir mounted at /var/log in both containers", "shipper's logs show lines written by app"]}, "12": {"pass": false, "earned": 0, "total": 4, "failed": ["Service shop-svc is NodePort, port 80 -> targetPort 80", "nodePort is 30080", "Service has endpoints"]}, "13": {"pass": false, "earned": 0, "total": 5, "failed": ["ingressClassName nginx and host shop.example.com", "/api (Prefix) -> api:8080", "/ (Prefix) -> web:80"]}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "17": {"pass": false, "earned": 0, "total": 8, "failed": ["node is Ready", "kubelet is active", "kubelet is enabled (survives reboot)"]}, "18": {"pass": false, "earned": 0, "total": 7, "failed": ["all payments replicas available", "Service payments has endpoints (selector fixed)", "Service targetPort matches container port 80"]}, "19": {"pass": false, "earned": 1, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure", "pod crasher Running with DB_HOST set"]}, "20": {"pass": false, "earned": 0, "total": 8, "failed": ["kube-scheduler static pod manifest repaired", "new pods get scheduled"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries workload=batch:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "30": {"pass": false, "earned": 0, "total": 4, "failed": ["container port 80 named app-port", "Service port 8080 -> targetPort app-port (by name)", "readiness probe uses the named port; endpoints ready"]}, "33": {"pass": false, "earned": 1, "total": 5, "failed": ["ops-44 has an aggregationRule with the right label", "aggregation populated rules from the two labelled roles", "lee can list pods and services cluster-wide"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~2h"]}, "42": {"pass": false, "earned": 0, "total": 4, "failed": ["applies to app=client-47, Egress only", "DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP", "TCP 443 to 192.168.100.0/24 only"]}}} {"at": "2026-09-08T19:59", "mode": "exam", "hard": false, "ids": [2, 3, 4, 8, 12, 13, 15, 16, 17, 18, 19, 20, 25, 28, 29, 30, 33, 37, 40, 42], "score_pct": 14.3, "earned": 15, "total": 105, "time_used": 2902, "results": {"2": {"pass": true, "earned": 7, "total": 7, "failed": []}, "3": {"pass": false, "earned": 0, "total": 5, "failed": ["overlay kustomization.yaml exists", "resources deployed into namespace store-prod", "store-api has 3 replicas", "Deployment and Service carry label env=prod"]}, "4": {"pass": true, "earned": 4, "total": 4, "failed": []}, "8": {"pass": false, "earned": 0, "total": 4, "failed": ["pod log-shipper Running with containers app and shipper", "volume logs is an emptyDir mounted at /var/log in both containers", "shipper's logs show lines written by app"]}, "12": {"pass": false, "earned": 0, "total": 4, "failed": ["Service shop-svc is NodePort, port 80 -> targetPort 80", "nodePort is 30080", "Service has endpoints"]}, "13": {"pass": false, "earned": 0, "total": 5, "failed": ["ingressClassName nginx and host shop.example.com", "/api (Prefix) -> api:8080", "/ (Prefix) -> web:80"]}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "17": {"pass": false, "earned": 0, "total": 8, "failed": ["node is Ready", "kubelet is active", "kubelet is enabled (survives reboot)"]}, "18": {"pass": false, "earned": 0, "total": 7, "failed": ["all payments replicas available", "Service payments has endpoints (selector fixed)", "Service targetPort matches container port 80"]}, "19": {"pass": false, "earned": 1, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure", "pod crasher Running with DB_HOST set"]}, "20": {"pass": false, "earned": 0, "total": 8, "failed": ["kube-scheduler static pod manifest repaired", "new pods get scheduled"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries workload=batch:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "30": {"pass": false, "earned": 0, "total": 4, "failed": ["container port 80 named app-port", "Service port 8080 -> targetPort app-port (by name)", "readiness probe uses the named port; endpoints ready"]}, "33": {"pass": false, "earned": 1, "total": 5, "failed": ["ops-44 has an aggregationRule with the right label", "aggregation populated rules from the two labelled roles", "lee can list pods and services cluster-wide"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~2h"]}, "42": {"pass": false, "earned": 0, "total": 4, "failed": ["applies to app=client-47, Egress only", "DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP", "TCP 443 to 192.168.100.0/24 only"]}}}
{"at": "2026-09-10T21:07", "mode": "drill", "hard": true, "ids": [106, 112, 115, 120, 121], "score_pct": 6.5, "earned": 2, "total": 31, "time_used": 30, "results": {"106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "115": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "120": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "121": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-14 in ac-omega (the context's namespace), 2 available, owner=ac", "Service ledger-14 port 9000 in the same namespace", "ns.txt has the namespace"]}}} {"at": "2026-09-10T21:07", "mode": "drill", "hard": true, "ids": [49, 55, 58, 63, 64], "score_pct": 6.5, "earned": 2, "total": 31, "time_used": 30, "results": {"49": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "55": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "58": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "63": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "64": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-14 in ac-omega (the context's namespace), 2 available, owner=ac", "Service ledger-14 port 9000 in the same namespace", "ns.txt has the namespace"]}}}
{"at": "2026-09-12T22:17", "mode": "drill", "hard": false, "ids": [1, 6, 21, 34, 41], "score_pct": 87.5, "earned": 21, "total": 24, "time_used": 534, "results": {"1": {"pass": true, "earned": 6, "total": 6, "failed": []}, "6": {"pass": false, "earned": 3, "total": 6, "failed": ["CSR jane exists with signer kubernetes.io/kube-apiserver-client", "CSR jane approved and certificate issued"]}, "21": {"pass": true, "earned": 4, "total": 4, "failed": []}, "34": {"pass": true, "earned": 4, "total": 4, "failed": []}, "41": {"pass": true, "earned": 4, "total": 4, "failed": []}}} {"at": "2026-09-12T22:17", "mode": "drill", "hard": false, "ids": [1, 6, 21, 34, 41], "score_pct": 87.5, "earned": 21, "total": 24, "time_used": 534, "results": {"1": {"pass": true, "earned": 6, "total": 6, "failed": []}, "6": {"pass": false, "earned": 3, "total": 6, "failed": ["CSR jane exists with signer kubernetes.io/kube-apiserver-client", "CSR jane approved and certificate issued"]}, "21": {"pass": true, "earned": 4, "total": 4, "failed": []}, "34": {"pass": true, "earned": 4, "total": 4, "failed": []}, "41": {"pass": true, "earned": 4, "total": 4, "failed": []}}}
{"at": "2026-09-12T22:59", "mode": "drill", "hard": false, "ids": [17, 19, 37, 38, 39], "score_pct": 29.4, "earned": 10, "total": 34, "time_used": 1987, "results": {"17": {"pass": false, "earned": 4, "total": 8, "failed": ["kubelet is active", "kubelet is enabled (survives reboot)"]}, "19": {"pass": false, "earned": 1, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure", "pod crasher Running with DB_HOST set"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "38": {"pass": false, "earned": 0, "total": 7, "failed": ["kube-controller-manager manifest repaired and pod Running", "a new Deployment gets a ReplicaSet and running pods"]}, "39": {"pass": false, "earned": 4, "total": 7, "failed": ["kubelet config points at a CA file that exists", "kubelet active and enabled"]}}} {"at": "2026-09-12T22:59", "mode": "drill", "hard": false, "ids": [17, 19, 37, 38, 39], "score_pct": 29.4, "earned": 10, "total": 34, "time_used": 1987, "results": {"17": {"pass": false, "earned": 4, "total": 8, "failed": ["kubelet is active", "kubelet is enabled (survives reboot)"]}, "19": {"pass": false, "earned": 1, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure", "pod crasher Running with DB_HOST set"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "38": {"pass": false, "earned": 0, "total": 7, "failed": ["kube-controller-manager manifest repaired and pod Running", "a new Deployment gets a ReplicaSet and running pods"]}, "39": {"pass": false, "earned": 4, "total": 7, "failed": ["kubelet config points at a CA file that exists", "kubelet active and enabled"]}}}
{"at": "2026-09-13T01:06", "mode": "drill", "hard": false, "ids": [17, 19, 37, 38, 39], "score_pct": 29.4, "earned": 10, "total": 34, "time_used": 6649, "results": {"17": {"pass": false, "earned": 4, "total": 8, "failed": ["kubelet is active", "kubelet is enabled (survives reboot)"]}, "19": {"pass": false, "earned": 1, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure", "pod crasher Running with DB_HOST set"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "38": {"pass": false, "earned": 0, "total": 7, "failed": ["kube-controller-manager manifest repaired and pod Running", "a new Deployment gets a ReplicaSet and running pods"]}, "39": {"pass": false, "earned": 4, "total": 7, "failed": ["kubelet config points at a CA file that exists", "kubelet active and enabled"]}}} {"at": "2026-09-13T01:06", "mode": "drill", "hard": false, "ids": [17, 19, 37, 38, 39], "score_pct": 29.4, "earned": 10, "total": 34, "time_used": 6649, "results": {"17": {"pass": false, "earned": 4, "total": 8, "failed": ["kubelet is active", "kubelet is enabled (survives reboot)"]}, "19": {"pass": false, "earned": 1, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure", "pod crasher Running with DB_HOST set"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "38": {"pass": false, "earned": 0, "total": 7, "failed": ["kube-controller-manager manifest repaired and pod Running", "a new Deployment gets a ReplicaSet and running pods"]}, "39": {"pass": false, "earned": 4, "total": 7, "failed": ["kubelet config points at a CA file that exists", "kubelet active and enabled"]}}}
{"at": "2026-09-14T16:49", "mode": "drill", "hard": false, "ids": [3, 5, 6, 8, 12, 13, 15, 16, 17, 18, 19, 20, 25, 28, 29, 30, 33, 37, 38, 39, 40, 42, 106, 112, 115, 120, 121], "score_pct": 42.0, "earned": 63, "total": 150, "time_used": 510, "results": {"17": {"pass": true, "earned": 8, "total": 8, "failed": []}, "20": {"pass": true, "earned": 8, "total": 8, "failed": []}, "3": {"pass": true, "earned": 5, "total": 5, "failed": []}, "5": {"pass": true, "earned": 5, "total": 5, "failed": []}, "8": {"pass": true, "earned": 4, "total": 4, "failed": []}, "12": {"pass": true, "earned": 4, "total": 4, "failed": []}, "38": {"pass": true, "earned": 7, "total": 7, "failed": []}, "18": {"pass": true, "earned": 7, "total": 7, "failed": []}, "19": {"pass": false, "earned": 4, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure"]}, "6": {"pass": true, "earned": 6, "total": 6, "failed": []}, "13": {"pass": false, "earned": 0, "total": 5, "failed": ["ingressClassName nginx and host shop.example.com", "/api (Prefix) -> api:8080", "/ (Prefix) -> web:80"]}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries workload=batch:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "30": {"pass": false, "earned": 0, "total": 4, "failed": ["container port 80 named svc-http", "Service port 8080 -> targetPort svc-http (by name)", "readiness probe uses the named port; endpoints ready"]}, "33": {"pass": false, "earned": 1, "total": 5, "failed": ["ops-19 has an aggregationRule with the right label", "aggregation populated rules from the two labelled roles", "sam can list pods and services cluster-wide"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "39": {"pass": false, "earned": 0, "total": 7, "failed": ["kubelet config points at a CA file that exists", "node Ready", "kubelet active and enabled"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~2h"]}, "42": {"pass": false, "earned": 0, "total": 4, "failed": ["applies to app=client-64, Egress only", "DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP", "TCP 443 to 172.16.0.0/16 only"]}, "106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret admin.lab.local-tls is kubernetes.io/tls with CN admin.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "115": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "120": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "121": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-73 in ac-bravo (the context's namespace), 2 available, owner=ac", "Service ledger-73 port 9000 in the same namespace", "ns.txt has the namespace"]}}} {"at": "2026-09-14T16:49", "mode": "drill", "hard": false, "ids": [3, 5, 6, 8, 12, 13, 15, 16, 17, 18, 19, 20, 25, 28, 29, 30, 33, 37, 38, 39, 40, 42, 49, 55, 58, 63, 64], "score_pct": 42.0, "earned": 63, "total": 150, "time_used": 510, "results": {"17": {"pass": true, "earned": 8, "total": 8, "failed": []}, "20": {"pass": true, "earned": 8, "total": 8, "failed": []}, "3": {"pass": true, "earned": 5, "total": 5, "failed": []}, "5": {"pass": true, "earned": 5, "total": 5, "failed": []}, "8": {"pass": true, "earned": 4, "total": 4, "failed": []}, "12": {"pass": true, "earned": 4, "total": 4, "failed": []}, "38": {"pass": true, "earned": 7, "total": 7, "failed": []}, "18": {"pass": true, "earned": 7, "total": 7, "failed": []}, "19": {"pass": false, "earned": 4, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure"]}, "6": {"pass": true, "earned": 6, "total": 6, "failed": []}, "13": {"pass": false, "earned": 0, "total": 5, "failed": ["ingressClassName nginx and host shop.example.com", "/api (Prefix) -> api:8080", "/ (Prefix) -> web:80"]}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries workload=batch:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "30": {"pass": false, "earned": 0, "total": 4, "failed": ["container port 80 named svc-http", "Service port 8080 -> targetPort svc-http (by name)", "readiness probe uses the named port; endpoints ready"]}, "33": {"pass": false, "earned": 1, "total": 5, "failed": ["ops-19 has an aggregationRule with the right label", "aggregation populated rules from the two labelled roles", "sam can list pods and services cluster-wide"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "39": {"pass": false, "earned": 0, "total": 7, "failed": ["kubelet config points at a CA file that exists", "node Ready", "kubelet active and enabled"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~2h"]}, "42": {"pass": false, "earned": 0, "total": 4, "failed": ["applies to app=client-64, Egress only", "DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP", "TCP 443 to 172.16.0.0/16 only"]}, "49": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "55": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret admin.lab.local-tls is kubernetes.io/tls with CN admin.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "58": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "63": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "64": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-73 in ac-bravo (the context's namespace), 2 available, owner=ac", "Service ledger-73 port 9000 in the same namespace", "ns.txt has the namespace"]}}}
{"at": "2026-09-15T20:55", "mode": "drill", "hard": false, "ids": [13, 15, 16, 19, 25, 28, 29, 30, 33, 37, 39, 40, 42, 106, 112, 115, 120, 121], "score_pct": 41.7, "earned": 40, "total": 96, "time_used": 608, "results": {"13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "19": {"pass": true, "earned": 6, "total": 6, "failed": []}, "33": {"pass": true, "earned": 5, "total": 5, "failed": []}, "30": {"pass": true, "earned": 4, "total": 4, "failed": []}, "37": {"pass": true, "earned": 6, "total": 6, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~6h"]}, "42": {"pass": true, "earned": 4, "total": 4, "failed": []}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries team=finance:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "115": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "120": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "121": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-21 in ac-echo (the context's namespace), 2 available, owner=ac", "Service ledger-21 port 9000 in the same namespace", "ns.txt has the namespace"]}}} {"at": "2026-09-15T20:55", "mode": "drill", "hard": false, "ids": [13, 15, 16, 19, 25, 28, 29, 30, 33, 37, 39, 40, 42, 49, 55, 58, 63, 64], "score_pct": 41.7, "earned": 40, "total": 96, "time_used": 608, "results": {"13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "19": {"pass": true, "earned": 6, "total": 6, "failed": []}, "33": {"pass": true, "earned": 5, "total": 5, "failed": []}, "30": {"pass": true, "earned": 4, "total": 4, "failed": []}, "37": {"pass": true, "earned": 6, "total": 6, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~6h"]}, "42": {"pass": true, "earned": 4, "total": 4, "failed": []}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries team=finance:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "49": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "55": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "58": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "63": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "64": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-21 in ac-echo (the context's namespace), 2 available, owner=ac", "Service ledger-21 port 9000 in the same namespace", "ns.txt has the namespace"]}}}
{"at": "2026-09-16T19:40", "mode": "drill", "hard": false, "ids": [15, 16, 25, 28, 29, 40], "score_pct": 50.0, "earned": 14, "total": 28, "time_used": 1258, "results": {"15": {"pass": true, "earned": 6, "total": 6, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "29": {"pass": true, "earned": 4, "total": 4, "failed": []}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~24h"]}}} {"at": "2026-09-16T19:40", "mode": "drill", "hard": false, "ids": [15, 16, 25, 28, 29, 40], "score_pct": 50.0, "earned": 14, "total": 28, "time_used": 1258, "results": {"15": {"pass": true, "earned": 6, "total": 6, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "29": {"pass": true, "earned": 4, "total": 4, "failed": []}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~24h"]}}}
{"at": "2026-09-19T12:27", "mode": "drill", "hard": false, "ids": [106, 217, 34, 202, 102, 301, 39, 13, 32, 204, 25, 2, 8, 112, 4, 109, 313, 9], "score_pct": 39.7, "earned": 48, "total": 121, "time_used": 4197, "results": {"8": {"pass": true, "earned": 4, "total": 4, "failed": []}, "34": {"pass": true, "earned": 4, "total": 4, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "301": {"pass": true, "earned": 9, "total": 9, "failed": []}, "4": {"pass": true, "earned": 4, "total": 4, "failed": []}, "9": {"pass": true, "earned": 4, "total": 4, "failed": []}, "106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "217": {"pass": false, "earned": 0, "total": 10, "failed": ["Pod tiger-probe Running with both labels", "container.txt holds a real container ID for that Pod", "container.txt records the runtimeType", "container.log holds the httpd startup output"]}, "202": {"pass": false, "earned": 3, "total": 8, "failed": ["watchAllNamespaces=true reached the pod env", "Ledger main-ledger has currency USD and retentionDays 30"]}, "102": {"pass": false, "earned": 2, "total": 10, "failed": ["ConfigMap restore-proof-67 exists again with its original data", "etcd uses a restored data directory (manifest changed)"]}, "32": {"pass": false, "earned": 0, "total": 4, "failed": ["ResourceQuota limits pods/6, requests.cpu 1, limits.memory 2Gi", "LimitRange defaults set", "worker pods Running with defaulted requests"]}, "204": {"pass": false, "earned": 0, "total": 8, "failed": ["evicted-first.txt lists exactly the BestEffort pods", "api-qos.txt says Burstable", "guaranteed.txt names orion-ledger"]}, "2": {"pass": false, "earned": 0, "total": 7, "failed": ["/opt/cka/etcd-snapshot.db exists on the control-plane node", "snapshot is a real etcd backup (> 100 KB)"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret secure.lab.local-tls is kubernetes.io/tls with CN secure.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "109": {"pass": false, "earned": 2, "total": 8, "failed": ["native sidecar: initContainer agent with restartPolicy Always", "probes on app as specified", "Running, shared log written by agent, grace period 5"]}, "313": {"pass": false, "earned": 0, "total": 11, "failed": ["Secret classic-tls is a TLS secret with CN classic.r700.io", "both host rules route to the right Services on /", "TLS covers both hosts with classic-tls", "defaultBackend points at web-classic:80, class is nginx"]}}} {"at": "2026-09-19T12:27", "mode": "drill", "hard": false, "ids": [49, 217, 34, 202, 45, 301, 39, 13, 32, 204, 25, 2, 8, 55, 4, 52, 313, 9], "score_pct": 39.7, "earned": 48, "total": 121, "time_used": 4197, "results": {"8": {"pass": true, "earned": 4, "total": 4, "failed": []}, "34": {"pass": true, "earned": 4, "total": 4, "failed": []}, "39": {"pass": true, "earned": 7, "total": 7, "failed": []}, "13": {"pass": true, "earned": 5, "total": 5, "failed": []}, "25": {"pass": true, "earned": 4, "total": 4, "failed": []}, "301": {"pass": true, "earned": 9, "total": 9, "failed": []}, "4": {"pass": true, "earned": 4, "total": 4, "failed": []}, "9": {"pass": true, "earned": 4, "total": 4, "failed": []}, "49": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "217": {"pass": false, "earned": 0, "total": 10, "failed": ["Pod tiger-probe Running with both labels", "container.txt holds a real container ID for that Pod", "container.txt records the runtimeType", "container.log holds the httpd startup output"]}, "202": {"pass": false, "earned": 3, "total": 8, "failed": ["watchAllNamespaces=true reached the pod env", "Ledger main-ledger has currency USD and retentionDays 30"]}, "45": {"pass": false, "earned": 2, "total": 10, "failed": ["ConfigMap restore-proof-67 exists again with its original data", "etcd uses a restored data directory (manifest changed)"]}, "32": {"pass": false, "earned": 0, "total": 4, "failed": ["ResourceQuota limits pods/6, requests.cpu 1, limits.memory 2Gi", "LimitRange defaults set", "worker pods Running with defaulted requests"]}, "204": {"pass": false, "earned": 0, "total": 8, "failed": ["evicted-first.txt lists exactly the BestEffort pods", "api-qos.txt says Burstable", "guaranteed.txt names orion-ledger"]}, "2": {"pass": false, "earned": 0, "total": 7, "failed": ["/opt/cka/etcd-snapshot.db exists on the control-plane node", "snapshot is a real etcd backup (> 100 KB)"]}, "55": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret secure.lab.local-tls is kubernetes.io/tls with CN secure.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "52": {"pass": false, "earned": 2, "total": 8, "failed": ["native sidecar: initContainer agent with restartPolicy Always", "probes on app as specified", "Running, shared log written by agent, grace period 5"]}, "313": {"pass": false, "earned": 0, "total": 11, "failed": ["Secret classic-tls is a TLS secret with CN classic.r700.io", "both host rules route to the right Services on /", "TLS covers both hosts with classic-tls", "defaultBackend points at web-classic:80, class is nginx"]}}}
{"at": "2026-09-20T17:32", "mode": "exam", "hard": true, "ids": [401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418], "score_pct": 100.0, "earned": 163, "total": 163, "time_used": 2909, "results": {"402": {"pass": true, "earned": 9, "total": 9, "failed": []}, "404": {"pass": true, "earned": 9, "total": 9, "failed": []}, "405": {"pass": true, "earned": 11, "total": 11, "failed": []}, "401": {"pass": true, "earned": 8, "total": 8, "failed": []}, "403": {"pass": true, "earned": 9, "total": 9, "failed": []}, "406": {"pass": true, "earned": 7, "total": 7, "failed": []}, "407": {"pass": true, "earned": 10, "total": 10, "failed": []}, "408": {"pass": true, "earned": 9, "total": 9, "failed": []}, "409": {"pass": true, "earned": 9, "total": 9, "failed": []}, "410": {"pass": true, "earned": 8, "total": 8, "failed": []}, "411": {"pass": true, "earned": 7, "total": 7, "failed": []}, "412": {"pass": true, "earned": 11, "total": 11, "failed": []}, "413": {"pass": true, "earned": 10, "total": 10, "failed": []}, "414": {"pass": true, "earned": 9, "total": 9, "failed": []}, "415": {"pass": true, "earned": 9, "total": 9, "failed": []}, "416": {"pass": true, "earned": 10, "total": 10, "failed": []}, "417": {"pass": true, "earned": 10, "total": 10, "failed": []}, "418": {"pass": true, "earned": 8, "total": 8, "failed": []}}, "synthetic": true, "note": "Scripted end-to-end verification of Set C (tasks solved programmatically), not a person sitting it"} {"at": "2026-09-20T17:32", "mode": "exam", "hard": true, "ids": [68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85], "score_pct": 100.0, "earned": 163, "total": 163, "time_used": 2909, "results": {"69": {"pass": true, "earned": 9, "total": 9, "failed": []}, "71": {"pass": true, "earned": 9, "total": 9, "failed": []}, "72": {"pass": true, "earned": 11, "total": 11, "failed": []}, "68": {"pass": true, "earned": 8, "total": 8, "failed": []}, "70": {"pass": true, "earned": 9, "total": 9, "failed": []}, "73": {"pass": true, "earned": 7, "total": 7, "failed": []}, "74": {"pass": true, "earned": 10, "total": 10, "failed": []}, "75": {"pass": true, "earned": 9, "total": 9, "failed": []}, "76": {"pass": true, "earned": 9, "total": 9, "failed": []}, "77": {"pass": true, "earned": 8, "total": 8, "failed": []}, "78": {"pass": true, "earned": 7, "total": 7, "failed": []}, "79": {"pass": true, "earned": 11, "total": 11, "failed": []}, "80": {"pass": true, "earned": 10, "total": 10, "failed": []}, "81": {"pass": true, "earned": 9, "total": 9, "failed": []}, "82": {"pass": true, "earned": 9, "total": 9, "failed": []}, "83": {"pass": true, "earned": 10, "total": 10, "failed": []}, "84": {"pass": true, "earned": 10, "total": 10, "failed": []}, "85": {"pass": true, "earned": 8, "total": 8, "failed": []}}, "synthetic": true, "note": "Scripted end-to-end verification of Set C (tasks solved programmatically), not a person sitting it"}

2
setup.sh
View File

@ -88,7 +88,7 @@ fi
# copied out of the running etcd pod (that image is distroless), so install them properly. # copied out of the running etcd pod (that image is distroless), so install them properly.
echo "==> Installing etcdctl/etcdutl on the control-plane node..." echo "==> Installing etcdctl/etcdutl on the control-plane node..."
printf ' ' printf ' '
CONTAINER_RUNTIME="$RT" bash ./etcd-tools.sh || echo " warn: no etcd client on the control-plane node; Q02/Q102 will say so when they seed" CONTAINER_RUNTIME="$RT" bash ./etcd-tools.sh || echo " warn: no etcd client on the control-plane node; Q02/Q45 will say so when they seed"
# remember the runtime so the CLI uses the matching exec command # remember the runtime so the CLI uses the matching exec command
mkdir -p .state && echo "$RT" > .state/runtime mkdir -p .state && echo "$RT" > .state/runtime

2
tracks/cka/bank_extra.py
View File

@ -282,7 +282,7 @@ $K -n %{ns} delete deploy %{dep} --ignore-not-found >/dev/null
}, },
{ {
"id": 26, "domain": "WORK", "title": "DaemonSet on every node", "id": 26, "domain": "WORK", "title": "DaemonSet on every node",
# needs every node schedulable: Q25/Q107 taint W1 and must keep the taint # needs every node schedulable: Q25/Q50 taint W1 and must keep the taint
"breaks": ["w1-taint"], "breaks": ["w1-taint"],
"params": lambda r: {"ns": ns_pick(r, "agents"), "ds": name_pick(r, "node-agent")}, "params": lambda r: {"ns": ns_pick(r, "agents"), "ds": name_pick(r, "node-agent")},
"task": """ "task": """

164
tracks/cka/bank_hard.py
View File

@ -1,4 +1,4 @@
"""Hard-difficulty CKA tasks (ids 101-122): multi-part, no hints, traps, exact output, """Hard-difficulty CKA tasks (ids 44-67): multi-part, no hints, traps, exact output,
compound failures, collateral-damage checks, exam-style contexts. Modelled on the kind of compound failures, collateral-damage checks, exam-style contexts. Modelled on the kind of
difficulty people report from killer.sh, not on its content.""" difficulty people report from killer.sh, not on its content."""
from tracks.cka.common import C, NS, ns_pick, name_pick from tracks.cka.common import C, NS, ns_pick, name_pick
@ -11,7 +11,7 @@ def ctx_pick(r):
QUESTIONS = [ QUESTIONS = [
# ================================================================ ARCH ==== # ================================================================ ARCH ====
{ {
"id": 101, "domain": "ARCH", "difficulty": "hard", "title": "Exact RBAC with an existing over-grant", "id": 44, "domain": "ARCH", "difficulty": "hard", "title": "Exact RBAC with an existing over-grant",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "ci"), "sa": name_pick(r, "ci-bot")}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "ci"), "sa": name_pick(r, "ci-bot")},
"task": CTX + """ "task": CTX + """
ServiceAccount `%{sa}` in namespace `%{ns}` is used by a CI system. Its permissions must be ServiceAccount `%{sa}` in namespace `%{ns}` is used by a CI system. Its permissions must be
@ -41,7 +41,7 @@ eq "$($K auth can-i create pods --subresource=exec -n %{ns} --as=system:servicea
], ],
}, },
{ {
"id": 102, "domain": "ARCH", "breaks": "etcd", "difficulty": "hard", "title": "etcd restore", "id": 45, "domain": "ARCH", "breaks": "etcd", "difficulty": "hard", "title": "etcd restore",
# solo: restoring rolls EVERY API object back to the snapshot - other tasks' seeds and answers too # solo: restoring rolls EVERY API object back to the snapshot - other tasks' seeds and answers too
"solo": True, "solo": True,
"params": lambda r: {"ctx": "k8s-c1-H", "cm": name_pick(r, "restore-proof")}, "params": lambda r: {"ctx": "k8s-c1-H", "cm": name_pick(r, "restore-proof")},
@ -65,13 +65,13 @@ $K -n default create cm %{cm} --from-literal=proof=$(date +%s)-%{cm} >/dev/null
out=$(nx "$CP" 'etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/cka/backup.db' 2>&1) \ out=$(nx "$CP" 'etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /opt/cka/backup.db' 2>&1) \
|| { echo "snapshot save on $CP failed: $out"; exit 1; } || { echo "snapshot save on $CP failed: $out"; exit 1; }
sleep 2; $K -n default delete cm %{cm} >/dev/null sleep 2; $K -n default delete cm %{cm} >/dev/null
nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' </dev/null > "$WS/.q102-datadir" nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' </dev/null > "$WS/.q45-datadir"
date +%s > "$WS/.q102-t0" date +%s > "$WS/.q45-t0"
""", """,
"checks": [ "checks": [
C("ConfigMap %{cm} exists again with its original data", 6, r"""j -n default cm %{cm} '{.data.proof}' | grep -q -- '-%{cm}$'"""), C("ConfigMap %{cm} exists again with its original data", 6, r"""j -n default cm %{cm} '{.data.proof}' | grep -q -- '-%{cm}$'"""),
C("etcd runs on a data directory restored from the snapshot", 2, r""" C("etcd runs on a data directory restored from the snapshot", 2, r"""
was=$(cat "$WS/.q102-datadir" 2>/dev/null); t0=$(cat "$WS/.q102-t0" 2>/dev/null); [ -n "$was" ] && [ -n "$t0" ] || exit 1 was=$(cat "$WS/.q45-datadir" 2>/dev/null); t0=$(cat "$WS/.q45-t0" 2>/dev/null); [ -n "$was" ] && [ -n "$t0" ] || exit 1
now=$(nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' </dev/null) now=$(nx "$CP" 'awk "/hostPath:/{getline; p=\$2} /name: etcd-data/{if (p) print p}" /etc/kubernetes/manifests/etcd.yaml' </dev/null)
[ -n "$now" ] || exit 1 [ -n "$now" ] || exit 1
[ "$now" != "$was" ] && exit 0 [ "$now" != "$was" ] && exit 0
@ -82,7 +82,7 @@ for c in etcd kube-apiserver kube-scheduler kube-controller-manager; do eq "$(j
], ],
}, },
{ {
"id": 103, "domain": "ARCH", "breaks": ["w3-kubelet", "w3-staticpath"], "difficulty": "hard", "title": "Static pod on a relocated manifest path", "id": 46, "domain": "ARCH", "breaks": ["w3-kubelet", "w3-staticpath"], "difficulty": "hard", "title": "Static pod on a relocated manifest path",
"params": lambda r: {"ctx": "k8s-c1-H", "ns": ns_pick(r, "edge"), "pod": name_pick(r, "edge-proxy")}, "params": lambda r: {"ctx": "k8s-c1-H", "ns": ns_pick(r, "edge"), "pod": name_pick(r, "edge-proxy")},
"task": CTX + """ "task": CTX + """
Exactly one worker node is labelled `tier=edge`. On that node create a **static pod**: Exactly one worker node is labelled `tier=edge`. On that node create a **static pod**:
@ -108,30 +108,30 @@ systemctl restart kubelet'
], ],
}, },
{ {
"id": 104, "domain": "ARCH", "difficulty": "hard", "title": "User onboarding to a working kubeconfig", "id": 47, "domain": "ARCH", "difficulty": "hard", "title": "User onboarding to a working kubeconfig",
"params": lambda r: {"ctx": "k8s-c1-H", "user": r.choice(["maya", "theo", "iris"]), "ns": ns_pick(r, "dev")}, "params": lambda r: {"ctx": "k8s-c1-H", "user": r.choice(["maya", "theo", "iris"]), "ns": ns_pick(r, "dev")},
"task": CTX + """ "task": CTX + """
Onboard user `%{user}` (group `developers`) end to end: Onboard user `%{user}` (group `developers`) end to end:
1. Key + CSR with CN=%{user}, O=developers; Kubernetes CSR object `%{user}`, approved, certificate issued. 1. Key + CSR with CN=%{user}, O=developers; Kubernetes CSR object `%{user}`, approved, certificate issued.
2. A **self-contained** kubeconfig at `{WS}/q104/%{user}.kubeconfig` (certs embedded, not file paths) 2. A **self-contained** kubeconfig at `{WS}/q47/%{user}.kubeconfig` (certs embedded, not file paths)
with a context `%{user}@lab` whose default namespace is `%{ns}`. with a context `%{user}@lab` whose default namespace is `%{ns}`.
3. `%{user}` may do everything to pods, deployments and services in `%{ns}` (edit-level), nothing in other namespaces. 3. `%{user}` may do everything to pods, deployments and services in `%{ns}` (edit-level), nothing in other namespaces.
`kubectl --kubeconfig {WS}/q104/%{user}.kubeconfig get pods` must work with no extra flags. `kubectl --kubeconfig {WS}/q47/%{user}.kubeconfig get pods` must work with no extra flags.
""", """,
"setup": NS + r"""mkns %{ns}; rm -rf "$WS/q104"; mkdir -p "$WS/q104"; $K delete csr %{user} --ignore-not-found >/dev/null; $K -n %{ns} delete rolebinding --all >/dev/null 2>&1""", "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q47"; mkdir -p "$WS/q47"; $K delete csr %{user} --ignore-not-found >/dev/null; $K -n %{ns} delete rolebinding --all >/dev/null 2>&1""",
"checks": [ "checks": [
C("CSR %{user} approved with certificate, O=developers", 2, r"""[ -n "$(j csr %{user} '{.status.certificate}')" ] && j csr %{user} '{.status.certificate}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'O *= *developers'"""), C("CSR %{user} approved with certificate, O=developers", 2, r"""[ -n "$(j csr %{user} '{.status.certificate}')" ] && j csr %{user} '{.status.certificate}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'O *= *developers'"""),
C("kubeconfig is self-contained and works without flags", 3, r"""f="$WS/q104/%{user}.kubeconfig"; [ -f "$f" ] || exit 1; grep -q 'client-certificate-data' "$f" && grep -q 'certificate-authority-data' "$f" && ! grep -q 'client-certificate:' "$f" && kubectl --kubeconfig "$f" get pods >/dev/null 2>&1"""), C("kubeconfig is self-contained and works without flags", 3, r"""f="$WS/q47/%{user}.kubeconfig"; [ -f "$f" ] || exit 1; grep -q 'client-certificate-data' "$f" && grep -q 'certificate-authority-data' "$f" && ! grep -q 'client-certificate:' "$f" && kubectl --kubeconfig "$f" get pods >/dev/null 2>&1"""),
C("context %{user}@lab defaults to %{ns}", 1, r"""eq "$(kubectl --kubeconfig "$WS/q104/%{user}.kubeconfig" config view -o jsonpath='{.contexts[?(@.name=="%{user}@lab")].context.namespace}')" %{ns}"""), C("context %{user}@lab defaults to %{ns}", 1, r"""eq "$(kubectl --kubeconfig "$WS/q47/%{user}.kubeconfig" config view -o jsonpath='{.contexts[?(@.name=="%{user}@lab")].context.namespace}')" %{ns}"""),
C("edit rights in %{ns}, none elsewhere", 2, r"""eq "$($K auth can-i create deployments -n %{ns} --as=%{user} --as-group=developers 2>/dev/null)" yes && eq "$($K auth can-i delete pods -n %{ns} --as=%{user} --as-group=developers 2>/dev/null)" yes && eq "$($K auth can-i get pods -n default --as=%{user} --as-group=developers 2>/dev/null || true)" no && eq "$($K auth can-i get secrets -n kube-system --as=%{user} --as-group=developers 2>/dev/null || true)" no"""), C("edit rights in %{ns}, none elsewhere", 2, r"""eq "$($K auth can-i create deployments -n %{ns} --as=%{user} --as-group=developers 2>/dev/null)" yes && eq "$($K auth can-i delete pods -n %{ns} --as=%{user} --as-group=developers 2>/dev/null)" yes && eq "$($K auth can-i get pods -n default --as=%{user} --as-group=developers 2>/dev/null || true)" no && eq "$($K auth can-i get secrets -n kube-system --as=%{user} --as-group=developers 2>/dev/null || true)" no"""),
], ],
}, },
{ {
"id": 105, "domain": "ARCH", "difficulty": "hard", "title": "Kustomize with generators and patches", "id": 48, "domain": "ARCH", "difficulty": "hard", "title": "Kustomize with generators and patches",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "kz"), "rep": r.choice([3, 4]), "lvl": r.choice(["debug", "trace", "warn"])}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "kz"), "rep": r.choice([3, 4]), "lvl": r.choice(["debug", "trace", "warn"])},
"task": CTX + """ "task": CTX + """
Base at `{WS}/q105/base` (Deployment `web`, Service `web`). Build overlay `{WS}/q105/overlays/prod` that: Base at `{WS}/q48/base` (Deployment `web`, Service `web`). Build overlay `{WS}/q48/overlays/prod` that:
- targets namespace `%{ns}` and adds label `tier=prod` to all resources (labels only, not selectors), - targets namespace `%{ns}` and adds label `tier=prod` to all resources (labels only, not selectors),
- sets replicas to %{rep} via a **strategic merge patch** file `replicas.yaml`, - sets replicas to %{rep} via a **strategic merge patch** file `replicas.yaml`,
- adds a `configMapGenerator` named `web-config` with literal `LOG_LEVEL=%{lvl}` (hash suffix enabled), - adds a `configMapGenerator` named `web-config` with literal `LOG_LEVEL=%{lvl}` (hash suffix enabled),
@ -140,11 +140,11 @@ Base at `{WS}/q105/base` (Deployment `web`, Service `web`). Build overlay `{WS}/
Apply with `kubectl apply -k`. The Deployment must reference the hashed ConfigMap name. Apply with `kubectl apply -k`. The Deployment must reference the hashed ConfigMap name.
""", """,
"setup": r""" "setup": r"""
rm -rf "$WS/q105"; mkdir -p "$WS/q105/base" rm -rf "$WS/q48"; mkdir -p "$WS/q48/base"
cat > "$WS/q105/base/kustomization.yaml" <<'EOF' cat > "$WS/q48/base/kustomization.yaml" <<'EOF'
resources: [deployment.yaml, service.yaml] resources: [deployment.yaml, service.yaml]
EOF EOF
cat > "$WS/q105/base/deployment.yaml" <<'EOF' cat > "$WS/q48/base/deployment.yaml" <<'EOF'
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
metadata: {name: web} metadata: {name: web}
@ -156,7 +156,7 @@ spec:
spec: spec:
containers: [{name: web, image: nginx:1.25, ports: [{containerPort: 80}]}] containers: [{name: web, image: nginx:1.25, ports: [{containerPort: 80}]}]
EOF EOF
cat > "$WS/q105/base/service.yaml" <<'EOF' cat > "$WS/q48/base/service.yaml" <<'EOF'
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
metadata: {name: web} metadata: {name: web}
@ -165,14 +165,14 @@ EOF
$K delete ns %{ns} --ignore-not-found --wait=false >/dev/null $K delete ns %{ns} --ignore-not-found --wait=false >/dev/null
""", """,
"checks": [ "checks": [
C("overlay has replicas.yaml (SMP) and envfrom.yaml (JSON6902)", 1, r"""[ -f "$WS/q105/overlays/prod/replicas.yaml" ] && [ -f "$WS/q105/overlays/prod/envfrom.yaml" ] && grep -q 'op:' "$WS/q105/overlays/prod/envfrom.yaml" """), C("overlay has replicas.yaml (SMP) and envfrom.yaml (JSON6902)", 1, r"""[ -f "$WS/q48/overlays/prod/replicas.yaml" ] && [ -f "$WS/q48/overlays/prod/envfrom.yaml" ] && grep -q 'op:' "$WS/q48/overlays/prod/envfrom.yaml" """),
C("deployed to %{ns}: %{rep} replicas, tier=prod label, selector untouched", 2, r"""d="$K -n %{ns} get deploy web"; eq "$($d -o jsonpath='{.spec.replicas}' 2>/dev/null)" %{rep} && eq "$($d -o jsonpath='{.metadata.labels.tier}')" prod && [ -z "$($d -o jsonpath='{.spec.selector.matchLabels.tier}')" ]"""), C("deployed to %{ns}: %{rep} replicas, tier=prod label, selector untouched", 2, r"""d="$K -n %{ns} get deploy web"; eq "$($d -o jsonpath='{.spec.replicas}' 2>/dev/null)" %{rep} && eq "$($d -o jsonpath='{.metadata.labels.tier}')" prod && [ -z "$($d -o jsonpath='{.spec.selector.matchLabels.tier}')" ]"""),
C("hashed ConfigMap with LOG_LEVEL=%{lvl}, referenced via envFrom", 3, r"""cm=$(j -n %{ns} deploy web '{.spec.template.spec.containers[0].envFrom[0].configMapRef.name}'); [ -n "$cm" ] && echo "$cm" | grep -Eq '^web-config-[a-z0-9]{6,}$' && eq "$(j -n %{ns} cm "$cm" '{.data.LOG_LEVEL}')" %{lvl}"""), C("hashed ConfigMap with LOG_LEVEL=%{lvl}, referenced via envFrom", 3, r"""cm=$(j -n %{ns} deploy web '{.spec.template.spec.containers[0].envFrom[0].configMapRef.name}'); [ -n "$cm" ] && echo "$cm" | grep -Eq '^web-config-[a-z0-9]{6,}$' && eq "$(j -n %{ns} cm "$cm" '{.data.LOG_LEVEL}')" %{lvl}"""),
C("pods available and env visible", 1, r"""eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" %{rep} && $K -n %{ns} exec deploy/web -- env 2>/dev/null | grep -q '^LOG_LEVEL=%{lvl}$'"""), C("pods available and env visible", 1, r"""eq "$(j -n %{ns} deploy web '{.status.availableReplicas}')" %{rep} && $K -n %{ns} exec deploy/web -- env 2>/dev/null | grep -q '^LOG_LEVEL=%{lvl}$'"""),
], ],
}, },
{ {
"id": 106, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults", "id": 49, "domain": "ARCH", "difficulty": "hard", "title": "CRD with validation and defaults",
"params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])}, "params": lambda r: {"ctx": ctx_pick(r), "kind": r.choice(["Rollout", "Canary", "Release"]), "env": r.choice(["prod", "stage"])},
"task": CTX + """ "task": CTX + """
Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the kind in lowercase plus Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the kind in lowercase plus
@ -182,44 +182,44 @@ Create CRD `%{kind}` (group `deploy.example.com`, v1, Namespaced, plural = the k
- `spec.image` string, required. - `spec.image` string, required.
Then: Then:
1. Create `%{kind}` `good` in `default` with replicas 3, image `nginx:1.27`, environment `%{env}`. 1. Create `%{kind}` `good` in `default` with replicas 3, image `nginx:1.27`, environment `%{env}`.
2. Save the API server's rejection message for a `%{kind}` with `replicas: 0` to `{WS}/q106/rejected.txt`. 2. Save the API server's rejection message for a `%{kind}` with `replicas: 0` to `{WS}/q49/rejected.txt`.
3. Create `%{kind}` `defaulted` in `default` with only replicas 2 and image set; its environment 3. Create `%{kind}` `defaulted` in `default` with only replicas 2 and image set; its environment
must be `dev` after creation. must be `dev` after creation.
This context's default namespace is not necessarily `default` — say `-n default` explicitly. This context's default namespace is not necessarily `default` — say `-n default` explicitly.
""", """,
"setup": r"""rm -rf "$WS/q106"; mkdir -p "$WS/q106"; k=$(echo %{kind} | tr A-Z a-z); $K delete crd ${k}s.deploy.example.com --ignore-not-found >/dev/null""", "setup": r"""rm -rf "$WS/q49"; mkdir -p "$WS/q49"; k=$(echo %{kind} | tr A-Z a-z); $K delete crd ${k}s.deploy.example.com --ignore-not-found >/dev/null""",
"checks": [ "checks": [
C("CRD exists, Namespaced, replicas has min/max and is required", 2, r"""s=""; k=$(echo %{kind} | tr A-Z a-z); c="$K get crd ${k}s.deploy.example.com"; eq "$($c -o jsonpath='{.spec.scope}' 2>/dev/null)" Namespaced && s=$($c -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec}'); echo "$s" | grep -q '"minimum":1' && echo "$s" | grep -q '"maximum":10' && echo "$s" | grep -q '"required":\[' """), C("CRD exists, Namespaced, replicas has min/max and is required", 2, r"""s=""; k=$(echo %{kind} | tr A-Z a-z); c="$K get crd ${k}s.deploy.example.com"; eq "$($c -o jsonpath='{.spec.scope}' 2>/dev/null)" Namespaced && s=$($c -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec}'); echo "$s" | grep -q '"minimum":1' && echo "$s" | grep -q '"maximum":10' && echo "$s" | grep -q '"required":\[' """),
C("good created with environment %{env}", 1, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com good '{.spec.environment}')" %{env}"""), C("good created with environment %{env}", 1, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com good '{.spec.environment}')" %{env}"""),
C("replicas 0 is rejected and the message was saved", 2, r"""grep -qiE 'minimum|greater than|>= 1|invalid' "$WS/q106/rejected.txt" 2>/dev/null"""), C("replicas 0 is rejected and the message was saved", 2, r"""grep -qiE 'minimum|greater than|>= 1|invalid' "$WS/q49/rejected.txt" 2>/dev/null"""),
C("defaulting works: defaulted.spec.environment == dev", 2, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com defaulted '{.spec.environment}')" dev"""), C("defaulting works: defaulted.spec.environment == dev", 2, r"""k=$(echo %{kind} | tr A-Z a-z); eq "$(j -n default ${k}s.deploy.example.com defaulted '{.spec.environment}')" dev"""),
], ],
}, },
{ {
"id": 122, "domain": "ARCH", "breaks": "certs", "difficulty": "hard", "title": "Renew control-plane certificates", "id": 65, "domain": "ARCH", "breaks": "certs", "difficulty": "hard", "title": "Renew control-plane certificates",
"params": lambda r: {"ctx": "k8s-c1-H"}, "params": lambda r: {"ctx": "k8s-c1-H"},
"task": CTX + """ "task": CTX + """
Security policy requires the control-plane certificates to be renewed now. Security policy requires the control-plane certificates to be renewed now.
On the control-plane node, renew **all** kubeadm-managed certificates, make every control-plane On the control-plane node, renew **all** kubeadm-managed certificates, make every control-plane
component pick up the new certificates, and update the admin kubeconfig on the node component pick up the new certificates, and update the admin kubeconfig on the node
(`/etc/kubernetes/admin.conf`) so it uses the renewed client certificate. (`/etc/kubernetes/admin.conf`) so it uses the renewed client certificate.
Write the new `apiserver` certificate's expiry (as printed by `kubeadm certs check-expiration`) to `{WS}/q122/expiry.txt`. Write the new `apiserver` certificate's expiry (as printed by `kubeadm certs check-expiration`) to `{WS}/q65/expiry.txt`.
""", """,
"setup": r"""rm -rf "$WS/q122"; mkdir -p "$WS/q122"; nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' > "$WS/.q122-before" """, "setup": r"""rm -rf "$WS/q65"; mkdir -p "$WS/q65"; nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' > "$WS/.q65-before" """,
"checks": [ "checks": [
C("apiserver certificate was renewed (newer notBefore)", 3, r"""b=$(sed 's/notBefore=//' "$WS/.q122-before"); a=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' | sed 's/notBefore=//'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""), C("apiserver certificate was renewed (newer notBefore)", 3, r"""b=$(sed 's/notBefore=//' "$WS/.q65-before"); a=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -startdate' | sed 's/notBefore=//'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""),
C("all components restarted and Running with the new certs (apiserver serving cert matches file)", 3, r""" C("all components restarted and Running with the new certs (apiserver serving cert matches file)", 3, r"""
for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done
srv=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial -startdate'); f=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial -startdate'); [ -n "$srv" ] && eq "$srv" "$f" || exit 1 srv=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -serial -startdate'); f=$(nx "$CP" 'openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -serial -startdate'); [ -n "$srv" ] && eq "$srv" "$f" || exit 1
# ...and it is the RENEWED certificate: an untouched cluster also serves the file it has on disk # ...and it is the RENEWED certificate: an untouched cluster also serves the file it has on disk
b=$(sed 's/notBefore=//' "$WS/.q122-before"); a=$(echo "$srv" | sed -n 's/notBefore=//p'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""), b=$(sed 's/notBefore=//' "$WS/.q65-before"); a=$(echo "$srv" | sed -n 's/notBefore=//p'); [ "$(date -d "$a" +%s)" -gt "$(date -d "$b" +%s)" ]"""),
C("admin.conf uses the renewed client cert", 2, r"""nx "$CP" 'grep client-certificate-data /etc/kubernetes/admin.conf | awk "{print \$2}" | base64 -d | openssl x509 -noout -startdate' | sed 's/notBefore=//' > /tmp/q122a; b=$(sed 's/notBefore=//' "$WS/.q122-before"); [ "$(date -d "$(cat /tmp/q122a)" +%s)" -gt "$(date -d "$b" +%s)" ]"""), C("admin.conf uses the renewed client cert", 2, r"""nx "$CP" 'grep client-certificate-data /etc/kubernetes/admin.conf | awk "{print \$2}" | base64 -d | openssl x509 -noout -startdate' | sed 's/notBefore=//' > /tmp/q65a; b=$(sed 's/notBefore=//' "$WS/.q65-before"); [ "$(date -d "$(cat /tmp/q65a)" +%s)" -gt "$(date -d "$b" +%s)" ]"""),
C("expiry.txt matches kubeadm's report for apiserver", 1, r"""exp=$(nx "$CP" 'kubeadm certs check-expiration 2>/dev/null' | awk '$1=="apiserver"{print $2,$3,$4,$5,$6}'); [ -n "$exp" ] && grep -qF "$exp" "$WS/q122/expiry.txt" 2>/dev/null"""), C("expiry.txt matches kubeadm's report for apiserver", 1, r"""exp=$(nx "$CP" 'kubeadm certs check-expiration 2>/dev/null' | awk '$1=="apiserver"{print $2,$3,$4,$5,$6}'); [ -n "$exp" ] && grep -qF "$exp" "$WS/q65/expiry.txt" 2>/dev/null"""),
], ],
}, },
{ {
"id": 124, "domain": "ARCH", "difficulty": "hard", "title": "Prepare a node join without shell access to the control plane", "id": 67, "domain": "ARCH", "difficulty": "hard", "title": "Prepare a node join without shell access to the control plane",
"params": lambda r: {"ctx": "k8s-c1-H"}, "params": lambda r: {"ctx": "k8s-c1-H"},
"task": CTX + """ "task": CTX + """
A new worker node is about to join this cluster, but you can only hand the networking team a A new worker node is about to join this cluster, but you can only hand the networking team a
@ -232,7 +232,7 @@ A new worker node is about to join this cluster, but you can only hand the netwo
3. Determine the API server's real address as seen **from inside the cluster network** — not the 3. Determine the API server's real address as seen **from inside the cluster network** — not the
`127.0.0.1:<port>` your local kubeconfig uses to reach it from this machine, which a joining `127.0.0.1:<port>` your local kubeconfig uses to reach it from this machine, which a joining
node cannot use. node cannot use.
4. Write exactly these three lines to `{WS}/q124/node-join.env`: 4. Write exactly these three lines to `{WS}/q67/node-join.env`:
``` ```
JOIN_TOKEN=<token> JOIN_TOKEN=<token>
CA_HASH=sha256:<hash> CA_HASH=sha256:<hash>
@ -242,27 +242,27 @@ Every value must be genuine: plugged into `kubeadm join $API_ENDPOINT --token $J
--discovery-token-ca-cert-hash $CA_HASH` right now, it would work. --discovery-token-ca-cert-hash $CA_HASH` right now, it would work.
""", """,
"setup": r""" "setup": r"""
rm -rf "$WS/q124"; mkdir -p "$WS/q124" rm -rf "$WS/q67"; mkdir -p "$WS/q67"
for t in $($K -n kube-system get secrets -o jsonpath='{.items[?(@.type=="bootstrap.kubernetes.io/token")].metadata.name}'); do $K -n kube-system delete secret $t >/dev/null; done for t in $($K -n kube-system get secrets -o jsonpath='{.items[?(@.type=="bootstrap.kubernetes.io/token")].metadata.name}'); do $K -n kube-system delete secret $t >/dev/null; done
""", """,
"checks": [ "checks": [
C("a real, unexpired bootstrap token", 2, r""" C("a real, unexpired bootstrap token", 2, r"""
tok=$(grep -oE '^JOIN_TOKEN=[a-z0-9]{6}\.[a-z0-9]{16}' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2); [ -n "$tok" ] || exit 1 tok=$(grep -oE '^JOIN_TOKEN=[a-z0-9]{6}\.[a-z0-9]{16}' "$WS/q67/node-join.env" 2>/dev/null | cut -d= -f2); [ -n "$tok" ] || exit 1
id=${tok%%.*}; exp=$(j -n kube-system secret bootstrap-token-$id '{.data.expiration}' | base64 -d 2>/dev/null); [ -n "$exp" ] || exit 1 id=${tok%%.*}; exp=$(j -n kube-system secret bootstrap-token-$id '{.data.expiration}' | base64 -d 2>/dev/null); [ -n "$exp" ] || exit 1
[ "$(date -d "$exp" +%s)" -gt "$(date +%s)" ]"""), [ "$(date -d "$exp" +%s)" -gt "$(date +%s)" ]"""),
C("CA_HASH computed correctly (matches the real CA's SPKI SHA-256)", 3, r""" C("CA_HASH computed correctly (matches the real CA's SPKI SHA-256)", 3, r"""
want=$(nx "$CP" "openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'") want=$(nx "$CP" "openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'")
got=$(grep -oE '^CA_HASH=sha256:[0-9a-f]{64}' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2 | cut -d: -f2) got=$(grep -oE '^CA_HASH=sha256:[0-9a-f]{64}' "$WS/q67/node-join.env" 2>/dev/null | cut -d= -f2 | cut -d: -f2)
[ -n "$want" ] && eq "$got" "$want" """), [ -n "$want" ] && eq "$got" "$want" """),
C("API_ENDPOINT is the real in-cluster address, not the local kubeconfig's", 2, r""" C("API_ENDPOINT is the real in-cluster address, not the local kubeconfig's", 2, r"""
want=$($K get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}' 2>/dev/null) want=$($K get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}' 2>/dev/null)
got=$(grep -oE '^API_ENDPOINT=.*' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2 | tr -d '[:space:]') got=$(grep -oE '^API_ENDPOINT=.*' "$WS/q67/node-join.env" 2>/dev/null | cut -d= -f2 | tr -d '[:space:]')
[ -n "$want" ] && eq "$got" "$want" """), [ -n "$want" ] && eq "$got" "$want" """),
], ],
}, },
# ================================================================ WORK ==== # ================================================================ WORK ====
{ {
"id": 107, "domain": "WORK", "breaks": "w1-taint", "difficulty": "hard", "title": "Spread, tolerate, prioritise — under a quota", "id": 50, "domain": "WORK", "breaks": "w1-taint", "difficulty": "hard", "title": "Spread, tolerate, prioritise — under a quota",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "spread"), "dep": name_pick(r, "gateway"), "key": r.choice(["pool", "lane"]), "val": r.choice(["ingress", "edge"])}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "spread"), "dep": name_pick(r, "gateway"), "key": r.choice(["pool", "lane"]), "val": r.choice(["ingress", "edge"])},
"task": CTX + """ "task": CTX + """
Node `{W1}` is tainted `%{key}=%{val}:NoSchedule`. In namespace `%{ns}` create Deployment `%{dep}` Node `{W1}` is tainted `%{key}=%{val}:NoSchedule`. In namespace `%{ns}` create Deployment `%{dep}`
@ -294,7 +294,7 @@ EOF
], ],
}, },
{ {
"id": 108, "domain": "WORK", "difficulty": "hard", "title": "Rollback to a specific revision, then pause", "id": 51, "domain": "WORK", "difficulty": "hard", "title": "Rollback to a specific revision, then pause",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "rel"), "dep": name_pick(r, "portal")}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "rel"), "dep": name_pick(r, "portal")},
"task": CTX + """ "task": CTX + """
Deployment `%{dep}` in `%{ns}` has gone through several releases; the current one is broken. Deployment `%{dep}` in `%{ns}` has gone through several releases; the current one is broken.
@ -330,7 +330,7 @@ sys.exit(1)'"""),
], ],
}, },
{ {
"id": 109, "domain": "WORK", "difficulty": "hard", "title": "Native sidecar with probes and an exact report", "id": 52, "domain": "WORK", "difficulty": "hard", "title": "Native sidecar with probes and an exact report",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "side"), "pod": name_pick(r, "collector")}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "side"), "pod": name_pick(r, "collector")},
"task": CTX + """ "task": CTX + """
Create pod `%{pod}` in `%{ns}`: Create pod `%{pod}` in `%{ns}`:
@ -342,33 +342,33 @@ Create pod `%{pod}` in `%{ns}`:
own logs under `/var/log/nginx`, and an emptyDir mounted over `/var/log` hides that directory), own logs under `/var/log/nginx`, and an emptyDir mounted over `/var/log` hides that directory),
- `terminationGracePeriodSeconds: 5`. - `terminationGracePeriodSeconds: 5`.
When it is Running, write the names of **all** containers of the pod (init and regular), sorted, When it is Running, write the names of **all** containers of the pod (init and regular), sorted,
one per line, to `{WS}/q109/containers.txt`, and the pod's `.status.podIP` to `{WS}/q109/ip.txt`. one per line, to `{WS}/q52/containers.txt`, and the pod's `.status.podIP` to `{WS}/q52/ip.txt`.
""", """,
"setup": NS + r"""mkns %{ns}; rm -rf "$WS/q109"; mkdir -p "$WS/q109"; $K -n %{ns} delete pod %{pod} --ignore-not-found --force --grace-period=0 --wait=true >/dev/null 2>&1""", "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q52"; mkdir -p "$WS/q52"; $K -n %{ns} delete pod %{pod} --ignore-not-found --force --grace-period=0 --wait=true >/dev/null 2>&1""",
"checks": [ "checks": [
C("native sidecar: initContainer agent with restartPolicy Always", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.spec.initContainers[?(@.name=="agent")].restartPolicy}')" Always"""), C("native sidecar: initContainer agent with restartPolicy Always", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.spec.initContainers[?(@.name=="agent")].restartPolicy}')" Always"""),
C("probes on app as specified", 2, r"""c='{.spec.containers[?(@.name=="app")]'; eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.httpGet.path}")" / && eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.periodSeconds}")" 5 && eq "$(j -n %{ns} pod %{pod} "$c.livenessProbe.tcpSocket.port}")" 80"""), C("probes on app as specified", 2, r"""c='{.spec.containers[?(@.name=="app")]'; eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.httpGet.path}")" / && eq "$(j -n %{ns} pod %{pod} "$c.readinessProbe.periodSeconds}")" 5 && eq "$(j -n %{ns} pod %{pod} "$c.livenessProbe.tcpSocket.port}")" 80"""),
C("Running, shared log written by agent, grace period 5", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.terminationGracePeriodSeconds}')" 5 && [ -n "$($K -n %{ns} exec %{pod} -c app -- sh -c 'tail -1 /var/log/agent/agent.log' 2>/dev/null)" ]"""), C("Running, shared log written by agent, grace period 5", 2, r"""eq "$(j -n %{ns} pod %{pod} '{.status.phase}')" Running && eq "$(j -n %{ns} pod %{pod} '{.spec.terminationGracePeriodSeconds}')" 5 && [ -n "$($K -n %{ns} exec %{pod} -c app -- sh -c 'tail -1 /var/log/agent/agent.log' 2>/dev/null)" ]"""),
C("report files exact", 2, r"""want=$( (j -n %{ns} pod %{pod} '{.spec.initContainers[*].name}'; echo; j -n %{ns} pod %{pod} '{.spec.containers[*].name}') | tr ' ' '\n' | grep . | sort); [ -n "$want" ] && [ "$(grep . "$WS/q109/containers.txt" 2>/dev/null | sort)" = "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q109/ip.txt" 2>/dev/null)" "$(j -n %{ns} pod %{pod} '{.status.podIP}')" """), C("report files exact", 2, r"""want=$( (j -n %{ns} pod %{pod} '{.spec.initContainers[*].name}'; echo; j -n %{ns} pod %{pod} '{.spec.containers[*].name}') | tr ' ' '\n' | grep . | sort); [ -n "$want" ] && [ "$(grep . "$WS/q52/containers.txt" 2>/dev/null | sort)" = "$want" ] && eq "$(tr -d '[:space:]' < "$WS/q52/ip.txt" 2>/dev/null)" "$(j -n %{ns} pod %{pod} '{.status.podIP}')" """),
], ],
}, },
{ {
# Scoped to its own namespace: the cluster-wide version compared a file written mid-exam with the # Scoped to its own namespace: the cluster-wide version compared a file written mid-exam with the
# whole cluster at submit time, so every pod another task (or a grader probe) created or deleted # whole cluster at submit time, so every pod another task (or a grader probe) created or deleted
# in between failed it. killer.sh-style exact-output tasks work on data the task owns. # in between failed it. killer.sh-style exact-output tasks work on data the task owns.
"id": 110, "domain": "WORK", "difficulty": "hard", "title": "Namespace audit to exact files", "id": 53, "domain": "WORK", "difficulty": "hard", "title": "Namespace audit to exact files",
"params": lambda r: {"ctx": "k8s-c1-H", "ns": ns_pick(r, "audit")}, "params": lambda r: {"ctx": "k8s-c1-H", "ns": ns_pick(r, "audit")},
"task": CTX + """ "task": CTX + """
Namespace `%{ns}` runs a mix of workloads. Produce three files with contents **exactly** as Namespace `%{ns}` runs a mix of workloads. Produce three files with contents **exactly** as
specified — no headers, one entry per line, sorted with `sort`: specified — no headers, one entry per line, sorted with `sort`:
1. `{WS}/q110/no-readiness.txt` — the name of every Pod in `%{ns}` whose **first** container has 1. `{WS}/q53/no-readiness.txt` — the name of every Pod in `%{ns}` whose **first** container has
no readiness probe (whatever the Pod's phase). no readiness probe (whatever the Pod's phase).
2. `{WS}/q110/per-node.txt` — `<node> <count>`: how many Pods in `%{ns}` are **Running** on each 2. `{WS}/q53/per-node.txt` — `<node> <count>`: how many Pods in `%{ns}` are **Running** on each
node, listing only nodes that have at least one. node, listing only nodes that have at least one.
3. `{WS}/q110/images.txt` — every distinct container image used by Pods in `kube-system`. 3. `{WS}/q53/images.txt` — every distinct container image used by Pods in `kube-system`.
""", """,
"setup": NS + r""" "setup": NS + r"""
mkns %{ns}; rm -rf "$WS/q110"; mkdir -p "$WS/q110" mkns %{ns}; rm -rf "$WS/q53"; mkdir -p "$WS/q53"
$K -n %{ns} delete pod --all --force --grace-period=0 --wait=true >/dev/null 2>&1 $K -n %{ns} delete pod --all --force --grace-period=0 --wait=true >/dev/null 2>&1
$K apply -f - >/dev/null <<EOF $K apply -f - >/dev/null <<EOF
apiVersion: v1 apiVersion: v1
@ -422,16 +422,16 @@ for i in $(seq 1 60); do [ "$($K -n %{ns} get pod once-a -o jsonpath='{.status.p
C("no-readiness.txt exact (first container only, every phase)", 3, r"""want=$($K -n %{ns} get pods -o json 2>/dev/null | python3 -c ' C("no-readiness.txt exact (first container only, every phase)", 3, r"""want=$($K -n %{ns} get pods -o json 2>/dev/null | python3 -c '
import sys,json import sys,json
for p in json.load(sys.stdin)["items"]: for p in json.load(sys.stdin)["items"]:
if not p["spec"]["containers"][0].get("readinessProbe"): print(p["metadata"]["name"])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q110/no-readiness.txt" 2>/dev/null)" = "$want" ]"""), if not p["spec"]["containers"][0].get("readinessProbe"): print(p["metadata"]["name"])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q53/no-readiness.txt" 2>/dev/null)" = "$want" ]"""),
C("per-node.txt exact (Running pods only)", 2, r"""want=$($K -n %{ns} get pods -o json 2>/dev/null | python3 -c ' C("per-node.txt exact (Running pods only)", 2, r"""want=$($K -n %{ns} get pods -o json 2>/dev/null | python3 -c '
import sys,json,collections import sys,json,collections
c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"] if p["status"]["phase"]=="Running" and p["spec"].get("nodeName")) c=collections.Counter(p["spec"].get("nodeName") for p in json.load(sys.stdin)["items"] if p["status"]["phase"]=="Running" and p["spec"].get("nodeName"))
for n in c: print(n, c[n])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q110/per-node.txt" 2>/dev/null)" = "$want" ]"""), for n in c: print(n, c[n])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q53/per-node.txt" 2>/dev/null)" = "$want" ]"""),
C("images.txt exact", 2, r"""want=$(j -n kube-system pods '{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u); [ -n "$want" ] && [ "$(grep . "$WS/q110/images.txt" 2>/dev/null)" = "$want" ]"""), C("images.txt exact", 2, r"""want=$(j -n kube-system pods '{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u); [ -n "$want" ] && [ "$(grep . "$WS/q53/images.txt" 2>/dev/null)" = "$want" ]"""),
], ],
}, },
{ {
"id": 121, "domain": "WORK", "difficulty": "hard", "title": "The context's namespace", "id": 64, "domain": "WORK", "difficulty": "hard", "title": "The context's namespace",
# ctx_ns: this task reads the context's own default namespace instead of being told one, so # ctx_ns: this task reads the context's own default namespace instead of being told one, so
# k8s-c2-AC must be created pointing at *this* task's ns (see context_namespace in ./exam). # k8s-c2-AC must be created pointing at *this* task's ns (see context_namespace in ./exam).
"ctx_ns": True, "ctx_ns": True,
@ -440,19 +440,19 @@ for n in c: print(n, c[n])' | sort); [ -n "$want" ] && [ "$(grep . "$WS/q110/per
In the **current namespace of that context** (do not pass `-n`), create Deployment `%{dep}` In the **current namespace of that context** (do not pass `-n`), create Deployment `%{dep}`
(busybox:1.36, `sleep 3600`, 2 replicas, label `owner=ac`) and a ClusterIP Service `%{dep}` (busybox:1.36, `sleep 3600`, 2 replicas, label `owner=ac`) and a ClusterIP Service `%{dep}`
on port 9000 targeting port 9000. Then write the namespace name you deployed into, as reported on port 9000 targeting port 9000. Then write the namespace name you deployed into, as reported
by `kubectl config view --minify`, to `{WS}/q121/ns.txt`. by `kubectl config view --minify`, to `{WS}/q64/ns.txt`.
""", """,
"setup": NS + r"""mkns %{ns}; rm -rf "$WS/q121"; mkdir -p "$WS/q121"; $K -n %{ns} delete deploy,svc %{dep} --ignore-not-found >/dev/null; $K -n default delete deploy,svc %{dep} --ignore-not-found >/dev/null""", "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q64"; mkdir -p "$WS/q64"; $K -n %{ns} delete deploy,svc %{dep} --ignore-not-found >/dev/null; $K -n default delete deploy,svc %{dep} --ignore-not-found >/dev/null""",
"checks": [ "checks": [
C("Deployment %{dep} in %{ns} (the context's namespace), 2 available, owner=ac", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && { eq "$(j -n %{ns} deploy %{dep} '{.metadata.labels.owner}')" ac || eq "$(j -n %{ns} deploy %{dep} '{.spec.template.metadata.labels.owner}')" ac; }"""), C("Deployment %{dep} in %{ns} (the context's namespace), 2 available, owner=ac", 3, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 2 && { eq "$(j -n %{ns} deploy %{dep} '{.metadata.labels.owner}')" ac || eq "$(j -n %{ns} deploy %{dep} '{.spec.template.metadata.labels.owner}')" ac; }"""),
C("Service %{dep} port 9000 in the same namespace", 1, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].port}')" 9000"""), C("Service %{dep} port 9000 in the same namespace", 1, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].port}')" 9000"""),
C("nothing was created in namespace default by mistake", 1, r"""! $K -n default get deploy %{dep} >/dev/null 2>&1"""), C("nothing was created in namespace default by mistake", 1, r"""! $K -n default get deploy %{dep} >/dev/null 2>&1"""),
C("ns.txt has the namespace", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q121/ns.txt" 2>/dev/null)" %{ns}"""), C("ns.txt has the namespace", 1, r"""eq "$(tr -d '[:space:]' < "$WS/q64/ns.txt" 2>/dev/null)" %{ns}"""),
], ],
}, },
# ================================================================= NET ==== # ================================================================= NET ====
{ {
"id": 111, "domain": "NET", "difficulty": "hard", "title": "Zero-trust namespace", "id": 54, "domain": "NET", "difficulty": "hard", "title": "Zero-trust namespace",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "zt"), "from": ns_pick(r, "front")}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "zt"), "from": ns_pick(r, "front")},
"task": CTX + """ "task": CTX + """
Namespace `%{ns}` runs `api` (label `app=api`, port 8080) and `db` (label `app=db`, port 5432). Namespace `%{ns}` runs `api` (label `app=api`, port 8080) and `db` (label `app=db`, port 5432).
@ -499,7 +499,7 @@ sys.exit(0 if eg and ing else 1)'"""),
], ],
}, },
{ {
"id": 112, "domain": "NET", "difficulty": "hard", "title": "TLS Ingress with a generated certificate", "id": 55, "domain": "NET", "difficulty": "hard", "title": "TLS Ingress with a generated certificate",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "tls"), "host": r.choice(["secure.lab.local", "portal.lab.local", "admin.lab.local"])}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "tls"), "host": r.choice(["secure.lab.local", "portal.lab.local", "admin.lab.local"])},
"task": CTX + """ "task": CTX + """
Namespace `%{ns}` has Deployment+Service `web` (port 80). Namespace `%{ns}` has Deployment+Service `web` (port 80).
@ -507,17 +507,17 @@ Namespace `%{ns}` has Deployment+Service `web` (port 80).
Secret `%{host}-tls` of type `kubernetes.io/tls`. Secret `%{host}-tls` of type `kubernetes.io/tls`.
2. Create Ingress `web` (class `nginx`) serving host `%{host}` with that TLS secret, path `/` Prefix → `web:80`, 2. Create Ingress `web` (class `nginx`) serving host `%{host}` with that TLS secret, path `/` Prefix → `web:80`,
and the annotation `nginx.ingress.kubernetes.io/ssl-redirect: "true"`. and the annotation `nginx.ingress.kubernetes.io/ssl-redirect: "true"`.
3. Write the certificate's SHA-256 fingerprint (`openssl x509 -fingerprint -sha256`, the value only) to `{WS}/q112/fp.txt`. 3. Write the certificate's SHA-256 fingerprint (`openssl x509 -fingerprint -sha256`, the value only) to `{WS}/q55/fp.txt`.
""", """,
"setup": NS + r"""mkns %{ns}; rm -rf "$WS/q112"; mkdir -p "$WS/q112"; $K -n %{ns} create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} delete secret %{host}-tls --ignore-not-found >/dev/null; $K -n %{ns} delete ingress web --ignore-not-found >/dev/null""", "setup": NS + r"""mkns %{ns}; rm -rf "$WS/q55"; mkdir -p "$WS/q55"; $K -n %{ns} create deploy web --image=nginx:1.25 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} expose deploy web --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null; $K -n %{ns} delete secret %{host}-tls --ignore-not-found >/dev/null; $K -n %{ns} delete ingress web --ignore-not-found >/dev/null""",
"checks": [ "checks": [
C("Secret %{host}-tls is kubernetes.io/tls with CN %{host}", 2, r"""eq "$(j -n %{ns} secret %{host}-tls '{.type}')" kubernetes.io/tls && j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'CN *= *%{host}'"""), C("Secret %{host}-tls is kubernetes.io/tls with CN %{host}", 2, r"""eq "$(j -n %{ns} secret %{host}-tls '{.type}')" kubernetes.io/tls && j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject 2>/dev/null | grep -q 'CN *= *%{host}'"""),
C("Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", 3, r"""i="$K -n %{ns} get ingress web"; eq "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" nginx && eq "$($i -o jsonpath='{.spec.tls[0].secretName}')" %{host}-tls && eq "$($i -o jsonpath='{.spec.tls[0].hosts[0]}')" %{host} && eq "$($i -o jsonpath='{.spec.rules[0].host}')" %{host} && eq "$($i -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}')" web && eq "$($i -o jsonpath='{.metadata.annotations.nginx\.ingress\.kubernetes\.io/ssl-redirect}')" true"""), C("Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", 3, r"""i="$K -n %{ns} get ingress web"; eq "$($i -o jsonpath='{.spec.ingressClassName}' 2>/dev/null)" nginx && eq "$($i -o jsonpath='{.spec.tls[0].secretName}')" %{host}-tls && eq "$($i -o jsonpath='{.spec.tls[0].hosts[0]}')" %{host} && eq "$($i -o jsonpath='{.spec.rules[0].host}')" %{host} && eq "$($i -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}')" web && eq "$($i -o jsonpath='{.metadata.annotations.nginx\.ingress\.kubernetes\.io/ssl-redirect}')" true"""),
C("fp.txt matches the secret's certificate fingerprint", 2, r"""fp=$(j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//'); [ -n "$fp" ] && grep -qiF "$fp" "$WS/q112/fp.txt" 2>/dev/null"""), C("fp.txt matches the secret's certificate fingerprint", 2, r"""fp=$(j -n %{ns} secret %{host}-tls '{.data.tls\.crt}' | base64 -d | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//'); [ -n "$fp" ] && grep -qiF "$fp" "$WS/q55/fp.txt" 2>/dev/null"""),
], ],
}, },
{ {
"id": 113, "domain": "NET", "difficulty": "hard", "title": "Gateway API canary with header match", "id": 56, "domain": "NET", "difficulty": "hard", "title": "Gateway API canary with header match",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "canary"), "host": r.choice(["shop", "pay", "cart"]), "w": r.choice([10, 20, 25])}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "canary"), "host": r.choice(["shop", "pay", "cart"]), "w": r.choice([10, 20, 25])},
"task": CTX + """ "task": CTX + """
Gateway API CRDs are installed. Namespace `%{ns}` has Services `v1` and `v2` (port 80). Gateway API CRDs are installed. Namespace `%{ns}` has Services `v1` and `v2` (port 80).
@ -549,7 +549,7 @@ sys.exit(0 if ok else 1)'"""),
], ],
}, },
{ {
"id": 114, "domain": "NET", "breaks": "coredns", "difficulty": "hard", "title": "DNS down: three faults", "disruptive": True, "id": 57, "domain": "NET", "breaks": "coredns", "difficulty": "hard", "title": "DNS down: three faults", "disruptive": True,
"params": lambda r: {"ctx": "k8s-c1-H"}, "params": lambda r: {"ctx": "k8s-c1-H"},
"task": CTX + """ "task": CTX + """
Nothing in the cluster resolves DNS. Restore it. There is more than one cause. Nothing in the cluster resolves DNS. Restore it. There is more than one cause.
@ -559,9 +559,9 @@ succeed from a pod in `default` (use the FQDN: busybox's nslookup does not apply
`kube-system`. `kube-system`.
""", """,
"setup": r""" "setup": r"""
$K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > /tmp/Corefile.q114 $K -n kube-system get cm coredns -o jsonpath='{.data.Corefile}' > /tmp/Corefile.q57
sed -i 's|forward \. /etc/resolv.conf|forward . 10.255.255.254|' /tmp/Corefile.q114 sed -i 's|forward \. /etc/resolv.conf|forward . 10.255.255.254|' /tmp/Corefile.q57
$K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.q114 --dry-run=client -o yaml | $K apply -f - >/dev/null $K -n kube-system create cm coredns --from-file=Corefile=/tmp/Corefile.q57 --dry-run=client -o yaml | $K apply -f - >/dev/null
$K -n kube-system scale deploy coredns --replicas=0 >/dev/null $K -n kube-system scale deploy coredns --replicas=0 >/dev/null
$K apply -f - >/dev/null <<EOF $K apply -f - >/dev/null <<EOF
apiVersion: networking.k8s.io/v1 apiVersion: networking.k8s.io/v1
@ -583,7 +583,7 @@ $K -n default delete pod dns-probe --wait=false >/dev/null; exit 1"""),
}, },
# ================================================================ STOR ==== # ================================================================ STOR ====
{ {
"id": 115, "domain": "STOR", "difficulty": "hard", "title": "Bind the right local volume", "id": 58, "domain": "STOR", "difficulty": "hard", "title": "Bind the right local volume",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "lv"), "pvc": name_pick(r, "fast-data")}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "lv"), "pvc": name_pick(r, "fast-data")},
"task": CTX + """ "task": CTX + """
Two PersistentVolumes of class `local-fast` exist. Only one of them lives on the node labelled Two PersistentVolumes of class `local-fast` exist. Only one of them lives on the node labelled
@ -631,7 +631,7 @@ done
], ],
}, },
{ {
"id": 116, "domain": "STOR", "difficulty": "hard", "title": "Migrate data between claims", "id": 59, "domain": "STOR", "difficulty": "hard", "title": "Migrate data between claims",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "mig"), "n": r.choice([7, 11, 13])}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "mig"), "n": r.choice([7, 11, 13])},
"task": CTX + """ "task": CTX + """
In `%{ns}`, PVC `old-data` holds files a running pod `legacy` wrote (`/data`). Storage is being In `%{ns}`, PVC `old-data` holds files a running pod `legacy` wrote (`/data`). Storage is being
@ -660,17 +660,17 @@ spec:
EOF EOF
for i in $(seq 1 30); do [ "$(j -n %{ns} pod legacy '{.status.phase}')" = Running ] && break; sleep 2; done for i in $(seq 1 30); do [ "$(j -n %{ns} pod legacy '{.status.phase}')" = Running ] && break; sleep 2; done
pv=$(j -n %{ns} pvc old-data '{.spec.volumeName}'); [ -n "$pv" ] && $K patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' >/dev/null pv=$(j -n %{ns} pvc old-data '{.spec.volumeName}'); [ -n "$pv" ] && $K patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' >/dev/null
echo "$pv" > "$WS/.q116-pv" echo "$pv" > "$WS/.q59-pv"
""", """,
"checks": [ "checks": [
C("Job migrate completed", 2, r"""eq "$(j -n %{ns} job migrate '{.status.succeeded}')" 1"""), C("Job migrate completed", 2, r"""eq "$(j -n %{ns} job migrate '{.status.succeeded}')" 1"""),
C("legacy Running on new-data with all %{n} files intact", 3, r"""eq "$(j -n %{ns} pod legacy '{.status.phase}')" Running && eq "$(j -n %{ns} pod legacy '{.spec.volumes[?(@.name!="kube-api-access")].persistentVolumeClaim.claimName}' | tr ' ' '\n' | grep -v '^$' | head -1)" new-data && eq "$($K -n %{ns} exec legacy -- sh -c 'ls /data/file-*.txt | wc -l' 2>/dev/null | tr -d ' ')" %{n} && eq "$($K -n %{ns} exec legacy -- cat /data/file-%{n}.txt 2>/dev/null)" record-%{n}"""), C("legacy Running on new-data with all %{n} files intact", 3, r"""eq "$(j -n %{ns} pod legacy '{.status.phase}')" Running && eq "$(j -n %{ns} pod legacy '{.spec.volumes[?(@.name!="kube-api-access")].persistentVolumeClaim.claimName}' | tr ' ' '\n' | grep -v '^$' | head -1)" new-data && eq "$($K -n %{ns} exec legacy -- sh -c 'ls /data/file-*.txt | wc -l' 2>/dev/null | tr -d ' ')" %{n} && eq "$($K -n %{ns} exec legacy -- cat /data/file-%{n}.txt 2>/dev/null)" record-%{n}"""),
C("old-data deleted; its PV Released (not deleted)", 2, r"""! $K -n %{ns} get pvc old-data >/dev/null 2>&1 && pv=$(cat "$WS/.q116-pv" 2>/dev/null) && eq "$(j pv $pv '{.status.phase}')" Released"""), C("old-data deleted; its PV Released (not deleted)", 2, r"""! $K -n %{ns} get pvc old-data >/dev/null 2>&1 && pv=$(cat "$WS/.q59-pv" 2>/dev/null) && eq "$(j pv $pv '{.status.phase}')" Released"""),
], ],
}, },
# ============================================================== TSHOOT ==== # ============================================================== TSHOOT ====
{ {
"id": 117, "domain": "TSHOOT", "breaks": "w2-kubelet", "difficulty": "hard", "title": "Worker node: three things wrong", "disruptive": True, "id": 60, "domain": "TSHOOT", "breaks": "w2-kubelet", "difficulty": "hard", "title": "Worker node: three things wrong", "disruptive": True,
"params": lambda r: {"ctx": "k8s-c1-H"}, "params": lambda r: {"ctx": "k8s-c1-H"},
"task": CTX + """ "task": CTX + """
Node `{W2}` is unusable. Make it fully usable again: `Ready`, schedulable, kubelet running and Node `{W2}` is unusable. Make it fully usable again: `Ready`, schedulable, kubelet running and
@ -694,7 +694,7 @@ $K -n default delete pod node-probe --wait=false >/dev/null 2>&1; exit $ok"""),
], ],
}, },
{ {
"id": 118, "domain": "TSHOOT", "breaks": ["scheduler", "controller-manager"], "difficulty": "hard", "title": "Two control-plane components down", "disruptive": True, "id": 61, "domain": "TSHOOT", "breaks": ["scheduler", "controller-manager"], "difficulty": "hard", "title": "Two control-plane components down", "disruptive": True,
"params": lambda r: {"ctx": "k8s-c1-H"}, "params": lambda r: {"ctx": "k8s-c1-H"},
"task": CTX + """ "task": CTX + """
New Deployments create no pods, and pods that do get created stay `Pending`. Two different New Deployments create no pods, and pods that do get created stay `Pending`. Two different
@ -718,7 +718,7 @@ $K -n default delete deploy cp-probe --wait=false >/dev/null 2>&1; exit $ok"""),
], ],
}, },
{ {
"id": 119, "domain": "TSHOOT", "difficulty": "hard", "title": "Application down for four reasons", "id": 62, "domain": "TSHOOT", "difficulty": "hard", "title": "Application down for four reasons",
"params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "checkout"), "dep": name_pick(r, "checkout")}, "params": lambda r: {"ctx": ctx_pick(r), "ns": ns_pick(r, "checkout"), "dep": name_pick(r, "checkout")},
"task": CTX + """ "task": CTX + """
Deployment `%{dep}` in `%{ns}` should have 3 Available replicas and Service `%{dep}` should route Deployment `%{dep}` in `%{ns}` should have 3 Available replicas and Service `%{dep}` should route
@ -762,21 +762,21 @@ spec:
selector: {app: %{dep}-web} selector: {app: %{dep}-web}
ports: [{port: 80, targetPort: 8080}] ports: [{port: 80, targetPort: 8080}]
EOF EOF
j -n %{ns} deploy %{dep} '{.metadata.uid}' > "$WS/.q119-uid"; j -n %{ns} svc %{dep} '{.spec.clusterIP}' > "$WS/.q119-ip" j -n %{ns} deploy %{dep} '{.metadata.uid}' > "$WS/.q62-uid"; j -n %{ns} svc %{dep} '{.spec.clusterIP}' > "$WS/.q62-ip"
""", """,
"checks": [ "checks": [
C("3 replicas Available", 3, r"""for i in $(seq 1 6); do [ "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" = 3 ] && exit 0; sleep 5; done; exit 1"""), C("3 replicas Available", 3, r"""for i in $(seq 1 6); do [ "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" = 3 ] && exit 0; sleep 5; done; exit 1"""),
C("Service routes: endpoints ready, targetPort 80", 2, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].targetPort}')" 80 && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""), C("Service routes: endpoints ready, targetPort 80", 2, r"""eq "$(j -n %{ns} svc %{dep} '{.spec.ports[0].targetPort}')" 80 && [ -n "$(j -n %{ns} endpointslices -l kubernetes.io/service-name=%{dep} '{.items[*].endpoints[?(@.conditions.ready==true)].addresses[*]}')" ]"""),
C("Deployment UID and Service ClusterIP unchanged (fixed in place)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 || exit 1 # only once it is actually fixed C("Deployment UID and Service ClusterIP unchanged (fixed in place)", 2, r"""eq "$(j -n %{ns} deploy %{dep} '{.status.availableReplicas}')" 3 || exit 1 # only once it is actually fixed
eq "$(j -n %{ns} deploy %{dep} '{.metadata.uid}')" "$(cat "$WS/.q119-uid")" && eq "$(j -n %{ns} svc %{dep} '{.spec.clusterIP}')" "$(cat "$WS/.q119-ip")" """), eq "$(j -n %{ns} deploy %{dep} '{.metadata.uid}')" "$(cat "$WS/.q62-uid")" && eq "$(j -n %{ns} svc %{dep} '{.spec.clusterIP}')" "$(cat "$WS/.q62-ip")" """),
C("quota still present and permits the 3 pods", 1, r"""p=$(j -n %{ns} quota tight '{.spec.hard.pods}'); [ -n "$p" ] && [ "$p" -ge 3 ]"""), C("quota still present and permits the 3 pods", 1, r"""p=$(j -n %{ns} quota tight '{.spec.hard.pods}'); [ -n "$p" ] && [ "$p" -ge 3 ]"""),
], ],
}, },
{ {
"id": 120, "domain": "TSHOOT", "difficulty": "hard", "title": "Cluster facts, exactly", "id": 63, "domain": "TSHOOT", "difficulty": "hard", "title": "Cluster facts, exactly",
"params": lambda r: {"ctx": "k8s-c1-H", "node": r.choice(["{W1}", "{W2}", "{W3}"])}, "params": lambda r: {"ctx": "k8s-c1-H", "node": r.choice(["{W1}", "{W2}", "{W3}"])},
"task": CTX + """ "task": CTX + """
Write `{WS}/q120/facts.txt` with these lines, values taken from the running cluster: Write `{WS}/q63/facts.txt` with these lines, values taken from the running cluster:
``` ```
pod-cidr=<cluster pod CIDR as configured on kube-controller-manager> pod-cidr=<cluster pod CIDR as configured on kube-controller-manager>
service-cidr=<service CIDR as configured on kube-apiserver> service-cidr=<service CIDR as configured on kube-apiserver>
@ -785,15 +785,15 @@ proxy-mode=<kube-proxy mode from its ConfigMap (empty means iptables — write i
%{node}-pod-cidr=<.spec.podCIDR of that node> %{node}-pod-cidr=<.spec.podCIDR of that node>
``` ```
""", """,
"setup": r"""rm -rf "$WS/q120"; mkdir -p "$WS/q120" """, "setup": r"""rm -rf "$WS/q63"; mkdir -p "$WS/q63" """,
"checks": [ "checks": [
C("pod-cidr and service-cidr", 2, r"""f="$WS/q120/facts.txt"; pc=$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'cluster-cidr=[^"]*' | cut -d= -f2); sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2); grep -qx "pod-cidr=$pc" "$f" 2>/dev/null && grep -qx "service-cidr=$sc" "$f" """), C("pod-cidr and service-cidr", 2, r"""f="$WS/q63/facts.txt"; pc=$(j -n kube-system pod -l component=kube-controller-manager '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'cluster-cidr=[^"]*' | cut -d= -f2); sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2); grep -qx "pod-cidr=$pc" "$f" 2>/dev/null && grep -qx "service-cidr=$sc" "$f" """),
C("dns-ip and proxy-mode", 2, r"""f="$WS/q120/facts.txt"; d=$(j -n kube-system svc kube-dns '{.spec.clusterIP}'); m=$(j -n kube-system cm kube-proxy '{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"'); [ -n "$m" ] || m=iptables; grep -qx "dns-ip=$d" "$f" 2>/dev/null && grep -qx "proxy-mode=$m" "$f" """), C("dns-ip and proxy-mode", 2, r"""f="$WS/q63/facts.txt"; d=$(j -n kube-system svc kube-dns '{.spec.clusterIP}'); m=$(j -n kube-system cm kube-proxy '{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"'); [ -n "$m" ] || m=iptables; grep -qx "dns-ip=$d" "$f" 2>/dev/null && grep -qx "proxy-mode=$m" "$f" """),
C("%{node} pod CIDR", 1, r"""grep -qx "%{node}-pod-cidr=$(j node %{node} '{.spec.podCIDR}')" "$WS/q120/facts.txt" 2>/dev/null"""), C("%{node} pod CIDR", 1, r"""grep -qx "%{node}-pod-cidr=$(j node %{node} '{.spec.podCIDR}')" "$WS/q63/facts.txt" 2>/dev/null"""),
], ],
}, },
{ {
"id": 123, "domain": "TSHOOT", "breaks": "apiserver", "difficulty": "hard", "title": "Nothing works: kube-apiserver itself is down", "disruptive": True, "id": 66, "domain": "TSHOOT", "breaks": "apiserver", "difficulty": "hard", "title": "Nothing works: kube-apiserver itself is down", "disruptive": True,
"params": lambda r: {"ctx": "k8s-c1-H"}, "params": lambda r: {"ctx": "k8s-c1-H"},
"task": CTX + """ "task": CTX + """
Every `kubectl` command from every context times out or refuses to connect. This is not a Every `kubectl` command from every context times out or refuses to connect. This is not a
@ -824,6 +824,6 @@ $K -n default delete pod api-probe --wait=false >/dev/null 2>&1; exit $ok"""),
# derived params: w2 = 100 - w for the canary split # derived params: w2 = 100 - w for the canary split
for _q in QUESTIONS: for _q in QUESTIONS:
if _q["id"] == 113: if _q["id"] == 56:
_orig = _q["params"] _orig = _q["params"]
_q["params"] = lambda r, _o=_orig: (lambda p: {**p, "w2": 100 - p["w"]})(_o(r)) _q["params"] = lambda r, _o=_orig: (lambda p: {**p, "w2": 100 - p["w"]})(_o(r))

192
tracks/cka/bank_sim_c.py
View File

@ -7,8 +7,8 @@ repair, PSA-free securityContext work, HPA behavior blocks, and "write the answe
data-extraction tasks. data-extraction tasks.
Two scenarios are deliberately adapted so they cannot wreck the practice cluster: Two scenarios are deliberately adapted so they cannot wreck the practice cluster:
- 410 audits the installed CNI instead of installing Calico over kindnet. - 77 audits the installed CNI instead of installing Calico over kindnet.
- 413 plans a control-plane upgrade instead of running `kubeadm upgrade apply` for real. - 80 plans a control-plane upgrade instead of running `kubeadm upgrade apply` for real.
Run with: ./exam sim c Run with: ./exam sim c
""" """
@ -23,9 +23,9 @@ def task(**kw):
Q.append(kw) Q.append(kw)
# ------------------------------------------------------------------ 401 ---- # ------------------------------------------------------------------ 68 ----
task( task(
id=401, domain="STOR", title="Provision geo-fence persistent storage", id=68, domain="STOR", title="Provision geo-fence persistent storage",
task=""" task="""
In Namespace `geo-fence`: In Namespace `geo-fence`:
@ -100,31 +100,31 @@ k -n geo-fence get pvc
""", """,
) )
# ------------------------------------------------------------------ 402 ---- # ------------------------------------------------------------------ 69 ----
task( task(
id=402, domain="ARCH", title="Render a chart, then install it", id=69, domain="ARCH", title="Render a chart, then install it",
task=""" task="""
A chart is at `{WS}/q402/chart`. A chart is at `{WS}/q69/chart`.
1. Render the chart **without installing it** as release name `ride-argocd`, namespace 1. Render the chart **without installing it** as release name `ride-argocd`, namespace
`ride-gitops`, with `replicaCount=3`, saving the rendered manifests to `ride-gitops`, with `replicaCount=3`, saving the rendered manifests to
`{WS}/q402/manifest.yaml`. `{WS}/q69/manifest.yaml`.
2. Then install that same chart into Namespace `ride-gitops` (create it) as release 2. Then install that same chart into Namespace `ride-gitops` (create it) as release
`ride-argocd` with `replicaCount=3`. `ride-argocd` with `replicaCount=3`.
3. Save the output of listing the releases in that Namespace to `{WS}/q402/releases.txt`. 3. Save the output of listing the releases in that Namespace to `{WS}/q69/releases.txt`.
""", """,
setup=r""" setup=r"""
rm -rf "$WS/q402"; mkdir -p "$WS/q402/chart/templates" rm -rf "$WS/q69"; mkdir -p "$WS/q69/chart/templates"
cat > "$WS/q402/chart/Chart.yaml" <<'EOF' cat > "$WS/q69/chart/Chart.yaml" <<'EOF'
apiVersion: v2 apiVersion: v2
name: ride name: ride
version: 0.2.0 version: 0.2.0
EOF EOF
cat > "$WS/q402/chart/values.yaml" <<'EOF' cat > "$WS/q69/chart/values.yaml" <<'EOF'
replicaCount: 1 replicaCount: 1
image: nginx:1.27 image: nginx:1.27
EOF EOF
cat > "$WS/q402/chart/templates/deploy.yaml" <<'EOF' cat > "$WS/q69/chart/templates/deploy.yaml" <<'EOF'
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
metadata: {name: {{ .Release.Name }}-server, namespace: {{ .Release.Namespace }}} metadata: {name: {{ .Release.Name }}-server, namespace: {{ .Release.Namespace }}}
@ -141,29 +141,29 @@ command -v helm >/dev/null || echo "warn: helm is not installed on this machine"
""", """,
checks=[ checks=[
C("manifest.yaml holds rendered manifests, not an installed release", 3, r""" C("manifest.yaml holds rendered manifests, not an installed release", 3, r"""
f="$WS/q402/manifest.yaml"; [ -s "$f" ] || exit 1 f="$WS/q69/manifest.yaml"; [ -s "$f" ] || exit 1
grep -q 'kind: Deployment' "$f" && grep -q 'ride-argocd-server' "$f" && grep -Eq 'replicas: 3' "$f" """), grep -q 'kind: Deployment' "$f" && grep -q 'ride-argocd-server' "$f" && grep -Eq 'replicas: 3' "$f" """),
C("release ride-argocd is deployed in ride-gitops with 3 replicas", 3, r""" C("release ride-argocd is deployed in ride-gitops with 3 replicas", 3, r"""
helm -n ride-gitops status ride-argocd -o json 2>/dev/null \ helm -n ride-gitops status ride-argocd -o json 2>/dev/null \
| python3 -c 'import sys,json;sys.exit(0 if json.load(sys.stdin)["info"]["status"]=="deployed" else 1)' || exit 1 | python3 -c 'import sys,json;sys.exit(0 if json.load(sys.stdin)["info"]["status"]=="deployed" else 1)' || exit 1
for i in $(seq 1 20); do [ "$(j -n ride-gitops deploy ride-argocd-server '{.status.availableReplicas}')" = 3 ] && exit 0; sleep 5; done; exit 1"""), for i in $(seq 1 20); do [ "$(j -n ride-gitops deploy ride-argocd-server '{.status.availableReplicas}')" = 3 ] && exit 0; sleep 5; done; exit 1"""),
C("releases.txt lists the ride-argocd release", 3, r""" C("releases.txt lists the ride-argocd release", 3, r"""
grep -q 'ride-argocd' "$WS/q402/releases.txt" 2>/dev/null && grep -qi 'deployed' "$WS/q402/releases.txt" """), grep -q 'ride-argocd' "$WS/q69/releases.txt" 2>/dev/null && grep -qi 'deployed' "$WS/q69/releases.txt" """),
], ],
solution=""" solution="""
helm template ride-argocd {WS}/q402/chart -n ride-gitops --set replicaCount=3 > {WS}/q402/manifest.yaml helm template ride-argocd {WS}/q69/chart -n ride-gitops --set replicaCount=3 > {WS}/q69/manifest.yaml
k create ns ride-gitops k create ns ride-gitops
helm -n ride-gitops install ride-argocd {WS}/q402/chart --set replicaCount=3 helm -n ride-gitops install ride-argocd {WS}/q69/chart --set replicaCount=3
helm -n ride-gitops list > {WS}/q402/releases.txt helm -n ride-gitops list > {WS}/q69/releases.txt
# `helm template` renders locally and talks to no cluster - it is how you review what a chart # `helm template` renders locally and talks to no cluster - it is how you review what a chart
# would do, and how GitOps pipelines produce committed YAML. `helm install` is the one that # would do, and how GitOps pipelines produce committed YAML. `helm install` is the one that
# creates a release. --create-namespace on install saves the separate `k create ns`. # creates a release. --create-namespace on install saves the separate `k create ns`.
""", """,
) )
# ------------------------------------------------------------------ 403 ---- # ------------------------------------------------------------------ 70 ----
task( task(
id=403, domain="TSHOOT", title="Fix the driver onboarding app", id=70, domain="TSHOOT", title="Fix the driver onboarding app",
task=""" task="""
Deployment `onboard-app` in Namespace `driver-onboard` never becomes Ready. It has **three** Deployment `onboard-app` in Namespace `driver-onboard` never becomes Ready. It has **three**
separate faults. separate faults.
@ -225,21 +225,21 @@ k -n driver-onboard rollout status deploy onboard-app
""", """,
) )
# ------------------------------------------------------------------ 404 ---- # ------------------------------------------------------------------ 71 ----
task( task(
id=404, domain="NET", title="Expose and verify the dispatch API", id=71, domain="NET", title="Expose and verify the dispatch API",
task=""" task="""
Deployment `dispatch-api` is running in Namespace `dispatch-hub` and serves on container port 80. Deployment `dispatch-api` is running in Namespace `dispatch-hub` and serves on container port 80.
1. Expose it as a **ClusterIP** Service named `dispatch-service`, port 80 to target port 80. 1. Expose it as a **ClusterIP** Service named `dispatch-service`, port 80 to target port 80.
It must actually have endpoints. It must actually have endpoints.
2. From inside the cluster, fetch `http://dispatch-service.dispatch-hub.svc:80` and save the 2. From inside the cluster, fetch `http://dispatch-service.dispatch-hub.svc:80` and save the
response body to `{WS}/q404/test-result.txt`. response body to `{WS}/q71/test-result.txt`.
3. Write the Service's ClusterIP — the address only, nothing else — to `{WS}/q404/service-ip.txt`. 3. Write the Service's ClusterIP — the address only, nothing else — to `{WS}/q71/service-ip.txt`.
""", """,
setup=NS + r""" setup=NS + r"""
mkns dispatch-hub mkns dispatch-hub
rm -rf "$WS/q404"; mkdir -p "$WS/q404" rm -rf "$WS/q71"; mkdir -p "$WS/q71"
$K -n dispatch-hub delete svc dispatch-service --ignore-not-found >/dev/null 2>&1 $K -n dispatch-hub delete svc dispatch-service --ignore-not-found >/dev/null 2>&1
$K -n dispatch-hub delete deploy dispatch-api --ignore-not-found --wait=true >/dev/null 2>&1 $K -n dispatch-hub delete deploy dispatch-api --ignore-not-found --wait=true >/dev/null 2>&1
$K -n dispatch-hub create deploy dispatch-api --image=nginx:1.27 --replicas=2 --port=80 >/dev/null $K -n dispatch-hub create deploy dispatch-api --image=nginx:1.27 --replicas=2 --port=80 >/dev/null
@ -254,26 +254,26 @@ eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" ClusterIP \
C("the Service has active endpoints", 3, r""" C("the Service has active endpoints", 3, r"""
[ -n "$(j -n dispatch-hub endpointslices -l kubernetes.io/service-name=dispatch-service '{.items[*].endpoints[*].addresses[*]}')" ]"""), [ -n "$(j -n dispatch-hub endpointslices -l kubernetes.io/service-name=dispatch-service '{.items[*].endpoints[*].addresses[*]}')" ]"""),
C("test-result.txt holds the response from nginx", 2, r""" C("test-result.txt holds the response from nginx", 2, r"""
grep -qi 'nginx\|<html' "$WS/q404/test-result.txt" 2>/dev/null"""), grep -qi 'nginx\|<html' "$WS/q71/test-result.txt" 2>/dev/null"""),
C("service-ip.txt contains exactly the Service ClusterIP", 2, r""" C("service-ip.txt contains exactly the Service ClusterIP", 2, r"""
want=$(j -n dispatch-hub svc dispatch-service '{.spec.clusterIP}') want=$(j -n dispatch-hub svc dispatch-service '{.spec.clusterIP}')
got=$(tr -d '[:space:]' < "$WS/q404/service-ip.txt" 2>/dev/null) got=$(tr -d '[:space:]' < "$WS/q71/service-ip.txt" 2>/dev/null)
[ -n "$want" ] && eq "$got" "$want" """), [ -n "$want" ] && eq "$got" "$want" """),
], ],
solution=""" solution="""
k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80 k -n dispatch-hub expose deploy dispatch-api --name=dispatch-service --port=80 --target-port=80
k -n dispatch-hub get endpointslices -l kubernetes.io/service-name=dispatch-service # must not be empty k -n dispatch-hub get endpointslices -l kubernetes.io/service-name=dispatch-service # must not be empty
k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- \\ k -n dispatch-hub run probe --image=busybox:1.36 --restart=Never -i --rm -- \\
wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q404/test-result.txt wget -qO- http://dispatch-service.dispatch-hub.svc:80 > {WS}/q71/test-result.txt
k -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q404/service-ip.txt k -n dispatch-hub get svc dispatch-service -o jsonpath='{.spec.clusterIP}' > {WS}/q71/service-ip.txt
# `expose` copies the Deployment's selector for you, which is what makes the endpoints appear. # `expose` copies the Deployment's selector for you, which is what makes the endpoints appear.
# Write the IP with jsonpath, not by eyeballing `get svc` - the grader compares it literally. # Write the IP with jsonpath, not by eyeballing `get svc` - the grader compares it literally.
""", """,
) )
# ------------------------------------------------------------------ 405 ---- # ------------------------------------------------------------------ 72 ----
task( task(
id=405, domain="STOR", title="Migrate ride history to a Retain volume", id=72, domain="STOR", title="Migrate ride history to a Retain volume",
task=""" task="""
In Namespace `ride-history`, Deployment `ride-archive` stores data on PVC `ride-archive-pvc`, In Namespace `ride-history`, Deployment `ride-archive` stores data on PVC `ride-archive-pvc`,
whose PersistentVolume has reclaim policy `Delete` — losing the claim would destroy the data. whose PersistentVolume has reclaim policy `Delete` — losing the claim would destroy the data.
@ -407,9 +407,9 @@ k -n ride-history scale deploy ride-archive --replicas=1
""", """,
) )
# ------------------------------------------------------------------ 406 ---- # ------------------------------------------------------------------ 73 ----
task( task(
id=406, domain="TSHOOT", title="Fix ETA service connectivity", id=73, domain="TSHOOT", title="Fix ETA service connectivity",
task=""" task="""
In Namespace `eta-service`, Deployment `eta-calculator` is healthy but Service `eta-backend` In Namespace `eta-service`, Deployment `eta-calculator` is healthy but Service `eta-backend`
returns nothing. There are **two** faults in the Service. returns nothing. There are **two** faults in the Service.
@ -457,9 +457,9 @@ k -n eta-service get endpointslices -l kubernetes.io/service-name=eta-backend
""", """,
) )
# ------------------------------------------------------------------ 407 ---- # ------------------------------------------------------------------ 74 ----
task( task(
id=407, domain="NET", title="Fleet dashboard canary with Gateway API", id=74, domain="NET", title="Fleet dashboard canary with Gateway API",
task=""" task="""
Namespace `fleet-ops` has Services `fleet-dashboard-v1` and `fleet-dashboard-v2` (both port 80) Namespace `fleet-ops` has Services `fleet-dashboard-v1` and `fleet-dashboard-v2` (both port 80)
and a GatewayClass `fleet-gateway-class` already exists. and a GatewayClass `fleet-gateway-class` already exists.
@ -551,9 +551,9 @@ EOF
""", """,
) )
# ------------------------------------------------------------------ 408 ---- # ------------------------------------------------------------------ 75 ----
task( task(
id=408, domain="ARCH", title="Cluster-wide read access for the auth team", id=75, domain="ARCH", title="Cluster-wide read access for the auth team",
task=""" task="""
The rider auth team needs a read-only identity that works in **every** Namespace. The rider auth team needs a read-only identity that works in **every** Namespace.
@ -562,11 +562,11 @@ The rider auth team needs a read-only identity that works in **every** Namespace
`pods`, `deployments`, `services` and `events`. `pods`, `deployments`, `services` and `events`.
3. Bind it cluster-wide to that ServiceAccount with ClusterRoleBinding `auth-viewer-binding`. 3. Bind it cluster-wide to that ServiceAccount with ClusterRoleBinding `auth-viewer-binding`.
4. Verify the ServiceAccount can list pods in `kube-system` and write the single-word answer to 4. Verify the ServiceAccount can list pods in `kube-system` and write the single-word answer to
`{WS}/q408/auth-check.txt`. `{WS}/q75/auth-check.txt`.
""", """,
setup=NS + r""" setup=NS + r"""
mkns rider-auth mkns rider-auth
rm -rf "$WS/q408"; mkdir -p "$WS/q408" rm -rf "$WS/q75"; mkdir -p "$WS/q75"
$K delete clusterrolebinding auth-viewer-binding --ignore-not-found >/dev/null 2>&1 $K delete clusterrolebinding auth-viewer-binding --ignore-not-found >/dev/null 2>&1
$K delete clusterrole auth-cluster-viewer --ignore-not-found >/dev/null 2>&1 $K delete clusterrole auth-cluster-viewer --ignore-not-found >/dev/null 2>&1
$K -n rider-auth delete sa auth-auditor --ignore-not-found >/dev/null 2>&1 $K -n rider-auth delete sa auth-auditor --ignore-not-found >/dev/null 2>&1
@ -587,22 +587,22 @@ eq "$($b -o jsonpath='{.roleRef.name}' 2>/dev/null)" auth-cluster-viewer \
&& $b -o jsonpath='{.subjects[*].name}' | grep -qw auth-auditor \ && $b -o jsonpath='{.subjects[*].name}' | grep -qw auth-auditor \
&& eq "$($K auth can-i list pods -n kube-system --as=system:serviceaccount:rider-auth:auth-auditor 2>/dev/null)" yes"""), && eq "$($K auth can-i list pods -n kube-system --as=system:serviceaccount:rider-auth:auth-auditor 2>/dev/null)" yes"""),
C("auth-check.txt records the verification result", 2, r""" C("auth-check.txt records the verification result", 2, r"""
grep -qx 'yes' "$WS/q408/auth-check.txt" 2>/dev/null"""), grep -qx 'yes' "$WS/q75/auth-check.txt" 2>/dev/null"""),
], ],
solution=""" solution="""
k -n rider-auth create sa auth-auditor k -n rider-auth create sa auth-auditor
k create clusterrole auth-cluster-viewer --verb=get,list,watch --resource=pods,deployments,services,events k create clusterrole auth-cluster-viewer --verb=get,list,watch --resource=pods,deployments,services,events
k create clusterrolebinding auth-viewer-binding --clusterrole=auth-cluster-viewer \\ k create clusterrolebinding auth-viewer-binding --clusterrole=auth-cluster-viewer \\
--serviceaccount=rider-auth:auth-auditor --serviceaccount=rider-auth:auth-auditor
k auth can-i list pods -n kube-system --as=system:serviceaccount:rider-auth:auth-auditor > {WS}/q408/auth-check.txt k auth can-i list pods -n kube-system --as=system:serviceaccount:rider-auth:auth-auditor > {WS}/q75/auth-check.txt
# A ClusterRole bound with a RoleBinding only applies in that one Namespace - it is the # A ClusterRole bound with a RoleBinding only applies in that one Namespace - it is the
# ClusterRoleBinding that makes it cluster-wide. The --serviceaccount flag takes ns:name. # ClusterRoleBinding that makes it cluster-wide. The --serviceaccount flag takes ns:name.
""", """,
) )
# ------------------------------------------------------------------ 409 ---- # ------------------------------------------------------------------ 76 ----
task( task(
id=409, domain="WORK", title="Add an init container to the trip logger", id=76, domain="WORK", title="Add an init container to the trip logger",
task=""" task="""
Deployment `trip-logger` in Namespace `trip-tracker` must be given a startup step. Deployment `trip-logger` in Namespace `trip-tracker` must be given a startup step.
@ -669,15 +669,15 @@ k -n trip-tracker rollout status deploy trip-logger
""", """,
) )
# ------------------------------------------------------------------ 410 ---- # ------------------------------------------------------------------ 77 ----
task( task(
id=410, domain="NET", title="Audit the CNI, then lock a namespace down", id=77, domain="NET", title="Audit the CNI, then lock a namespace down",
task=""" task="""
Before rolling out NetworkPolicies, confirm what the cluster's CNI can actually enforce. Before rolling out NetworkPolicies, confirm what the cluster's CNI can actually enforce.
1. Write the name of the DaemonSet in `kube-system` that provides pod networking to 1. Write the name of the DaemonSet in `kube-system` that provides pod networking to
`{WS}/q410/cni.txt`. `{WS}/q77/cni.txt`.
2. Write `yes` or `no` to `{WS}/q410/enforced.txt`: does this CNI enforce NetworkPolicy? 2. Write `yes` or `no` to `{WS}/q77/enforced.txt`: does this CNI enforce NetworkPolicy?
3. In Namespace `zero-trust`, create NetworkPolicy `default-deny-all` that denies **all** ingress 3. In Namespace `zero-trust`, create NetworkPolicy `default-deny-all` that denies **all** ingress
and egress for every Pod in the Namespace, with no allow rules of any kind. and egress for every Pod in the Namespace, with no allow rules of any kind.
@ -686,14 +686,14 @@ happily and never enforce them, and a CNI's behaviour can change between release
""", """,
setup=NS + r""" setup=NS + r"""
mkns zero-trust mkns zero-trust
rm -rf "$WS/q410"; mkdir -p "$WS/q410" rm -rf "$WS/q77"; mkdir -p "$WS/q77"
$K -n zero-trust delete netpol default-deny-all --ignore-not-found >/dev/null 2>&1 $K -n zero-trust delete netpol default-deny-all --ignore-not-found >/dev/null 2>&1
""", """,
checks=[ checks=[
C("cni.txt names the CNI DaemonSet in kube-system", 2, r""" C("cni.txt names the CNI DaemonSet in kube-system", 2, r"""
want=$($K -n kube-system get ds -o name 2>/dev/null | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1) want=$($K -n kube-system get ds -o name 2>/dev/null | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1)
[ -n "$want" ] || want=kindnet [ -n "$want" ] || want=kindnet
grep -qi "$want" "$WS/q410/cni.txt" 2>/dev/null"""), grep -qi "$want" "$WS/q77/cni.txt" 2>/dev/null"""),
# The right answer depends on the build, so derive it: recent kindnet embeds a policy engine # The right answer depends on the build, so derive it: recent kindnet embeds a policy engine
# (nftables table `kindnet-network-policies` on every node - verified enforcing on this # (nftables table `kindnet-network-policies` on every node - verified enforcing on this
# cluster), calico/cilium/weave enforce, flannel does not. # cluster), calico/cilium/weave enforce, flannel does not.
@ -704,7 +704,7 @@ case "$ds" in
*kindnet*) nx "$W1" 'nft list table inet kindnet-network-policies >/dev/null 2>&1' </dev/null && want=yes || want=no ;; *kindnet*) nx "$W1" 'nft list table inet kindnet-network-policies >/dev/null 2>&1' </dev/null && want=yes || want=no ;;
*) want=no ;; *) want=no ;;
esac esac
grep -qix "$want" "$WS/q410/enforced.txt" 2>/dev/null"""), grep -qix "$want" "$WS/q77/enforced.txt" 2>/dev/null"""),
C("default-deny-all selects every pod in both directions", 4, r""" C("default-deny-all selects every pod in both directions", 4, r"""
n="$K -n zero-trust get netpol default-deny-all" n="$K -n zero-trust get netpol default-deny-all"
eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \ eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \
@ -713,7 +713,7 @@ eq "$($n -o jsonpath='{.spec.podSelector}' 2>/dev/null)" '{}' \
&& [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ]"""), && [ -z "$($n -o jsonpath='{.spec.ingress}')" ] && [ -z "$($n -o jsonpath='{.spec.egress}')" ]"""),
], ],
solution=""" solution="""
k -n kube-system get ds # kindnet -> {WS}/q410/cni.txt k -n kube-system get ds # kindnet -> {WS}/q77/cni.txt
# Don't answer from memory - prove it. Two pods, one deny policy, one wget: # Don't answer from memory - prove it. Two pods, one deny policy, one wget:
k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv; k -n zero-trust expose pod srv --port 80 k -n zero-trust run srv --image=nginx:1.27 --labels=app=srv; k -n zero-trust expose pod srv --port 80
k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600 k -n zero-trust run cli --image=busybox:1.36 -- sleep 3600
@ -727,7 +727,7 @@ spec:
policyTypes: [Ingress, Egress] policyTypes: [Ingress, Egress]
EOF EOF
k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null || echo BLOCKED # -> BLOCKED here k -n zero-trust exec cli -- wget -qO- -T3 http://srv >/dev/null || echo BLOCKED # -> BLOCKED here
echo yes > {WS}/q410/enforced.txt echo yes > {WS}/q77/enforced.txt
k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv # leave only the policy k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv # leave only the policy
# podSelector {} means every pod in the namespace; naming both policyTypes with no ingress/egress # podSelector {} means every pod in the namespace; naming both policyTypes with no ingress/egress
# blocks listed means "deny everything". Recent kindnet builds (this cluster's included) embed a # blocks listed means "deny everything". Recent kindnet builds (this cluster's included) embed a
@ -737,9 +737,9 @@ k -n zero-trust delete pod srv cli; k -n zero-trust delete svc srv
""", """,
) )
# ------------------------------------------------------------------ 411 ---- # ------------------------------------------------------------------ 78 ----
task( task(
id=411, domain="WORK", title="Priority for the rider support portal", id=78, domain="WORK", title="Priority for the rider support portal",
task=""" task="""
1. Create PriorityClass `rider-support-priority` with value `1000000`, **not** the global default, 1. Create PriorityClass `rider-support-priority` with value `1000000`, **not** the global default,
and description `Priority for rider support services`. and description `Priority for rider support services`.
@ -784,9 +784,9 @@ k -n rider-support get pod -l app=support-portal -o jsonpath='{.items[0].spec.pr
""", """,
) )
# ------------------------------------------------------------------ 412 ---- # ------------------------------------------------------------------ 79 ----
task( task(
id=412, domain="WORK", title="Surge pricing configuration", id=79, domain="WORK", title="Surge pricing configuration",
task=""" task="""
In Namespace `surge-pricing`: In Namespace `surge-pricing`:
@ -849,33 +849,33 @@ k -n surge-pricing rollout status deploy surge-engine
""", """,
) )
# ------------------------------------------------------------------ 413 ---- # ------------------------------------------------------------------ 80 ----
task( task(
id=413, domain="ARCH", title="Plan the control-plane upgrade", id=80, domain="ARCH", title="Plan the control-plane upgrade",
task=""" task="""
The control-plane node is scheduled for a minor-version upgrade. Prepare the change — **do not The control-plane node is scheduled for a minor-version upgrade. Prepare the change — **do not
execute an upgrade**. execute an upgrade**.
1. Write the cluster's current server version, as `kubectl` reports it, to `{WS}/q413/current.txt`. 1. Write the cluster's current server version, as `kubectl` reports it, to `{WS}/q80/current.txt`.
2. Write every node's name and kubelet version, one node per line, to `{WS}/q413/kubelets.txt`. 2. Write every node's name and kubelet version, one node per line, to `{WS}/q80/kubelets.txt`.
3. Write to `{WS}/q413/steps.sh` the ordered commands to upgrade the **control-plane** node — the 3. Write to `{WS}/q80/steps.sh` the ordered commands to upgrade the **control-plane** node — the
kubeadm package install, the kubeadm upgrade step, draining, the kubelet/kubectl package kubeadm package install, the kubeadm upgrade step, draining, the kubelet/kubectl package
install, restarting the kubelet, and uncordoning — one command per line, in execution order. install, restarting the kubelet, and uncordoning — one command per line, in execution order.
4. Write to `{WS}/q413/worker.txt` the single kubeadm subcommand that upgrades a **worker** node 4. Write to `{WS}/q80/worker.txt` the single kubeadm subcommand that upgrades a **worker** node
instead (it is not `upgrade apply`). instead (it is not `upgrade apply`).
""", """,
setup=r"""rm -rf "$WS/q413"; mkdir -p "$WS/q413" """, setup=r"""rm -rf "$WS/q80"; mkdir -p "$WS/q80" """,
checks=[ checks=[
C("current.txt matches the live server version", 2, r""" C("current.txt matches the live server version", 2, r"""
v=$($K version -o json 2>/dev/null | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])') v=$($K version -o json 2>/dev/null | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])')
[ -n "$v" ] && grep -qF "$v" "$WS/q413/current.txt" 2>/dev/null"""), [ -n "$v" ] && grep -qF "$v" "$WS/q80/current.txt" 2>/dev/null"""),
C("kubelets.txt lists every node with its kubelet version", 2, r""" C("kubelets.txt lists every node with its kubelet version", 2, r"""
f="$WS/q413/kubelets.txt"; [ -s "$f" ] || exit 1 f="$WS/q80/kubelets.txt"; [ -s "$f" ] || exit 1
for n in $($K get nodes -o jsonpath='{.items[*].metadata.name}'); do grep -q "$n" "$f" || exit 1; done for n in $($K get nodes -o jsonpath='{.items[*].metadata.name}'); do grep -q "$n" "$f" || exit 1; done
v=$($K get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}') v=$($K get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}')
grep -qF "$v" "$f" """), grep -qF "$v" "$f" """),
C("steps.sh has the upgrade commands in a workable order", 4, r""" C("steps.sh has the upgrade commands in a workable order", 4, r"""
f="$WS/q413/steps.sh"; [ -f "$f" ] || exit 1 f="$WS/q80/steps.sh"; [ -f "$f" ] || exit 1
grep -q 'kubeadm upgrade apply' "$f" || exit 1 grep -q 'kubeadm upgrade apply' "$f" || exit 1
grep -Eq '(apt|yum|dnf).*install.*kubeadm' "$f" || exit 1 grep -Eq '(apt|yum|dnf).*install.*kubeadm' "$f" || exit 1
grep -Eq '(apt|yum|dnf).*install.*kubelet' "$f" || exit 1 grep -Eq '(apt|yum|dnf).*install.*kubelet' "$f" || exit 1
@ -887,13 +887,13 @@ kl=$(grep -nE '(apt|yum|dnf).*install.*kubelet' "$f" | head -1 | cut -d: -f1)
un=$(grep -n 'uncordon' "$f" | head -1 | cut -d: -f1) un=$(grep -n 'uncordon' "$f" | head -1 | cut -d: -f1)
[ "$ka" -lt "$ap" ] && [ "$ap" -lt "$kl" ] && [ "$kl" -lt "$un" ]"""), [ "$ka" -lt "$ap" ] && [ "$ap" -lt "$kl" ] && [ "$kl" -lt "$un" ]"""),
C("worker.txt names `kubeadm upgrade node`", 2, r""" C("worker.txt names `kubeadm upgrade node`", 2, r"""
grep -Eq 'kubeadm[[:space:]]+upgrade[[:space:]]+node' "$WS/q413/worker.txt" 2>/dev/null \ grep -Eq 'kubeadm[[:space:]]+upgrade[[:space:]]+node' "$WS/q80/worker.txt" 2>/dev/null \
&& ! grep -q 'upgrade apply' "$WS/q413/worker.txt" """), && ! grep -q 'upgrade apply' "$WS/q80/worker.txt" """),
], ],
solution=""" solution="""
k version -o json | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])' > {WS}/q413/current.txt k version -o json | python3 -c 'import sys,json;print(json.load(sys.stdin)["serverVersion"]["gitVersion"])' > {WS}/q80/current.txt
k get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion --no-headers > {WS}/q413/kubelets.txt k get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion --no-headers > {WS}/q80/kubelets.txt
cat > {WS}/q413/steps.sh <<'EOF' cat > {WS}/q80/steps.sh <<'EOF'
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.34.1-1.1 && apt-mark hold kubeadm apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.34.1-1.1 && apt-mark hold kubeadm
kubeadm upgrade plan kubeadm upgrade plan
kubeadm upgrade apply v1.34.1 -y kubeadm upgrade apply v1.34.1 -y
@ -903,7 +903,7 @@ systemctl daemon-reload
systemctl restart kubelet systemctl restart kubelet
kubectl uncordon <control-plane> kubectl uncordon <control-plane>
EOF EOF
echo 'kubeadm upgrade node' > {WS}/q413/worker.txt echo 'kubeadm upgrade node' > {WS}/q80/worker.txt
# kubeadm goes first because it drives the whole thing; the control plane is upgraded by # kubeadm goes first because it drives the whole thing; the control plane is upgraded by
# `upgrade apply`, and only then do the node binaries (kubelet/kubectl) follow, ending with # `upgrade apply`, and only then do the node binaries (kubelet/kubectl) follow, ending with
# uncordon. Workers run `kubeadm upgrade node` - they host no control-plane static pods, so # uncordon. Workers run `kubeadm upgrade node` - they host no control-plane static pods, so
@ -912,9 +912,9 @@ echo 'kubeadm upgrade node' > {WS}/q413/worker.txt
""", """,
) )
# ------------------------------------------------------------------ 414 ---- # ------------------------------------------------------------------ 81 ----
task( task(
id=414, domain="TSHOOT", title="Fix the fare calculation pods", id=81, domain="TSHOOT", title="Fix the fare calculation pods",
task=""" task="""
Deployment `fare-calculator` in Namespace `fare-engine` has no running pods. There are **two** Deployment `fare-calculator` in Namespace `fare-engine` has no running pods. There are **two**
independent reasons, one storage and one scheduling. independent reasons, one storage and one scheduling.
@ -1001,9 +1001,9 @@ k -n fare-engine rollout status deploy fare-calculator
""", """,
) )
# ------------------------------------------------------------------ 415 ---- # ------------------------------------------------------------------ 82 ----
task( task(
id=415, domain="WORK", title="Autoscale the route planner on memory", id=82, domain="WORK", title="Autoscale the route planner on memory",
task=""" task="""
Create HorizontalPodAutoscaler `route-planner-hpa` in Namespace `route-planner` for Deployment Create HorizontalPodAutoscaler `route-planner-hpa` in Namespace `route-planner` for Deployment
`route-planner`: `route-planner`:
@ -1073,9 +1073,9 @@ k -n route-planner get hpa route-planner-hpa
""", """,
) )
# ------------------------------------------------------------------ 416 ---- # ------------------------------------------------------------------ 83 ----
task( task(
id=416, domain="NET", title="NetworkPolicy for the safety monitor", id=83, domain="NET", title="NetworkPolicy for the safety monitor",
task=""" task="""
In Namespace `safety-monitor`, create NetworkPolicy `safety-api-policy` for the Pods labelled In Namespace `safety-monitor`, create NetworkPolicy `safety-api-policy` for the Pods labelled
`app=safety-api`: `app=safety-api`:
@ -1146,9 +1146,9 @@ EOF
""", """,
) )
# ------------------------------------------------------------------ 417 ---- # ------------------------------------------------------------------ 84 ----
task( task(
id=417, domain="ARCH", title="Enforce TLS 1.3 on the booking portal", id=84, domain="ARCH", title="Enforce TLS 1.3 on the booking portal",
task=""" task="""
Namespace `booking-tls` runs Deployment `booking-nginx`, which serves HTTPS using the nginx Namespace `booking-tls` runs Deployment `booking-nginx`, which serves HTTPS using the nginx
configuration in ConfigMap `booking-nginx-config`. It currently accepts TLS 1.2 **and** 1.3. configuration in ConfigMap `booking-nginx-config`. It currently accepts TLS 1.2 **and** 1.3.
@ -1156,13 +1156,13 @@ configuration in ConfigMap `booking-nginx-config`. It currently accepts TLS 1.2
1. Change the ConfigMap so only TLS 1.3 is accepted. 1. Change the ConfigMap so only TLS 1.3 is accepted.
2. Make the running pods pick up the change, without editing the Deployment's pod template by hand. 2. Make the running pods pick up the change, without editing the Deployment's pod template by hand.
3. Write the nginx directive line you ended up with, as it appears in the ConfigMap, to 3. Write the nginx directive line you ended up with, as it appears in the ConfigMap, to
`{WS}/q417/ssl-line.txt`. `{WS}/q84/ssl-line.txt`.
A TLS 1.2 handshake against the service must be refused afterwards. A TLS 1.2 handshake against the service must be refused afterwards.
""", """,
setup=NS + r""" setup=NS + r"""
mkns booking-tls mkns booking-tls
rm -rf "$WS/q417"; mkdir -p "$WS/q417" rm -rf "$WS/q84"; mkdir -p "$WS/q84"
$K -n booking-tls delete deploy booking-nginx --ignore-not-found --wait=true >/dev/null 2>&1 $K -n booking-tls delete deploy booking-nginx --ignore-not-found --wait=true >/dev/null 2>&1
$K -n booking-tls delete cm booking-nginx-config --ignore-not-found >/dev/null 2>&1 $K -n booking-tls delete cm booking-nginx-config --ignore-not-found >/dev/null 2>&1
$K -n booking-tls delete secret booking-tls-cert --ignore-not-found >/dev/null 2>&1 $K -n booking-tls delete secret booking-tls-cert --ignore-not-found >/dev/null 2>&1
@ -1223,7 +1223,7 @@ cm_t=$($K -n booking-tls get cm booking-nginx-config -o json --show-managed-fiel
ts=$(j -n booking-tls pod -l app=booking-nginx '{.items[*].metadata.creationTimestamp}'); [ -n "$cm_t" ] && [ -n "$ts" ] || exit 1 ts=$(j -n booking-tls pod -l app=booking-nginx '{.items[*].metadata.creationTimestamp}'); [ -n "$cm_t" ] && [ -n "$ts" ] || exit 1
for t in $ts; do [[ "$t" > "$cm_t" ]] || exit 1; done"""), for t in $ts; do [[ "$t" > "$cm_t" ]] || exit 1; done"""),
C("ssl-line.txt records the directive", 2, r""" C("ssl-line.txt records the directive", 2, r"""
f="$WS/q417/ssl-line.txt" f="$WS/q84/ssl-line.txt"
grep -q 'ssl_protocols' "$f" 2>/dev/null && grep -q 'TLSv1\.3' "$f" && ! grep -q 'TLSv1\.2' "$f" """), grep -q 'ssl_protocols' "$f" 2>/dev/null && grep -q 'TLSv1\.3' "$f" && ! grep -q 'TLSv1\.2' "$f" """),
C("TLS 1.3 is accepted and TLS 1.2 is refused", 3, r""" C("TLS 1.3 is accepted and TLS 1.2 is refused", 3, r"""
for i in $(seq 1 20); do [ "$(j -n booking-tls deploy booking-nginx '{.status.readyReplicas}')" = 1 ] && break; sleep 5; done for i in $(seq 1 20); do [ "$(j -n booking-tls deploy booking-nginx '{.status.readyReplicas}')" = 1 ] && break; sleep 5; done
@ -1240,7 +1240,7 @@ k -n booking-tls edit cm booking-nginx-config
# ssl_protocols TLSv1.2 TLSv1.3; -> ssl_protocols TLSv1.3; # ssl_protocols TLSv1.2 TLSv1.3; -> ssl_protocols TLSv1.3;
k -n booking-tls rollout restart deploy booking-nginx k -n booking-tls rollout restart deploy booking-nginx
k -n booking-tls rollout status deploy booking-nginx k -n booking-tls rollout status deploy booking-nginx
echo 'ssl_protocols TLSv1.3;' > {WS}/q417/ssl-line.txt echo 'ssl_protocols TLSv1.3;' > {WS}/q84/ssl-line.txt
# verify from ON a node (the Service is published as NodePort 30443): # verify from ON a node (the Service is published as NodePort 30443):
# {SHELL_W1} # {SHELL_W1}
# echo | openssl s_client -connect 127.0.0.1:30443 -tls1_3 # connects # echo | openssl s_client -connect 127.0.0.1:30443 -tls1_3 # connects
@ -1253,20 +1253,20 @@ echo 'ssl_protocols TLSv1.3;' > {WS}/q417/ssl-line.txt
""", """,
) )
# ------------------------------------------------------------------ 418 ---- # ------------------------------------------------------------------ 85 ----
task( task(
id=418, domain="TSHOOT", title="Investigate vehicle tracker health", id=85, domain="TSHOOT", title="Investigate vehicle tracker health",
task=""" task="""
Namespace `vehicle-track` has a misbehaving workload. Collect evidence: Namespace `vehicle-track` has a misbehaving workload. Collect evidence:
1. List **all** pods in **all** namespaces sorted by restart count, and save the output to 1. List **all** pods in **all** namespaces sorted by restart count, and save the output to
`{WS}/q418/most-restarts.txt`. `{WS}/q85/most-restarts.txt`.
2. Save the last 50 log lines from the `app=vehicle-tracker` pods to `{WS}/q418/logs.txt`. 2. Save the last 50 log lines from the `app=vehicle-tracker` pods to `{WS}/q85/logs.txt`.
3. Save the Warning events from `vehicle-track`, sorted by time, to `{WS}/q418/warnings.txt`. 3. Save the Warning events from `vehicle-track`, sorted by time, to `{WS}/q85/warnings.txt`.
""", """,
setup=NS + r""" setup=NS + r"""
mkns vehicle-track mkns vehicle-track
rm -rf "$WS/q418"; mkdir -p "$WS/q418" rm -rf "$WS/q85"; mkdir -p "$WS/q85"
$K -n vehicle-track delete deploy vehicle-tracker --ignore-not-found --wait=true >/dev/null 2>&1 $K -n vehicle-track delete deploy vehicle-tracker --ignore-not-found --wait=true >/dev/null 2>&1
$K apply -f - >/dev/null <<'EOF' $K apply -f - >/dev/null <<'EOF'
apiVersion: apps/v1 apiVersion: apps/v1
@ -1286,19 +1286,19 @@ EOF
""", """,
checks=[ checks=[
C("most-restarts.txt is a sorted all-namespace pod listing", 3, r""" C("most-restarts.txt is a sorted all-namespace pod listing", 3, r"""
f="$WS/q418/most-restarts.txt"; [ -s "$f" ] || exit 1 f="$WS/q85/most-restarts.txt"; [ -s "$f" ] || exit 1
grep -qi 'NAMESPACE' "$f" && grep -q 'kube-system' "$f" && grep -q 'vehicle-track' "$f" \ grep -qi 'NAMESPACE' "$f" && grep -q 'kube-system' "$f" && grep -q 'vehicle-track' "$f" \
&& [ "$(wc -l < "$f")" -ge 5 ]"""), && [ "$(wc -l < "$f")" -ge 5 ]"""),
C("logs.txt holds the tracker's log output", 2, r""" C("logs.txt holds the tracker's log output", 2, r"""
grep -q 'tracker starting\|gps fix lost' "$WS/q418/logs.txt" 2>/dev/null"""), grep -q 'tracker starting\|gps fix lost' "$WS/q85/logs.txt" 2>/dev/null"""),
C("warnings.txt holds Warning events from the namespace", 3, r""" C("warnings.txt holds Warning events from the namespace", 3, r"""
f="$WS/q418/warnings.txt"; [ -s "$f" ] || exit 1 f="$WS/q85/warnings.txt"; [ -s "$f" ] || exit 1
grep -q 'Warning' "$f" && ! grep -q '^Normal' "$f" """), grep -q 'Warning' "$f" && ! grep -q '^Normal' "$f" """),
], ],
solution=""" solution="""
k get pods -A --sort-by='.status.containerStatuses[0].restartCount' > {WS}/q418/most-restarts.txt k get pods -A --sort-by='.status.containerStatuses[0].restartCount' > {WS}/q85/most-restarts.txt
k -n vehicle-track logs -l app=vehicle-tracker --tail=50 > {WS}/q418/logs.txt k -n vehicle-track logs -l app=vehicle-tracker --tail=50 > {WS}/q85/logs.txt
k -n vehicle-track get events --field-selector type=Warning --sort-by='.lastTimestamp' > {WS}/q418/warnings.txt k -n vehicle-track get events --field-selector type=Warning --sort-by='.lastTimestamp' > {WS}/q85/warnings.txt
# --sort-by takes a JSONPath into the object and sorts ascending, so the worst offenders land at # --sort-by takes a JSONPath into the object and sorts ascending, so the worst offenders land at
# the BOTTOM of that first file. `logs -l` reads every pod matching the label, which is what you # the BOTTOM of that first file. `logs -l` reads every pod matching the label, which is what you
# want for a Deployment whose pod name keeps changing under a CrashLoopBackOff. # want for a Deployment whose pod name keeps changing under a CrashLoopBackOff.

238
tracks/cka/bank_sim_d.py
View File

@ -30,12 +30,12 @@ def task(**kw):
Q.append(kw) Q.append(kw)
# ------------------------------------------------------------------ 501 ---- # ------------------------------------------------------------------ 86 ----
task( task(
id=501, domain="ARCH", title="How the control plane is run", id=86, domain="ARCH", title="How the control plane is run",
task=""" task="""
The operations handbook has to say how each part of this cluster's control plane is run. Write The operations handbook has to say how each part of this cluster's control plane is run. Write
`{WS}/q501/controlplane.txt` with exactly these six lines, each value one of `process`, `{WS}/q86/controlplane.txt` with exactly these six lines, each value one of `process`,
`static-pod`, `pod`, `deployment` or `daemonset`: `static-pod`, `pod`, `deployment` or `daemonset`:
``` ```
@ -49,21 +49,21 @@ dns: <value> <name of that object>
`dns` is whatever serves cluster DNS, for example `dns: daemonset some-name`. `dns` is whatever serves cluster DNS, for example `dns: daemonset some-name`.
""", """,
setup=r"""rm -rf "$WS/q501"; mkdir -p "$WS/q501" """, setup=r"""rm -rf "$WS/q86"; mkdir -p "$WS/q86" """,
checks=[ checks=[
C("kubelet is a process", 1, r""" C("kubelet is a process", 1, r"""
f="$WS/q501/controlplane.txt"; [ -s "$f" ] || exit 1 f="$WS/q86/controlplane.txt"; [ -s "$f" ] || exit 1
want=process; [ "$(nx "$CP" 'systemctl is-active kubelet' </dev/null)" = active ] || want=unknown want=process; [ "$(nx "$CP" 'systemctl is-active kubelet' </dev/null)" = active ] || want=unknown
eq "$(sed -n 's/^ *kubelet *: *//p' "$f" | head -1 | tr -d '[:space:]' | tr A-Z a-z)" "$want" """), eq "$(sed -n 's/^ *kubelet *: *//p' "$f" | head -1 | tr -d '[:space:]' | tr A-Z a-z)" "$want" """),
C("apiserver, scheduler, controller-manager and etcd are static pods", 3, r""" C("apiserver, scheduler, controller-manager and etcd are static pods", 3, r"""
f="$WS/q501/controlplane.txt" f="$WS/q86/controlplane.txt"
# a kubeadm control plane: the manifests live in the kubelet's staticPodPath on the node # a kubeadm control plane: the manifests live in the kubelet's staticPodPath on the node
nx "$CP" 'ls /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null >/dev/null 2>&1 || exit 1 nx "$CP" 'ls /etc/kubernetes/manifests/kube-apiserver.yaml' </dev/null >/dev/null 2>&1 || exit 1
for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do for c in kube-apiserver kube-scheduler kube-controller-manager etcd; do
eq "$(sed -n "s/^ *$c *: *//p" "$f" | head -1 | tr -d '[:space:]' | tr A-Z a-z)" static-pod || exit 1 eq "$(sed -n "s/^ *$c *: *//p" "$f" | head -1 | tr -d '[:space:]' | tr A-Z a-z)" static-pod || exit 1
done"""), done"""),
C("dns names the right kind of object and its name", 2, r""" C("dns names the right kind of object and its name", 2, r"""
f="$WS/q501/controlplane.txt" f="$WS/q86/controlplane.txt"
sel=$(j -n kube-system svc kube-dns '{.spec.selector.k8s-app}') sel=$(j -n kube-system svc kube-dns '{.spec.selector.k8s-app}')
own=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].kind}') own=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].kind}')
name=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].name}') name=$(j -n kube-system pod -l k8s-app=$sel '{.items[0].metadata.ownerReferences[0].name}')
@ -79,7 +79,7 @@ exit
k -n kube-system get pod -o wide # static pods carry the node name as a suffix... k -n kube-system get pod -o wide # static pods carry the node name as a suffix...
k -n kube-system get pod kube-apiserver-{CP} -o jsonpath='{.metadata.ownerReferences[0].kind}{"\\n"}' # ...and are owned by the Node k -n kube-system get pod kube-apiserver-{CP} -o jsonpath='{.metadata.ownerReferences[0].kind}{"\\n"}' # ...and are owned by the Node
k -n kube-system get deploy,ds # coredns is a Deployment (kube-proxy, kindnet are DaemonSets) k -n kube-system get deploy,ds # coredns is a Deployment (kube-proxy, kindnet are DaemonSets)
cat > {WS}/q501/controlplane.txt <<'EOF' cat > {WS}/q86/controlplane.txt <<'EOF'
kubelet: process kubelet: process
kube-apiserver: static-pod kube-apiserver: static-pod
kube-scheduler: static-pod kube-scheduler: static-pod
@ -92,11 +92,11 @@ EOF
""", """,
) )
# ------------------------------------------------------------------ 502 ---- # ------------------------------------------------------------------ 87 ----
task( task(
id=502, domain="TSHOOT", title="Cluster facts to an exact file", id=87, domain="TSHOOT", title="Cluster facts to an exact file",
task=""" task="""
Write `{WS}/q502/cluster-info.txt` — answers only, exactly this format: Write `{WS}/q87/cluster-info.txt` — answers only, exactly this format:
``` ```
1: <number of control-plane nodes> 1: <number of control-plane nodes>
@ -106,22 +106,22 @@ Write `{WS}/q502/cluster-info.txt` — answers only, exactly this format:
5: <the suffix the names of static Pods running on node {W3} end with> 5: <the suffix the names of static Pods running on node {W3} end with>
``` ```
""", """,
setup=r"""rm -rf "$WS/q502"; mkdir -p "$WS/q502" """, setup=r"""rm -rf "$WS/q87"; mkdir -p "$WS/q87" """,
checks=[ checks=[
C("node counts (lines 1 and 2)", 2, r""" C("node counts (lines 1 and 2)", 2, r"""
f="$WS/q502/cluster-info.txt"; [ -s "$f" ] || exit 1 f="$WS/q87/cluster-info.txt"; [ -s "$f" ] || exit 1
cp=$($K get nodes -l node-role.kubernetes.io/control-plane --no-headers 2>/dev/null | wc -l); all=$($K get nodes --no-headers | wc -l) cp=$($K get nodes -l node-role.kubernetes.io/control-plane --no-headers 2>/dev/null | wc -l); all=$($K get nodes --no-headers | wc -l)
eq "$(sed -n 's/^ *1 *: *//p' "$f" | tr -d '[:space:]')" "$cp" && eq "$(sed -n 's/^ *2 *: *//p' "$f" | tr -d '[:space:]')" "$((all - cp))" """), eq "$(sed -n 's/^ *1 *: *//p' "$f" | tr -d '[:space:]')" "$cp" && eq "$(sed -n 's/^ *2 *: *//p' "$f" | tr -d '[:space:]')" "$((all - cp))" """),
C("Service CIDR (line 3)", 2, r""" C("Service CIDR (line 3)", 2, r"""
sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2) sc=$(j -n kube-system pod -l component=kube-apiserver '{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep -o 'service-cluster-ip-range=[^"]*' | cut -d= -f2)
[ -n "$sc" ] && eq "$(sed -n 's/^ *3 *: *//p' "$WS/q502/cluster-info.txt" | tr -d '[:space:]')" "$sc" """), [ -n "$sc" ] && eq "$(sed -n 's/^ *3 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]')" "$sc" """),
C("CNI plugin and its config file (line 4)", 2, r""" C("CNI plugin and its config file (line 4)", 2, r"""
l=$(sed -n 's/^ *4 *: *//p' "$WS/q502/cluster-info.txt" | tr -d '[:space:]') l=$(sed -n 's/^ *4 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]')
cni=$($K -n kube-system get ds -o name | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1) cni=$($K -n kube-system get ds -o name | sed 's|.*/||' | grep -Ei 'kindnet|calico|cilium|flannel|weave' | head -1)
cfg=$(nx "$W1" 'ls /etc/cni/net.d/*.conflist /etc/cni/net.d/*.conf 2>/dev/null | head -1' </dev/null) cfg=$(nx "$W1" 'ls /etc/cni/net.d/*.conflist /etc/cni/net.d/*.conf 2>/dev/null | head -1' </dev/null)
[ -n "$cni" ] && [ -n "$cfg" ] && echo "$l" | grep -qi "$cni" && echo "$l" | grep -qF "$cfg" """), [ -n "$cni" ] && [ -n "$cfg" ] && echo "$l" | grep -qi "$cni" && echo "$l" | grep -qF "$cfg" """),
C("static Pod name suffix on {W3} (line 5)", 1, r""" C("static Pod name suffix on {W3} (line 5)", 1, r"""
v=$(sed -n 's/^ *5 *: *//p' "$WS/q502/cluster-info.txt" | tr -d '[:space:]'); [ "$v" = "-$W3" ] || [ "$v" = "$W3" ]"""), v=$(sed -n 's/^ *5 *: *//p' "$WS/q87/cluster-info.txt" | tr -d '[:space:]'); [ "$v" = "-$W3" ] || [ "$v" = "$W3" ]"""),
], ],
solution=""" solution="""
k get nodes # ROLES shows control-plane; count the rest k get nodes # ROLES shows control-plane; count the rest
@ -130,7 +130,7 @@ k -n kube-system get ds # kindnet
{SHELL_W1} {SHELL_W1}
ls /etc/cni/net.d/ # 10-kindnet.conflist ls /etc/cni/net.d/ # 10-kindnet.conflist
exit exit
cat > {WS}/q502/cluster-info.txt <<'EOF' cat > {WS}/q87/cluster-info.txt <<'EOF'
1: 1 1: 1
2: 3 2: 3
3: 10.96.0.0/16 3: 10.96.0.0/16
@ -142,9 +142,9 @@ EOF
""", """,
) )
# ------------------------------------------------------------------ 503 ---- # ------------------------------------------------------------------ 88 ----
task( task(
id=503, domain="NET", title="Ready only when its dependency answers", id=88, domain="NET", title="Ready only when its dependency answers",
task=""" task="""
In Namespace `oslo-web`: In Namespace `oslo-web`:
@ -191,125 +191,125 @@ k -n oslo-web get pod gatekeeper -w # 1/1 after the next probe
""", """,
) )
# ------------------------------------------------------------------ 504 ---- # ------------------------------------------------------------------ 89 ----
task( task(
id=504, domain="WORK", title="Sorting done by kubectl", id=89, domain="WORK", title="Sorting done by kubectl",
task=""" task="""
Write two executable scripts that call `kubectl` (spelled out — aliases do not exist in a fresh Write two executable scripts that call `kubectl` (spelled out — aliases do not exist in a fresh
shell) and let **kubectl itself** do the sorting with `--sort-by`: shell) and let **kubectl itself** do the sorting with `--sort-by`:
1. `{WS}/q504/pods-by-age.sh` — every Pod in every Namespace, oldest first (by creation time). 1. `{WS}/q89/pods-by-age.sh` — every Pod in every Namespace, oldest first (by creation time).
2. `{WS}/q504/pods-by-uid.sh` — the Pods in Namespace `kube-system`, sorted by `metadata.uid`. 2. `{WS}/q89/pods-by-uid.sh` — the Pods in Namespace `kube-system`, sorted by `metadata.uid`.
""", """,
setup=r"""rm -rf "$WS/q504"; mkdir -p "$WS/q504" """, setup=r"""rm -rf "$WS/q89"; mkdir -p "$WS/q89" """,
checks=[ checks=[
C("pods-by-age.sh: all namespaces, --sort-by creationTimestamp, runs", 2, r""" C("pods-by-age.sh: all namespaces, --sort-by creationTimestamp, runs", 2, r"""
f="$WS/q504/pods-by-age.sh"; [ -x "$f" ] || exit 1 f="$WS/q89/pods-by-age.sh"; [ -x "$f" ] || exit 1
grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \ grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \
&& grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.creationTimestamp" "$f" || exit 1 && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.creationTimestamp" "$f" || exit 1
out=$(timeout 60 bash "$f" 2>/dev/null); [ "$(echo "$out" | wc -l)" -ge 5 ] && echo "$out" | grep -q kube-system"""), out=$(timeout 60 bash "$f" 2>/dev/null); [ "$(echo "$out" | wc -l)" -ge 5 ] && echo "$out" | grep -q kube-system"""),
C("pods-by-uid.sh: kube-system, --sort-by metadata.uid, runs", 2, r""" C("pods-by-uid.sh: kube-system, --sort-by metadata.uid, runs", 2, r"""
f="$WS/q504/pods-by-uid.sh"; [ -x "$f" ] || exit 1 f="$WS/q89/pods-by-uid.sh"; [ -x "$f" ] || exit 1
grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-n|--namespace)[= ]*kube-system' "$f" \ grep -Eq 'kubectl\b.*\bget\b.*\b(po|pods?)\b' "$f" && grep -Eq -- '(-n|--namespace)[= ]*kube-system' "$f" \
&& grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.uid" "$f" || exit 1 && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?metadata\.uid" "$f" || exit 1
timeout 60 bash "$f" 2>/dev/null | grep -q etcd"""), timeout 60 bash "$f" 2>/dev/null | grep -q etcd"""),
C("no aliases in either script", 1, r"""[ -s "$WS/q504/pods-by-age.sh" ] && [ -s "$WS/q504/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q504/pods-by-age.sh" "$WS/q504/pods-by-uid.sh" """), C("no aliases in either script", 1, r"""[ -s "$WS/q89/pods-by-age.sh" ] && [ -s "$WS/q89/pods-by-uid.sh" ] && ! grep -Eq '(^|[;|&(] *| )k +' "$WS/q89/pods-by-age.sh" "$WS/q89/pods-by-uid.sh" """),
], ],
solution=""" solution="""
cat > {WS}/q504/pods-by-age.sh <<'EOF' cat > {WS}/q89/pods-by-age.sh <<'EOF'
kubectl get pods -A --sort-by=.metadata.creationTimestamp kubectl get pods -A --sort-by=.metadata.creationTimestamp
EOF EOF
cat > {WS}/q504/pods-by-uid.sh <<'EOF' cat > {WS}/q89/pods-by-uid.sh <<'EOF'
kubectl -n kube-system get pods --sort-by=.metadata.uid kubectl -n kube-system get pods --sort-by=.metadata.uid
EOF EOF
chmod +x {WS}/q504/*.sh chmod +x {WS}/q89/*.sh
bash {WS}/q504/pods-by-age.sh | head -3 # the control-plane pods come first - they are oldest bash {WS}/q89/pods-by-age.sh | head -3 # the control-plane pods come first - they are oldest
# --sort-by takes a JSONPath into each item; ascending only, so "newest first" is `| tac` or # --sort-by takes a JSONPath into each item; ascending only, so "newest first" is `| tac` or
# sorting by a field that grows the other way. # sorting by a field that grows the other way.
""", """,
) )
# ------------------------------------------------------------------ 505 ---- # ------------------------------------------------------------------ 90 ----
task( task(
id=505, domain="TSHOOT", title="Event forensics around kube-proxy", breaks="w2-kube-proxy", id=90, domain="TSHOOT", title="Event forensics around kube-proxy", breaks="w2-kube-proxy",
task=""" task="""
1. Write `{WS}/q505/cluster-events.sh`: a script that prints the latest events of the **whole** 1. Write `{WS}/q90/cluster-events.sh`: a script that prints the latest events of the **whole**
cluster (all Namespaces) sorted by creation time, using `kubectl`. Make it executable. cluster (all Namespaces) sorted by creation time, using `kubectl`. Make it executable.
2. Delete the kube-proxy Pod that runs on node `{W2}`. Write the events this caused to 2. Delete the kube-proxy Pod that runs on node `{W2}`. Write the events this caused to
`{WS}/q505/pod-deleted.log`. `{WS}/q90/pod-deleted.log`.
3. On node `{W2}`, stop **and remove** the kube-proxy **container** with `crictl` — leave the Pod 3. On node `{W2}`, stop **and remove** the kube-proxy **container** with `crictl` — leave the Pod
object alone. Write the events this caused to `{WS}/q505/container-killed.log`. object alone. Write the events this caused to `{WS}/q90/container-killed.log`.
4. Which of the two actions made the Pod's `RESTARTS` counter go up? Write `pod` or `container` to 4. Which of the two actions made the Pod's `RESTARTS` counter go up? Write `pod` or `container` to
`{WS}/q505/restarts.txt`. `{WS}/q90/restarts.txt`.
Do step 2 before step 3. Do step 2 before step 3.
""", """,
setup=r""" setup=r"""
rm -rf "$WS/q505"; mkdir -p "$WS/q505" rm -rf "$WS/q90"; mkdir -p "$WS/q90"
$K -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 -o jsonpath='{.items[0].metadata.uid}' > "$WS/.q505-uid" $K -n kube-system get pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 -o jsonpath='{.items[0].metadata.uid}' > "$WS/.q90-uid"
""", """,
checks=[ checks=[
C("cluster-events.sh: all namespaces, sorted by time, runs", 2, r""" C("cluster-events.sh: all namespaces, sorted by time, runs", 2, r"""
f="$WS/q505/cluster-events.sh"; [ -x "$f" ] || exit 1 f="$WS/q90/cluster-events.sh"; [ -x "$f" ] || exit 1
grep -Eq 'kubectl\b.*\bget\b.*\b(ev|events?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \ grep -Eq 'kubectl\b.*\bget\b.*\b(ev|events?)\b' "$f" && grep -Eq -- '(-A|--all-namespaces)' "$f" \
&& grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?(metadata\.creationTimestamp|lastTimestamp|eventTime|firstTimestamp)" "$f" || exit 1 && grep -Eq -- "--sort-by[= ]*['\"]?\{?\.?(metadata\.creationTimestamp|lastTimestamp|eventTime|firstTimestamp)" "$f" || exit 1
timeout 60 bash "$f" >/dev/null 2>&1"""), timeout 60 bash "$f" >/dev/null 2>&1"""),
C("the kube-proxy Pod on {W2} was deleted and replaced", 2, r""" C("the kube-proxy Pod on {W2} was deleted and replaced", 2, r"""
old=$(cat "$WS/.q505-uid" 2>/dev/null); [ -n "$old" ] || exit 1 old=$(cat "$WS/.q90-uid" 2>/dev/null); [ -n "$old" ] || exit 1
now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}') now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}')
[ -n "$now" ] && [ "$now" != "$old" ] && eq "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.phase}')" Running"""), [ -n "$now" ] && [ "$now" != "$old" ] && eq "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.phase}')" Running"""),
# counted on the REPLACEMENT pod, which starts at 0 - after a node reboot the original pod # counted on the REPLACEMENT pod, which starts at 0 - after a node reboot the original pod
# often already shows a restart, which would otherwise hand these points out for free # often already shows a restart, which would otherwise hand these points out for free
C("then its container was killed (restart counter >= 1)", 2, r""" C("then its container was killed (restart counter >= 1)", 2, r"""
old=$(cat "$WS/.q505-uid" 2>/dev/null); now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}') old=$(cat "$WS/.q90-uid" 2>/dev/null); now=$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].metadata.uid}')
[ -n "$old" ] && [ -n "$now" ] && [ "$now" != "$old" ] || exit 1 [ -n "$old" ] && [ -n "$now" ] && [ "$now" != "$old" ] || exit 1
[ "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null"""), [ "$(j -n kube-system pod -l k8s-app=kube-proxy --field-selector spec.nodeName=$W2 '{.items[0].status.containerStatuses[0].restartCount}')" -ge 1 ] 2>/dev/null"""),
C("pod-deleted.log and container-killed.log hold the events", 2, r""" C("pod-deleted.log and container-killed.log hold the events", 2, r"""
for f in pod-deleted.log container-killed.log; do for f in pod-deleted.log container-killed.log; do
grep -q 'kube-proxy' "$WS/q505/$f" 2>/dev/null && grep -Eq 'Killing|SuccessfulCreate|SuccessfulDelete|Scheduled|Pulled|Created|Started|BackOff' "$WS/q505/$f" || exit 1 grep -q 'kube-proxy' "$WS/q90/$f" 2>/dev/null && grep -Eq 'Killing|SuccessfulCreate|SuccessfulDelete|Scheduled|Pulled|Created|Started|BackOff' "$WS/q90/$f" || exit 1
done"""), done"""),
C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q505/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""), C("restarts.txt says container", 2, r"""eq "$(tr -d '[:space:]' < "$WS/q90/restarts.txt" 2>/dev/null | tr A-Z a-z)" container"""),
], ],
solution=""" solution="""
cat > {WS}/q505/cluster-events.sh <<'EOF' cat > {WS}/q90/cluster-events.sh <<'EOF'
kubectl get events -A --sort-by=.metadata.creationTimestamp kubectl get events -A --sort-by=.metadata.creationTimestamp
EOF EOF
chmod +x {WS}/q505/cluster-events.sh chmod +x {WS}/q90/cluster-events.sh
k -n kube-system get pod -l k8s-app=kube-proxy -o wide # the one on {W2} k -n kube-system get pod -l k8s-app=kube-proxy -o wide # the one on {W2}
k -n kube-system delete pod <kube-proxy-on-{W2}> k -n kube-system delete pod <kube-proxy-on-{W2}>
k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q505/pod-deleted.log k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -8 > {WS}/q90/pod-deleted.log
# Killing (old pod), SuccessfulCreate (DaemonSet), Scheduled, Pulled, Created, Started (new pod) # Killing (old pod), SuccessfulCreate (DaemonSet), Scheduled, Pulled, Created, Started (new pod)
{SHELL_W2} {SHELL_W2}
crictl ps --name kube-proxy # container id crictl ps --name kube-proxy # container id
crictl stop <id> && crictl rm <id> crictl stop <id> && crictl rm <id>
exit exit
k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q505/container-killed.log k -n kube-system get events --sort-by=.metadata.creationTimestamp | grep kube-proxy | tail -4 > {WS}/q90/container-killed.log
# Pulled, Created, Started - same Pod, new container # Pulled, Created, Started - same Pod, new container
k -n kube-system get pod -l k8s-app=kube-proxy -o wide # RESTARTS 1 on {W2} k -n kube-system get pod -l k8s-app=kube-proxy -o wide # RESTARTS 1 on {W2}
echo container > {WS}/q505/restarts.txt echo container > {WS}/q90/restarts.txt
# Deleting the Pod makes the DaemonSet create a brand-new Pod (RESTARTS 0). Killing only the # Deleting the Pod makes the DaemonSet create a brand-new Pod (RESTARTS 0). Killing only the
# container makes the kubelet start a new container inside the SAME Pod - that is a restart. # container makes the kubelet start a new container inside the SAME Pod - that is a restart.
""", """,
) )
# ------------------------------------------------------------------ 506 ---- # ------------------------------------------------------------------ 91 ----
def _census(r): def _census(r):
n = r.sample(range(2, 10), 4) n = r.sample(range(2, 10), 4)
return {"ctx": "k8s-c1-H", "north": n[0], "south": n[1], "east": n[2], "west": n[3]} return {"ctx": "k8s-c1-H", "north": n[0], "south": n[1], "east": n[2], "west": n[3]}
task( task(
id=506, domain="ARCH", title="API resources and a Role census", params=_census, id=91, domain="ARCH", title="API resources and a Role census", params=_census,
task=""" task="""
1. Write the names of every **namespaced** resource type this API server offers to 1. Write the names of every **namespaced** resource type this API server offers to
`{WS}/q506/namespaced.txt`, one per line — plural names, as `kubectl api-resources` prints them. `{WS}/q91/namespaced.txt`, one per line — plural names, as `kubectl api-resources` prints them.
2. Of the Namespaces `census-north`, `census-south`, `census-east` and `census-west`, which one holds 2. Of the Namespaces `census-north`, `census-south`, `census-east` and `census-west`, which one holds
the most Roles? Write `<namespace> <number of Roles>` to `{WS}/q506/most-roles.txt`. the most Roles? Write `<namespace> <number of Roles>` to `{WS}/q91/most-roles.txt`.
""", """,
setup=NS + r""" setup=NS + r"""
rm -rf "$WS/q506"; mkdir -p "$WS/q506" rm -rf "$WS/q91"; mkdir -p "$WS/q91"
for pair in north:%{north} south:%{south} east:%{east} west:%{west}; do for pair in north:%{north} south:%{south} east:%{east} west:%{west}; do
ns=census-${pair%%:*}; n=${pair##*:} ns=census-${pair%%:*}; n=${pair##*:}
mkns $ns mkns $ns
@ -319,37 +319,37 @@ done
""", """,
checks=[ checks=[
C("namespaced.txt lists exactly the namespaced resource types", 3, r""" C("namespaced.txt lists exactly the namespaced resource types", 3, r"""
f="$WS/q506/namespaced.txt"; [ -s "$f" ] || exit 1 f="$WS/q91/namespaced.txt"; [ -s "$f" ] || exit 1
want=$($K api-resources --namespaced=true -o name 2>/dev/null | sed 's/\..*//' | sort -u) want=$($K api-resources --namespaced=true -o name 2>/dev/null | sed 's/\..*//' | sort -u)
got=$(awk '{print $1}' "$f" | sed 's/\..*//' | grep . | sort -u) got=$(awk '{print $1}' "$f" | sed 's/\..*//' | grep . | sort -u)
[ -n "$want" ] && [ "$got" = "$want" ]"""), [ -n "$want" ] && [ "$got" = "$want" ]"""),
C("most-roles.txt names the namespace and its Role count", 3, r""" C("most-roles.txt names the namespace and its Role count", 3, r"""
best=$(for ns in census-north census-south census-east census-west; do echo "$ns $($K -n $ns get role --no-headers 2>/dev/null | wc -l)"; done | sort -k2 -n | tail -1) best=$(for ns in census-north census-south census-east census-west; do echo "$ns $($K -n $ns get role --no-headers 2>/dev/null | wc -l)"; done | sort -k2 -n | tail -1)
eq "$(tr -s ' ' < "$WS/q506/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """), eq "$(tr -s ' ' < "$WS/q91/most-roles.txt" 2>/dev/null | sed 's/^ *//; s/ *$//')" "$best" """),
], ],
solution=""" solution="""
k api-resources --namespaced=true -o name > /tmp/r.txt # deployments.apps, ... (group suffix) k api-resources --namespaced=true -o name > /tmp/r.txt # deployments.apps, ... (group suffix)
k api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q506/namespaced.txt k api-resources --namespaced=true --no-headers | awk '{print $1}' > {WS}/q91/namespaced.txt
for ns in census-north census-south census-east census-west; do echo "$ns $(k -n $ns get role --no-headers | wc -l)"; done for ns in census-north census-south census-east census-west; do echo "$ns $(k -n $ns get role --no-headers | wc -l)"; done
# pick the largest, e.g. echo "census-east 8" > {WS}/q506/most-roles.txt # pick the largest, e.g. echo "census-east 8" > {WS}/q91/most-roles.txt
# --namespaced=false gives the cluster-scoped half (nodes, pv, storageclasses, clusterroles...). # --namespaced=false gives the cluster-scoped half (nodes, pv, storageclasses, clusterroles...).
# `k get role -A --no-headers | awk '{print $1}' | sort | uniq -c | sort -n` counts every # `k get role -A --no-headers | awk '{print $1}' | sort | uniq -c | sort -n` counts every
# namespace at once. # namespace at once.
""", """,
) )
# ------------------------------------------------------------------ 507 ---- # ------------------------------------------------------------------ 92 ----
task( task(
id=507, domain="TSHOOT", title="Kubelet unit points at nothing", breaks="w1-kubelet", disruptive=True, id=92, domain="TSHOOT", title="Kubelet unit points at nothing", breaks="w1-kubelet", disruptive=True,
task=""" task="""
Node `{W1}` went `NotReady` after someone "tidied up" its kubelet installation. Bring it back to Node `{W1}` went `NotReady` after someone "tidied up" its kubelet installation. Bring it back to
`Ready`; the kubelet has to run as the systemd service it always was. Then write the full path of the `Ready`; the kubelet has to run as the systemd service it always was. Then write the full path of the
file you had to fix to `{WS}/q507/fixed-file.txt`. file you had to fix to `{WS}/q92/fixed-file.txt`.
Reach the node with `{SHELL_W1}`. Pods on a node that stays NotReady are evicted after five minutes. Reach the node with `{SHELL_W1}`. Pods on a node that stays NotReady are evicted after five minutes.
""", """,
setup=r""" setup=r"""
rm -rf "$WS/q507"; mkdir -p "$WS/q507" rm -rf "$WS/q92"; mkdir -p "$WS/q92"
nx "$W1" ' nx "$W1" '
d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf d=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
grep -q "^ExecStart=/usr/bin/kubelet " $d && sed -i "s|^ExecStart=/usr/bin/kubelet |ExecStart=/usr/local/bin/kubelet |" $d grep -q "^ExecStart=/usr/bin/kubelet " $d && sed -i "s|^ExecStart=/usr/bin/kubelet |ExecStart=/usr/local/bin/kubelet |" $d
@ -360,7 +360,7 @@ exit 0
C("{W1} is Ready", 4, r"""for i in $(seq 1 15); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 4; done; exit 1"""), C("{W1} is Ready", 4, r"""for i in $(seq 1 15); do [ "$(j node "$W1" '{.status.conditions[?(@.type=="Ready")].status}')" = True ] && exit 0; sleep 4; done; exit 1"""),
C("kubelet active, and its unit runs a kubelet binary that exists", 2, r""" C("kubelet active, and its unit runs a kubelet binary that exists", 2, r"""
nx "$W1" 'systemctl is-active --quiet kubelet && b=$(sed -n "s|^ExecStart=\([^ ]*kubelet\) .*|\1|p" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | tail -1) && [ -x "$b" ]' </dev/null"""), nx "$W1" 'systemctl is-active --quiet kubelet && b=$(sed -n "s|^ExecStart=\([^ ]*kubelet\) .*|\1|p" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | tail -1) && [ -x "$b" ]' </dev/null"""),
C("fixed-file.txt names the drop-in", 2, r"""grep -q '/etc/systemd/system/kubelet.service.d/10-kubeadm.conf' "$WS/q507/fixed-file.txt" 2>/dev/null"""), C("fixed-file.txt names the drop-in", 2, r"""grep -q '/etc/systemd/system/kubelet.service.d/10-kubeadm.conf' "$WS/q92/fixed-file.txt" 2>/dev/null"""),
], ],
solution=""" solution="""
k get nodes # {W1} NotReady k get nodes # {W1} NotReady
@ -375,21 +375,21 @@ systemctl daemon-reload # systemd caches unit files - with
systemctl restart kubelet systemctl restart kubelet
systemctl status kubelet # active (running) systemctl status kubelet # active (running)
exit exit
echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q507/fixed-file.txt echo /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > {WS}/q92/fixed-file.txt
k get nodes -w k get nodes -w
# status=203/EXEC means systemd could not exec the binary at all - a path problem, not a kubelet # status=203/EXEC means systemd could not exec the binary at all - a path problem, not a kubelet
# configuration problem. `ExecStart=` (empty) in a drop-in clears the unit's own ExecStart first. # configuration problem. `ExecStart=` (empty) in a drop-in clears the unit's own ExecStart first.
""", """,
) )
# ------------------------------------------------------------------ 508 ---- # ------------------------------------------------------------------ 93 ----
task( task(
id=508, domain="WORK", title="Secrets as a read-only volume and renamed env vars", id=93, domain="WORK", title="Secrets as a read-only volume and renamed env vars",
params=lambda r: {"ctx": "k8s-c1-H", "pw": "ledger-" + "".join(r.choice("abcdefghjkmnpqrstuvwxyz23456789") for _ in range(10))}, params=lambda r: {"ctx": "k8s-c1-H", "pw": "ledger-" + "".join(r.choice("abcdefghjkmnpqrstuvwxyz23456789") for _ in range(10))},
task=""" task="""
In Namespace `oslo-secure`: In Namespace `oslo-secure`:
1. Create Secret `bundle` from the file `{WS}/q508/tls-bundle.pem`; the key must be the file's name. 1. Create Secret `bundle` from the file `{WS}/q93/tls-bundle.pem`; the key must be the file's name.
2. Create Secret `db-login` with `user=svc-ledger` and `pass=%{pw}`. 2. Create Secret `db-login` with `user=svc-ledger` and `pass=%{pw}`.
3. Create Pod `vault-reader` (image `busybox:1.36`, command `sleep 86400`) that 3. Create Pod `vault-reader` (image `busybox:1.36`, command `sleep 86400`) that
- mounts Secret `bundle` **read-only** at `/etc/bundle`, - mounts Secret `bundle` **read-only** at `/etc/bundle`,
@ -397,14 +397,14 @@ In Namespace `oslo-secure`:
""", """,
setup=NS + r""" setup=NS + r"""
mkns oslo-secure mkns oslo-secure
rm -rf "$WS/q508"; mkdir -p "$WS/q508" rm -rf "$WS/q93"; mkdir -p "$WS/q93"
openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=ledger.oslo.internal" -keyout /dev/null -out "$WS/q508/tls-bundle.pem" >/dev/null 2>&1 openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj "/CN=ledger.oslo.internal" -keyout /dev/null -out "$WS/q93/tls-bundle.pem" >/dev/null 2>&1
$K -n oslo-secure delete pod vault-reader --ignore-not-found --force --grace-period=0 >/dev/null 2>&1 $K -n oslo-secure delete pod vault-reader --ignore-not-found --force --grace-period=0 >/dev/null 2>&1
$K -n oslo-secure delete secret bundle db-login --ignore-not-found >/dev/null $K -n oslo-secure delete secret bundle db-login --ignore-not-found >/dev/null
""", """,
checks=[ checks=[
C("Secrets bundle (key tls-bundle.pem, the file's content) and db-login", 2, r""" C("Secrets bundle (key tls-bundle.pem, the file's content) and db-login", 2, r"""
[ "$(j -n oslo-secure secret bundle '{.data.tls-bundle\.pem}' | base64 -d 2>/dev/null)" = "$(cat "$WS/q508/tls-bundle.pem")" ] \ [ "$(j -n oslo-secure secret bundle '{.data.tls-bundle\.pem}' | base64 -d 2>/dev/null)" = "$(cat "$WS/q93/tls-bundle.pem")" ] \
&& eq "$(j -n oslo-secure secret db-login '{.data.user}' | base64 -d)" svc-ledger && eq "$(j -n oslo-secure secret db-login '{.data.pass}' | base64 -d)" '%{pw}'"""), && eq "$(j -n oslo-secure secret db-login '{.data.user}' | base64 -d)" svc-ledger && eq "$(j -n oslo-secure secret db-login '{.data.pass}' | base64 -d)" '%{pw}'"""),
C("vault-reader Running with bundle mounted read-only at /etc/bundle", 3, r""" C("vault-reader Running with bundle mounted read-only at /etc/bundle", 3, r"""
p="$K -n oslo-secure get pod vault-reader" p="$K -n oslo-secure get pod vault-reader"
@ -420,7 +420,7 @@ env=$($K -n oslo-secure exec vault-reader -- env 2>/dev/null)
echo "$env" | grep -qx 'APP_USER=svc-ledger' && echo "$env" | grep -qx 'APP_PASS=%{pw}'"""), echo "$env" | grep -qx 'APP_USER=svc-ledger' && echo "$env" | grep -qx 'APP_PASS=%{pw}'"""),
], ],
solution=""" solution="""
k -n oslo-secure create secret generic bundle --from-file={WS}/q508/tls-bundle.pem # key = file name k -n oslo-secure create secret generic bundle --from-file={WS}/q93/tls-bundle.pem # key = file name
k -n oslo-secure create secret generic db-login --from-literal=user=svc-ledger --from-literal=pass='%{pw}' k -n oslo-secure create secret generic db-login --from-literal=user=svc-ledger --from-literal=pass='%{pw}'
k -n oslo-secure run vault-reader --image=busybox:1.36 $do -- sleep 86400 > vr.yaml k -n oslo-secure run vault-reader --image=busybox:1.36 $do -- sleep 86400 > vr.yaml
# in vr.yaml, pod level: # in vr.yaml, pod level:
@ -440,18 +440,18 @@ k -n oslo-secure exec vault-reader -- ls /etc/bundle
""", """,
) )
# ------------------------------------------------------------------ 509 ---- # ------------------------------------------------------------------ 94 ----
task( task(
id=509, domain="ARCH", title="Static Pod on the control plane behind a Service", id=94, domain="ARCH", title="Static Pod on the control plane behind a Service",
task=""" task="""
1. On the control-plane node (`{SHELL_CP}`) create a **static Pod** `ops-beacon` in Namespace 1. On the control-plane node (`{SHELL_CP}`) create a **static Pod** `ops-beacon` in Namespace
`default`: image `nginx:1.27`, requests `cpu: 10m` and `memory: 20Mi`, label `app=ops-beacon`. `default`: image `nginx:1.27`, requests `cpu: 10m` and `memory: 20Mi`, label `app=ops-beacon`.
2. Expose it with a Service `ops-beacon-svc` of type **NodePort** (port 80 → 80) whose endpoints 2. Expose it with a Service `ops-beacon-svc` of type **NodePort** (port 80 → 80) whose endpoints
contain that static Pod. contain that static Pod.
3. Write the node port the Service was given to `{WS}/q509/nodeport.txt`. 3. Write the node port the Service was given to `{WS}/q94/nodeport.txt`.
""", """,
setup=r""" setup=r"""
rm -rf "$WS/q509"; mkdir -p "$WS/q509" rm -rf "$WS/q94"; mkdir -p "$WS/q94"
nx "$CP" 'rm -f /etc/kubernetes/manifests/ops-beacon*.yaml' </dev/null nx "$CP" 'rm -f /etc/kubernetes/manifests/ops-beacon*.yaml' </dev/null
$K -n default delete svc ops-beacon-svc --ignore-not-found >/dev/null $K -n default delete svc ops-beacon-svc --ignore-not-found >/dev/null
# the kubelet drops the mirror pod on its next sync of the manifest directory # the kubelet drops the mirror pod on its next sync of the manifest directory
@ -469,7 +469,7 @@ s="$K -n default get svc ops-beacon-svc"
eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" NodePort && eq "$($s -o jsonpath='{.spec.ports[0].port}')" 80 && eq "$($s -o jsonpath='{.spec.ports[0].targetPort}')" 80 || exit 1 eq "$($s -o jsonpath='{.spec.type}' 2>/dev/null)" NodePort && eq "$($s -o jsonpath='{.spec.ports[0].port}')" 80 && eq "$($s -o jsonpath='{.spec.ports[0].targetPort}')" 80 || exit 1
ip=$(j -n default pod "ops-beacon-$CP" '{.status.podIP}'); [ -n "$ip" ] || exit 1 ip=$(j -n default pod "ops-beacon-$CP" '{.status.podIP}'); [ -n "$ip" ] || exit 1
j -n default endpointslices -l kubernetes.io/service-name=ops-beacon-svc '{.items[*].endpoints[*].addresses[*]}' | tr ' ' '\n' | grep -qx "$ip" """), j -n default endpointslices -l kubernetes.io/service-name=ops-beacon-svc '{.items[*].endpoints[*].addresses[*]}' | tr ' ' '\n' | grep -qx "$ip" """),
C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q509/nodeport.txt" 2>/dev/null)" "$np" """), C("nodeport.txt holds the Service's node port", 1, r"""np=$(j -n default svc ops-beacon-svc '{.spec.ports[0].nodePort}'); [ -n "$np" ] && eq "$(tr -d '[:space:]' < "$WS/q94/nodeport.txt" 2>/dev/null)" "$np" """),
], ],
solution=""" solution="""
k run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon $do > beacon.yaml k run ops-beacon -n default --image=nginx:1.27 --labels=app=ops-beacon $do > beacon.yaml
@ -481,19 +481,19 @@ exit
k -n default get pod ops-beacon-{CP} -o wide # mirror pod, named <pod>-<node> k -n default get pod ops-beacon-{CP} -o wide # mirror pod, named <pod>-<node>
k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80 k -n default expose pod ops-beacon-{CP} --name=ops-beacon-svc --type=NodePort --port=80 --target-port=80
k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc # the mirror pod's IP k -n default get endpointslices -l kubernetes.io/service-name=ops-beacon-svc # the mirror pod's IP
k -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q509/nodeport.txt k -n default get svc ops-beacon-svc -o jsonpath='{.spec.ports[0].nodePort}' > {WS}/q94/nodeport.txt
# The control-plane taint only keeps the SCHEDULER from placing pods there; the kubelet runs its # The control-plane taint only keeps the SCHEDULER from placing pods there; the kubelet runs its
# static pods regardless. `expose pod` copies the pod's labels into the selector. # static pods regardless. `expose pod` copies the pod's labels into the selector.
""", """,
) )
# ------------------------------------------------------------------ 510 ---- # ------------------------------------------------------------------ 95 ----
task( task(
id=510, domain="ARCH", title="The kubelet's two certificates", id=95, domain="ARCH", title="The kubelet's two certificates",
task=""" task="""
On node `{W2}` find the certificate the kubelet uses as a **client** (to talk to the API server) On node `{W2}` find the certificate the kubelet uses as a **client** (to talk to the API server)
and the one it **serves** its own API with (port 10250). Write exactly this to and the one it **serves** its own API with (port 10250). Write exactly this to
`{WS}/q510/kubelet-certs.txt`: `{WS}/q95/kubelet-certs.txt`:
``` ```
client-issuer: <issuer, as openssl prints it> client-issuer: <issuer, as openssl prints it>
@ -504,15 +504,15 @@ server-eku: <the extended key usage>
Reach the node with `{SHELL_W2}`. Reach the node with `{SHELL_W2}`.
""", """,
setup=r"""rm -rf "$WS/q510"; mkdir -p "$WS/q510" """, setup=r"""rm -rf "$WS/q95"; mkdir -p "$WS/q95" """,
checks=[ checks=[
C("client certificate: issuer and extended key usage", 3, r""" C("client certificate: issuer and extended key usage", 3, r"""
f="$WS/q510/kubelet-certs.txt"; [ -s "$f" ] || exit 1 f="$WS/q95/kubelet-certs.txt"; [ -s "$f" ] || exit 1
iss=$(nx "$W2" 'openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -issuer' </dev/null | sed 's/^issuer= *//' | tr -d ' ' | tr A-Z a-z) iss=$(nx "$W2" 'openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -issuer' </dev/null | sed 's/^issuer= *//' | tr -d ' ' | tr A-Z a-z)
[ -n "$iss" ] && sed -n 's/^ *client-issuer *: *//p' "$f" | tr -d ' ' | tr A-Z a-z | grep -qF "$iss" \ [ -n "$iss" ] && sed -n 's/^ *client-issuer *: *//p' "$f" | tr -d ' ' | tr A-Z a-z | grep -qF "$iss" \
&& sed -n 's/^ *client-eku *: *//p' "$f" | grep -qi 'client auth'"""), && sed -n 's/^ *client-eku *: *//p' "$f" | grep -qi 'client auth'"""),
C("serving certificate: issuer and extended key usage", 3, r""" C("serving certificate: issuer and extended key usage", 3, r"""
f="$WS/q510/kubelet-certs.txt" f="$WS/q95/kubelet-certs.txt"
iss=$(nx "$W2" 'openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -issuer' </dev/null | sed 's/^issuer= *//' | tr -d ' ' | tr A-Z a-z) iss=$(nx "$W2" 'openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -issuer' </dev/null | sed 's/^issuer= *//' | tr -d ' ' | tr A-Z a-z)
[ -n "$iss" ] && sed -n 's/^ *server-issuer *: *//p' "$f" | tr -d ' ' | tr A-Z a-z | grep -qF "$iss" \ [ -n "$iss" ] && sed -n 's/^ *server-issuer *: *//p' "$f" | tr -d ' ' | tr A-Z a-z | grep -qF "$iss" \
&& sed -n 's/^ *server-eku *: *//p' "$f" | grep -qi 'server auth'"""), && sed -n 's/^ *server-eku *: *//p' "$f" | grep -qi 'server auth'"""),
@ -537,9 +537,9 @@ exit
""", """,
) )
# ------------------------------------------------------------------ 511 ---- # ------------------------------------------------------------------ 96 ----
task( task(
id=511, domain="WORK", title="A Pod for the control plane only", id=96, domain="WORK", title="A Pod for the control plane only",
task=""" task="""
Create Pod `cp-watch` in Namespace `default`: image `httpd:2-alpine`, container name `watch`. Create Pod `cp-watch` in Namespace `default`: image `httpd:2-alpine`, container name `watch`.
@ -578,9 +578,9 @@ k -n default get pod cp-watch -o wide
""", """,
) )
# ------------------------------------------------------------------ 512 ---- # ------------------------------------------------------------------ 97 ----
task( task(
id=512, domain="TSHOOT", title="Stop the scheduler, schedule by hand", breaks="scheduler", id=97, domain="TSHOOT", title="Stop the scheduler, schedule by hand", breaks="scheduler",
task=""" task="""
1. Stop the kube-scheduler **temporarily** — it runs as a static Pod on the control-plane node — and 1. Stop the kube-scheduler **temporarily** — it runs as a static Pod on the control-plane node — and
confirm it is gone. confirm it is gone.
@ -634,7 +634,7 @@ k -n default get pod manual-2 -o wide # a worke
""", """,
) )
# ------------------------------------------------------------------ 513 ---- # ------------------------------------------------------------------ 98 ----
_CHART = r''' _CHART = r'''
mkchart() { # dir name version image replicas mkchart() { # dir name version image replicas
mkdir -p "$1/templates" mkdir -p "$1/templates"
@ -656,30 +656,30 @@ YAML
''' '''
task( task(
id=513, domain="ARCH", title="Helm release housekeeping", id=98, domain="ARCH", title="Helm release housekeeping",
task=""" task="""
Namespace `oslo-helm` holds several Helm releases; local charts are under `{WS}/q513/charts`. Namespace `oslo-helm` holds several Helm releases; local charts are under `{WS}/q98/charts`.
1. Delete the release `web-legacy`. 1. Delete the release `web-legacy`.
2. Upgrade the release `web-main` to chart version **1.1.0** (`{WS}/q513/charts/nginx-lite-1.1.0`). 2. Upgrade the release `web-main` to chart version **1.1.0** (`{WS}/q98/charts/nginx-lite-1.1.0`).
3. Install a new release `cache` from `{WS}/q513/charts/redis-lite` with **2** replicas (the chart's 3. Install a new release `cache` from `{WS}/q98/charts/redis-lite` with **2** replicas (the chart's
`replicaCount` value). `replicaCount` value).
4. One release in the Namespace never finished installing. Find it and remove it. 4. One release in the Namespace never finished installing. Find it and remove it.
""", """,
setup=NS + _CHART + r""" setup=NS + _CHART + r"""
mkns oslo-helm mkns oslo-helm
C="$WS/q513/charts"; rm -rf "$WS/q513"; mkdir -p "$C" C="$WS/q98/charts"; rm -rf "$WS/q98"; mkdir -p "$C"
mkchart "$C/nginx-lite-1.0.0" nginx-lite 1.0.0 nginx:1.25 1 mkchart "$C/nginx-lite-1.0.0" nginx-lite 1.0.0 nginx:1.25 1
mkchart "$C/nginx-lite-1.1.0" nginx-lite 1.1.0 nginx:1.27 1 mkchart "$C/nginx-lite-1.1.0" nginx-lite 1.1.0 nginx:1.27 1
mkchart "$C/redis-lite" redis-lite 0.3.0 redis:7-alpine 1 mkchart "$C/redis-lite" redis-lite 0.3.0 redis:7-alpine 1
mkchart /tmp/q513-stuck cache-edge 0.1.0 busybox:0.0-does-not-exist 1 mkchart /tmp/q98-stuck cache-edge 0.1.0 busybox:0.0-does-not-exist 1
for r in $(helm -n oslo-helm list -q 2>/dev/null); do helm -n oslo-helm uninstall "$r" >/dev/null 2>&1; done for r in $(helm -n oslo-helm list -q 2>/dev/null); do helm -n oslo-helm uninstall "$r" >/dev/null 2>&1; done
helm -n oslo-helm install web-legacy "$C/nginx-lite-1.0.0" >/dev/null helm -n oslo-helm install web-legacy "$C/nginx-lite-1.0.0" >/dev/null
helm -n oslo-helm install web-main "$C/nginx-lite-1.0.0" >/dev/null helm -n oslo-helm install web-main "$C/nginx-lite-1.0.0" >/dev/null
# helm records the release as pending-install BEFORE it starts waiting; kill it mid-wait and it # helm records the release as pending-install BEFORE it starts waiting; kill it mid-wait and it
# stays that way - the classic result of an interrupted install or a CI job that timed out # stays that way - the classic result of an interrupted install or a CI job that timed out
timeout -s KILL 6 helm -n oslo-helm install cache-broken /tmp/q513-stuck --wait --timeout 120s >/dev/null 2>&1 timeout -s KILL 6 helm -n oslo-helm install cache-broken /tmp/q98-stuck --wait --timeout 120s >/dev/null 2>&1
rm -rf /tmp/q513-stuck rm -rf /tmp/q98-stuck
helm -n oslo-helm status cache-broken -o json 2>/dev/null | grep -q '"status":"pending-install"' || echo "warn: cache-broken did not end up pending-install" helm -n oslo-helm status cache-broken -o json 2>/dev/null | grep -q '"status":"pending-install"' || echo "warn: cache-broken did not end up pending-install"
""", """,
checks=[ checks=[
@ -702,8 +702,8 @@ for i in $(seq 1 20); do [ "$(j -n oslo-helm deploy cache '{.status.availableRep
helm -n oslo-helm list # Helm 4 shows every status; Helm 3 hides pending ones helm -n oslo-helm list # Helm 4 shows every status; Helm 3 hides pending ones
helm -n oslo-helm list --pending # works in both: cache-broken pending-install helm -n oslo-helm list --pending # works in both: cache-broken pending-install
helm -n oslo-helm uninstall web-legacy helm -n oslo-helm uninstall web-legacy
helm -n oslo-helm upgrade web-main {WS}/q513/charts/nginx-lite-1.1.0 helm -n oslo-helm upgrade web-main {WS}/q98/charts/nginx-lite-1.1.0
helm -n oslo-helm install cache {WS}/q513/charts/redis-lite --set replicaCount=2 helm -n oslo-helm install cache {WS}/q98/charts/redis-lite --set replicaCount=2
helm -n oslo-helm uninstall cache-broken # also removes the Deployment it half-created helm -n oslo-helm uninstall cache-broken # also removes the Deployment it half-created
helm -n oslo-helm list # web-main (1.1.0, rev 2) and cache, both deployed helm -n oslo-helm list # web-main (1.1.0, rev 2) and cache, both deployed
# A pending-install release blocks a plain re-install under the same name ("cannot re-use a name # A pending-install release blocks a plain re-install under the same name ("cannot re-use a name
@ -711,9 +711,9 @@ helm -n oslo-helm list # web-main (1.1.0, rev 2) and cac
""", """,
) )
# ------------------------------------------------------------------ 514 ---- # ------------------------------------------------------------------ 99 ----
task( task(
id=514, domain="STOR", title="Per-replica storage for a StatefulSet", id=99, domain="STOR", title="Per-replica storage for a StatefulSet",
task=""" task="""
In Namespace `lima-data`: In Namespace `lima-data`:
@ -723,11 +723,11 @@ In Namespace `lima-data`:
`/data`. `/data`.
2. Once all 3 are Ready, scale the StatefulSet down to **2**. 2. Once all 3 are Ready, scale the StatefulSet down to **2**.
3. Write the names of the PersistentVolumeClaims that belong to the StatefulSet and still exist 3. Write the names of the PersistentVolumeClaims that belong to the StatefulSet and still exist
after the scale-down to `{WS}/q514/pvcs.txt`, one per line, sorted. after the scale-down to `{WS}/q99/pvcs.txt`, one per line, sorted.
""", """,
setup=NS + r""" setup=NS + r"""
mkns lima-data mkns lima-data
rm -rf "$WS/q514"; mkdir -p "$WS/q514" rm -rf "$WS/q99"; mkdir -p "$WS/q99"
$K -n lima-data delete sts ledger-db --ignore-not-found --wait=true >/dev/null 2>&1 $K -n lima-data delete sts ledger-db --ignore-not-found --wait=true >/dev/null 2>&1
$K -n lima-data delete svc ledger-db --ignore-not-found >/dev/null $K -n lima-data delete svc ledger-db --ignore-not-found >/dev/null
$K -n lima-data delete pvc -l app=ledger-db --ignore-not-found >/dev/null 2>&1 $K -n lima-data delete pvc -l app=ledger-db --ignore-not-found >/dev/null 2>&1
@ -745,7 +745,7 @@ for i in 0 1; do eq "$($K -n lima-data exec ledger-db-$i -- cat /data/id.txt 2>/
[ "$(j -n lima-data pod ledger-db-0 '{.spec.volumes[?(@.name=="data")].persistentVolumeClaim.claimName}')" = data-ledger-db-0 ]"""), [ "$(j -n lima-data pod ledger-db-0 '{.spec.volumes[?(@.name=="data")].persistentVolumeClaim.claimName}')" = data-ledger-db-0 ]"""),
C("pvcs.txt lists every claim that survived the scale-down", 2, r""" C("pvcs.txt lists every claim that survived the scale-down", 2, r"""
want=$($K -n lima-data get pvc --no-headers -o custom-columns=N:.metadata.name 2>/dev/null | grep '^data-ledger-db-' | sort) want=$($K -n lima-data get pvc --no-headers -o custom-columns=N:.metadata.name 2>/dev/null | grep '^data-ledger-db-' | sort)
[ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q514/pvcs.txt" 2>/dev/null)" = "$want" ]"""), [ -n "$want" ] && [ "$(echo "$want" | wc -l)" -ge 3 ] && [ "$(grep . "$WS/q99/pvcs.txt" 2>/dev/null)" = "$want" ]"""),
], ],
solution=""" solution="""
cat <<EOF | k apply -f - cat <<EOF | k apply -f -
@ -775,17 +775,17 @@ spec:
EOF EOF
k -n lima-data rollout status sts ledger-db k -n lima-data rollout status sts ledger-db
k -n lima-data scale sts ledger-db --replicas=2 k -n lima-data scale sts ledger-db --replicas=2
k -n lima-data get pvc -o custom-columns=N:.metadata.name --no-headers | grep '^data-ledger-db-' | sort > {WS}/q514/pvcs.txt k -n lima-data get pvc -o custom-columns=N:.metadata.name --no-headers | grep '^data-ledger-db-' | sort > {WS}/q99/pvcs.txt
cat {WS}/q514/pvcs.txt # all THREE: data-ledger-db-0, -1, -2 cat {WS}/q99/pvcs.txt # all THREE: data-ledger-db-0, -1, -2
# Claims are named <template>-<statefulset>-<ordinal> and outlive their pods: scaling down (or # Claims are named <template>-<statefulset>-<ordinal> and outlive their pods: scaling down (or
# deleting the StatefulSet) keeps them unless spec.persistentVolumeClaimRetentionPolicy says # deleting the StatefulSet) keeps them unless spec.persistentVolumeClaimRetentionPolicy says
# otherwise, so scaling back up re-attaches ledger-db-2 to its old data. # otherwise, so scaling back up re-attaches ledger-db-2 to its old data.
""", """,
) )
# ------------------------------------------------------------------ 515 ---- # ------------------------------------------------------------------ 100 ----
task( task(
id=515, domain="NET", title="Namespace AND pod selector, enforced", id=100, domain="NET", title="Namespace AND pod selector, enforced",
task=""" task="""
Pods labelled `app=db` in Namespace `oslo-db` must accept connections on TCP port 80 **only** from Pods labelled `app=db` in Namespace `oslo-db` must accept connections on TCP port 80 **only** from
Pods labelled `role=backend` that run in Namespaces labelled `env=prod`. Every other source must be Pods labelled `role=backend` that run in Namespaces labelled `env=prod`. Every other source must be
@ -863,9 +863,9 @@ k -n lima-api exec deploy/api -- wget -qO- -T3 http://db.oslo-db
""", """,
) )
# ------------------------------------------------------------------ 516 ---- # ------------------------------------------------------------------ 101 ----
task( task(
id=516, domain="NET", title="Resolves everything except the cluster", id=101, domain="NET", title="Resolves everything except the cluster",
task=""" task="""
Deployment `reporter` in Namespace `lima-client` cannot reach Service `ledger` — its log says Deployment `reporter` in Namespace `lima-client` cannot reach Service `ledger` — its log says
`bad address`. The Service, its endpoints and cluster DNS are all fine: other Pods in the Namespace `bad address`. The Service, its endpoints and cluster DNS are all fine: other Pods in the Namespace
@ -873,11 +873,11 @@ resolve `ledger` without trouble.
Fix the `reporter` Deployment (not the Service), without changing its image or command, so that its Fix the `reporter` Deployment (not the Service), without changing its image or command, so that its
requests succeed. Write the field you changed and its new value, as `<field>=<value>`, to requests succeed. Write the field you changed and its new value, as `<field>=<value>`, to
`{WS}/q516/fix.txt`. `{WS}/q101/fix.txt`.
""", """,
setup=NS + r""" setup=NS + r"""
mkns lima-client mkns lima-client
rm -rf "$WS/q516"; mkdir -p "$WS/q516" rm -rf "$WS/q101"; mkdir -p "$WS/q101"
$K -n lima-client create deploy ledger --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null $K -n lima-client create deploy ledger --image=nginx:1.27 --port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
$K -n lima-client expose deploy ledger --port=8080 --target-port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null $K -n lima-client expose deploy ledger --port=8080 --target-port=80 --dry-run=client -o yaml | $K apply -f - >/dev/null
$K -n lima-client delete deploy reporter --ignore-not-found --wait=true >/dev/null 2>&1 $K -n lima-client delete deploy reporter --ignore-not-found --wait=true >/dev/null 2>&1
@ -907,7 +907,7 @@ for i in $(seq 1 15); do
p=$($K -n lima-client get pod -l app=reporter --field-selector status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) p=$($K -n lima-client get pod -l app=reporter --field-selector status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
[ -n "$p" ] && $K -n lima-client logs "$p" --tail=3 2>/dev/null | grep -q 'ledger ok' && exit 0; sleep 3 [ -n "$p" ] && $K -n lima-client logs "$p" --tail=3 2>/dev/null | grep -q 'ledger ok' && exit 0; sleep 3
done; exit 1"""), done; exit 1"""),
C("fix.txt names dnsPolicy and its new value", 1, r"""grep -Eqi '^ *dnsPolicy *= *ClusterFirst *$' "$WS/q516/fix.txt" 2>/dev/null"""), C("fix.txt names dnsPolicy and its new value", 1, r"""grep -Eqi '^ *dnsPolicy *= *ClusterFirst *$' "$WS/q101/fix.txt" 2>/dev/null"""),
], ],
solution=""" solution="""
k -n lima-client logs deploy/reporter --tail=3 # wget: bad address 'ledger:8080' k -n lima-client logs deploy/reporter --tail=3 # wget: bad address 'ledger:8080'
@ -917,23 +917,23 @@ k -n lima-client get deploy reporter -o yaml | grep -n dns
k -n lima-client exec deploy/reporter -- cat /etc/resolv.conf # the podman/host resolver, no cluster search domains k -n lima-client exec deploy/reporter -- cat /etc/resolv.conf # the podman/host resolver, no cluster search domains
k -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}' k -n lima-client patch deploy reporter -p '{"spec":{"template":{"spec":{"dnsPolicy":"ClusterFirst"}}}}'
k -n lima-client logs deploy/reporter -f # "ledger ok" k -n lima-client logs deploy/reporter -f # "ledger ok"
echo dnsPolicy=ClusterFirst > {WS}/q516/fix.txt echo dnsPolicy=ClusterFirst > {WS}/q101/fix.txt
# The trap is the name: the DEFAULT dnsPolicy is ClusterFirst, while the policy literally called # The trap is the name: the DEFAULT dnsPolicy is ClusterFirst, while the policy literally called
# "Default" inherits the node's /etc/resolv.conf. ClusterFirstWithHostNet is for hostNetwork pods. # "Default" inherits the node's /etc/resolv.conf. ClusterFirstWithHostNet is for hostNetwork pods.
""", """,
) )
# ------------------------------------------------------------------ 517 ---- # ------------------------------------------------------------------ 102 ----
task( task(
id=517, domain="STOR", title="A claim that can never bind", id=102, domain="STOR", title="A claim that can never bind",
task=""" task="""
Deployment `uploader` in Namespace `oslo-files` has no running Pod. Find out why and fix it, so its Deployment `uploader` in Namespace `oslo-files` has no running Pod. Find out why and fix it, so its
Pod runs with the claim mounted at `/files`. Keep the claim's name (`shared-files`), its size and its Pod runs with the claim mounted at `/files`. Keep the claim's name (`shared-files`), its size and its
StorageClass. Write the root cause, in one line, to `{WS}/q517/reason.txt`. StorageClass. Write the root cause, in one line, to `{WS}/q102/reason.txt`.
""", """,
setup=NS + r""" setup=NS + r"""
mkns oslo-files mkns oslo-files
rm -rf "$WS/q517"; mkdir -p "$WS/q517" rm -rf "$WS/q102"; mkdir -p "$WS/q102"
$K -n oslo-files delete deploy uploader --ignore-not-found --wait=true >/dev/null 2>&1 $K -n oslo-files delete deploy uploader --ignore-not-found --wait=true >/dev/null 2>&1
$K -n oslo-files delete pvc shared-files --ignore-not-found --wait=true >/dev/null 2>&1 $K -n oslo-files delete pvc shared-files --ignore-not-found --wait=true >/dev/null 2>&1
$K apply -f - >/dev/null <<'EOF' $K apply -f - >/dev/null <<'EOF'
@ -965,7 +965,7 @@ for i in $(seq 1 20); do [ "$(j -n oslo-files deploy uploader '{.status.availabl
eq "$(j -n oslo-files deploy uploader '{.status.availableReplicas}')" 1 \ eq "$(j -n oslo-files deploy uploader '{.status.availableReplicas}')" 1 \
&& eq "$(j -n oslo-files deploy uploader '{.spec.template.spec.volumes[?(@.persistentVolumeClaim.claimName=="shared-files")].name}')" files \ && eq "$(j -n oslo-files deploy uploader '{.spec.template.spec.volumes[?(@.persistentVolumeClaim.claimName=="shared-files")].name}')" files \
&& j -n oslo-files deploy uploader '{.spec.template.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /files"""), && j -n oslo-files deploy uploader '{.spec.template.spec.containers[0].volumeMounts[*].mountPath}' | grep -qw /files"""),
C("reason.txt names the unsupported access mode", 2, r"""grep -Eqi 'ReadWriteMany|RWX|access ?mode' "$WS/q517/reason.txt" 2>/dev/null"""), C("reason.txt names the unsupported access mode", 2, r"""grep -Eqi 'ReadWriteMany|RWX|access ?mode' "$WS/q102/reason.txt" 2>/dev/null"""),
], ],
solution=""" solution="""
k -n oslo-files get pod # Pending k -n oslo-files get pod # Pending
@ -979,7 +979,7 @@ k -n oslo-files delete pvc shared-files
k apply -f pvc.yaml k apply -f pvc.yaml
k -n oslo-files rollout restart deploy uploader # or just wait - the pending pod retries k -n oslo-files rollout restart deploy uploader # or just wait - the pending pod retries
k -n oslo-files get pvc,pod k -n oslo-files get pvc,pod
echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q517/reason.txt echo 'PVC requested ReadWriteMany, which the local-path provisioner cannot provide (RWO/RWOP only)' > {WS}/q102/reason.txt
# RWX needs a filesystem many nodes can mount at once (NFS, CephFS, a cloud file service). # RWX needs a filesystem many nodes can mount at once (NFS, CephFS, a cloud file service).
# ReadWriteOnce is per NODE, not per pod - ReadWriteOncePod is the per-pod one. # ReadWriteOnce is per NODE, not per pod - ReadWriteOncePod is the per-pod one.
""", """,

6
tracks/cka/questions.py
View File

@ -178,7 +178,7 @@ $K delete ns store-prod --ignore-not-found --wait=false >/dev/null
], ],
}, },
{ {
"id": 5, "domain": "ARCH", "title": "Static pod", "breaks": ["w3-staticpath"], # Q103 relocates W3's staticPodPath "id": 5, "domain": "ARCH", "title": "Static pod", "breaks": ["w3-staticpath"], # Q46 relocates W3's staticPodPath
"task": """ "task": """
Create a **static pod** named `static-web` running image `nginx:1.27` Create a **static pod** named `static-web` running image `nginx:1.27`
on node `{W3}` (namespace `default`). on node `{W3}` (namespace `default`).
@ -631,8 +631,8 @@ $K -n default delete pod sched-probe --wait=false >/dev/null 2>&1
}, },
] ]
# ---- aggregate the banks: 1-20 here, 21-43 in bank_extra, 101-124 (hard) in bank_hard, # ---- aggregate the banks: 1-20 here, 21-43 in bank_extra, 44-67 (hard) in bank_hard,
# ---- 401-418 / 501-517 in the two fixed simulator sets # ---- 68-85 / 86-102 in the two fixed simulator sets. Numbering is contiguous - keep it that way.
from tracks.cka import bank_extra, bank_hard, bank_sim_c, bank_sim_d from tracks.cka import bank_extra, bank_hard, bank_sim_c, bank_sim_d
QUESTIONS = (QUESTIONS + bank_extra.QUESTIONS + bank_hard.QUESTIONS QUESTIONS = (QUESTIONS + bank_extra.QUESTIONS + bank_hard.QUESTIONS
+ bank_sim_c.QUESTIONS + bank_sim_d.QUESTIONS) + bank_sim_c.QUESTIONS + bank_sim_d.QUESTIONS)

84
tracks/cka/solutions.py
View File

@ -245,7 +245,7 @@ k -n kube-system get pod -l component=etcd -o jsonpath='{.items[0].spec.containe
""" """
# Hard Architecture Tasks # Hard Architecture Tasks
S[101] = """# Step 1: Inspect bindings for the ServiceAccount to find and remove over-privileged bindings S[44] = """# Step 1: Inspect bindings for the ServiceAccount to find and remove over-privileged bindings
k get clusterrolebinding -o wide | grep %{sa} k get clusterrolebinding -o wide | grep %{sa}
k delete clusterrolebinding %{sa}-legacy-admin k delete clusterrolebinding %{sa}-legacy-admin
@ -268,7 +268,7 @@ k auth can-i --list -n %{ns} --as=system:serviceaccount:%{ns}:%{sa}
k auth can-i create pods --subresource=exec -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} # yes k auth can-i create pods --subresource=exec -n %{ns} --as=system:serviceaccount:%{ns}:%{sa} # yes
""" """
S[102] = """# Step 1: Access the control plane node S[45] = """# Step 1: Access the control plane node
{SHELL_CP} {SHELL_CP}
# Step 2: Restore snapshot into a new data directory # Step 2: Restore snapshot into a new data directory
@ -288,7 +288,7 @@ watch crictl ps
kubectl -n default get cm %{cm} kubectl -n default get cm %{cm}
""" """
S[103] = """# Step 1: Identify the edge node matching the label selector S[46] = """# Step 1: Identify the edge node matching the label selector
k get nodes -l tier=edge k get nodes -l tier=edge
# Step 2: Access the target node # Step 2: Access the target node
@ -313,7 +313,7 @@ EOF
# k get pods -n %{ns} -o wide # k get pods -n %{ns} -o wide
""" """
S[104] = """# Step 1: Generate user private key and Certificate Signing Request S[47] = """# Step 1: Generate user private key and Certificate Signing Request
openssl genrsa -out %{user}.key 2048 openssl genrsa -out %{user}.key 2048
openssl req -new -key %{user}.key -subj "/CN=%{user}/O=developers" -out %{user}.csr openssl req -new -key %{user}.key -subj "/CN=%{user}/O=developers" -out %{user}.csr
@ -331,7 +331,7 @@ k get csr %{user} -o jsonpath='{.status.certificate}' | base64 -d > %{user}.crt
k -n %{ns} create rolebinding %{user}-edit --clusterrole=edit --user=%{user} k -n %{ns} create rolebinding %{user}-edit --clusterrole=edit --user=%{user}
# Step 4: Construct self-contained kubeconfig with embedded certificates # Step 4: Construct self-contained kubeconfig with embedded certificates
KC={WS}/q104/%{user}.kubeconfig KC={WS}/q47/%{user}.kubeconfig
k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt k config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
SERVER=$(k config view --minify -o jsonpath='{.clusters[0].cluster.server}') SERVER=$(k config view --minify -o jsonpath='{.clusters[0].cluster.server}')
@ -344,8 +344,8 @@ kubectl --kubeconfig $KC config use-context %{user}@lab
kubectl --kubeconfig $KC get pods kubectl --kubeconfig $KC get pods
""" """
S[105] = """# Step 1: Create directory for prod overlay S[48] = """# Step 1: Create directory for prod overlay
mkdir -p {WS}/q105/overlays/prod && cd {WS}/q105/overlays/prod mkdir -p {WS}/q48/overlays/prod && cd {WS}/q48/overlays/prod
# Step 2: Create replica patch manifest # Step 2: Create replica patch manifest
cat > replicas.yaml <<EOF cat > replicas.yaml <<EOF
@ -383,7 +383,7 @@ k create ns %{ns}
k apply -k . k apply -k .
""" """
S[106] = """# Step 0: This context does not default to the `default` namespace - check, then pass -n default S[49] = """# Step 0: This context does not default to the `default` namespace - check, then pass -n default
k config view --minify -o jsonpath='{..namespace}{"\\n"}' k config view --minify -o jsonpath='{..namespace}{"\\n"}'
# Step 1: CRD with validation (required fields, integer bounds, enum) and a server-side default # Step 1: CRD with validation (required fields, integer bounds, enum) and a server-side default
@ -425,15 +425,15 @@ spec: {replicas: 3, image: nginx:1.27, environment: %{env}}
EOF EOF
# Step 3: The rejection goes to STDERR - redirect 2>, not >, or the file stays empty # Step 3: The rejection goes to STDERR - redirect 2>, not >, or the file stays empty
printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: bad}\\nspec: {replicas: 0, image: x}\\n' | k -n default apply -f - 2> {WS}/q106/rejected.txt printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: bad}\\nspec: {replicas: 0, image: x}\\n' | k -n default apply -f - 2> {WS}/q49/rejected.txt
cat {WS}/q106/rejected.txt # ...spec.replicas in body should be greater than or equal to 1 cat {WS}/q49/rejected.txt # ...spec.replicas in body should be greater than or equal to 1
# Step 4: Omit environment and let the API server fill in the default # Step 4: Omit environment and let the API server fill in the default
printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: defaulted}\\nspec: {replicas: 2, image: nginx:1.27}\\n' | k -n default apply -f - printf 'apiVersion: deploy.example.com/v1\\nkind: %{kind}\\nmetadata: {name: defaulted}\\nspec: {replicas: 2, image: nginx:1.27}\\n' | k -n default apply -f -
k -n default get $(echo %{kind} | tr A-Z a-z) defaulted -o jsonpath='{.spec.environment}{"\\n"}' # dev k -n default get $(echo %{kind} | tr A-Z a-z) defaulted -o jsonpath='{.spec.environment}{"\\n"}' # dev
""" """
S[122] = """# Step 1: Access control plane node S[65] = """# Step 1: Access control plane node
{SHELL_CP} {SHELL_CP}
# Step 2: Renew all cluster certificates via kubeadm # Step 2: Renew all cluster certificates via kubeadm
@ -448,10 +448,10 @@ exit
# Step 5: Back on your own machine now - {WS} is a path on your machine, not the node, # Step 5: Back on your own machine now - {WS} is a path on your machine, not the node,
# so capture the expiry report from out here and redirect it into the required file # so capture the expiry report from out here and redirect it into the required file
podman exec cka-sim-control-plane kubeadm certs check-expiration | grep '^apiserver ' > {WS}/q122/expiry.txt podman exec cka-sim-control-plane kubeadm certs check-expiration | grep '^apiserver ' > {WS}/q65/expiry.txt
""" """
S[124] = """# Step 1: On the control-plane node, create a fresh bootstrap token S[67] = """# Step 1: On the control-plane node, create a fresh bootstrap token
{SHELL_CP} {SHELL_CP}
kubeadm token create --ttl 2h kubeadm token create --ttl 2h
# -> prints a token like abcdef.0123456789abcdef # -> prints a token like abcdef.0123456789abcdef
@ -470,7 +470,7 @@ exit
kubectl get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}{"\\n"}' kubectl get endpointslices -l kubernetes.io/service-name=kubernetes -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}{"\\n"}'
# Step 4: Assemble the handoff file # Step 4: Assemble the handoff file
cat > {WS}/q124/node-join.env <<EOF cat > {WS}/q67/node-join.env <<EOF
JOIN_TOKEN=<token from step 1> JOIN_TOKEN=<token from step 1>
CA_HASH=sha256:<hash from step 2> CA_HASH=sha256:<hash from step 2>
API_ENDPOINT=<ip:port from step 3> API_ENDPOINT=<ip:port from step 3>
@ -793,7 +793,7 @@ k -n %{ns} get pod -l app=worker -o jsonpath='{.items[0].spec.containers[0].reso
""" """
# Hard Workload Tasks # Hard Workload Tasks
S[107] = """# Step 1: Read the constraints before writing anything S[50] = """# Step 1: Read the constraints before writing anything
k get node {W1} -o jsonpath='{.spec.taints}{"\\n"}' # %{key}=%{val}:NoSchedule k get node {W1} -o jsonpath='{.spec.taints}{"\\n"}' # %{key}=%{val}:NoSchedule
k -n %{ns} describe quota # pods: 2, requests.cpu: 250m k -n %{ns} describe quota # pods: 2, requests.cpu: 250m
@ -836,7 +836,7 @@ k -n %{ns} get pod -l app=%{dep} -o wide
k -n %{ns} describe quota %{ns}-quota # pods 3/3, requests.cpu 300m/300m k -n %{ns} describe quota %{ns}-quota # pods 3/3, requests.cpu 300m/300m
""" """
S[108] = """# Step 1: Find WHICH revision used nginx:1.25 - the task asks for that one specifically, which S[51] = """# Step 1: Find WHICH revision used nginx:1.25 - the task asks for that one specifically, which
# is not necessarily the previous one. --revision=N shows a single revision's pod template. # is not necessarily the previous one. --revision=N shows a single revision's pod template.
k -n %{ns} rollout history deploy %{dep} k -n %{ns} rollout history deploy %{dep}
for r in $(k -n %{ns} rollout history deploy %{dep} | awk '/^[0-9]/{print $1}'); do for r in $(k -n %{ns} rollout history deploy %{dep} | awk '/^[0-9]/{print $1}'); do
@ -863,7 +863,7 @@ k -n %{ns} get pod -l app=%{dep} -o jsonpath='{.items[*].spec.containers[0].imag
k -n %{ns} get rs -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image k -n %{ns} get rs -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image
""" """
S[109] = """# Step 1: Deploy Pod with a native sidecar (restartPolicy: Always inside initContainers), S[52] = """# Step 1: Deploy Pod with a native sidecar (restartPolicy: Always inside initContainers),
# a shared emptyDir the sidecar writes into and the main container reads from. Mount it at # a shared emptyDir the sidecar writes into and the main container reads from. Mount it at
# /var/log/agent, NOT /var/log: an emptyDir over /var/log hides nginx's /var/log/nginx and # /var/log/agent, NOT /var/log: an emptyDir over /var/log hides nginx's /var/log/nginx and
# nginx crash-loops with "could not open error log file" # nginx crash-loops with "could not open error log file"
@ -897,30 +897,30 @@ EOF
# Step 2: Once Running, write the exact report files # Step 2: Once Running, write the exact report files
k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name} {.spec.containers[*].name}' \\ k -n %{ns} get pod %{pod} -o jsonpath='{.spec.initContainers[*].name} {.spec.containers[*].name}' \\
| tr ' ' '\\n' | sort > {WS}/q109/containers.txt | tr ' ' '\\n' | sort > {WS}/q52/containers.txt
k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' > {WS}/q109/ip.txt k -n %{ns} get pod %{pod} -o jsonpath='{.status.podIP}' > {WS}/q52/ip.txt
""" """
S[110] = r"""# Step 0: look before you script - which pods, which phases, which nodes S[53] = r"""# Step 0: look before you script - which pods, which phases, which nodes
k -n %{ns} get pods -o wide k -n %{ns} get pods -o wide
# Step 1: FIRST container without a readinessProbe, any phase (the Completed pod counts too). # Step 1: FIRST container without a readinessProbe, any phase (the Completed pod counts too).
# jsonpath prints nothing for a missing field, so emit "<name> <probe-or-empty>" and keep the bare names. # jsonpath prints nothing for a missing field, so emit "<name> <probe-or-empty>" and keep the bare names.
k -n %{ns} get pods -o jsonpath='{range .items[*]}{.metadata.name} {.spec.containers[0].readinessProbe}{"\n"}{end}' \ k -n %{ns} get pods -o jsonpath='{range .items[*]}{.metadata.name} {.spec.containers[0].readinessProbe}{"\n"}{end}' \
| awk 'NF==1{print $1}' | sort > {WS}/q110/no-readiness.txt | awk 'NF==1{print $1}' | sort > {WS}/q53/no-readiness.txt
# Step 2: Running pods per node, "<node> <count>". uniq -c prints "count node", so swap the columns. # Step 2: Running pods per node, "<node> <count>". uniq -c prints "count node", so swap the columns.
k -n %{ns} get pods --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' \ k -n %{ns} get pods --field-selector=status.phase=Running -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' \
| sort | uniq -c | awk '{print $2, $1}' | sort > {WS}/q110/per-node.txt | sort | uniq -c | awk '{print $2, $1}' | sort > {WS}/q53/per-node.txt
# Step 3: distinct images in kube-system # Step 3: distinct images in kube-system
k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u > {WS}/q110/images.txt k -n kube-system get pods -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u > {WS}/q53/images.txt
# Traps: cache-a has a readiness probe - on its SECOND container, so it belongs in file 1; # Traps: cache-a has a readiness probe - on its SECOND container, so it belongs in file 1;
# once-a is Completed, so it is in file 1 but not in file 2. `cat -A` shows stray spaces or \r. # once-a is Completed, so it is in file 1 but not in file 2. `cat -A` shows stray spaces or \r.
""" """
S[121] = """# Step 1: Switch context and find out which namespace it defaults to S[64] = """# Step 1: Switch context and find out which namespace it defaults to
k config use-context %{ctx} k config use-context %{ctx}
k config view --minify -o jsonpath='{..namespace}{"\\n"}' # -> %{ns} k config view --minify -o jsonpath='{..namespace}{"\\n"}' # -> %{ns}
@ -930,7 +930,7 @@ k label deploy %{dep} owner=ac
k expose deploy %{dep} --port=9000 --target-port=9000 # ClusterIP is the default type k expose deploy %{dep} --port=9000 --target-port=9000 # ClusterIP is the default type
# Step 3: Record the namespace exactly as kubectl reports it # Step 3: Record the namespace exactly as kubectl reports it
k config view --minify -o jsonpath='{..namespace}' > {WS}/q121/ns.txt k config view --minify -o jsonpath='{..namespace}' > {WS}/q64/ns.txt
# Step 4: Verify it all landed there - and nothing in default # Step 4: Verify it all landed there - and nothing in default
k get deploy,svc %{dep} k get deploy,svc %{dep}
@ -1156,7 +1156,7 @@ k -n %{ns} get svc legacy # EXTERNAL-IP column shows %{ext} - it i
""" """
# Hard Networking Tasks # Hard Networking Tasks
S[111] = """# Step 1: Label the source namespace for ingress matching S[54] = """# Step 1: Label the source namespace for ingress matching
k label ns %{from} team=%{from} k label ns %{from} team=%{from}
# Step 2: Apply default-deny (ingress + egress), allow-dns egress, and tiered service policies # Step 2: Apply default-deny (ingress + egress), allow-dns egress, and tiered service policies
@ -1208,7 +1208,7 @@ spec:
EOF EOF
""" """
S[112] = """# Step 1: Generate self-signed TLS certificate and private key S[55] = """# Step 1: Generate self-signed TLS certificate and private key
openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=%{host}" -keyout tls.key -out tls.crt openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=%{host}" -keyout tls.key -out tls.crt
# Step 2: Create TLS secret in target namespace # Step 2: Create TLS secret in target namespace
@ -1219,10 +1219,10 @@ k -n %{ns} create ingress web --class=nginx --rule="%{host}/*=web:80,tls=%{host}
--annotation nginx.ingress.kubernetes.io/ssl-redirect=true --annotation nginx.ingress.kubernetes.io/ssl-redirect=true
# Step 4: Extract SHA256 fingerprint to workspace file # Step 4: Extract SHA256 fingerprint to workspace file
openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > {WS}/q112/fp.txt openssl x509 -in tls.crt -noout -fingerprint -sha256 | cut -d= -f2 > {WS}/q55/fp.txt
""" """
S[113] = """# Step 1: Create GatewayClass, Gateway with multi-host listeners, and canary HTTPRoute S[56] = """# Step 1: Create GatewayClass, Gateway with multi-host listeners, and canary HTTPRoute
cat <<EOF | k apply -f - cat <<EOF | k apply -f -
apiVersion: gateway.networking.k8s.io/v1 apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass kind: GatewayClass
@ -1252,7 +1252,7 @@ spec:
EOF EOF
""" """
S[114] = """# Step 1: Remove stray blocking NetworkPolicy in kube-system S[57] = """# Step 1: Remove stray blocking NetworkPolicy in kube-system
k -n kube-system get netpol k -n kube-system get netpol
k -n kube-system delete netpol maintenance-lockdown k -n kube-system delete netpol maintenance-lockdown
@ -1402,7 +1402,7 @@ k -n %{ns} exec reader -- cat /data/marker.txt
""" """
# Hard Storage Tasks # Hard Storage Tasks
S[115] = """# Step 1: Identify PV labels and corresponding node affinity S[58] = """# Step 1: Identify PV labels and corresponding node affinity
k get pv -l media --show-labels k get pv -l media --show-labels
k get node -l disk=nvme k get node -l disk=nvme
@ -1430,7 +1430,7 @@ EOF
k -n %{ns} get pod consumer -o wide k -n %{ns} get pod consumer -o wide
""" """
S[116] = """# Step 1: Create target PVC for migration S[59] = """# Step 1: Create target PVC for migration
k -n %{ns} apply -f - <<EOF k -n %{ns} apply -f - <<EOF
apiVersion: v1 apiVersion: v1
kind: PersistentVolumeClaim kind: PersistentVolumeClaim
@ -1611,7 +1611,7 @@ systemctl status kubelet
""" """
# Hard Troubleshooting Tasks # Hard Troubleshooting Tasks
S[117] = """# Step 1: Check node status in cluster S[60] = """# Step 1: Check node status in cluster
k get node {W2} # NotReady,SchedulingDisabled k get node {W2} # NotReady,SchedulingDisabled
# Step 2: Access the node # Step 2: Access the node
@ -1633,7 +1633,7 @@ k get pod test-pod -o wide
k delete pod test-pod --now k delete pod test-pod --now
""" """
S[118] = """# Step 1: Check control plane pods in kube-system S[61] = """# Step 1: Check control plane pods in kube-system
k -n kube-system get pods k -n kube-system get pods
# scheduler CrashLoop, controller-manager ImagePullBackOff # scheduler CrashLoop, controller-manager ImagePullBackOff
@ -1655,7 +1655,7 @@ k get deploy probe
k delete deploy probe k delete deploy probe
""" """
S[119] = """# Step 1: Diagnose namespace quota and deployment events S[62] = """# Step 1: Diagnose namespace quota and deployment events
k -n %{ns} get all k -n %{ns} get all
k -n %{ns} describe deploy %{dep} k -n %{ns} describe deploy %{dep}
k -n %{ns} describe rs -l app=%{dep} # Quota exceeded: pods limit is 1 k -n %{ns} describe rs -l app=%{dep} # Quota exceeded: pods limit is 1
@ -1681,24 +1681,24 @@ k -n %{ns} get deploy %{dep}
k -n %{ns} get endpointslices -l kubernetes.io/service-name=%{dep} k -n %{ns} get endpointslices -l kubernetes.io/service-name=%{dep}
""" """
S[120] = r"""# Gather each fact and append it directly to the report file, one key=value line at a time S[63] = r"""# Gather each fact and append it directly to the report file, one key=value line at a time
pc=$(k -n kube-system get pod -l component=kube-controller-manager -o jsonpath='{.items[0].spec.containers[0].command}' | grep -o 'cluster-cidr=[^,"]*' | cut -d= -f2) pc=$(k -n kube-system get pod -l component=kube-controller-manager -o jsonpath='{.items[0].spec.containers[0].command}' | grep -o 'cluster-cidr=[^,"]*' | cut -d= -f2)
echo "pod-cidr=$pc" > {WS}/q120/facts.txt echo "pod-cidr=$pc" > {WS}/q63/facts.txt
sc=$(k -n kube-system get pod -l component=kube-apiserver -o jsonpath='{.items[0].spec.containers[0].command}' | grep -o 'service-cluster-ip-range=[^,"]*' | cut -d= -f2) sc=$(k -n kube-system get pod -l component=kube-apiserver -o jsonpath='{.items[0].spec.containers[0].command}' | grep -o 'service-cluster-ip-range=[^,"]*' | cut -d= -f2)
echo "service-cidr=$sc" >> {WS}/q120/facts.txt echo "service-cidr=$sc" >> {WS}/q63/facts.txt
d=$(k -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}') d=$(k -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}')
echo "dns-ip=$d" >> {WS}/q120/facts.txt echo "dns-ip=$d" >> {WS}/q63/facts.txt
# Empty mode in the ConfigMap means kube-proxy defaults to iptables # Empty mode in the ConfigMap means kube-proxy defaults to iptables
m=$(k -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"') m=$(k -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep '^mode:' | awk '{print $2}' | tr -d '"')
echo "proxy-mode=${m:-iptables}" >> {WS}/q120/facts.txt echo "proxy-mode=${m:-iptables}" >> {WS}/q63/facts.txt
echo "%{node}-pod-cidr=$(k get node %{node} -o jsonpath='{.spec.podCIDR}')" >> {WS}/q120/facts.txt echo "%{node}-pod-cidr=$(k get node %{node} -o jsonpath='{.spec.podCIDR}')" >> {WS}/q63/facts.txt
""" """
S[123] = """# Step 1: Confirm nothing works through kubectl at all S[66] = """# Step 1: Confirm nothing works through kubectl at all
kubectl get nodes # connection refused / timeout -- the API server itself is down kubectl get nodes # connection refused / timeout -- the API server itself is down
# Step 2: Access the control-plane node directly (kubectl can't help you here) # Step 2: Access the control-plane node directly (kubectl can't help you here)

132
weak-spots-cka.md
View File

@ -66,36 +66,36 @@ _Generated by exam from 10 attempt(s); regenerated on every `end`._
### Q42 · NET · Egress-only NetworkPolicy (2/3 failed, last 4/4 pts, last failed 2026-09-14T16:49) ### Q42 · NET · Egress-only NetworkPolicy (2/3 failed, last 4/4 pts, last failed 2026-09-14T16:49)
- passed last time (4/4) - one more clean pass takes it off this list - passed last time (4/4) - one more clean pass takes it off this list
### Q102 · ARCH · etcd restore (1/1 failed, last 2/10 pts, last failed 2026-09-19T12:27) ### Q45 · ARCH · etcd restore (1/1 failed, last 2/10 pts, last failed 2026-09-19T12:27)
- ✗ ConfigMap restore-proof-67 exists again with its original data - ✗ ConfigMap restore-proof-67 exists again with its original data
- ✗ etcd uses a restored data directory (manifest changed) - ✗ etcd uses a restored data directory (manifest changed)
### Q106 · ARCH · CRD with validation and defaults (4/4 failed, last 0/7 pts, last failed 2026-09-19T12:27) ### Q49 · ARCH · CRD with validation and defaults (4/4 failed, last 0/7 pts, last failed 2026-09-19T12:27)
- ✗ CRD exists, Namespaced, replicas has min/max and is required - ✗ CRD exists, Namespaced, replicas has min/max and is required
- ✗ good created with environment prod - ✗ good created with environment prod
- ✗ replicas 0 is rejected and the message was saved - ✗ replicas 0 is rejected and the message was saved
- ✗ defaulting works: defaulted.spec.environment == dev - ✗ defaulting works: defaulted.spec.environment == dev
### Q109 · WORK · Native sidecar with probes and an exact report (1/1 failed, last 2/8 pts, last failed 2026-09-19T12:27) ### Q52 · WORK · Native sidecar with probes and an exact report (1/1 failed, last 2/8 pts, last failed 2026-09-19T12:27)
- ✗ native sidecar: initContainer agent with restartPolicy Always - ✗ native sidecar: initContainer agent with restartPolicy Always
- ✗ probes on app as specified - ✗ probes on app as specified
- ✗ Running, shared log written by agent, grace period 5 - ✗ Running, shared log written by agent, grace period 5
### Q112 · NET · TLS Ingress with a generated certificate (4/4 failed, last 0/7 pts, last failed 2026-09-19T12:27) ### Q55 · NET · TLS Ingress with a generated certificate (4/4 failed, last 0/7 pts, last failed 2026-09-19T12:27)
- ✗ Secret secure.lab.local-tls is kubernetes.io/tls with CN secure.lab.local - ✗ Secret secure.lab.local-tls is kubernetes.io/tls with CN secure.lab.local
- ✗ Ingress web: class nginx, host, TLS secret, ssl-redirect annotation - ✗ Ingress web: class nginx, host, TLS secret, ssl-redirect annotation
- ✗ fp.txt matches the secret's certificate fingerprint - ✗ fp.txt matches the secret's certificate fingerprint
### Q115 · STOR · Bind the right local volume (3/3 failed, last 1/6 pts, last failed 2026-09-15T20:55) ### Q58 · STOR · Bind the right local volume (3/3 failed, last 1/6 pts, last failed 2026-09-15T20:55)
- ✗ PVC uses a selector and is Bound to lv-nvme - ✗ PVC uses a selector and is Bound to lv-nvme
- ✗ consumer Running on the nvme node with /data mounted - ✗ consumer Running on the nvme node with /data mounted
### Q120 · TSHOOT · Cluster facts, exactly (3/3 failed, last 0/5 pts, last failed 2026-09-15T20:55) ### Q63 · TSHOOT · Cluster facts, exactly (3/3 failed, last 0/5 pts, last failed 2026-09-15T20:55)
- ✗ pod-cidr and service-cidr - ✗ pod-cidr and service-cidr
- ✗ dns-ip and proxy-mode - ✗ dns-ip and proxy-mode
- ✗ cka-sim-worker3 pod CIDR - ✗ cka-sim-worker3 pod CIDR
### Q121 · WORK · The context's namespace (3/3 failed, last 1/6 pts, last failed 2026-09-15T20:55) ### Q64 · WORK · The context's namespace (3/3 failed, last 1/6 pts, last failed 2026-09-15T20:55)
- ✗ Deployment ledger-21 in ac-echo (the context's namespace), 2 available, owner=ac - ✗ Deployment ledger-21 in ac-echo (the context's namespace), 2 available, owner=ac
- ✗ Service ledger-21 port 9000 in the same namespace - ✗ Service ledger-21 port 9000 in the same namespace
- ✗ ns.txt has the namespace - ✗ ns.txt has the namespace
@ -147,64 +147,64 @@ _Generated by exam from 10 attempt(s); regenerated on every `end`._
| 41 | ARCH | Component versions | 0/4 | · | · | ✓ | · | · | · | · | · | · | | 41 | ARCH | Component versions | 0/4 | · | · | ✓ | · | · | · | · | · | · |
| 42 | NET | Egress-only NetworkPolicy | · | 0/4 | · | · | · | · | 0/4 | ✓ | · | · | | 42 | NET | Egress-only NetworkPolicy | · | 0/4 | · | · | · | · | 0/4 | ✓ | · | · |
| 43 | NET | Headless and ExternalName services | · | · | · | · | · | · | · | · | · | · | | 43 | NET | Headless and ExternalName services | · | · | · | · | · | · | · | · | · | · |
| 101 | ARCH | Exact RBAC with an existing over-grant | · | · | · | · | · | · | · | · | · | · | | 44 | ARCH | Exact RBAC with an existing over-grant | · | · | · | · | · | · | · | · | · | · |
| 102 | ARCH | etcd restore | · | · | · | · | · | · | · | · | · | 2/10 | | 45 | ARCH | etcd restore | · | · | · | · | · | · | · | · | · | 2/10 |
| 103 | ARCH | Static pod on a relocated manifest path | · | · | · | · | · | · | · | · | · | · | | 46 | ARCH | Static pod on a relocated manifest path | · | · | · | · | · | · | · | · | · | · |
| 104 | ARCH | User onboarding to a working kubeconfig | · | · | · | · | · | · | · | · | · | · | | 47 | ARCH | User onboarding to a working kubeconfig | · | · | · | · | · | · | · | · | · | · |
| 105 | ARCH | Kustomize with generators and patches | · | · | · | · | · | · | · | · | · | · | | 48 | ARCH | Kustomize with generators and patches | · | · | · | · | · | · | · | · | · | · |
| 106 | ARCH | CRD with validation and defaults | · | · | 0/7 | · | · | · | 0/7 | 0/7 | · | 0/7 | | 49 | ARCH | CRD with validation and defaults | · | · | 0/7 | · | · | · | 0/7 | 0/7 | · | 0/7 |
| 107 | WORK | Spread, tolerate, prioritise — under a quota | · | · | · | · | · | · | · | · | · | · | | 50 | WORK | Spread, tolerate, prioritise — under a quota | · | · | · | · | · | · | · | · | · | · |
| 108 | WORK | Rollback to a specific revision, then pause | · | · | · | · | · | · | · | · | · | · | | 51 | WORK | Rollback to a specific revision, then pause | · | · | · | · | · | · | · | · | · | · |
| 109 | WORK | Native sidecar with probes and an exact report | · | · | · | · | · | · | · | · | · | 2/8 | | 52 | WORK | Native sidecar with probes and an exact report | · | · | · | · | · | · | · | · | · | 2/8 |
| 110 | WORK | Namespace audit to exact files | · | · | · | · | · | · | · | · | · | · | | 53 | WORK | Namespace audit to exact files | · | · | · | · | · | · | · | · | · | · |
| 111 | NET | Zero-trust namespace | · | · | · | · | · | · | · | · | · | · | | 54 | NET | Zero-trust namespace | · | · | · | · | · | · | · | · | · | · |
| 112 | NET | TLS Ingress with a generated certificate | · | · | 0/7 | · | · | · | 0/7 | 0/7 | · | 0/7 | | 55 | NET | TLS Ingress with a generated certificate | · | · | 0/7 | · | · | · | 0/7 | 0/7 | · | 0/7 |
| 113 | NET | Gateway API canary with header match | · | · | · | · | · | · | · | · | · | · | | 56 | NET | Gateway API canary with header match | · | · | · | · | · | · | · | · | · | · |
| 114 | NET | DNS down: three faults | · | · | · | · | · | · | · | · | · | · | | 57 | NET | DNS down: three faults | · | · | · | · | · | · | · | · | · | · |
| 115 | STOR | Bind the right local volume | · | · | 1/6 | · | · | · | 1/6 | 1/6 | · | · | | 58 | STOR | Bind the right local volume | · | · | 1/6 | · | · | · | 1/6 | 1/6 | · | · |
| 116 | STOR | Migrate data between claims | · | · | · | · | · | · | · | · | · | · | | 59 | STOR | Migrate data between claims | · | · | · | · | · | · | · | · | · | · |
| 117 | TSHOOT | Worker node: three things wrong | · | · | · | · | · | · | · | · | · | · | | 60 | TSHOOT | Worker node: three things wrong | · | · | · | · | · | · | · | · | · | · |
| 118 | TSHOOT | Two control-plane components down | · | · | · | · | · | · | · | · | · | · | | 61 | TSHOOT | Two control-plane components down | · | · | · | · | · | · | · | · | · | · |
| 119 | TSHOOT | Application down for four reasons | · | · | · | · | · | · | · | · | · | · | | 62 | TSHOOT | Application down for four reasons | · | · | · | · | · | · | · | · | · | · |
| 120 | TSHOOT | Cluster facts, exactly | · | · | 0/5 | · | · | · | 0/5 | 0/5 | · | · | | 63 | TSHOOT | Cluster facts, exactly | · | · | 0/5 | · | · | · | 0/5 | 0/5 | · | · |
| 121 | WORK | The context's namespace | · | · | 1/6 | · | · | · | 1/6 | 1/6 | · | · | | 64 | WORK | The context's namespace | · | · | 1/6 | · | · | · | 1/6 | 1/6 | · | · |
| 122 | ARCH | Renew control-plane certificates | · | · | · | · | · | · | · | · | · | · | | 65 | ARCH | Renew control-plane certificates | · | · | · | · | · | · | · | · | · | · |
| 123 | TSHOOT | Nothing works: kube-apiserver itself is down | · | · | · | · | · | · | · | · | · | · | | 66 | TSHOOT | Nothing works: kube-apiserver itself is down | · | · | · | · | · | · | · | · | · | · |
| 124 | ARCH | Prepare a node join without shell access to the control plane | · | · | · | · | · | · | · | · | · | · | | 67 | ARCH | Prepare a node join without shell access to the control plane | · | · | · | · | · | · | · | · | · | · |
| 401 | STOR | Provision geo-fence persistent storage | · | · | · | · | · | · | · | · | · | · | | 68 | STOR | Provision geo-fence persistent storage | · | · | · | · | · | · | · | · | · | · |
| 402 | ARCH | Render a chart, then install it | · | · | · | · | · | · | · | · | · | · | | 69 | ARCH | Render a chart, then install it | · | · | · | · | · | · | · | · | · | · |
| 403 | TSHOOT | Fix the driver onboarding app | · | · | · | · | · | · | · | · | · | · | | 70 | TSHOOT | Fix the driver onboarding app | · | · | · | · | · | · | · | · | · | · |
| 404 | NET | Expose and verify the dispatch API | · | · | · | · | · | · | · | · | · | · | | 71 | NET | Expose and verify the dispatch API | · | · | · | · | · | · | · | · | · | · |
| 405 | STOR | Migrate ride history to a Retain volume | · | · | · | · | · | · | · | · | · | · | | 72 | STOR | Migrate ride history to a Retain volume | · | · | · | · | · | · | · | · | · | · |
| 406 | TSHOOT | Fix ETA service connectivity | · | · | · | · | · | · | · | · | · | · | | 73 | TSHOOT | Fix ETA service connectivity | · | · | · | · | · | · | · | · | · | · |
| 407 | NET | Fleet dashboard canary with Gateway API | · | · | · | · | · | · | · | · | · | · | | 74 | NET | Fleet dashboard canary with Gateway API | · | · | · | · | · | · | · | · | · | · |
| 408 | ARCH | Cluster-wide read access for the auth team | · | · | · | · | · | · | · | · | · | · | | 75 | ARCH | Cluster-wide read access for the auth team | · | · | · | · | · | · | · | · | · | · |
| 409 | WORK | Add an init container to the trip logger | · | · | · | · | · | · | · | · | · | · | | 76 | WORK | Add an init container to the trip logger | · | · | · | · | · | · | · | · | · | · |
| 410 | NET | Audit the CNI, then lock a namespace down | · | · | · | · | · | · | · | · | · | · | | 77 | NET | Audit the CNI, then lock a namespace down | · | · | · | · | · | · | · | · | · | · |
| 411 | WORK | Priority for the rider support portal | · | · | · | · | · | · | · | · | · | · | | 78 | WORK | Priority for the rider support portal | · | · | · | · | · | · | · | · | · | · |
| 412 | WORK | Surge pricing configuration | · | · | · | · | · | · | · | · | · | · | | 79 | WORK | Surge pricing configuration | · | · | · | · | · | · | · | · | · | · |
| 413 | ARCH | Plan the control-plane upgrade | · | · | · | · | · | · | · | · | · | · | | 80 | ARCH | Plan the control-plane upgrade | · | · | · | · | · | · | · | · | · | · |
| 414 | TSHOOT | Fix the fare calculation pods | · | · | · | · | · | · | · | · | · | · | | 81 | TSHOOT | Fix the fare calculation pods | · | · | · | · | · | · | · | · | · | · |
| 415 | WORK | Autoscale the route planner on memory | · | · | · | · | · | · | · | · | · | · | | 82 | WORK | Autoscale the route planner on memory | · | · | · | · | · | · | · | · | · | · |
| 416 | NET | NetworkPolicy for the safety monitor | · | · | · | · | · | · | · | · | · | · | | 83 | NET | NetworkPolicy for the safety monitor | · | · | · | · | · | · | · | · | · | · |
| 417 | ARCH | Enforce TLS 1.3 on the booking portal | · | · | · | · | · | · | · | · | · | · | | 84 | ARCH | Enforce TLS 1.3 on the booking portal | · | · | · | · | · | · | · | · | · | · |
| 418 | TSHOOT | Investigate vehicle tracker health | · | · | · | · | · | · | · | · | · | · | | 85 | TSHOOT | Investigate vehicle tracker health | · | · | · | · | · | · | · | · | · | · |
| 501 | ARCH | How the control plane is run | · | · | · | · | · | · | · | · | · | · | | 86 | ARCH | How the control plane is run | · | · | · | · | · | · | · | · | · | · |
| 502 | TSHOOT | Cluster facts to an exact file | · | · | · | · | · | · | · | · | · | · | | 87 | TSHOOT | Cluster facts to an exact file | · | · | · | · | · | · | · | · | · | · |
| 503 | NET | Ready only when its dependency answers | · | · | · | · | · | · | · | · | · | · | | 88 | NET | Ready only when its dependency answers | · | · | · | · | · | · | · | · | · | · |
| 504 | WORK | Sorting done by kubectl | · | · | · | · | · | · | · | · | · | · | | 89 | WORK | Sorting done by kubectl | · | · | · | · | · | · | · | · | · | · |
| 505 | TSHOOT | Event forensics around kube-proxy | · | · | · | · | · | · | · | · | · | · | | 90 | TSHOOT | Event forensics around kube-proxy | · | · | · | · | · | · | · | · | · | · |
| 506 | ARCH | API resources and a Role census | · | · | · | · | · | · | · | · | · | · | | 91 | ARCH | API resources and a Role census | · | · | · | · | · | · | · | · | · | · |
| 507 | TSHOOT | Kubelet unit points at nothing | · | · | · | · | · | · | · | · | · | · | | 92 | TSHOOT | Kubelet unit points at nothing | · | · | · | · | · | · | · | · | · | · |
| 508 | WORK | Secrets as a read-only volume and renamed env vars | · | · | · | · | · | · | · | · | · | · | | 93 | WORK | Secrets as a read-only volume and renamed env vars | · | · | · | · | · | · | · | · | · | · |
| 509 | ARCH | Static Pod on the control plane behind a Service | · | · | · | · | · | · | · | · | · | · | | 94 | ARCH | Static Pod on the control plane behind a Service | · | · | · | · | · | · | · | · | · | · |
| 510 | ARCH | The kubelet's two certificates | · | · | · | · | · | · | · | · | · | · | | 95 | ARCH | The kubelet's two certificates | · | · | · | · | · | · | · | · | · | · |
| 511 | WORK | A Pod for the control plane only | · | · | · | · | · | · | · | · | · | · | | 96 | WORK | A Pod for the control plane only | · | · | · | · | · | · | · | · | · | · |
| 512 | TSHOOT | Stop the scheduler, schedule by hand | · | · | · | · | · | · | · | · | · | · | | 97 | TSHOOT | Stop the scheduler, schedule by hand | · | · | · | · | · | · | · | · | · | · |
| 513 | ARCH | Helm release housekeeping | · | · | · | · | · | · | · | · | · | · | | 98 | ARCH | Helm release housekeeping | · | · | · | · | · | · | · | · | · | · |
| 514 | STOR | Per-replica storage for a StatefulSet | · | · | · | · | · | · | · | · | · | · | | 99 | STOR | Per-replica storage for a StatefulSet | · | · | · | · | · | · | · | · | · | · |
| 515 | NET | Namespace AND pod selector, enforced | · | · | · | · | · | · | · | · | · | · | | 100 | NET | Namespace AND pod selector, enforced | · | · | · | · | · | · | · | · | · | · |
| 516 | NET | Resolves everything except the cluster | · | · | · | · | · | · | · | · | · | · | | 101 | NET | Resolves everything except the cluster | · | · | · | · | · | · | · | · | · | · |
| 517 | STOR | A claim that can never bind | · | · | · | · | · | · | · | · | · | · | | 102 | STOR | A claim that can never bind | · | · | · | · | · | · | · | · | · | · |
_E = full exam, D = drill, · = not in that session, n/m = partial credit._ _E = full exam, D = drill, · = not in that session, n/m = partial credit._