From 58a357d7d233acaf6e2b8a853dcefb838b6d4c49 Mon Sep 17 00:00:00 2001 From: vumast3r Date: Thu, 10 Sep 2026 21:25:18 -0600 Subject: [PATCH] feat: add kube-apiserver failure and node-join-verification tasks Close two of the four CKA-relevant gaps identified against Killercoda's scenario catalog: - Q123 (TSHOOT, hard): kube-apiserver itself crash-loops on a static-pod manifest typo, same pattern as the existing scheduler/controller-manager failures (Q20/Q118) but with the added twist that kubectl is completely unusable until it's fixed, forcing node-level diagnosis (crictl, journalctl). - Q124 (ARCH, hard): produce a real node-join handoff file by independently computing the CA discovery hash kubeadm join verifies against, and finding the API server's actual in-cluster address rather than the local kubeconfig's port-forwarded one. Cluster-upgrade and operator-authoring remain unimplemented and are documented as genuine infrastructure gaps in CKA_SOLUTIONS_EXPLAINED.md (kind's node images have no apt-installable multi-version kubelet/kubeadm, and operator authoring needs a real built/pushed image) rather than faked. Also updates the Killercoda cross-reference table with the gaps closed, and logs an ended drill session to history-cka.jsonl/weak-spots-cka.md. Co-Authored-By: Claude Sonnet 5 --- CKA_SOLUTIONS_EXPLAINED.md | 100 +++++++++++++++- history-cka.jsonl | 1 + tracks/cka/bank_hard.py | 70 ++++++++++++ tracks/cka/solutions.py | 46 ++++++++ weak-spots-cka.md | 229 ++++++++++++++++++++----------------- 5 files changed, 343 insertions(+), 103 deletions(-) diff --git a/CKA_SOLUTIONS_EXPLAINED.md b/CKA_SOLUTIONS_EXPLAINED.md index 41ac572..7f19fe9 100644 --- a/CKA_SOLUTIONS_EXPLAINED.md +++ b/CKA_SOLUTIONS_EXPLAINED.md @@ -2,6 +2,59 @@ This guide explains how to approach and solve every question in the CKA exam bank. Every entry gives you the **underlying Kubernetes concept** (why the mechanism behaves the way it does), the **exact commands/manifests** that solve the task, and the **gotchas the grader is actually checking for** — the small details that separate a passing answer from a plausible-looking one. `k` is used throughout as the `kubectl` alias (already configured on the exam VMs). +## Practicing the same ground on Killercoda + +This bank mirrors the real CKA curriculum weights (25/15/20/10/30% across the five domains), and so does Killercoda's free, browser-based CKA scenario set. Killercoda groups its own scenarios into five domains too — the mapping below cross-references its actual scenario titles against the equivalent questions in this bank, so you know exactly which free scenario to run when a topic here feels shaky. + +| Killercoda scenario | Killercoda domain | Equivalent question(s) here | +|---|---|---| +| StorageClass and Dynamic Provisioning | Storage | Q16 | +| PV Access Modes | Storage | Q15 | +| Volume Can't Be Mounted | Storage | Q35, Q115 (troubleshooting a binding/mount failure) | +| PVC Data Migration | Storage | Q116 | +| Deployment and StatefulSet Rollouts | Workloads & Scheduling | Q07, Q108 | +| Init Container and Secret | Workloads & Scheduling | Q28 + Q10/Q29 (init container *and* a broken Secret reference, combined) | +| DaemonSet HostPath Configurator | Workloads & Scheduling | Q26 | +| Node Selector and Scheduling | Workloads & Scheduling | Q09 (this bank uses required nodeAffinity instead of nodeSelector — worth doing both) | +| Node Taints and Tolerations | Workloads & Scheduling | Q25, Q107 | +| Scheduling Pod Affinity | Workloads & Scheduling | *no direct equivalent here* — this bank only exercises anti-affinity (Q107); do this one to cover positive affinity too | +| Scheduling Pod Anti Affinity | Workloads & Scheduling | Q107 | +| Scheduling Priority | Workloads & Scheduling | Q24, Q107 | +| NetworkPolicy Isolation | Services & Networking | Q11, Q111 | +| Troubleshoot NetworkPolicies | Services & Networking | Q114 (a blocking NetworkPolicy is one of its three faults) | +| App Goes Public | Services & Networking | Q12 (NodePort here; LoadBalancer+autoscale adds Q23's HPA angle) | +| Gateway API Setup | Services & Networking | Q22, Q113 | +| CoreDNS Misconfigured | Services & Networking | Q37, Q114 | +| TLS Secret and HTTPS | Services & Networking | Q112 | +| Apiserver Crash | Troubleshooting | Q123 | +| Apiserver Misconfigured | Troubleshooting | Q123 | +| Deployment Scaling Issue | Troubleshooting | Q23 | +| Application Multi Container Issue | Troubleshooting | Q08, Q109 (troubleshooting angle on a multi-container pod) | +| Applications Misconfigured | Troubleshooting | Q18, Q119 | +| Components Misconfigured | Troubleshooting | Q20, Q38, Q118 | +| RBAC for CronJob | Cluster Architecture | Q01/Q101's RBAC pattern, applied to a CronJob's ServiceAccount specifically | +| RBAC User Permissions | Cluster Architecture | Q06, Q33, Q104 | +| Operator Setup and CRD | Cluster Architecture | Q04, Q34, Q106 (CRD half only — this bank never writes an actual operator/controller) | +| Operator Namespace Creator | Cluster Architecture | *no equivalent here* — this bank doesn't cover writing controller logic (arguably CKAD/dev territory more than CKA) | +| Static Pod Move | Cluster Architecture | Q05, Q103 | +| Cluster Setup and Node Join | Cluster Architecture | Q40 (token) + Q124 (independently verifying the CA discovery hash and real in-cluster API endpoint) — the token/crypto half only, see gap note | +| Cluster Upgrade | Cluster Architecture | *no equivalent here* — see gap note below | +| Cluster Certificate Management | Cluster Architecture | Q06, Q122 | + +### Real gaps worth knowing about + +Two topics from Killercoda's list still have **no faithful equivalent in this simulator**, for concrete infrastructure reasons rather than oversight: + +- **Cluster Upgrade** — `kubeadm upgrade plan`/`apply`, draining nodes, upgrading kubelet/kubectl package versions in order. Not reproducible on this rig: the kind node images bake kubelet/kubeadm in as fixed binaries with no `pkgs.k8s.io` apt repo configured, so there's no real multi-version binary swap to perform here — faking it would just be theater. Practice this one on Killercoda, which provisions real upgradeable VMs. +- **Operator Namespace Creator** — writing actual controller/reconciliation logic and shipping it as a running operator. Doing this properly would mean building and pushing a real container image, and arguably strays into CKAD/development territory more than CKA administration — but it's on the exam-relevant list, so don't be caught flat-footed if it comes up. + +Two other gaps this bank previously had are now closed: + +- **Apiserver Crash / Apiserver Misconfigured** → **Q123**. Same diagnostic method as the existing scheduler/controller-manager failures (Q20, Q38, Q118), just applied to kube-apiserver itself — with the added twist that *every* `kubectl` command is unusable until you fix it, so you have to work from the node directly (`crictl`, `journalctl`) rather than leaning on `kubectl describe`. +- **Cluster Setup and Node Join** → partially closed by **Q124**. A full `kubeadm init`/`join` end-to-end on a fresh node isn't safe to automate here (it would mean deleting a live Node object on your shared 3-worker cluster, which several other questions pin pods to by label/taint) — but Q124 drills the cryptographic half a real join actually depends on: computing the CA's discovery hash yourself and finding the real in-cluster API endpoint, rather than trusting whatever `kubeadm token create --print-join-command` or your local kubeconfig hands you. + +The exam-day skill this table is really pointing at, regardless of which platform you practice on: the diagnostic *method* is identical everywhere — read the actual error (`describe`, `logs`, `journalctl -u kubelet`), don't guess from memory, and verify with the tool the grader would use (`auth can-i`, `get endpointslices`, `rollout status`) rather than assuming a manifest applied cleanly means it's actually working. + --- ## Domain 1: Cluster Architecture, Installation & Configuration (25%) @@ -445,6 +498,30 @@ This guide explains how to approach and solve every question in the CKA exam ban --- +### Q124 · [HARD] Prepare a Node Join Without Shell Access to the Control Plane +- **Concept**: `kubeadm join` doesn't trust the control plane on faith — it verifies the server it's talking to by hashing the CA certificate's SubjectPublicKeyInfo with SHA-256 and comparing it against the `--discovery-token-ca-cert-hash` you supply, which is exactly the computation this task has you reproduce by hand instead of trusting whatever `kubeadm token create --print-join-command` prints. Separately, the address a *joining node* needs is the API server's real address on the cluster network — not the `127.0.0.1:` your own kubeconfig uses, which only works because kind port-forwards that specifically for your host machine. +- **Solution**: + ```bash + {SHELL_CP} + kubeadm token create --ttl 2h + + # the exact hash kubeadm join itself verifies against + openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \ + | openssl rsa -pubin -outform der 2>/dev/null \ + | openssl dgst -sha256 -hex | sed 's/^.* //' + + exit + kubectl get endpoints kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}:{.subsets[0].ports[0].port}' + ``` + ``` + JOIN_TOKEN= + CA_HASH=sha256: + API_ENDPOINT=: + ``` +- **Gotchas**: the `kubernetes` Service's own `Endpoints` object (`kubectl get endpoints kubernetes`) is the reliable source for the *real* advertised address — it's populated directly from kube-apiserver's `--advertise-address`, which is what any node on the cluster's own network actually reaches, as opposed to `kubectl cluster-info`/`config view`, which shows the address **your specific machine** uses (potentially through a completely different port-forward or load balancer). Recomputing the CA hash yourself rather than trusting a printed value is the actual security property `kubeadm join` is built around — a client blindly copying a hash it was handed defeats the whole point of a discovery hash. + +--- + ## Domain 2: Workloads & Scheduling (15%) ### Q07 · Rolling Update Strategy & Scaling @@ -753,7 +830,7 @@ This guide explains how to approach and solve every question in the CKA exam ban - labelSelector: {matchLabels: {app: %{dep}}} topologyKey: kubernetes.io/hostname tolerations: - - {key: %{key}, operator: Equal, value: "%{val}", effect: NoSchedule} + - {key: %{key}, operator: {Equal, value: "%{val}", effect: NoSchedule}} containers: - name: app image: nginx:1.27 @@ -1536,3 +1613,24 @@ This guide explains how to approach and solve every question in the CKA exam ban -pod-cidr= ``` - **Gotchas**: kube-proxy's ConfigMap often has `mode: ""` (an explicit empty string) rather than the key being absent entirely — both mean "use the default, iptables" and the file must say `iptables` either way, not literally blank. Write exactly the `key=value` format the task specifies, one per line, no extra whitespace — this is graded with exact string matches, not fuzzy comparison. + +--- + +### Q123 · [HARD] Nothing Works: kube-apiserver Itself Is Down +- **Concept**: Every other control-plane failure in this bank (scheduler, controller-manager, even etcd mid-restore) leaves `kubectl` itself usable, because kube-apiserver is still up to talk to. A broken kube-apiserver is categorically different: it's the single component every `kubectl` command depends on, so once it's down, `kubectl describe`/`logs`/`get` all stop working simultaneously and give you nothing to work with. The kubelet, however, doesn't need the API server to manage **static pods** — it reads the manifest directly off disk — so the fix is the same class of move as Q20/Q118 (repair a static pod manifest), just reached without `kubectl` in hand. +- **Solution**: + ```bash + kubectl get nodes # connection refused / timeout -- confirms it's the apiserver, not your network + + {SHELL_CP} + crictl ps -a | grep apiserver # see it crash-looping + journalctl -u kubelet -n 50 --no-pager | grep -i apiserver + + vi /etc/kubernetes/manifests/kube-apiserver.yaml + # fix whatever's broken in the command/args (e.g. a typo'd binary name) + + exit + kubectl get nodes # back to normal + kubectl -n kube-system get pods -l component=kube-apiserver + ``` +- **Gotchas**: don't reach for `kubectl describe`/`logs` first here — they need the very component that's down, so they'll just hang or refuse to connect. `crictl` talks to containerd directly on the node and works regardless of API server state, making it (along with `journalctl -u kubelet`) your only real diagnostic tool until the fix lands. Once fixed, double-check the *other* static pods (etcd, scheduler, controller-manager) are still healthy — it's easy to nick something else while editing manifests under pressure with no `kubectl` to sanity-check against. diff --git a/history-cka.jsonl b/history-cka.jsonl index fb7ed80..0c7a921 100644 --- a/history-cka.jsonl +++ b/history-cka.jsonl @@ -1,2 +1,3 @@ {"at": "2026-09-06T22:07", "mode": "drill", "hard": false, "ids": [1, 5, 6, 34, 41], "score_pct": 8.0, "earned": 2, "total": 25, "time_used": 432, "pauses": 0, "paused_seconds": 0, "results": {"1": {"pass": false, "earned": 2, "total": 6, "failed": ["ServiceAccount rbac-app/deploy-bot exists", "Role deploy-manager and RoleBinding deploy-bot-binding exist", "deploy-bot can get/list/watch/create/update/patch/delete deployments"]}, "5": {"pass": false, "earned": 0, "total": 5, "failed": ["manifest containing static-web is in /etc/kubernetes/manifests on the node", "mirror pod static-web- is Running", "image is nginx:1.27"]}, "6": {"pass": false, "earned": 0, "total": 6, "failed": ["apiserver-expiry.txt records the real expiry month and year", "CSR jane exists with signer kubernetes.io/kube-apiserver-client", "CSR jane approved and certificate issued", "jane can list pods in default (view) but not delete them"]}, "34": {"pass": false, "earned": 0, "total": 4, "failed": ["path.txt has the correct field path", "Gadget/sample created with the field = 4"]}, "41": {"pass": false, "earned": 0, "total": 4, "failed": ["apiserver and kubelet versions correct", "etcd and runtime versions correct"]}}} {"at": "2026-09-08T19:59", "mode": "exam", "hard": false, "ids": [2, 3, 4, 8, 12, 13, 15, 16, 17, 18, 19, 20, 25, 28, 29, 30, 33, 37, 40, 42], "score_pct": 14.3, "earned": 15, "total": 105, "time_used": 2902, "pauses": 4, "paused_seconds": 113639, "results": {"2": {"pass": true, "earned": 7, "total": 7, "failed": []}, "3": {"pass": false, "earned": 0, "total": 5, "failed": ["overlay kustomization.yaml exists", "resources deployed into namespace store-prod", "store-api has 3 replicas", "Deployment and Service carry label env=prod"]}, "4": {"pass": true, "earned": 4, "total": 4, "failed": []}, "8": {"pass": false, "earned": 0, "total": 4, "failed": ["pod log-shipper Running with containers app and shipper", "volume logs is an emptyDir mounted at /var/log in both containers", "shipper's logs show lines written by app"]}, "12": {"pass": false, "earned": 0, "total": 4, "failed": ["Service shop-svc is NodePort, port 80 -> targetPort 80", "nodePort is 30080", "Service has endpoints"]}, "13": {"pass": false, "earned": 0, "total": 5, "failed": ["ingressClassName nginx and host shop.example.com", "/api (Prefix) -> api:8080", "/ (Prefix) -> web:80"]}, "15": {"pass": false, "earned": 0, "total": 6, "failed": ["PV pv-data: 1Gi hostPath /mnt/data, RWO, Retain, class manual", "PVC pvc-data Bound to pv-data", "data-pod Running with claim mounted at /usr/share/data"]}, "16": {"pass": false, "earned": 0, "total": 5, "failed": ["StorageClass fast-local: local-path, WaitForFirstConsumer, Delete", "fast-local is default and standard is not", "fast-claim picked up fast-local and is Bound", "fast-pod Running with claim at /data"]}, "17": {"pass": false, "earned": 0, "total": 8, "failed": ["node is Ready", "kubelet is active", "kubelet is enabled (survives reboot)"]}, "18": {"pass": false, "earned": 0, "total": 7, "failed": ["all payments replicas available", "Service payments has endpoints (selector fixed)", "Service targetPort matches container port 80"]}, "19": {"pass": false, "earned": 1, "total": 6, "failed": ["crasher.log captured and contains the DB_HOST failure", "pod crasher Running with DB_HOST set"]}, "20": {"pass": false, "earned": 0, "total": 8, "failed": ["kube-scheduler static pod manifest repaired", "new pods get scheduled"]}, "25": {"pass": false, "earned": 0, "total": 4, "failed": ["cka-sim-worker carries workload=batch:NoSchedule", "deployment tolerates the taint", "both pods Running on cka-sim-worker"]}, "28": {"pass": false, "earned": 0, "total": 4, "failed": ["pod Running, init container named setup completed", "emptyDir 'content' shared by both containers", "nginx serves the init-written content"]}, "29": {"pass": false, "earned": 1, "total": 4, "failed": ["deployment now references a different ConfigMap", "all pods available and see mode=new"]}, "30": {"pass": false, "earned": 0, "total": 4, "failed": ["container port 80 named app-port", "Service port 8080 -> targetPort app-port (by name)", "readiness probe uses the named port; endpoints ready"]}, "33": {"pass": false, "earned": 1, "total": 5, "failed": ["ops-44 has an aggregationRule with the right label", "aggregation populated rules from the two labelled roles", "lee can list pods and services cluster-wide"]}, "37": {"pass": false, "earned": 1, "total": 6, "failed": ["CoreDNS Corefile forwards to the node resolver again", "in-cluster and external names resolve from a pod"]}, "40": {"pass": false, "earned": 0, "total": 5, "failed": ["join.sh contains a kubeadm join with token and CA hash", "the token in the file exists in the cluster with TTL ~2h"]}, "42": {"pass": false, "earned": 0, "total": 4, "failed": ["applies to app=client-47, Egress only", "DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP", "TCP 443 to 192.168.100.0/24 only"]}}} +{"at": "2026-09-10T21:07", "mode": "drill", "hard": true, "ids": [106, 112, 115, 120, 121], "score_pct": 6.5, "earned": 2, "total": 31, "time_used": 30, "pauses": 1, "paused_seconds": 175907, "results": {"106": {"pass": false, "earned": 0, "total": 7, "failed": ["CRD exists, Namespaced, replicas has min/max and is required", "good created with environment prod", "replicas 0 is rejected and the message was saved", "defaulting works: defaulted.spec.environment == dev"]}, "112": {"pass": false, "earned": 0, "total": 7, "failed": ["Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local", "Ingress web: class nginx, host, TLS secret, ssl-redirect annotation", "fp.txt matches the secret's certificate fingerprint"]}, "115": {"pass": false, "earned": 1, "total": 6, "failed": ["PVC uses a selector and is Bound to lv-nvme", "consumer Running on the nvme node with /data mounted"]}, "120": {"pass": false, "earned": 0, "total": 5, "failed": ["pod-cidr and service-cidr", "dns-ip and proxy-mode", "cka-sim-worker3 pod CIDR"]}, "121": {"pass": false, "earned": 1, "total": 6, "failed": ["Deployment ledger-14 in ac-omega (the context's namespace), 2 available, owner=ac", "Service ledger-14 port 9000 in the same namespace", "ns.txt has the namespace"]}}} diff --git a/tracks/cka/bank_hard.py b/tracks/cka/bank_hard.py index 2af5ff5..8e3697e 100644 --- a/tracks/cka/bank_hard.py +++ b/tracks/cka/bank_hard.py @@ -196,6 +196,48 @@ srv=$(nx "$CP" 'echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | op C("expiry.txt matches kubeadm's report for apiserver", 1, r"""exp=$(nx "$CP" 'kubeadm certs check-expiration 2>/dev/null' | awk '$1=="apiserver"{print $2,$3,$4,$5,$6}'); [ -n "$exp" ] && grep -qF "$exp" "$WS/q122/expiry.txt" 2>/dev/null"""), ], }, +{ + "id": 124, "domain": "ARCH", "difficulty": "hard", "title": "Prepare a node join without shell access to the control plane", + "params": lambda r: {"ctx": "k8s-c1-H"}, + "task": CTX + """ +A new worker node is about to join this cluster, but you can only hand the networking team a +**file** — no shell access to the control plane. + +1. On the control-plane node, create a **new** bootstrap token with TTL `2h`. +2. Compute the CA certificate's discovery hash **yourself** with openssl — the same + SHA-256-of-the-SubjectPublicKeyInfo value `kubeadm join --discovery-token-ca-cert-hash` + checks against. Do not just copy whatever `kubeadm token create --print-join-command` prints. +3. Determine the API server's real address as seen **from inside the cluster network** — not the + `127.0.0.1:` your local kubeconfig uses to reach it from this machine, which a joining + node cannot use. +4. Write exactly these three lines to `{WS}/q124/node-join.env`: +``` +JOIN_TOKEN= +CA_HASH=sha256: +API_ENDPOINT=: +``` +Every value must be genuine: plugged into `kubeadm join $API_ENDPOINT --token $JOIN_TOKEN +--discovery-token-ca-cert-hash $CA_HASH` right now, it would work. +""", + "setup": r""" +rm -rf "$WS/q124"; mkdir -p "$WS/q124" +for t in $($K -n kube-system get secrets -o jsonpath='{.items[?(@.type=="bootstrap.kubernetes.io/token")].metadata.name}'); do $K -n kube-system delete secret $t >/dev/null; done +""", + "checks": [ + C("a real, unexpired bootstrap token", 2, r""" +tok=$(grep -oE '^JOIN_TOKEN=[a-z0-9]{6}\.[a-z0-9]{16}' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2); [ -n "$tok" ] || exit 1 +id=${tok%%.*}; exp=$(j -n kube-system secret bootstrap-token-$id '{.data.expiration}' | base64 -d 2>/dev/null); [ -n "$exp" ] || exit 1 +[ "$(date -d "$exp" +%s)" -gt "$(date +%s)" ]"""), + C("CA_HASH computed correctly (matches the real CA's SPKI SHA-256)", 3, r""" +want=$(nx "$CP" "openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'") +got=$(grep -oE '^CA_HASH=sha256:[0-9a-f]{64}' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2 | cut -d: -f2) +[ -n "$want" ] && eq "$got" "$want" """), + C("API_ENDPOINT is the real in-cluster address, not the local kubeconfig's", 2, r""" +want=$($K get endpoints kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}:{.subsets[0].ports[0].port}' 2>/dev/null) +got=$(grep -oE '^API_ENDPOINT=.*' "$WS/q124/node-join.env" 2>/dev/null | cut -d= -f2 | tr -d '[:space:]') +[ -n "$want" ] && eq "$got" "$want" """), + ], +}, # ================================================================ WORK ==== { "id": 107, "domain": "WORK", "breaks": "w1-taint", "difficulty": "hard", "title": "Spread, tolerate, prioritise — under a quota", @@ -650,6 +692,34 @@ proxy-mode=/dev/null"""), ], }, +{ + "id": 123, "domain": "TSHOOT", "breaks": "apiserver", "difficulty": "hard", "title": "Nothing works: kube-apiserver itself is down", "disruptive": True, + "params": lambda r: {"ctx": "k8s-c1-H"}, + "task": CTX + """ +Every `kubectl` command from every context times out or refuses to connect. This is not a +network problem — a control-plane component itself is failing to start. `kubectl` is unusable +until you fix it, so investigate directly on the control-plane node's containers. + +Repair the failing component on the control-plane node (`{SHELL_CP}`). Afterwards, `kubectl get +nodes` must work again, and every other control-plane component must still be healthy. +""", + "setup": r""" +nx "$CP" ' +f=/etc/kubernetes/manifests/kube-apiserver.yaml +grep -q "^ *- kube-apiserver$" $f && sed -i "s|^\( *\)- kube-apiserver$|\1- kube-apiserverr|" $f +' +""", + "checks": [ + C("kube-apiserver manifest repaired and Running", 4, r"""nx "$CP" 'grep -q "^ *- kube-apiserver$" /etc/kubernetes/manifests/kube-apiserver.yaml' && eq "$(j -n kube-system pod -l component=kube-apiserver '{.items[0].status.phase}')" Running"""), + C("cluster fully reachable again (get nodes, create a pod)", 4, r""" +$K get nodes >/dev/null 2>&1 || exit 1 +$K -n default delete pod api-probe --ignore-not-found --wait=true >/dev/null 2>&1 +$K -n default run api-probe --image=busybox:1.36 --restart=Never -- sleep 60 >/dev/null 2>&1 +ok=1; for i in $(seq 1 20); do [ "$(j -n default pod api-probe '{.status.phase}')" = Running ] && { ok=0; break; }; sleep 3; done +$K -n default delete pod api-probe --wait=false >/dev/null 2>&1; exit $ok"""), + C("etcd, scheduler and controller-manager unaffected (no collateral damage)", 2, r"""for c in etcd kube-scheduler kube-controller-manager; do eq "$(j -n kube-system pod -l component=$c '{.items[0].status.phase}')" Running || exit 1; done"""), + ], +}, ] # derived params: w2 = 100 - w for the canary split diff --git a/tracks/cka/solutions.py b/tracks/cka/solutions.py index e3d000d..9a5265c 100644 --- a/tracks/cka/solutions.py +++ b/tracks/cka/solutions.py @@ -445,6 +445,32 @@ cp /etc/kubernetes/admin.conf ~/.kube/config kubeadm certs check-expiration | grep '^apiserver ' """ +S[124] = """# Step 1: On the control-plane node, create a fresh bootstrap token +{SHELL_CP} +kubeadm token create --ttl 2h +# -> prints a token like abcdef.0123456789abcdef + +# Step 2: Compute the CA discovery hash yourself. This is exactly what +# `kubeadm join --discovery-token-ca-cert-hash` verifies against, so you should never have to +# trust a value someone else hands you without being able to check it independently. +openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | \\ + openssl rsa -pubin -outform der 2>/dev/null | \\ + openssl dgst -sha256 -hex | sed 's/^.* //' + +# Step 3: Find the real in-cluster API endpoint -- NOT what your local kubeconfig uses. +# kind maps the API server to 127.0.0.1: for YOUR machine only; a node +# joining over the cluster network needs the address the apiserver actually advertises. +exit +kubectl get endpoints kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}:{.subsets[0].ports[0].port}{"\\n"}' + +# Step 4: Assemble the handoff file +cat > {WS}/q124/node-join.env < +CA_HASH=sha256: +API_ENDPOINT= +EOF +""" + # ================================================================= WORK ==== # Domain 2: Workloads & Scheduling (15%) @@ -1543,3 +1569,23 @@ k -n kube-system get cm kube-proxy -o yaml | grep '^ *mode:' # Empty string # Step 5: Query individual node Pod CIDR k get node %{node} -o jsonpath='{.spec.podCIDR}' """ + +S[123] = """# Step 1: Confirm nothing works through kubectl at all +kubectl get nodes # connection refused / timeout -- the API server itself is down + +# Step 2: Access the control-plane node directly (kubectl can't help you here) +{SHELL_CP} + +# Step 3: Check what containerd/kubelet is trying (and failing) to run +crictl ps -a | grep apiserver +journalctl -u kubelet -n 50 --no-pager | grep -i apiserver + +# Step 4: Fix the static pod manifest +vi /etc/kubernetes/manifests/kube-apiserver.yaml +# kube-apiserverr -> kube-apiserver + +# Step 5: Confirm recovery from your own terminal +exit +kubectl get nodes +kubectl -n kube-system get pods -l component=kube-apiserver +""" diff --git a/weak-spots-cka.md b/weak-spots-cka.md index dd88cfb..e8538c3 100644 --- a/weak-spots-cka.md +++ b/weak-spots-cka.md @@ -1,6 +1,6 @@ # CKA weak spots -_Generated by exam from 2 attempt(s); regenerated on every `end`._ +_Generated by exam from 3 attempt(s); regenerated on every `end`._ ## Domain scores (latest full exam, 2026-09-08T19:59, 14.3%) @@ -123,108 +123,133 @@ _Generated by exam from 2 attempt(s); regenerated on every `end`._ - ✗ DNS rule: kube-system namespace + k8s-app=kube-dns, ports 53 UDP and TCP - ✗ TCP 443 to 192.168.100.0/24 only +### Q106 · ARCH · CRD with validation and defaults (1/1 failed, last 0/7 pts, 2026-09-10T21:07) +- ✗ CRD exists, Namespaced, replicas has min/max and is required +- ✗ good created with environment prod +- ✗ replicas 0 is rejected and the message was saved +- ✗ defaulting works: defaulted.spec.environment == dev + +### Q112 · NET · TLS Ingress with a generated certificate (1/1 failed, last 0/7 pts, 2026-09-10T21:07) +- ✗ Secret portal.lab.local-tls is kubernetes.io/tls with CN portal.lab.local +- ✗ Ingress web: class nginx, host, TLS secret, ssl-redirect annotation +- ✗ fp.txt matches the secret's certificate fingerprint + +### Q115 · STOR · Bind the right local volume (1/1 failed, last 1/6 pts, 2026-09-10T21:07) +- ✗ PVC uses a selector and is Bound to lv-nvme +- ✗ consumer Running on the nvme node with /data mounted + +### Q120 · TSHOOT · Cluster facts, exactly (1/1 failed, last 0/5 pts, 2026-09-10T21:07) +- ✗ pod-cidr and service-cidr +- ✗ dns-ip and proxy-mode +- ✗ cka-sim-worker3 pod CIDR + +### Q121 · WORK · The context's namespace (1/1 failed, last 1/6 pts, 2026-09-10T21:07) +- ✗ Deployment ledger-14 in ac-omega (the context's namespace), 2 available, owner=ac +- ✗ Service ledger-14 port 9000 in the same namespace +- ✗ ns.txt has the namespace + ## Attempt matrix -| Q | Domain | Title | D1 | E2 | -|---|---|---|---|---| -| 1 | ARCH | RBAC for a deployment bot | 2/6 | · | -| 2 | ARCH | etcd snapshot | · | ✓ | -| 3 | ARCH | Kustomize overlay | · | 0/5 | -| 4 | ARCH | CustomResourceDefinition | · | ✓ | -| 5 | ARCH | Static pod | 0/5 | · | -| 6 | ARCH | Certificates: expiry check and user CSR | 0/6 | · | -| 7 | WORK | Rolling update & scaling | · | · | -| 8 | WORK | Multi-container pod with shared volume | · | 0/4 | -| 9 | WORK | Scheduling with node affinity and resources | · | · | -| 10 | WORK | ConfigMap and Secret consumption | · | · | -| 11 | NET | NetworkPolicy | · | · | -| 12 | NET | NodePort Service | · | 0/4 | -| 13 | NET | Ingress routing | · | 0/5 | -| 14 | NET | Cluster DNS | · | · | -| 15 | STOR | PersistentVolume, claim and pod | · | 0/6 | -| 16 | STOR | Default StorageClass and dynamic provisioning | · | 0/5 | -| 17 | TSHOOT | Node NotReady | · | 0/8 | -| 18 | TSHOOT | Broken deployment and service | · | 0/7 | -| 19 | TSHOOT | CrashLoopBackOff diagnosis | · | 1/6 | -| 20 | TSHOOT | Control plane component failure | · | 0/8 | -| 21 | ARCH | Helm release install and upgrade | · | · | -| 22 | NET | Gateway API HTTPRoute | · | · | -| 23 | WORK | metrics-server repair and HPA | · | · | -| 24 | WORK | PriorityClass | · | · | -| 25 | WORK | Taints and tolerations | · | 0/4 | -| 26 | WORK | DaemonSet on every node | · | · | -| 27 | WORK | CronJob with a manual run | · | · | -| 28 | WORK | Init container prepares content | · | 0/4 | -| 29 | WORK | Immutable ConfigMap rotation | · | 1/4 | -| 30 | NET | Named ports | · | 0/4 | -| 31 | WORK | Parallel Job | · | · | -| 32 | WORK | ResourceQuota and LimitRange | · | · | -| 33 | ARCH | Aggregated ClusterRole | · | 1/5 | -| 34 | ARCH | Discover a CRD field with kubectl explain | 0/4 | · | -| 35 | STOR | Re-bind a Released PersistentVolume | · | · | -| 36 | STOR | Data survives the pod | · | · | -| 37 | TSHOOT | Cluster DNS is broken | · | 1/6 | -| 38 | TSHOOT | Deployments never create pods | · | · | -| 39 | TSHOOT | kubelet misconfiguration | · | · | -| 40 | ARCH | kubeadm join token | · | 0/5 | -| 41 | ARCH | Component versions | 0/4 | · | -| 42 | NET | Egress-only NetworkPolicy | · | 0/4 | -| 43 | NET | Headless and ExternalName services | · | · | -| 101 | ARCH | Exact RBAC with an existing over-grant | · | · | -| 102 | ARCH | etcd restore | · | · | -| 103 | ARCH | Static pod on a relocated manifest path | · | · | -| 104 | ARCH | User onboarding to a working kubeconfig | · | · | -| 105 | ARCH | Kustomize with generators and patches | · | · | -| 106 | ARCH | CRD with validation and defaults | · | · | -| 107 | WORK | Spread, tolerate, prioritise — under a quota | · | · | -| 108 | WORK | Rollback to a specific revision, then pause | · | · | -| 109 | WORK | Native sidecar with probes and an exact report | · | · | -| 110 | WORK | Cluster audit to exact files | · | · | -| 111 | NET | Zero-trust namespace | · | · | -| 112 | NET | TLS Ingress with a generated certificate | · | · | -| 113 | NET | Gateway API canary with header match | · | · | -| 114 | NET | DNS down: three faults | · | · | -| 115 | STOR | Bind the right local volume | · | · | -| 116 | STOR | Migrate data between claims | · | · | -| 117 | TSHOOT | Worker node: three things wrong | · | · | -| 118 | TSHOOT | Two control-plane components down | · | · | -| 119 | TSHOOT | Application down for four reasons | · | · | -| 120 | TSHOOT | Cluster facts, exactly | · | · | -| 121 | WORK | The context's namespace | · | · | -| 122 | ARCH | Renew control-plane certificates | · | · | -| 201 | ARCH | kubeconfig forensics | · | · | -| 202 | ARCH | Helm chart with CRDs, then a custom resource | · | · | -| 203 | WORK | Find the controller, then scale it | · | · | -| 204 | WORK | Which Pods get evicted first | · | · | -| 205 | ARCH | Kustomize: drop a ConfigMap, add an HPA, override in prod | · | · | -| 206 | STOR | PV, PVC and a Deployment that mounts it | · | · | -| 207 | TSHOOT | Resource usage scripts | · | · | -| 208 | ARCH | Drain a node for maintenance | · | · | -| 209 | ARCH | Query the API from inside a Pod | · | · | -| 210 | ARCH | ServiceAccount limited to two resources | · | · | -| 211 | WORK | DaemonSet on every node | · | · | -| 212 | WORK | One Pod per node, on purpose | · | · | -| 213 | NET | Convert an Ingress to Gateway API, then split on a header | · | · | -| 214 | ARCH | Certificate expiry, two ways | · | · | -| 215 | NET | Egress policy with paired destinations | · | · | -| 216 | NET | CoreDNS: back up, extend, verify | · | · | -| 217 | TSHOOT | Inspect a container with crictl | · | · | -| 301 | ARCH | etcd configuration facts | · | · | -| 302 | NET | Service rules on the node | · | · | -| 303 | NET | A second Service CIDR | · | · | -| 304 | ARCH | Bound tokens versus legacy tokens | · | · | -| 305 | WORK | Schedule without the scheduler | · | · | -| 306 | STOR | Grow a volume in place | · | · | -| 307 | TSHOOT | Node accepts nothing new | · | · | -| 308 | ARCH | API resource reference card | · | · | -| 309 | WORK | A PodDisruptionBudget that blocks maintenance | · | · | -| 310 | NET | Route to something outside the cluster | · | · | -| 311 | ARCH | Image inventory across the cluster | · | · | -| 312 | WORK | A Job that fails, then one that doesn't | · | · | -| 313 | NET | Two hosts, TLS, and a default backend | · | · | -| 314 | TSHOOT | A static pod that never starts | · | · | -| 315 | STOR | Keep the data when the claim goes | · | · | -| 316 | ARCH | Plan a control-plane upgrade | · | · | -| 317 | TSHOOT | Certificates stop being issued | · | · | +| Q | Domain | Title | D1 | E2 | D3 | +|---|---|---|---|---|---| +| 1 | ARCH | RBAC for a deployment bot | 2/6 | · | · | +| 2 | ARCH | etcd snapshot | · | ✓ | · | +| 3 | ARCH | Kustomize overlay | · | 0/5 | · | +| 4 | ARCH | CustomResourceDefinition | · | ✓ | · | +| 5 | ARCH | Static pod | 0/5 | · | · | +| 6 | ARCH | Certificates: expiry check and user CSR | 0/6 | · | · | +| 7 | WORK | Rolling update & scaling | · | · | · | +| 8 | WORK | Multi-container pod with shared volume | · | 0/4 | · | +| 9 | WORK | Scheduling with node affinity and resources | · | · | · | +| 10 | WORK | ConfigMap and Secret consumption | · | · | · | +| 11 | NET | NetworkPolicy | · | · | · | +| 12 | NET | NodePort Service | · | 0/4 | · | +| 13 | NET | Ingress routing | · | 0/5 | · | +| 14 | NET | Cluster DNS | · | · | · | +| 15 | STOR | PersistentVolume, claim and pod | · | 0/6 | · | +| 16 | STOR | Default StorageClass and dynamic provisioning | · | 0/5 | · | +| 17 | TSHOOT | Node NotReady | · | 0/8 | · | +| 18 | TSHOOT | Broken deployment and service | · | 0/7 | · | +| 19 | TSHOOT | CrashLoopBackOff diagnosis | · | 1/6 | · | +| 20 | TSHOOT | Control plane component failure | · | 0/8 | · | +| 21 | ARCH | Helm release install and upgrade | · | · | · | +| 22 | NET | Gateway API HTTPRoute | · | · | · | +| 23 | WORK | metrics-server repair and HPA | · | · | · | +| 24 | WORK | PriorityClass | · | · | · | +| 25 | WORK | Taints and tolerations | · | 0/4 | · | +| 26 | WORK | DaemonSet on every node | · | · | · | +| 27 | WORK | CronJob with a manual run | · | · | · | +| 28 | WORK | Init container prepares content | · | 0/4 | · | +| 29 | WORK | Immutable ConfigMap rotation | · | 1/4 | · | +| 30 | NET | Named ports | · | 0/4 | · | +| 31 | WORK | Parallel Job | · | · | · | +| 32 | WORK | ResourceQuota and LimitRange | · | · | · | +| 33 | ARCH | Aggregated ClusterRole | · | 1/5 | · | +| 34 | ARCH | Discover a CRD field with kubectl explain | 0/4 | · | · | +| 35 | STOR | Re-bind a Released PersistentVolume | · | · | · | +| 36 | STOR | Data survives the pod | · | · | · | +| 37 | TSHOOT | Cluster DNS is broken | · | 1/6 | · | +| 38 | TSHOOT | Deployments never create pods | · | · | · | +| 39 | TSHOOT | kubelet misconfiguration | · | · | · | +| 40 | ARCH | kubeadm join token | · | 0/5 | · | +| 41 | ARCH | Component versions | 0/4 | · | · | +| 42 | NET | Egress-only NetworkPolicy | · | 0/4 | · | +| 43 | NET | Headless and ExternalName services | · | · | · | +| 101 | ARCH | Exact RBAC with an existing over-grant | · | · | · | +| 102 | ARCH | etcd restore | · | · | · | +| 103 | ARCH | Static pod on a relocated manifest path | · | · | · | +| 104 | ARCH | User onboarding to a working kubeconfig | · | · | · | +| 105 | ARCH | Kustomize with generators and patches | · | · | · | +| 106 | ARCH | CRD with validation and defaults | · | · | 0/7 | +| 107 | WORK | Spread, tolerate, prioritise — under a quota | · | · | · | +| 108 | WORK | Rollback to a specific revision, then pause | · | · | · | +| 109 | WORK | Native sidecar with probes and an exact report | · | · | · | +| 110 | WORK | Cluster audit to exact files | · | · | · | +| 111 | NET | Zero-trust namespace | · | · | · | +| 112 | NET | TLS Ingress with a generated certificate | · | · | 0/7 | +| 113 | NET | Gateway API canary with header match | · | · | · | +| 114 | NET | DNS down: three faults | · | · | · | +| 115 | STOR | Bind the right local volume | · | · | 1/6 | +| 116 | STOR | Migrate data between claims | · | · | · | +| 117 | TSHOOT | Worker node: three things wrong | · | · | · | +| 118 | TSHOOT | Two control-plane components down | · | · | · | +| 119 | TSHOOT | Application down for four reasons | · | · | · | +| 120 | TSHOOT | Cluster facts, exactly | · | · | 0/5 | +| 121 | WORK | The context's namespace | · | · | 1/6 | +| 122 | ARCH | Renew control-plane certificates | · | · | · | +| 201 | ARCH | kubeconfig forensics | · | · | · | +| 202 | ARCH | Helm chart with CRDs, then a custom resource | · | · | · | +| 203 | WORK | Find the controller, then scale it | · | · | · | +| 204 | WORK | Which Pods get evicted first | · | · | · | +| 205 | ARCH | Kustomize: drop a ConfigMap, add an HPA, override in prod | · | · | · | +| 206 | STOR | PV, PVC and a Deployment that mounts it | · | · | · | +| 207 | TSHOOT | Resource usage scripts | · | · | · | +| 208 | ARCH | Drain a node for maintenance | · | · | · | +| 209 | ARCH | Query the API from inside a Pod | · | · | · | +| 210 | ARCH | ServiceAccount limited to two resources | · | · | · | +| 211 | WORK | DaemonSet on every node | · | · | · | +| 212 | WORK | One Pod per node, on purpose | · | · | · | +| 213 | NET | Convert an Ingress to Gateway API, then split on a header | · | · | · | +| 214 | ARCH | Certificate expiry, two ways | · | · | · | +| 215 | NET | Egress policy with paired destinations | · | · | · | +| 216 | NET | CoreDNS: back up, extend, verify | · | · | · | +| 217 | TSHOOT | Inspect a container with crictl | · | · | · | +| 301 | ARCH | etcd configuration facts | · | · | · | +| 302 | NET | Service rules on the node | · | · | · | +| 303 | NET | A second Service CIDR | · | · | · | +| 304 | ARCH | Bound tokens versus legacy tokens | · | · | · | +| 305 | WORK | Schedule without the scheduler | · | · | · | +| 306 | STOR | Grow a volume in place | · | · | · | +| 307 | TSHOOT | Node accepts nothing new | · | · | · | +| 308 | ARCH | API resource reference card | · | · | · | +| 309 | WORK | A PodDisruptionBudget that blocks maintenance | · | · | · | +| 310 | NET | Route to something outside the cluster | · | · | · | +| 311 | ARCH | Image inventory across the cluster | · | · | · | +| 312 | WORK | A Job that fails, then one that doesn't | · | · | · | +| 313 | NET | Two hosts, TLS, and a default backend | · | · | · | +| 314 | TSHOOT | A static pod that never starts | · | · | · | +| 315 | STOR | Keep the data when the claim goes | · | · | · | +| 316 | ARCH | Plan a control-plane upgrade | · | · | · | +| 317 | TSHOOT | Certificates stop being issued | · | · | · | _E = full exam, D = drill, · = not in that session, n/m = partial credit._