Every changed task was seeded, graded untouched, solved and re-graded on a
separate kind cluster. The four sim sets were rehearsed as full sittings (real
baseline, seed everything, solve in order, grade at the end): A 150/150,
B 176/176, C 163/163, D 122/122. Every random-pool task was either passed in a
real session or cycled the same way.
Graders that could never pass, or passed for free:
Q306 (local-path cannot resize; now graded as an in-place resize), Q314 (GNU
grep prefixed file names), Q305 (plain `get events` has no SOURCE column and
events expire; now managedFields), Q410 (kindnet DOES enforce NetworkPolicy;
the expected answer is now derived from the CNI), Q35, Q212 (missing `]`),
Q101 (`can-i create pods/exec` asks about a pod NAMED exec; multi-type get),
Q109 (an emptyDir over nginx's /var/log crash-loops it), Q40 (TTL was
measured at grading time), Q121, Q106, Q104, Q41, Q26, Q403, Q409, Q417,
Q113, Q22, Q24/Q108 (flaky right after a rollout), and free points on
untouched Q24/Q102/Q108/Q119/Q122.
Reference solutions: S[43], S[107], S[121], S[106] rewritten; S[116] (a
Completed Job pod still pins its PVC), S[119] (a pods quota equal to replicas
deadlocks the rollout); DNS checks use FQDNs; the explainer and exam guide
are corrected (etcd restore is etcdutl's job on 3.6+).
Cross-session leaks: the pre-session baseline now also repairs Q114 (the
kube-system NetworkPolicy and CoreDNS replicas), Q117/Q39 kubelet CA paths,
Q317, kubelet drop-in binary paths, a moved scheduler manifest, leftover
static pods and the default StorageClass. Q115 releases its fixed PVs from
claims left by earlier sessions.
Conflicts: `breaks` may be a list and `solo` keeps Q102's etcd restore out of
mixed sessions. Q118, Q5/Q103, Q26 vs the W1 taints, set A 203/204/208/212 and
set B 305/309 no longer sabotage each other. Q317 now disables only CSR
signing (the old break killed the whole controller-manager). Q110/Q311 no
longer compare against the whole cluster at submit time. Two delete/recreate
races in setups are fixed.
Harness: the weak list clears after two clean passes in a row, weak entries
show the latest attempt's failures, domain scores use the exam's own tasks,
history entries can be marked synthetic (entry 11 is), the web UI lists
every sim set, `./exam grade` refuses mid-exam (--force), preflight checks W1.
Set D (`./exam sim d`): 17 original killer.sh-style tasks, ids 501-517,
122 points - control-plane inventory, event forensics with crictl, a broken
kubelet unit, kubelet certificates, manual scheduling with the scheduler
stopped, Helm housekeeping with a stuck release, an enforced AND-vs-OR
NetworkPolicy, dnsPolicy and access-mode troubleshooting, and more.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two things in one regeneration of the data files:
- Stripped "pauses" and "paused_seconds" from all 10 existing entries, so the
data matches the code change in 01877f0 rather than leaving historical
telemetry behind. Nothing else in those entries was touched.
- Attempt #11: Simulator Set C, 163/163. Recorded for completeness, but it is
not a measure of skill - the tasks were completed programmatically to verify
the new bank end to end on a freshly seeded cluster, not solved by hand.
Set C tasks carry set="c" and are excluded from POOL, so this does not
affect `./exam drill weak` or `./exam drill all` selection; `./exam sim c`
re-runs the full set regardless of history.
The genuine result from this week is attempt #10: an 18-task drill at 39.7%,
8 of 11 attempted tasks at full credit, 7 never opened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NziZdvBvy7CpEAAbZNgp8i
Pausing is for life happening - a baby needing attention - not a metric worth
keeping, and the numbers were meaningless anyway: the last recorded attempt read
12 pauses / 90,755 seconds, almost all of it sessions left paused overnight.
Removed:
- the st["pauses"] counter in set_paused()
- "pauses" and "paused_seconds" from the history entry in record_attempt()
- "pauses" from the web API session payload
- the "· paused 3× (12:04)" segment on the results screen
Kept, because both are load-bearing rather than telemetry:
- paused_total, which remaining() adds back to the deadline so paused time is
not silently deducted from the clock, and which the web clock reads to do the
same countdown math
- paused_at, which is just "is the session paused right now" and drives the ⏸
indicator
Neither is written to history or surfaced in any report; they exist only for the
lifetime of a live session.
Note for anyone reading a long time_used later: it is computed as
ended_at - started_at - paused_total, and paused_total only accrues while
actually paused, so a session resumed and then left running overnight inflates
it. Attempt #11 logs 28:59:01 for that reason, not a timing bug.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NziZdvBvy7CpEAAbZNgp8i
`./exam sim c` - 18 tasks, 163 points, 120 minutes, exam rules. Built after a
23% result on a paid mock, so reruns cost nothing and the weak topics can be
drilled as often as needed.
Original scenarios, not copies, exercising the same techniques as an 18-question
sitting: hostPath PV/PVC bound by volumeName, data migration between claims onto
a Retain volume, Helm render-then-install, Gateway API header canary, RBAC with
the answer written to a file, probe/selector/targetPort repair, HPA v2 behavior
blocks, and three "write the output to an exact path" data-extraction tasks.
Ids 401-418, set="c", difficulty="hard". cmd_sim is already generic over SETS so
no harness change was needed; set-tagged tasks stay out of POOL and cannot leak
into `./exam start` or drills. CKA bank goes 101 -> 119 tasks.
Two scenarios are deliberately adapted, because the originals would destroy the
practice cluster:
- 410 audits the installed CNI and applies a default-deny, rather than
installing Calico over the running kindnet. The Flannel-vs-Calico
"does this CNI actually enforce NetworkPolicy" judgement is kept as a
written answer.
- 413 plans a control-plane upgrade - versions, ordered command sequence,
`kubeadm upgrade node` for workers - and never runs `kubeadm upgrade apply`.
The grader checks the ordering, modelled on Q316.
Verified end to end: every task seeded, solved with its own reference solution
and graded, 163/163 across all 18, in four batches. Checks were also confirmed
to fail cleanly against an unsolved cluster rather than erroring.
Three environment facts cost a round of debugging and are now encoded in the
tasks and their solutions:
- The StorageClass on this kind cluster is named `standard`, not `local-path`,
and is WaitForFirstConsumer - a PVC stays Pending until its pod can be
scheduled, which is what interlocks the two faults in 414.
- Under rootless podman the host cannot route to node IPs, so a host-side
openssl against a NodePort just hangs (rc=124). 417 probes TLS from inside a
node via nx against 127.0.0.1:<nodePort>. Nodes have openssl and curl but
not nc or wget.
- hostPath PV data must be seeded on every node (401, 405), since nothing
pins the pod to the node holding the directory.
Seeding all 18 creates ~24 pods, which on this host needs
kernel.keys.maxkeys raised above the 200 default or container starts fail with
"unable to join session keyring: disk quota exceeded".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NziZdvBvy7CpEAAbZNgp8i
Audit of the bank and harness after a 23% mock result. Two of these were
silently corrupting every practice session on this cluster.
restore_baseline:
- The w1-taint cleanup removed nothing at all. `kubectl taint node N a- b- c-`
aborts on the first key that is not present and applies none of the removals,
so with only `workload` set the command reported five "not found" lines,
exited 1, and left the taint in place - the precise failure the block's own
comment says it exists to prevent. W1 then rejected pods in every later
session. Now one call per key. Verified by re-seeding workload=gpu:NoSchedule
and confirming a single restore_baseline run clears it.
- metrics-server was never repaired. Q23 seeds its scenario by stripping
--kubelet-insecure-tls, but carried no "breaks" key, so nothing tracked it
and nothing undid it. On kind the kubelet serving certs have no IP SANs, so
without that flag metrics-server never goes Ready, v1beta1.metrics.k8s.io
has no endpoints, aggregated discovery goes stale, and EVERY namespace
deletion in the cluster hangs on NamespaceDeletionDiscoveryFailure - which
is where the ~40 stuck namespaces came from, plus a dead `kubectl top`.
Q23 now declares breaks: metrics-server and the baseline restores the flag.
Baseline runs before seeding, so a session containing Q23 re-breaks it a
moment later as intended. Verified by stripping the flag and confirming one
restore_baseline run puts it back.
Session composition:
- pick() dropped any domain whose tasks are all "hard" from a normal session.
CKS SUPPLY and RUNTIME hold one hard task each, so `./exam start` returned
6 of 9 tasks with 40% of the blueprint silently absent. It now tops up from
the hard pool when a domain's normal pool is short.
- compose_exam() iterated QUOTA in declaration order, so a one-task domain
lost every "breaks" tie to a bigger domain that claimed the shared slot
first: CKS RUNTIME (only Q9, apiserver-flag) was never drawn in ANY mode
because SETUP went first and took that slot with Q2. Now walks domains
scarcest-first, and warns on a shortfall instead of truncating in silence.
CKS now yields 8/9 covering all six domains, warning SETUP 1/2.
- session_questions() rebuilt the list by filtering QUESTIONS (sorted by id),
discarding the session's own order - so compose_exam's rng.shuffle never
reached the candidate and every exam arrived grouped by domain and
difficulty, base tasks then hard then sim sets. Fixed to preserve the stored
order, and cmd_drill now shuffles explicit-id drills after conflict
resolution, so which task gets dropped stays deterministic while the order
does not.
Bank correctness (verified by grading each reference solution):
- CKAD Q2 was unpassable, 5/8. Its init container waited on `until nslookup
backend`, but busybox nslookup exits 1 even on a successful lookup because
it walks the whole search list and the trailing NXDOMAINs set the status -
the loop never terminates. The premise was wrong too: a ClusterIP Service
resolves from creation, endpoints or not, so DNS could never have gated it.
Now waits on a real connection; 8/8, and the pod genuinely holds at Init:0/1
until the backend Deployment exists. Same bug class as e3f5ecd.
- CKS Q1 was unsatisfiable as written, 6/10. The text asked for three
NetworkPolicies but the grader required a fourth: one podSelector cannot
cover both the web egress half and the db ingress half. Task text now asks
for four and says why.
- CKS Q4 check 4 was a false positive. `... || r=0` read a failed exec as
"no token mounted", awarding 3 points when the candidate had done nothing
and the probe pod simply could not start - which happens often on this
cluster. Now requires the pod to reach Running and exec to work. Verified
it awards the points before the fix and fails after, with pod creation
blocked by a quota.
- CKAD Q8's solution asserted a Pod's image cannot be changed in place.
It can - spec.containers[].image is mutable; `kubectl set image pod/...`
works and grades 7/7. Rewritten, noting which fields really are immutable.
Also: the `sim` usage and help strings hardcoded "a|b" and a "17-task" count,
wrong for a third set and wrong for CKAD/CKS, which define no sets at all. Both
are now generated from SETS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NziZdvBvy7CpEAAbZNgp8i
Remaining findings from the repo audit (1, 2 and 4 landed in df04c56).
restore_baseline now undoes two more leaks, both of which cost real debugging
time this week:
- apiserver (Q123 renames the command to "kube-apiserverr"). This runs FIRST
and over podman/ssh rather than kubectl, because it is exactly the failure
that makes kubectl unusable - every later step in the block depends on the
API being up. It waits for /readyz before continuing.
- staticPodPath (Q103 relocates it to /etc/kubernetes/static on the tier=edge
node). The existing kubelet repair only fired when
/var/lib/kubelet/config.yaml.orig existed, and nothing ever creates that
backup, so a relocated path survived every later session and silently broke
Q05, whose check hardcodes /etc/kubernetes/manifests. Normalised
unconditionally, restarting the kubelet only when the file actually changed.
Verified against the live cluster: with both breaks applied the API was
unreachable, and one restore_baseline run brought all four nodes Ready, restored
the manifest, reset staticPodPath and let the kubelet re-adopt the Q05 mirror
pod. Re-running takes 1.1s versus 14.6s when repairing, confirming it is
idempotent and does not restart anything it has not changed.
etcd (Q102) and certs (Q122) are deliberately left alone, now documented in the
block: neither setup damages anything, so any damage comes from a half-finished
candidate attempt, and "mid-attempt" is indistinguishable from "done correctly"
without guessing - guessing wrong rewrites etcd's data-dir or re-issues certs on
a healthy cluster. proxy-rules (Q302) needs no repair at all; its "breaks" key
exists for no_conflicts() bookkeeping, not because it touches shared state.
Also:
- S[30], S[42], S[108] referenced params that do not exist (%{svc}/%{app},
%{np}/%{port}, %{rev}), so they rendered placeholders verbatim. Rewritten
against the real task text - S[42] was missing the DNS egress rule entirely
and S[108] stopped before the scale/pause/set-image steps, so both would
have failed 2 of 3 checks if followed literally. A param-reference scan over
the whole bank is now clean.
- Q124's check and S[124] used the deprecated v1 Endpoints API; switched to
EndpointSlice, verified both forms return 10.89.1.3:6443 on this cluster.
- q_by_id printed "Valid: 1-101" for non-contiguous ids; it now derives the
real ranges (1-43, 101-124, 201-217, 301-317).
- Untracked workspace/q03. Both q03 and q21 setups start with rm -rf and
rewrite every file from heredocs, so they are generated output, not seed
templates: tracking them only produced phantom diffs, since a re-seed
reverted edits to base/ and deleted anything left in overlays/. With nothing
tracked there, teardown.sh can go back to wiping workspace/ wholesale.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fqcio9TXkEXH35UHebTgoL
Both checks ran `nslookup kubernetes.default` inside a busybox:1.36 pod and
required it to succeed. busybox's nslookup does not apply the pod's resolv.conf
`search` list - it queries the name literally - so that lookup returns NXDOMAIN
no matter how correctly CoreDNS is configured. Verified in-cluster: from a pod
with a correct `search default.svc.cluster.local svc.cluster.local cluster.local`
and a healthy Corefile, `kubernetes`, `kubernetes.default` and
`kubernetes.default.svc` all return NXDOMAIN while the full
`kubernetes.default.svc.cluster.local` resolves to 10.96.0.1.
The result was a task that could never score more than 3/6: the local drill
history shows this check failing on 4 of 4 attempts with the Corefile fully
repaired.
Use the FQDN in both checks, and say so in the task text so the candidate
verifies the same way the grader does. Q216 in bank_sim_a already used an FQDN;
these two were the outliers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fqcio9TXkEXH35UHebTgoL
Three fixes from an audit of the repo:
1. setup_contexts() deleted all three k8s-c?-? contexts and recreated none
unless the session was started with --hard. But './exam drill <ids>' and
'./exam drill weak' don't filter by difficulty, so hard tasks land in
hard=False sessions - and 24 of them open with "kubectl config use-context
...", which then fails outright. Q121 was unpassable that way: it says to
deploy into the context's namespace without -n, and its check requires that
namespace, so the context was the only way to learn the name. Contexts are
now created whenever a task in the session references one.
Also fixes how k8s-c2-AC's namespace was chosen: it took the first ns in the
whole session, which is arbitrary when several tasks share a context. A
context has one default namespace, so it belongs to the task that actually
reads it - Q121, now marked "ctx_ns": True.
2. restore_baseline() cleared taint keys dedicated/workload/team, but Q107
draws from pool/lane and Q307 pins maintenance=inprogress, and nothing
uncordoned a node. A leaked cordon or taint silently fails every later task
that counts pods per node (Q26/Q211 DaemonSet coverage, Q212's spread) -
W1 had been cordoned and tainted across sessions for two days. Clears all
six keys now and uncordons all three workers.
3. ttyd was launched with -W (writable) and no interface binding, so it listened
on 0.0.0.0 while the API server correctly bound localhost. That exposed an
unauthenticated root-equivalent shell with the cluster kubeconfig to the whole
network. Pinned to loopback, overridable via CKA_TTYD_INTERFACE.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fqcio9TXkEXH35UHebTgoL
tracks/cka/solutions.py had drifted out of sync with the actual current
task text for several questions (Q06, 11, 12, 13, 14, 22, 32, 40, 109,
110, 120, 122) - wrong namespaces/resource names, missing required steps,
or output files that were never actually written. Rewrote each to match
its task's real requirements.
tracks/cka/bank_extra.py's Q33 grading check queried
`.spec.aggregationRule` on a ClusterRole, but ClusterRole has no `.spec`
field - `aggregationRule` and `rules` are top-level. Fixed the jsonpath
so a correct answer actually passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fqcio9TXkEXH35UHebTgoL
Task setup scripts regenerate workspace/qNN/ on every seed, so it's
runtime output, not source - same reasoning as the earlier q03/q21
untrack. Ignore workspace/* by default, keeping q03 and q21 as the
two already-tracked seed-template exceptions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Disruptive tasks corrupt shared state (static pod manifests, kubelet
config, CoreDNS) keyed by a "breaks" target, but a session's setup only
touches the questions it actually contains. If a task isn't re-picked in
a later session, its damage lingers indefinitely - e.g. Q39 wrecking W1's
kubelet config persisted across two unrelated TSHOOT drills that never
included Q39 again.
Add restore_baseline(), run unconditionally in begin_session() before
seeding, that repairs every known "breaks" target (scheduler,
controller-manager, CoreDNS, all three worker kubelets, W1's demo
taints) regardless of which questions are in the new session. Hard-mode-
only breaks (etcd, certs, apiserver) aren't covered yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
workspace/q03's Q03 setup script wipes and regenerates this base
every time the task is seeded (rm -rf "$WS/q03"; mkdir -p ...), so
these were runtime output that had been accidentally committed
rather than source files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The UI was light everywhere except the embedded terminal, which felt
disjointed. Switch to a cohesive dark palette with layered surfaces,
tinted state colors for pass/fail/warn/flag, and polish on buttons,
cards, overlays, and the question nav (hover/active states, shadows,
custom scrollbars).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kind load docker-image shells out to the docker CLI internally
regardless of KIND_EXPERIMENTAL_PROVIDER, so it can't see images that
only exist in podman's store and fails when the docker socket isn't
reachable. Save each image with podman save and load the tar via
kind load image-archive, which is runtime-agnostic.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Close two of the four CKA-relevant gaps identified against Killercoda's
scenario catalog:
- Q123 (TSHOOT, hard): kube-apiserver itself crash-loops on a static-pod
manifest typo, same pattern as the existing scheduler/controller-manager
failures (Q20/Q118) but with the added twist that kubectl is completely
unusable until it's fixed, forcing node-level diagnosis (crictl,
journalctl).
- Q124 (ARCH, hard): produce a real node-join handoff file by
independently computing the CA discovery hash kubeadm join verifies
against, and finding the API server's actual in-cluster address rather
than the local kubeconfig's port-forwarded one.
Cluster-upgrade and operator-authoring remain unimplemented and are
documented as genuine infrastructure gaps in CKA_SOLUTIONS_EXPLAINED.md
(kind's node images have no apt-installable multi-version kubelet/kubeadm,
and operator authoring needs a real built/pushed image) rather than faked.
Also updates the Killercoda cross-reference table with the gaps closed,
and logs an ended drill session to history-cka.jsonl/weak-spots-cka.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Expand every entry with a concept section explaining the underlying
mechanism, exact commands/YAML pulled from tracks/cka/solutions.py,
and gotchas tied to what each question's grader checks actually
verify. Doc grows from ~750 to ~1540 lines.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Filled six gaps in the solutions guide (Helm, Kustomize generators,
CRD validation, cluster audit, Gateway API canary, context namespace)
so every question id in the bank (1-43, 101-122) has a section.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Q02/Q102 seeds tried to copy etcdctl out of the running etcd pod with
`kubectl exec -- cat`, which cannot work: registry.k8s.io/etcd is distroless
(no cat, no sh, no tar). The copy failed, stderr went to /dev/null and seeding
still printed a green tick, so the snapshot task had no etcdctl on the node.
Q102 was worse: it also ran `exec -- sh -c 'etcdctl ... snapshot save'` inside
that container, so its /opt/cka/backup.db was never created at all.
- etcd-tools.sh installs both binaries on the control-plane node, taking them
from the etcd image already in the node's containerd (offline, exactly the
server version), falling back to the upstream release tarball, then apt.
- setup.sh runs it at bootstrap; ensure_etcd_tools in the exam prelude installs
them through CKA_NODE_ACCESS when a task is seeded, so ssh/vagrant clusters
get them too, and both etcd setups now fail with a reason instead of hiding it.
- Q102 takes its snapshot with the node's etcdctl; Q02 no longer documents a
kubectl cp fallback that cannot work on a distroless image.
Co-authored-by: Junie <junie@jetbrains.com>
